General

Written by Rambus Press

Frank Ferro, Senior Director Product Management at Rambus, and Shane Rau, Senior Research Executive at IDC, recently hosted a webinar that explores the role of tailored DRAM solutions in advancing artificial intelligence. Part one of this four-part series reviewed a range of topics including the interconnected system landscape, the impact of COVID-19 on the data center, and how AI is helping to make sense of the data deluge. This blog (part two), takes a closer look at how AI enables useful data processing, various examples of AI silicon, and the evolving role of DRAM in advancing artificial intelligence.

How AI Enables Useful Data Processing

According to Rau, the sheer amount of data that has been generated in recent years more than justifies the need for AI. As well, AI requires a significant amount of processing power to handle data complexity.

“When you have more distributed data across the landscape generated by different system types, you have different data types. Some of that data is more important than other data. For example, the non-entertainment imaging share of total data, think a lot of static images, is declining” he explains.

“In contrast, entertainment remains a huge part of the data being created and think a lot of moving images, think videos, think Netflix, where the data cannot be interrupted, no one wants the sound or the video of their Netflix movie to be interrupted. That is a form of critical data that cannot be interrupted. It is real time data.”

As Rau emphasizes, AI needs to know how to prioritize data as well as process it.

“You have a combination of AI dealing with a lot of data, a lot of distributed data across that landscape, having to assess whether it is critical or not, and then how sophisticated that data type is. But immediately, the utility for AI is identifying the useful data and bringing that data to the surface for human attention,” he elaborates. “AI is stepping in between our data that we can no longer process because of the amounts and the sophistication of that data and doing the job, and also bringing the data to our attention so we can make good decisions.”

Examples of AI Silicon

As Rau observes, the industry will be working for the next decade or more to advance AI algorithms and processors.

“They will also be developing memory, memory capacity, and various memory types to adapt to the needs that AI will have across this period and across this whole data landscape,” he states.

To more clearly illustrate the market growth of various chips, Rau points to the slide below that aggregates different types of silicon including FPGAs and GPUs, along with specialized AI ASICs/ASSPs.

“You can see significant growth in devices like PCs, phones and tablets. Phones drive a lot of early AI data processing because many of the phone manufacturers, think Apple, think Huawei or their chip providers like Qualcomm, have AI-specific processing capabilities they can put in a phone,” he elaborates.

“This is critical for a phone when it’s doing AI, it’s right in front of you and can process a lot of incoming data through your voice or the video that you’re creating. It can then determine what is important to process and what needs to move on further through edge infrastructure, communications infrastructure, and then into the data center.”

According to Rau, PCs, helped by GPUs, will also be processing AI applications and data, while PCs, phones and tablets are a huge driver of the AI data processing silicon opportunity. Networking infrastructure is “very small” at this point, says Rau, but networking infrastructure will also be processing AI. Specifically, packet processing, compression, and decompression of data as it moves through infrastructure will require AI.

“We have a silicon opportunity that’s pervasive across the AI landscape (or the system landscape) from the IoT endpoints into the data center and cloud. So, we have established the need for AI, the need for processing of AI, and the opportunity for the processing silicon to do AI,” he adds. “With these processors comes the need for memory and most often DRAM. TPUs, GPUs, FPGAs, and other data processing silicon types need DRAM attached to them to bring the data close to the processing so it can be done quickly, such as in real time processing of video.”

The Evolving Role of DRAM in Advancing Artificial Intelligence

As Rau points out, DRAM is a core technology that has proven itself to be extremely adaptive over time, enabling it to support a wide range of system types.

“DRAM has adapted to the needs of new systems, starting with the 90s when personal computing and PCs drove the need for processing and large amounts of memory to meet the general-purpose needs of PCs. When DRAM was, what we called a commodity DRAM, it was ubiquitous, but then as graphics and gaming and other applications came in DRAM adapted into more specialized forms of DRAM for servers, for graphics, and GDDR for example,” he explained.

“More recently with the advent of smartphones, specialized low power LPDRAM is used for those devices. As well, cloud servers sometimes use commodity DRAM or specialized DRAM in large quantities and even some memory modules with their own data buffer chips are used to put more intelligence on the DRAM, next to the DRAM on the module so that DRAM can make some decisions on its own and offload some functions from the CPU. In this way, DRAM is also adapting.”

In terms of the next decade, says Rau, DRAM will once again adapt to new applications across multiple verticals like automotive, video surveillance, and smart homes.

“These applications will need DRAM and processing and heavy amounts of AI. Think in the smart home, think of Alexa, for example, when you talk to Alexa right now, Alexa sends your requests back to the data center,” he elaborates. “In the future, we think that smart home systems will be more locally intelligent to process your request in real time. That means more memory, that means more DRAM.”

Although the smart home space is different than automotive, says Rau, vehicles will also need significant amounts of memory and processing to support AI systems with varying levels of autonomous driving capabilities. In terms of video surveillance, says Rau, AI processing is used to process the event, analyze its severity, and determine will happen next.

“Will that event need to be reported through the video surveillance endpoint, through infrastructure, into the data center in the cloud for some form of more detailed analysis and response?”

The Ubiquity of AI and DRAM

According to Rau, AI will ultimately become ubiquitous in electronic systems.

“As human beings, we use those systems, conceivably seven billion human beings on the planet. There will be seven billion types of AI solutions – especially when you consider all the potential combinations of training versus inferencing, different processing types, the different DRAM types and capacities and configurations that will support that AI,” he explains. “With all of this, DRAM technology continues to adapt. We have GDDR6, which formerly was just graphics DRAM, now being applied in automotive, for example. And HBM, which is high bandwidth memory. Again, formerly just for graphics, but now applied to applications outside of graphics that require very high performance, low latency, as well as a non-proprietary, cost-effective form of performance. There are also memory buffers that go on modules that help the DRAM be more intelligent to respond and offload the needs of CPUs.”

For the next decade and beyond, Rau concludes, DRAM will continue to adapt to new applications driven by AI and data processing.

In part one of this two-part blog series, Semiconductor Engineering editor in chief Ed Sperling spoke with Steven Woo, Rambus fellow and distinguished inventor, about the relationship between Moore’s Law and the thermal challenges faced by the semiconductor industry. Specifically, Woo highlighted how the breakdown of Dennard scaling around 2005 prompted GPU designers to place a major emphasis on thermal management and dissipation. In this blog post, we’ll take a closer look at thermal challenges and solutions across multiple verticals and applications including data centers, high performance computing (HPC), mobile devices and various memory types.

As Woo notes, power efficiency has become a top priority across the semiconductor industry in recent years.

“The amount of work that’s required [for thermal management] and the amount of space that’s required to actually deal with the heat is really growing over time – and is something that’s really unsustainable in our industry,” Woo states.

Commenting on battery-powered devices such as smartphones and tablets, Woo says that the mobile industry has developed multiple power-saving techniques like switching off specific system components that are not in use. These components can be quickly restarted when a change of operation is required.

“The ability to keep some silicon dark for certain amounts of time and to quickly turn it on while you turn off other parts of the silicon has been critical to managing the power and thermals for battery operated devices,” he adds.

Commenting on microfluidics, Woo observes that there is a lot of emphasis on using fluids to remove heat from certain systems.

“If you look at solutions like the latest version of the TPU from Google, there’s liquid cooling. If you look at supercomputing you see more and more machines at the high end that are using liquids to move the heat in a way that it can be dissipated more readily,” he explains. “In fact, there are inert liquids that you can even immerse whole boards into. Because they are inert, nothing shorts out. Having that direct liquid contact with something that is inert is by far the best way to remove the heat.”

Woo also emphasizes that DRAM is very sensitive to heat.

“If you are doing things correctly, then you [should be] designing [a system] so that the data movement doesn’t get affected [by the heat]. As a system designer, what you have to think about is how much power is going to be dissipated at the target data rate – and then think about everything you have to do to make sure the temperature remains relatively stable.”

As Woo points out, thermal challenges have prompted chip and memory architects to change the way systems are designed.

“For example, HBM memory uses stacking to try and reduce the distances that data has to move and reduce the amount of heat and power that needs to be dissipated. [The] graphics card [shown below] illustrates that even with those changes there’s still a lot of challenges with managing the thermals,” he elaborates.

“This is a modern graphics card that uses HBM memory and you can see once again it’s quite thick and quite big. As I remove the fan cover here what blows over the of the radiator structure we again see a really large really complicated kind of structure whose job it is to whisk the heat away from the processor.”

In this case, says Woo, the board itself (where the processing is done) is much smaller than what you see on a GDDR6 card.

“However, the thermal solution is just as beefy. So, you have the processor and two HBM RAMs on a very compact board. Again, most of the volume is really dedicated towards dealing with the thermals,” he continues.

“You can also see a copper plate whose job it is to whisk the heat away from the processor and the memory. As well, there is a heat pipe structure which moves that heat across a large set of bladed fins and air flows over those fins to dissipate the heat.”

According to Woo, engineers have formulated extensive design guidelines that detail how air should flow and where the heat should be moving to keep an entire system at a reasonable temperature.

“Over the years people have gone from relatively simple types of fan blade structures (like you can see in this card) to something that’s much more advanced,” he adds.

“There is also a big emphasis on efficiency of those fans, as well as having relatively large surface area of the blades, but also being quiet. The design and the angles and the way that the blades are shaped in heavily influences the ability to both move a lot of air and remain quiet.”