Written by Steven Woo for Rambus Press

In part 5 of this series, we discussed the most common memory systems that are used in the highest performance AI applications. These include on-chip memory, high bandwidth memory (HBM) and Graphics DDR SDRAM (GDDR SDRAM). In this blog post, we’ll take an in-depth look at on-chip memory, as well as explore some of the primary differences between HBM and GDDR6.

Suppose we set out to design a memory system that provides 256GB/s of memory bandwidth. The slide above illustrates the key differences when implementing such a memory system using HBM2 and GDDR6 DRAMs. As the table illustrates, data rates are one of the primary differences between the two memory types. Because there are fewer data wires, GDDR6 must transmit data at a higher data at a much higher data rate – 16 gigabits per second – as opposed to HBM2 which is running at two gigabits per second with many more wires. As such, the primary challenge associated with GDDR6 is signal integrity, although GDDR6 can enable a lower cost solution because it doesn’t involve intricate stacking like HBM2 does.

Now, let’s take a closer look at the PHY circuits which reside on the SoC and which connect the DRAMs to the SoC. With GDDR6, the amount of area that designers need to reserve on the SoC is between 1.5 and 1.75 times the area required for similar PHY circuits designed to communicate with HBM2 at the same total bandwidth of 256GB/s. In terms of power, it’s a more significant difference, as the GDDR6 PHYs consume anywhere between three-and-a-half to four-and-a-half times as much power as the PHY for an HBM2 implementation at the same bandwidth. From an engineering perspective, there is a definite SoC PHY area and power advantage for HBM2 memory systems.

But the area and power advantages of HBM2 PHYs comes at the expense of higher complexity and cost. This is because HBM2 requires an additional interposer, substrate and intricate stacking within the DRAM as well as between components in the system, all of which add extra cost and complexity, and which are currently exacerbated by the relative newness of the technology. Fundamentally, the decision to use HBM2 or GDDR6 comes down to how you trade off the most important design requirements against what matters most in your system. If you can handle the cost and complexity of an HBM2 implementation, then it is a very good solution. However, if your system requires either more standard manufacturing techniques, or can’t quite absorb the additional costs, then GDDR is a great solution.

On-chip Memory Benefits and Challenges

In the latter part of this blog post, we’ll be taking a closer look at the individual types of memories used in AI systems that I referenced earlier so we can walk through the benefits and challenges of each. On the slide above, we can see some of the benefits and challenges of on-chip memory. On-chip memory is used today in solutions such as Microsoft’s Brainwave, Graphcore’s IPU, and Cerebras’ Wafer-Scale Engine. As we’ve previously noted, one of the biggest benefits of on-chip memory is the extremely high bandwidth that it offers.

Today, on-chip memories can deliver on the order of tens of terabytes a second in the largest chips that are about the size of a reticle. On-chip memory also offers extremely low latency and very high-power efficiency, because the data doesn’t need to move very far. The low latency and high bandwidth enable high utilization of the compute engines, even when processing jobs with small batch sizes, which can be particularly important for cloud-based infrastructures.

In terms of challenges and disadvantages, on-chip memory has lower storage capacity compared to external DRAM, although systems designers are implementing some interesting techniques to extend the capacity. For example, instead of saving intermediate results during more complex calculations, results are simply recalculated because it’s cheaper and faster to perform computation than to store and retrieve data from on-chip and external memory. In addition to limited capacity, on-chip memory is challenged by scalability, especially when it comes to large AI models.

Multiple FPGAs at datacenter scale can form a persistent DNN HW microservice, enabling scale out of models at ultra-low latencies

On the bottom left of the slide above, the Brainwave approach to system scalability is illustrated by the chaining together of multiple FPGA cards to solve and implement some of their larger networks.

HBM Benefits and Challenges

Let’s take a closer look at HBM now. One of the primary benefits of HBM is that it achieves very high bandwidth, the highest bandwidth for any external memory device today, up to 256 gigabytes a second of bandwidth per HBM2 DRAM stack. HBM also offers very good power efficiency because it’s running many I/Os – 1,024 – at a relatively low data rate of two gigabits per second. There are a number of solutions, especially in the AI space, that are adopting HBM2, including nVidia’s Tesla V100 and the second-generation Google TPU.

HBM implementations are challenged by the high number of I/Os which requires a silicon interposer to place the large number of data wires in close proximity to each other. There’s additional cost and design complexity with all the stacking involved, both within the HBM DRAM stack, as well as stacking the components together on top of the interposer and the substrate before it goes onto the PCB. The system engineering challenges associated with this stacking, and the complexity with keeping the silicon cool to maintain reliability, all add to the difficulty of implementing HBM2 solutions. Still, if it’s something that you can tolerate in terms of the cost and the design complexity, HBM2 can be a great choice for external bandwidth solutions.

GDDR6 Benefits and Challenges

GDDR offers a good middle ground between performance, complexity and cost. So, you can get very high bandwidth and capacity because it is an external type of DRAM that’s already packaged. It’s also easier to integrate, while the system engineering is much simpler compared to a memory type like HBM. GDDR is a great solution for systems that are particularly cost-sensitive or when HBM can’t be easily implemented.

The primary challenge associated with GDDR is running at very high data rates. Although signal integrity becomes more difficult, GDDR is implementable at these speeds with well-designed and well-managed PHYs, packages and boards.

On the lower left of the slide above, you can see a screen capture and an eye diagram that illustrates some of the work Rambus did several years ago in preparation for achieving GDDR6 data rates. This work proved the ability to signal at these high data rates across standard PCBs and included technology development to enable SoC PHYs to achieve these speeds.

Additional resources:
White Paper: HBM2E and GDDR6: Memory Solutions for AI
Webinar: Memory Systems for AI and Leading-Edge Applications
Webinar: GDDR6 and HBM2E: Memory Solutions for AI

Read more of this series:
Memory Systems for AI: Part 1
Memory Systems for AI: Part 2
Memory Systems for AI: Part 3
Memory Systems for AI: Part 4
Memory Systems for AI: Part 5

Written by Steven Woo for Rambus Press

In part four of this series, we took a closer look at the Roofline model, a modern computer architecture tool that illustrates how applications like artificial intelligence (AI) programs perform on different processor architectures like Google’s tensor processing unit (TPU), NVIDIA’s K80 GPU and Intel’s Haswell CPU. In this blog post, we’ll discuss common memory systems that are used in the highest performance AI applications.

The slide above illustrates the most common types of memory used in high-performance AI applications: on-chip memory, high bandwidth memory (HBM) and Graphics DDR SDRAM (GDDR SDRAM). On-chip memory, in the form of SRAM, is by far the highest bandwidth and most power-efficient solution available today. On-chip memory provides on the order of tens of terabytes a second of memory bandwidth and by far the best power efficiency. However, the tradeoff with on-chip memory is that capacity is low, totaling only a few hundred megabytes per reticle in the best implementations today. If you’re running smaller neural networks, or if you can split your neural network between multiple chips and they can work together, then on-chip memory is a great solution. But if you can’t, external memory must be used.

Currently, the two highest-performing external memories are HBM and GDDR. HBM (shown in the middle of the slide above) is a newer type of memory technology that involves newer stacking techniques both within the memory device, and between components. HBM devices also have many data wires running at a relatively low data rate. Specifically, each HBM stack has 1,024 data wires running at two gigabits per second. That’s more wires than can really be supported on a standard PCB. Therefore, the DRAM device is connected to the SoC using a silicon interposer, where those 1,024 wires can be more finely spaced and etched in the same way that wires are etched into an SoC.

These fine pitch wires etched into a silicon interposer allow us to have a very wide I/O interface running at a very low data rate – and still achieve a very high bandwidth simply through sheer numbers. Put simply, what you can achieve with HBM memory is very high bandwidth at very good power-efficiency. Although not quite as high as the bandwidth provided by on-chip memory, you get much higher memory capacity (8GB today, going higher in the future) in return. So, HBM offers the best bandwidth and power-efficiency for any external memory option that’s available. HBM is used today in solutions from various graphics card manufacturers and some AI chip vendors.

On the right is GDDR memory, which was originally created for the graphics market some 20 years ago. GDDR has undergone several major evolutions, with the latest generation (GDDR6) offering a great trade-off between bandwidth and power efficiency, cost and reliability. GDDR DRAMs rely on the same high-volume manufacturing and assembly techniques used with DDR DRAMs, leveraging this infrastructure and offering a familiarity that reduces cost and implementation difficulty. GDDR DRAMs have 32 data wires (fewer than HBM DRAMs), but run them at much higher data rates, up to 16Gbps in the current GDDR6 standard. The main challenge with GDDR memory systems is the higher data rate, which makes signal integrity and power-efficiency more challenging. Even so, GDDR offers a nice tradeoff between performance, power-efficiency, capacity, cost, and reliability.

Read more of this series:
Memory Systems for AI: Part 1
Memory Systems for AI: Part 2
Memory Systems for AI: Part 3
Memory Systems for AI: Part 4

Written by Steven Woo for Rambus Press

In part three of this series, we discussed how a Roofline model can help system designers better understand if the performance of applications running on specific processors is limited more by compute resources, or by memory bandwidth. Rooflines are particularly useful when analyzing machine learning applications like neural networks running on artificial intelligence (AI) processors. In this blog post, we’ll be taking a closer look at a Roofline model that illustrates how AI applications perform on Google’s tensor processing unit (TPU), NVIDIA’s K80 GPU and Intel’s Haswell CPU.

The graph above is featured in a paper published by Google a couple of years ago detailing the first-generation Tensor Processing Unit (TPU). It’s a very insightful paper, because it compares the performance of Google’s TPU against two other processors. You can see three different Rooflines in the graph above: one in red, one in gold and one in blue. The blue Roofline represents the Google TPU, a special purpose-built piece of silicon that was specifically designed for AI inferencing. The NVIDIA K80 – a GPU designed to handle a larger class of operations – is in red. Represented in gold is the Roofline for the Intel Haswell CPU, a very general-purpose processor.

Source: N.Jouppi, et.al., “In-Datacenter Performance Analysis of a Tensor Processing Unit^TM,” https://arxiv.org/ftp/arxiv/papers/1704/1704.04760.pdf

In the Google paper, 6 neural networks were coded for each of the different architectures. The stars show the architectures of neural networks that were coded for the Google TPU. The triangles are the same neural network architectures coded for the NVIDIA K80, while the circles are the same neural network architectures coded for the Intel Haswell processor. When you look at the neural networks running on the Intel Haswell and the NVIDIA K80, you will observe that many of the circles and triangles are under the flat part of the respective Rooflines – and many of them are under the slanted part of these Rooflines. This means some of the neural networks are limited by memory bandwidth, while others are more limited by compute performance for these processor architectures.

However, when you look at the stars you see the majority lie on or under the sloped part of the Roofline for the Google TPU. This placement tells us that memory bandwidth is a critical bottleneck for machine learning applications running on these purpose-built AI processors. In fact, it’s part of the reason why Google used HBM (the highest bandwidth discrete DRAM memory available at the time) in future versions of the TPU. So, we can certainly infer that memory bandwidth is critical for AI silicon and applications.

Since memory is an important resource for AI chips, it is essential to make the best use of available bandwidth. Indeed, traditional scientific calculations are frequently executed as 32-bit and 64-bit floating point numbers. However, modern research has shown this level of precision isn’t needed to achieve good performance for machine learning applications. It turns out that many architectures use reduced precision, so instead of executing all calculations with 32-bit numbers, sometimes 16-bit, 8-bit, or even fewer bits are used. This approach of utilizing “reduced precision” numbers has the benefit of maximizing available bandwidth, because you can squeeze in more numbers, as each number now takes fewer bits to encode.

Written by Steven Woo for Rambus Press

In part two of this series, we took a closer look at how the upcoming deployment of 5G technology will enable processing at the edge, and how the industry is further refining the edge into the near edge and the far edge. The near edge is closer to the cloud, while the far edge is closer to the endpoints. As we noted, we expect to see a full range of AI solutions spanning the near and far edge. Specifically, at the near edge, closest to the cloud, AI solutions and memory systems will likely resemble what is seen in cloud data centers including on-chip memory, HBM and GDDR. At the far edge, AI memory solutions will likely be similar to those deployed in endpoint devices, including on-chip memory, LPDDR, and perhaps even DDR.

In this blog post, we’ll explore how to determine if specific AI architectures are limited by either their compute performance or memory bandwidth by introducing the Roofline model. Put simply, the Roofline model illustrates how an application performs on a given processor architecture by plotting performance (operations per second) on the y-axis  against the amount of data reuse (also called “operational intensity”) on the x-axis.

The operational intensity of an application on a particular processor is a measure of how many times each piece of data is reused for computations once it’s retrieved from the memory system. If an application has a high operational intensity, it means that that data is reused many, many times in various calculations once it’s retrieved from the memory system. Applications with high operational intensity put less stress on the memory system, because data can be reused often. In contrast, applications with low operational intensity can be bottlenecked by the memory systems of the processors they are running on, because they demand much more memory bandwidth to achieve high performance.

The Roofline is composed of two different line segments and is unique for each processor architecture. The first segment is a horizontal line which represents the peak performance of the processors. If every compute unit is running continuously at full speed, this is a measure of the peak performance that you can’t exceed. The other part of the Roofline is a slanted part of the line which shows when the architecture is limited by the amount of memory bandwidth that you can provide. If there isn’t enough memory bandwidth provided by the processor, then the architecture may be bottlenecked, preventing the compute units from running at peak performance as they wait for data from the memory system. As you can see in the image above, the Roofline itself is colored in solid green and is composed from a portion of the sloped line, as well as a part of the horizontal line.

Each processor architecture has its own Roofline model, because each has different peak compute performance and memory system bandwidths that they can provide. We can plot different applications against a Roofline curve to gain a better understanding for a specific architecture whether that application is limited more by memory bandwidth – or limited more by the peak compute performance of the processor. In this particular chart, application number one lies more underneath the slanted part of the line. This means the operational intensity, or the number of times each piece of data is reused in that application, is low enough that we can’t achieve peak performance. If more bandwidth was provided, we could move closer to peak performance (the horizontal part of the Roofline).

On the right is application number three, which lies underneath the flat part of the curve. This means application number three has enough memory bandwidth – and is limited more by available compute resources in that processor. If we had more compute resources (for example, more adders and multipliers) or could run them faster, then we could potentially achieve a higher level of performance.

Application number two is near where both the horizontal and sloped parts of the Roofline meet. This means that this application is partially limited by the peak performance of  the computational resources of the processor and partially limited by memory bandwidth. Application two could benefit if additional computational resources and memory bandwidth became available.

On final point should be made about Roofline. The corner in the Roofline image above is known as the ridge point. This represents the minimum amount of reuse – the minimum operational intensity needed to achieve the maximum processor performance. This is an important point, because it helps us understand how algorithms can be arranged to achieve peak performance for applications.