Memory PHYs

Written by Steven Woo

As we discussed in our previous blog post, there is a sense of growing concern in the high-performance computing (HPC) space over successive generations of supercomputers that have continued to move further and further away from architectural balance between compute and memory resources. This is because compute performance has improved at significantly faster rates than memory and I/O subsystems.

Amdahl’s Rule of Thumb

Gene Amdahl, who gained fame for his seminal work on diminishing returns that became known as Amdahl’s Law, formulated a lesser known second principle that is sometimes referred to as ‘Amdahl’s Rule of Thumb’ or ‘Amdahl’s Other Law.’ This second principle addresses the importance of architectural balance and stipulates a 1:1:1 ratio of FLOPs to Memory Bandwidth to IO bandwidth. With an ideal target of one byte of memory needed per FLOP (matched by one byte of transfer over an external interface), Amdahl believed that computer systems could remain in balance.

The move away from this 1:1:1 ratio, and hence from balanced architectures, has been an ongoing concern among high-performance computer system architects over the past two decades. In his keynote talk at Supercomputing 2016 (SC16), John McAlpin of the Texas Advanced Computing Center presented data illustrating how systems are becoming more unbalanced with each new generation.

Source: https://www.hpcwire.com/2016/11/07/mccalpin-traces-hpc-system-balance-trends/

Based on the period from 1990-2016, the trends shown in the image above illustrate the stark nature of this growing imbalance: the ratio of FLOPs to memory bandwidth is growing at a rate of 4.5X per decade, while the ratio of FLOPs to network bandwidth is increasing at a rate of 9X per decade.  Measured as a function of memory and network latency, the growth rates are even more severe: the ratio of FLOPs to memory latency is growing at a rate of more than 100X per decade, while the ratio of FLOPs to network latency is increasing at a rate of ~30X per decade.

Improving HPC Memory Hierarchies

With such astonishing rates of growth in architecture imbalance, the critical question has become how to feed the compute pipeline in the most economical, power-efficient manner to achieve the ExaFLOP performance targeted by next-generation supercomputers. Improved memory hierarchies and more power-efficient memory systems are the focus for addressing the growing imbalance relative to memory performance. Having fallen so far behind the ratios needed to maintain true balance, supercomputing memory systems could clearly benefit from any improvements in memory bandwidth and latency.

However, there is an industry-wide realization that simply keeping up with the historic trend of doubling memory bandwidth and improving power efficiency every generation will be even more challenging going forward. Clearly, new approaches are needed to improve memory hierarchies that move in the direction of restoring balance to high performance computing system architectures. The increased use of HBM memory in the HPC space is one such potential approach that could help address the demand for more memory system performance with improved power efficiency. While HBM memory is more challenging to design into systems than other memories such as DDR and GDDR, the benefits for HPC systems are so compelling that it is being introduced into future high-performance computing systems in Japan.

More specifically, the Post-K Supercomputer will be comprised of nodes powered by Fujitsu’s A64FX CPU connected to four HBM DRAMs serving as the processor main memory. The NEC SX-Aurora Tsubasa platform provides a scalable computing solution capable of supercomputer-class performance using the company’s new Vector Engine card. The A500-64 supercomputer, built around this platform, uses 64 Vector Engines that include a high-performance vector processor connected to six HBM DRAMs that provide unprecedented levels of memory bandwidth.

HPC Cores and Clock Frequencies

Although next-generation HPC processor architectures have yet to be finalized, preliminary investigations conducted in research under the US’s next-generation supercomputer program indicate that while the number of processing cores on a chip are expected to increase, the clock frequencies of the cores may potentially decrease to conserve power. Concurrency at the level of billion-way parallelism is also a possibility – and application-level parallelism will be critical to achieving Exascale performance on these machines. At this scale, reliability and resiliency will also be critical. HPC applications may run for weeks or months on these machines, so silent errors that propagate through iterative computations will render the machines useless if not identified in a timely manner. Consequently, methods for improving memory and link reliability and resilience will only grow in importance.

Written by Steven Woo

As we’ve discussed in previous Rambus blog posts, high-performance computing (HPC) memory bandwidth and capacity are continuing to fall further and further behind the performance of compute engines. Consequently, memory and I/O subsystems are becoming increasingly larger bottlenecks that must be carefully managed by applications and operating systems. There is also an increasing emphasis on improving HPC power efficiency, which in turn, is driving the demand for more power-efficient memory systems that achieve higher memory bandwidths and capacities – without increasing power.

Created by Horst Simon in a widely-recognized presentation about the difficulties and challenges with achieving Exascale computing targets, the image below illustrates the energy trends of computation and data movement. As of 2013, FLOPs were still more expensive than on-chip data movement in terms of energy per bit.

However, the projection for 2018 (at that time) was for the energy per bit of on-chip data movement to exceed that of computation. In 2013 (and projections for 2018), the energy per bit of off-chip/DRAM communication and long-distance data movement (local interconnects, longer-distance networking) exceeds that of computation by at least an order of magnitude.

Reassessing the HPC Memory Paradigm

The current memory hierarchy paradigm that supports massively parallel systems has become a major concern for the HPC sector. As we noted above, there is an increasing emphasis on improving HPC power efficiency, which in turn, is driving the demand for higher memory bandwidths and capacities without increasing power. The desire to increase memory capacity per processor core will put additional pressure on power budgets, which could in turn impact the way in which applications and architectures scale in the future. Scaling will be further strained as data movement continues to become a limiter in terms of power and performance. Moving forward, HPC architectures will need to minimize data movement, while also improving the power efficiency and performance of memory interfaces and chip-to-chip links.

HPC and HBM

Coupled with the high cost of power associated with data movement, supercomputer system architects are looking at new solutions for addressing the need for more memory system performance with improved power efficiency. Although HBM memory is more challenging to design into systems than other memories such as DDR and GDDR, it is being introduced into future high-performance computing systems in Japan to address the critical memory system needs of performance and power-efficiency.

For example, the Post-K Supercomputer will be comprised of nodes powered by Fujitsu’s A64FX CPU connected to four HBM DRAMs serving as the processor main memory. The NEC SX-Aurora Tsubasa platform provides a scalable computing solution capable of supercomputer-class performance using the company’s new Vector Engine card. The A500-64 supercomputer, built around this platform, uses 64 Vector Engines that include a high-performance vector processor connected to six HBM DRAMs that provide unprecedented levels of memory bandwidth. These systems illustrate that there are compelling advantages to using HBM in high-performance computing applications.

Written by Steven Woo

Dominated by the United States, Japan and China, the high-performance computing (HPC) space is driven by an insatiable demand for ever-higher performance and greater power efficiency. With each new supercomputer debut, the above-mentioned trio sets progressively higher bars with the goal of capturing the highest Top500 score.

Summit, Sierra and Sunway TaihuLight

In June 2018, the U.S. recaptured the Top500 performance crown with the launch of Oak Ridge National Laboratory’s Summit supercomputer which achieved 122.3 PFLOPs on the High Performance Linpack (HPL) benchmark. In November 2018, Summit widened its lead as the number one system, bolstering its HPL performance from 122.3 to 143.5 petaflops. Summit, an IBM AC922 system, packs more than three petabytes of DRAM memory and links over 27,000 Nvidia Volta GPUs with more than 9,000 IBM Power9 CPUs.

In addition to Summit clinching the number one spot, November 2018 saw the U.S. designed Sierra supercomputer moving from 71.6 to 94.6 petaflops, bumping it from the number three position to number two. Sierra is equipped with 1.38 petabytes of DRAM memory, 8,640 CPUs and 17,280 Nvidia Tesla V100 Tensor Core GPUs. At third place (as of November 2018) is China’s Sunway TaihuLight supercomputer, installed at the National Supercomputing Center in Wuxi. TaihuLight had previously held the top position on the Top500 list for two years, with an impressive HPL performance of 93.0 petaflops. Developed by China’s National Research Center of Parallel Computer Engineering & Technology (NRCPC), TaihuLight packs Sunway SW26010 260C 1.45GHz processors, 10,649,600 cores and 1,310,720 GB of memory.

The Exascale Supercomputer Program

Summit and Sierra are moving towards the U.S. Department of Energy’s (DoE) goal of developing an Exascale supercomputer capable of achieving 1 ExaFLOP performance within a power budget of 20-40 MegaWatts by 2020, an approximate 1000X performance increase over the previous generation of supercomputers. The US DoE’s Exascale program, launched several years ago to achieve the next level of supercomputing performance has been exploring new architectures and technologies to achieve these goals. With Moore’s Law slowing and Dennard scaling finished, at these levels of performance system power efficiency is a primary concern in the Exascale program. More specifically, a 5X-10X improvement in power efficiency is required compared to the previous generation of supercomputers, meaning that simply scaling up the previous generation’s architectures isn’t a viable option.

GPUs and Memory for HPC

Over the last 10 years, U.S., Chinese, and Japanese supercomputers have adopted compute accelerators based on graphics processor units (GPUs) to bolster performance, power efficiency and compute density (performance in a volume of space). Since Moore’s Law has provided a growing number of transistors over the years, an increasing number of compute pipelines offered by accelerators must be matched with memory bandwidth to feed them. Both HBM and GDDR have been employed by accelerators in the current generation of supercomputers, with Summit’s Tesla V100s using HBM2 and Stampede-2’s Xeon Phi accelerators packing GDDR5.

HPC Memory Bottlenecks

The design of more efficient memory hierarchies for the HPC sector is critical and only growing in importance, as many HPC applications are designed to manipulate large data sets. However, after decades of compute performance improving faster than memory and I/O subsystems, the supercomputing market now finds itself severely limited by bottlenecked memory, I/O performance and stagnating power-efficiency. Minimizing data movement and moving compute closer to where the data resides are two methods for dealing with this massive imbalance.

The term ‘neuromorphic computing’ can be traced back to the 1980s when Caltech researcher Carver Mead first proposed the concept of designing integrated circuits (ICs) to mimic the organization of living neuron cells. As the National Institute of Standards (NIST) states, neuromorphic computing promises to dramatically improve the efficiency of computational tasks such as perception and decision making.

Nanoscale oscillators and reconfigurable Josephson junctions

Although software and specialized hardware implementations of neural networks have made significant progress in recent years, such implementations are still are still ‘many orders of magnitude’ less energy efficient than the human brain. This is precisely why NIST researchers are working on several bio-inspired hardware implementations of neuromorphic networks. More specifically, one project is based on high-frequency room-temperature nanoscale oscillators utilizing the spin-torque effect, while the other is designed around dynamically reconfigurable magnetic Josephson junctions operating at liquid-helium temperatures.

Neuromorphic silicon: Loihi and Akida

On the silicon side, Intel introduced a self-learning neuromorphic chip dubbed ‘Loihi.’ The chip mimics
how the human brain functions by learning to operate based on various modes of feedback from its environment. The energy-efficient chip – which uses data to learn and make inferences – becomes smarter over time, without the need for traditional training methodologies. Rather, Loihi takes a novel approach to computing via asynchronous spiking.

Similarly, BrainChip developed the Akida NSoC, which the company describes as a neuromorphic chip engineered on a digital logic process. Inspired by the biological function of neurons, these spiking neural networks (SNNs) are inherently lower power than traditional convolutional neural networks (CNNs). Indeed, SNNs replace the math-intensive convolutions and back-propagation training methods of CNNs with neuron functions and feed-forward training methodologies.

IBM’s True North neuromorphic chip

A patent filed by IBM in late 2018 offers a rare insight into the company’s True North neuromorphic chip. According to Nicole Hemsoth of The Next Platform, one of the elements making IBM’s neuromorphic device more efficient and scalable is a new algorithm that determines placement, or how a neurosynaptic core in True North is mapped onto the chip (and even across chips). In addition, the patent details how the new device supports standard deep learning frameworks that can be used and converted during offline training. As Hemsoth explains, the trained networks are converted into corelets after training and subsequently converted into model files that can be loaded onto specific neurosynaptic substrates.

Nengo and algorithms

On the software side, Applied Brain Research developed Nengo, an advanced tool that allows programmers to more easily run AI algorithms on new cortical chips. According to Wired, the Python-based compiler is targeted at AI applications that will run on general purpose neuromorphic hardware. Nengo has already been used to develop vision systems, speech systems, motion control and adaptive robotic controllers.

Memory for neuromorphic systems

As a recent article in Advanced Material Technologies emphasizes, artificial neurons should closely mimic biological neurons, while artificial synapses should accurately emulate biological synapses. Both artificial implementations must also be power‐efficient, scalable and capable of implementing relevant learning rules to facilitate large‐scale neuromorphic functions. Thus far, a wide range of memory devices have been utilized to realize the goal of creating efficient artificial synapses. These include resistive random‐access memory (ReRAM), diffusive memristors, phase change memory (PCM), ferroelectric field‐effect transistors (FeFET), spintronics-based MRAM and synaptic transistors.

“Each of these devices has its own strengths and weaknesses. One type of device can be preferred over the others depending on the requirements for a specific application,” the article states. “Although attempts have been made to implement logic circuits using emerging devices, however, at least in the near future, neuromorphic systems still need the mature and reliable CMOS circuitry to implement peripheral components, which means neuromorphic systems and CMOS circuits will complement rather than replace each other.”

Interested in learning more about various types of memory? You can browse our article archive on the subject here.

5G is more memory hungry compared to previous generations, according to Gary Hilson, memory editor at EE Times.  In his recent article, he pulled together an industry consensus about the types of memory that might be considered to comply with 5G demands, the pros and the cons.

Hilson writes that it makes sense when you think about how much computing power people are carrying around in their hands compared to even the early days of the Blackberry. Mobile networks are just as much about transmitting 4K video as they are talk and text.

He adds, “Connected devices not only include smartphones, but sensors, parking meters, smart cars, wearables, and utilities. Telecom infrastructure is now networking and compute infrastructure—flash and DRAM are supplanting SRAM and TCAM, and there might be room for emerging memories, too.”

Hilson explains that a high speed TCAM (ternary content-addressable memory) can search its entire contents in a single clock cycle and is faster than RAM. It’s a mainstay of networking gear, such as high-performance routers and switches, to increase the speed of route look-up, packet classification, packet forwarding and access control list-based commands.

However, there are downsides to TCAM. Hilson quotes Jim Handy, principal at Objective Analysis, as saying, “despite its longevity, it is ‘exotic’ in that it’s quite expensive and there are limited suppliers. But there’s solid payback from using them. They streamline the routers. They make them far faster with less other processing hardware.”

Hilson also writes about resistive-RAM (ReRam) and MRAM as possible contenders to mate up with 5G requirements.  But as he notes, RRAM-based TCAM circuits can match the performance of CMOS-based SRAM circuits for multicore neuromorphic processor applications despite the performance and reliability tradeoffs.  However, the issue with both RRAM and the emerging magneto-resistive random-access memory (MRAM) is temperature.  Neither can handle high heat.

Hilson again quotes Handy saying, “Embedded MRAM has made some inroads.  (However,) the telephone companies want both high and low temperature stuff.”

Another viewpoint in Hilson’s article comes from Virtium vice president Hiep Pham, who oversees the company’s networking endeavors. For the immediate future, the transition from 4G means continuing to support legacy equipment. “The goal for 5G is basically to have a faster speed, a high capacity, and low latency,” he said. “But 5G will go together with some applications.”

“This is where the lines get blurry,” Hilson reports.  “The evolution from a telecom network to a technology platform means it’s not just about a better voice and data experience for handsets, but support for a wide range of edge computing and IoT scenarios, including industrial automation, as well as autonomous vehicles and smart cities.”

He states that Pham believes the amount of artificial intelligence (AI) in the network will increase, starting in data centers but also distributed through the network, right to the edge. “The edge will have to handle some of the AI, even though it goes through 4G or 5G.” He said there will remain bandwidth limitations, so the edge must be capable of storing and process information.

Lastly, Ryan Baxter, senior director for data center at Micron, rounds out Hilson’s piece by saying, “The distributed nature of a 5G platform means there’s been a proliferation of different memory technologies being considered and not just standard DDR3 and DDR4.”

Baxter is quoted as saying that low power double-data rate (LPDDR) and graphics double-date rate (GDDR)-based technology are being looked at and the requirements of such broad spectrum are driving a fair amount of fragmentation because 5G presents itself as a blend of telecom and computing networking.

“We’re seeing significantly larger memory footprints required to essentially support the compute elements of the 5G deployments,” he says in Hilson’s article.