Rambus

At Rambus, we create cutting-edge semiconductor and IP products, spanning memory and interfaces to security, smart sensors and lighting.

Home > Memory PHYs

Memory PHYs

Below is an archive of the blog posts related to Memory PHYs. To learn more about our memory interface IP solutions, click here.

Reduced-Precision Computation for Neural Network Training

by Leave a Comment

In our previous blog post, we discussed the process of training neural networks (NN) and briefly touched on NN training platforms and related memory bandwidth issues. As we noted, neural network training and inference performance are heavily contingent upon memory bandwidth. This is because the memory system is typically tasked with holding the neural network parameters – weights and biases – along with training data.

Stock image of brain and networks

With hardware optimized for performing fast computations, neural networks constantly stress the memory system as model parameters and data are fetched from memory. More specifically, moving data to and from the compute engines strains both memory capacity and bandwidth, as the training sets and model parameters are held in local memory to avoid data transfers over slower PCIe interconnects.

Reduced-Precision Computation for Neural Networks

One popular method of reducing memory bandwidth demand and increasing power-efficiency for neural networks focuses on reduced-precision computation. Instead of computing with 32-bit floating point numbers, many implementations are now using 16, 8, and in some cases fewer bits of precision. This is because neural networks are tolerant of reduced precision and the power saved by avoiding computing on the least significant digits is often considerable.

Indeed, NVIDIA has discussed the benefits of using 16-bit operands and 32-bit accumulates for neural network training. Neural network parameters can be reduced in precision to 8 bits after (or during) training. While this approach may reduce the accuracy of inferencing, networks can be further trained to regain the classification accuracy lost when reducing the precision of network parameters. Microsoft also discussed the benefits of reduced precision computation when introducing Brainwave, the company’s machine learning infrastructure, at Hot Chips 2017.

In addition to using 16-bit operands and 32-bit accumulates for neural network training, numerous other methods have been discussed in various industry and academic research papers to better utilize memory bandwidth and to improve power-efficiency, including pruning weights that are close to 0 and quantizing weights to reduce storage overhead.

New Formats for Reduced Precision Computation

Reduced precision computation has garnered significant industry interest over the past few years, with a range of new formats now supported in hardware for AI and related software frameworks. For example, bfloat16 is perhaps the most popular new format that is supported by Intel’s Nervana AI processor, Xeon processors and FPGAs, as well as Google’s TPUs and TensorFlow framework. Compared to the IEEE 754 single-precision floating point format, bfloat16 has the same number of exponent bits and covers the same numerical range (~1e-38 to ~3e38), but at lower numerical precision. The similarity between the two also allows fast conversion between the two formats.

Extreme Reduced Precision Computing

Xnor.ai is a company exploring extreme reduced precision computation by using single-bit model parameters. Essentially, the company has reworked training and inference algorithms so that interference operations are reduced to single-bit xnor operations. By doing so, these computations can be performed natively in hardware on almost any processor at high speed and with high power-efficiency, including the Raspberry Pi and smartphone CPUs. Among the many demos they have shown to date, the most impressive is real-time object recognition running locally on an iPhone.

Conclusion

Neural network training applications can more effectively utilize memory bandwidth by packing more numbers into each set of bytes – instead of defaulting to full-precision computation operations. In turn, these savings can be leveraged to implement additional neurons or reduce the total cost of operation (TCO). With the benefits of reduced precision computation being realized in modern AI systems, the industry must now turn its attention to additional innovations in order to  continue fueling improvements in future AI systems.

Interested in reading more about machine learning and neural networks? You can browse our article archive on the subject here.

Understanding ML and ANN memory requirements

by Leave a Comment

Written by Steven Woo

Artificial Neural Networks

First proposed in 1944 by Warren McCullough and Walter Pitts, an artificial neural network (ANN) or more commonly, a neural network (NN), can perhaps best be defined as a computational model that attempts to closely emulate the network of neurons present in the human brain. More specifically, neuromorphic computing aspires to replicate the results that biological brains can achieve using analogous – albeit artificial – mechanisms.

Unlike traditional von Neumann systems, neural networks are not limited by conventional bottlenecks between program memory, data memory and the CPU. As Dan Kara, research director at ABI Research notes, standard CPUs will continue to play an important role in data centers for the foreseeable future. However, they will be supported by an increasing number of specialized non-von-Neumann platforms, as well as graphics processing units (GPUs), custom application-specific integrated circuits (ASICs) and field-programmable gate arrays (FPGAs).

Image representing neural and connected networks

Machine Learning

Creation of the term ‘machine learning’ (ML) is widely credited to Arthur Samuel, an artificial intelligence (AI) pioneer. From 1949 through the late 1960s, Samuel focused on enabling computers to learn from their experience. One of Samuel’s most famous learning programs used Lee’s Guide to Checkers to adjust its moves and optimize gameplay. This is because Samuel viewed machine learning as a form of AI that enables a system to learn from raw data rather than explicit programming. The field of machine learning has significantly expanded in the 21st century and now spans multiple sub-categories including supervised learning, unsupervised learning, reinforcement learning and deep learning.

AI: 2019 & Beyond

Neural networks and machine learning are currently experiencing an exciting renaissance – with a resurgence of interest across the semiconductor industry, academia and the media. Driven by Moore’s Law, compute performance and transistor densities have improved by several orders of magnitude since the last wave of machine learning advances in 1980s. This has enabled neural networks and machine learning to flourish across a wide range of industries.

Numerous chips and cores are now being designed and deployed to implement neural network training and inference. Analysts estimate there are at least 50 neural network processors and 10 neural network IP cores under development or in production by internet giants, traditional semiconductor manufacturers and numerous start-up companies. These specialized chips and cores are targeted at a diverse set of use cases such as data centers, mobile processors and edge processing systems.

NN, ML & Memory Requirements

Neural networks require ultra-high bandwidth capabilities and power-efficiency for inference and training. Moreover, extensive data training sets often demand large memory capacities, especially for data center applications. To fulfill these requirements, some companies are leveraging on-chip memory, while others are using HBM2 or GDDR6. Although on-chip memory provides the highest bandwidth and power efficiency, this paradigm sacrifices capacity to achieve the former.

HBM2 memory systems offer NN and ML engineers the advantages of the highest bandwidth per device, as well as optimal power efficiency for high performance memory systems. The key challenges for HBM2 memory systems are the high cost of the memory device, the additional cost of using a silicon interposer and the inherent difficulty of interposer-based system designs.

Meanwhile, GDDR offers NN and ML engineers high bandwidth as well as high capacity. Like previous GDDR generations, GDDR6 DRAMs utilize single-ended signaling and die-down implementations to provide the cleanest signaling environments for the highest per-pin signaling rates. The primary advantages of GDDR6 are compatibility with existing package and board infrastructure, as well as the relatively lower cost of DRAMs and packages. The critical challenges for GDDR6 memory systems are maintaining signal integrity and achieving reasonable power efficiencies.

Conclusion

Driven by Moore’s Law, neural networks and machine learning are currently experiencing an exciting renaissance. Both require ultra-high bandwidth capabilities, large capacities and power-efficiency for inference and training. To fulfill these requirements, some companies have chosen on-chip memory, while others are using HBM2 or GDDR6. Each standard offers its own advantages and disadvantages, which are explored in detail above.

GDDR6 Crosstalk Article – Rambus

by Leave a Comment

Director of Signal Integrity Engineering at Rambus, Shawn Nikoukary, recently published a piece in eeweb.com/EE Times about crosstalk (xtalk) as being the most challenging signal integrity (SI) issue associated with GDDR6.

image of board

Nikoukary says, “the main reason is that the circuits used to remove crosstalk from signals are complicated and require a large power/area overhead to be effective. There are several types of active and passive crosstalk cancellation methods, such as passive mutual capacitive couplings and crossed continuous time linear equalization (CTLE) and crossed decision feedback equalization (DFE) circuits.”

But he reminds SoC and system designers to keep in mind that it is much easier to design a channel that is low in crosstalk, than trying to correct it in the receiver.

It’s important to have a good handle on and fully understand crosstalk in order to design a robust GDDR6 interface, he emphasizes.  “Crosstalk is due to capacitive and inductive coupling between two signals and it is categorized as Near End and Far End XTalk (NEXT/FEXT), Nikoukary explains.

Further, he adds it is highly recommended that all data lines (DQ signals) running at higher data rates be routed as a strip line. The largest contributor of the crosstalk in GDDR6 channel is from the vertical transitions such as vias, plated through holes (PTHs) and BGAs which are placed near each other.  Great care needs to be taken to shield vias and design a BGA that will have a low crosstalk pin assignment.

“The majority of the crosstalk in GDDR6 systems results from the ball arrangement and mis-assignment in a BGA package,” according to Nikoukary. “This has a major impact on crosstalk and is especially true for certain applications where the PTHs are placed under the BGA.  For example, networking systems tend to have large PCB stack-up, very thick PCBs that utilize through vias.  These long through vias placed under the BGA cause large amounts of coupling and crosstalk if designed properly.”

Summarizing his key points, Nikoukary explains that “ideally, to eliminate this crosstalk, a very low crosstalk BGA assignment for the package is essential.  In a good BGA ball map, ground and power shielding is placed between signals or blind and buried vias are used in the dense area under the BGA to minimize crosstalk.”

 

Uber eyes an AI future

by Leave a Comment

Rick Merritt of the EE Times recently reported that Uber is using banks of Nvidia GPUs for deep learning applications. The transportation network company (TNC) is also testing FPGAs from multiple startups such as Eyeris, Graphcore and Wave Computing. In addition, Uber maintains a dedicated AI research team that oversees more than a dozen deep-learning models including recommendation engines for Uber Eats, fraud detection services and estimated driver arrival times.

Stock image of chip

“The algorithms span a half-dozen varieties, implemented across a laundry list of mainly open-source frameworks and libraries,” writes Merritt. “The underlying AI hardware today consumes as much as 40 kW in a rack of systems — twice the power that standard servers use — and can require flows of more than 100 petabytes of data.”

According to Merritt, Uber has begun designing its own compute and storage servers, as well as a variant of a 19-inch rack to house them along with vertically mounted switches. More specifically, Uber’s “super-hot” storage server utilizes two Intel Cascade Lake processors, with each 1U system packing more than 16 TB of NAND flash. Meanwhile, Uber’s warm storage system is built around Broadwell processors, with up to 70 8-TB SATA hard-disk drives in a 4U design that puts 6 petabytes in a rack.

“[Uber’s] systems have served more than 10 billion Uber rides to date at a rate of about 15 million a day across 600 cities in 65 countries,” he adds.

Commenting on the above, Steven Woo, Fellow and Distinguished Inventor in Rambus Labs, told Rambus Pressthat neural networks, machine learning and autonomous vehicle processors typically demand silicon with the highest memory bandwidths and power-efficiencies for inference and training. The latter, says Woo, can also require large memory capacities, especially for data center applications.

“Some companies use on-chip memory to power their silicon, while others deploy HBM2 or GDDR6,” he explains. “On-chip memory provides the highest bandwidths and power efficiencies but sacrifices capacity to achieve this.”

HBM2, says Woo, is being used in many high-end designs today, as the memory standard provides high bandwidth and is power efficient — although it comes with higher costs and is more complex to design with. As noted above, GDDR6 is also seeing interest, with the memory standard providing high bandwidth as well as high capacity. However, GDDR6 has higher power and challenging signal integrity constraints.

“As with other markets that use high-performance memories, the minimum expectation is to at least follow the historic trend of doubling memory bandwidth and improving power efficiency with each succeeding DRAM generation,” he adds.

Interested in learning more about AI and machine learning? You can check out our article archive on the subject here.

IDC White Paper Lays Out Future of Memories and Interfaces

by Leave a Comment

A recently published IDC white paper calls attention to the growing significance of chip connectivity.  Entitled, “Hidden Signals: The Memories and Interfaces Enabling IoT, 5G, and AI,” the paper states that data created by electronic systems will grow 10x to 103 zettabytes between 2014 to 2023.  Moreover, it says the average server will support over 670 gigabytes of DRAM in 2023.

IDC Report Blog Image

The underpinnings to this market growth are not only interfaces or chip connectivity and where data is stored in real-time, but also the connectivity between systems and processors computing the data, according to IDC.  Technologies critical in this scenario include GDDR6, HBM, and memory buffers. These are the components that move, accelerate, and store data and enable IoT, 5G, and AI.  These are the trends driving new markets for the next 10 years, according to the IDC white paper.

The 10-page paper provides the reader a comprehensive market analysis of various product classifications, a historical perspective and market growths per category, trends, and detailed definitions.

As for memory and interface technology and associated trends, IDC explains that interfacesbetween high-performance data processors have been evolving because the market has recognized that monolithic, homogeneous microprocessors are insufficient to process and secure the amount and disparity of data flooding into high-performance systems.

The paper adds that CPUs, GPUs, FPGAs, and ASICs share a need for external memories—such as CPUs for main memory and GPUs for frame buffers—that hold data close to the processor. These needs thus put specific pressure on the interfaces between these processors and their memories. Thus, interfaces and memories are inextricably intertwined and the two are critical to making the often-hidden data signals flow efficiently.

IDC also explains that systems like servers will need increasing amounts of DRAM. However, such increasing capacities means increasing the burden on the system microprocessor to manage that memory across the memory bus. Memory buffer chips reduce that burden by bringing local intelligence to the memory on the memory modules.

Buffer chips on Load-Reduced DIMMs (LR-DIMMs), for example, reduce the load on the microprocessor’s memory controller and across the memory bus by handling all data, command, and control signals sent to memory and handling all reads and writes to the DRAM chips.

All in all, the IDC paper paints a favorable landscape and market outlook for IoT, 54G, and AI based on today’s Rambus memory and interface technologies.  GDDR6 PHYs, 32 gigabit per second (Gbps) SerDes for 5G, 112Gbps ADC/DSP-based SerDes for next generation servers, and DDR5 buffer chips are leading the charge to further advance those product categories.

It’s definitely a good read for any and all who follow or want to learn more about these industry advances and the technologies driving them. Get a hold of a copy of this IDC White Paper right now.

 

HBM2 gets an upgrade as semiconductor industry eyes HBM3

by Leave a Comment

JEDEC recently updated its JESD235 High Bandwidth Memory (HBM) DRAM standard. As we’ve previously discussed on Rambus Press, HBM DRAM supports a wide range of use cases and verticals including graphics (GPUs), high performance computing (HPC), servers, networking and client applications. Indeed, HBM is particularly suitable for hardware that demands peak bandwidth, bandwidth per watt and capacity per area.

HBM2 gets an upgrade as semiconductor industry eyes HBM3

Densities up to 24GB, speeds hit 307 GB/s

According to JEDEC, the updated HBM standard leverages wide I/O and TSV technologies to support densities up to 24 GB per device – at speeds up to 307 GB/s. This bandwidth is delivered across a 1024-bit wide device interface that is divided into 8 independent channels on each DRAM stack. In practical terms, this means the standard can support 2-high, 4-high, 8-high, and 12-high TSV stacks of DRAM at full bandwidth, thereby enabling system flexibility for capacity requirements from 1 GB – 24 GB per stack.

Per pin bandwidth hits 2.4 Gbps

The updated HBM standard also extends per pin bandwidth to 2.4 Gbps, adds a new footprint option to accommodate the 16 Gb-layer and 12-high configurations for higher density components, while refreshing the MISR polynomial options to support the new configurations. As JEDEC notes, the HBM standard was developed and updated with support from leading GPU and CPU developers to extend the system bandwidth growth curve beyond levels supported by traditional discrete packaged memory.

From 8 Gb-per-layer to 16 Gb-per-layer

“This update was really focused on extending the support and the design from an 8 Gb-per-layer definition to a 16 Gb-per-layer,” Barry Wagner, HBM task group chair for JEDEC, told the EE Times. “A lot of the demand for the high capacity is driven by very large data-set–type applications for high-performance computing.”

Wagner also told the publication that the recent update would add density to HBM2 before HBM3 rolled out to power a new generation of applications and devices. To be sure, work is already underway to develop the new HBM3 standard, with an emphasis on increasing bandwidth and density, as well as improving performance per watt.

Let’s talk about HBM

High-bandwidth memory (HBM) enables system design with lower power consumption per I/O, as well as higher bandwidth memory access in a condensed form factor. This paradigm is achieved by stacking memory dies on top of each other, while sharing the same package as an SoC or GPU core using a silicon interposer. Essentially, HBM takes existing DRAM with 2.5D technology and moves it closer to the processor using a fatter data pipe. This effectively accelerates data throughput, while reducing the amount of power required to drive a signal and cutting RC delay.

Perhaps not surprisingly, HBM was initially perceived by GPU companies as a clear step in the evolutionary direction of graphics-specific memory. However, the networking and data center industries soon realized HBM could add a new tier of memory to the hierarchy – while delivering more bandwidth for applications that demanded faster access, as well as lower latency and power consumption. From our perspective, HBM enables data center solution developers to bring high performance memory closer to the CPU, thereby reducing latency and improving system throughput.

Interested in learning more about HBM? You can check out our HBM PHY product page here.