Rambus

At Rambus, we create cutting-edge semiconductor and IP products, spanning memory and interfaces to security, smart sensors and lighting.

Home > Memory PHYs

Memory PHYs

Below is an archive of the blog posts related to Memory PHYs. To learn more about our memory interface IP solutions, click here.

5G Eats Up More Memory

by Leave a Comment

5G is more memory hungry compared to previous generations, according to Gary Hilson, memory editor at EE Times.  In his recent article, he pulled together an industry consensus about the types of memory that might be considered to comply with 5G demands, the pros and the cons.

Hilson writes that it makes sense when you think about how much computing power people are carrying around in their hands compared to even the early days of the Blackberry. Mobile networks are just as much about transmitting 4K video as they are talk and text.

He adds, “Connected devices not only include smartphones, but sensors, parking meters, smart cars, wearables, and utilities. Telecom infrastructure is now networking and compute infrastructure—flash and DRAM are supplanting SRAM and TCAM, and there might be room for emerging memories, too.”

Hilson explains that a high speed TCAM (ternary content-addressable memory) can search its entire contents in a single clock cycle and is faster than RAM. It’s a mainstay of networking gear, such as high-performance routers and switches, to increase the speed of route look-up, packet classification, packet forwarding and access control list-based commands.

However, there are downsides to TCAM. Hilson quotes Jim Handy, principal at Objective Analysis, as saying, “despite its longevity, it is ‘exotic’ in that it’s quite expensive and there are limited suppliers. But there’s solid payback from using them. They streamline the routers. They make them far faster with less other processing hardware.”

Hilson also writes about resistive-RAM (ReRam) and MRAM as possible contenders to mate up with 5G requirements.  But as he notes, RRAM-based TCAM circuits can match the performance of CMOS-based SRAM circuits for multicore neuromorphic processor applications despite the performance and reliability tradeoffs.  However, the issue with both RRAM and the emerging magneto-resistive random-access memory (MRAM) is temperature.  Neither can handle high heat.

Hilson again quotes Handy saying, “Embedded MRAM has made some inroads.  (However,) the telephone companies want both high and low temperature stuff.”

Another viewpoint in Hilson’s article comes from Virtium vice president Hiep Pham, who oversees the company’s networking endeavors. For the immediate future, the transition from 4G means continuing to support legacy equipment. “The goal for 5G is basically to have a faster speed, a high capacity, and low latency,” he said. “But 5G will go together with some applications.”

“This is where the lines get blurry,” Hilson reports.  “The evolution from a telecom network to a technology platform means it’s not just about a better voice and data experience for handsets, but support for a wide range of edge computing and IoT scenarios, including industrial automation, as well as autonomous vehicles and smart cities.”

He states that Pham believes the amount of artificial intelligence (AI) in the network will increase, starting in data centers but also distributed through the network, right to the edge. “The edge will have to handle some of the AI, even though it goes through 4G or 5G.” He said there will remain bandwidth limitations, so the edge must be capable of storing and process information.

Lastly, Ryan Baxter, senior director for data center at Micron, rounds out Hilson’s piece by saying, “The distributed nature of a 5G platform means there’s been a proliferation of different memory technologies being considered and not just standard DDR3 and DDR4.”

Baxter is quoted as saying that low power double-data rate (LPDDR) and graphics double-date rate (GDDR)-based technology are being looked at and the requirements of such broad spectrum are driving a fair amount of fragmentation because 5G presents itself as a blend of telecom and computing networking.

“We’re seeing significantly larger memory footprints required to essentially support the compute elements of the 5G deployments,” he says in Hilson’s article.

 

Reduced-Precision Computation for Neural Network Training

by Leave a Comment

In our previous blog post, we discussed the process of training neural networks (NN) and briefly touched on NN training platforms and related memory bandwidth issues. As we noted, neural network training and inference performance are heavily contingent upon memory bandwidth. This is because the memory system is typically tasked with holding the neural network parameters – weights and biases – along with training data.

Stock image of brain and networks

With hardware optimized for performing fast computations, neural networks constantly stress the memory system as model parameters and data are fetched from memory. More specifically, moving data to and from the compute engines strains both memory capacity and bandwidth, as the training sets and model parameters are held in local memory to avoid data transfers over slower PCIe interconnects.

Reduced-Precision Computation for Neural Networks

One popular method of reducing memory bandwidth demand and increasing power-efficiency for neural networks focuses on reduced-precision computation. Instead of computing with 32-bit floating point numbers, many implementations are now using 16, 8, and in some cases fewer bits of precision. This is because neural networks are tolerant of reduced precision and the power saved by avoiding computing on the least significant digits is often considerable.

Indeed, NVIDIA has discussed the benefits of using 16-bit operands and 32-bit accumulates for neural network training. Neural network parameters can be reduced in precision to 8 bits after (or during) training. While this approach may reduce the accuracy of inferencing, networks can be further trained to regain the classification accuracy lost when reducing the precision of network parameters. Microsoft also discussed the benefits of reduced precision computation when introducing Brainwave, the company’s machine learning infrastructure, at Hot Chips 2017.

In addition to using 16-bit operands and 32-bit accumulates for neural network training, numerous other methods have been discussed in various industry and academic research papers to better utilize memory bandwidth and to improve power-efficiency, including pruning weights that are close to 0 and quantizing weights to reduce storage overhead.

New Formats for Reduced Precision Computation

Reduced precision computation has garnered significant industry interest over the past few years, with a range of new formats now supported in hardware for AI and related software frameworks. For example, bfloat16 is perhaps the most popular new format that is supported by Intel’s Nervana AI processor, Xeon processors and FPGAs, as well as Google’s TPUs and TensorFlow framework. Compared to the IEEE 754 single-precision floating point format, bfloat16 has the same number of exponent bits and covers the same numerical range (~1e-38 to ~3e38), but at lower numerical precision. The similarity between the two also allows fast conversion between the two formats.

Extreme Reduced Precision Computing

Xnor.ai is a company exploring extreme reduced precision computation by using single-bit model parameters. Essentially, the company has reworked training and inference algorithms so that interference operations are reduced to single-bit xnor operations. By doing so, these computations can be performed natively in hardware on almost any processor at high speed and with high power-efficiency, including the Raspberry Pi and smartphone CPUs. Among the many demos they have shown to date, the most impressive is real-time object recognition running locally on an iPhone.

Conclusion

Neural network training applications can more effectively utilize memory bandwidth by packing more numbers into each set of bytes – instead of defaulting to full-precision computation operations. In turn, these savings can be leveraged to implement additional neurons or reduce the total cost of operation (TCO). With the benefits of reduced precision computation being realized in modern AI systems, the industry must now turn its attention to additional innovations in order to  continue fueling improvements in future AI systems.

Interested in reading more about machine learning and neural networks? You can browse our article archive on the subject here.

Understanding ML and ANN memory requirements

by Leave a Comment

Written by Steven Woo

Artificial Neural Networks

First proposed in 1944 by Warren McCullough and Walter Pitts, an artificial neural network (ANN) or more commonly, a neural network (NN), can perhaps best be defined as a computational model that attempts to closely emulate the network of neurons present in the human brain. More specifically, neuromorphic computing aspires to replicate the results that biological brains can achieve using analogous – albeit artificial – mechanisms.

Unlike traditional von Neumann systems, neural networks are not limited by conventional bottlenecks between program memory, data memory and the CPU. As Dan Kara, research director at ABI Research notes, standard CPUs will continue to play an important role in data centers for the foreseeable future. However, they will be supported by an increasing number of specialized non-von-Neumann platforms, as well as graphics processing units (GPUs), custom application-specific integrated circuits (ASICs) and field-programmable gate arrays (FPGAs).

Image representing neural and connected networks

Machine Learning

Creation of the term ‘machine learning’ (ML) is widely credited to Arthur Samuel, an artificial intelligence (AI) pioneer. From 1949 through the late 1960s, Samuel focused on enabling computers to learn from their experience. One of Samuel’s most famous learning programs used Lee’s Guide to Checkers to adjust its moves and optimize gameplay. This is because Samuel viewed machine learning as a form of AI that enables a system to learn from raw data rather than explicit programming. The field of machine learning has significantly expanded in the 21st century and now spans multiple sub-categories including supervised learning, unsupervised learning, reinforcement learning and deep learning.

AI: 2019 & Beyond

Neural networks and machine learning are currently experiencing an exciting renaissance – with a resurgence of interest across the semiconductor industry, academia and the media. Driven by Moore’s Law, compute performance and transistor densities have improved by several orders of magnitude since the last wave of machine learning advances in 1980s. This has enabled neural networks and machine learning to flourish across a wide range of industries.

Numerous chips and cores are now being designed and deployed to implement neural network training and inference. Analysts estimate there are at least 50 neural network processors and 10 neural network IP cores under development or in production by internet giants, traditional semiconductor manufacturers and numerous start-up companies. These specialized chips and cores are targeted at a diverse set of use cases such as data centers, mobile processors and edge processing systems.

NN, ML & Memory Requirements

Neural networks require ultra-high bandwidth capabilities and power-efficiency for inference and training. Moreover, extensive data training sets often demand large memory capacities, especially for data center applications. To fulfill these requirements, some companies are leveraging on-chip memory, while others are using HBM2 or GDDR6. Although on-chip memory provides the highest bandwidth and power efficiency, this paradigm sacrifices capacity to achieve the former.

HBM2 memory systems offer NN and ML engineers the advantages of the highest bandwidth per device, as well as optimal power efficiency for high performance memory systems. The key challenges for HBM2 memory systems are the high cost of the memory device, the additional cost of using a silicon interposer and the inherent difficulty of interposer-based system designs.

Meanwhile, GDDR offers NN and ML engineers high bandwidth as well as high capacity. Like previous GDDR generations, GDDR6 DRAMs utilize single-ended signaling and die-down implementations to provide the cleanest signaling environments for the highest per-pin signaling rates. The primary advantages of GDDR6 are compatibility with existing package and board infrastructure, as well as the relatively lower cost of DRAMs and packages. The critical challenges for GDDR6 memory systems are maintaining signal integrity and achieving reasonable power efficiencies.

Conclusion

Driven by Moore’s Law, neural networks and machine learning are currently experiencing an exciting renaissance. Both require ultra-high bandwidth capabilities, large capacities and power-efficiency for inference and training. To fulfill these requirements, some companies have chosen on-chip memory, while others are using HBM2 or GDDR6. Each standard offers its own advantages and disadvantages, which are explored in detail above.

GDDR6 Crosstalk Article – Rambus

by Leave a Comment

Director of Signal Integrity Engineering at Rambus, Shawn Nikoukary, recently published a piece in eeweb.com/EE Times about crosstalk (xtalk) as being the most challenging signal integrity (SI) issue associated with GDDR6.

image of board

Nikoukary says, “the main reason is that the circuits used to remove crosstalk from signals are complicated and require a large power/area overhead to be effective. There are several types of active and passive crosstalk cancellation methods, such as passive mutual capacitive couplings and crossed continuous time linear equalization (CTLE) and crossed decision feedback equalization (DFE) circuits.”

But he reminds SoC and system designers to keep in mind that it is much easier to design a channel that is low in crosstalk, than trying to correct it in the receiver.

It’s important to have a good handle on and fully understand crosstalk in order to design a robust GDDR6 interface, he emphasizes.  “Crosstalk is due to capacitive and inductive coupling between two signals and it is categorized as Near End and Far End XTalk (NEXT/FEXT), Nikoukary explains.

Further, he adds it is highly recommended that all data lines (DQ signals) running at higher data rates be routed as a strip line. The largest contributor of the crosstalk in GDDR6 channel is from the vertical transitions such as vias, plated through holes (PTHs) and BGAs which are placed near each other.  Great care needs to be taken to shield vias and design a BGA that will have a low crosstalk pin assignment.

“The majority of the crosstalk in GDDR6 systems results from the ball arrangement and mis-assignment in a BGA package,” according to Nikoukary. “This has a major impact on crosstalk and is especially true for certain applications where the PTHs are placed under the BGA.  For example, networking systems tend to have large PCB stack-up, very thick PCBs that utilize through vias.  These long through vias placed under the BGA cause large amounts of coupling and crosstalk if designed properly.”

Summarizing his key points, Nikoukary explains that “ideally, to eliminate this crosstalk, a very low crosstalk BGA assignment for the package is essential.  In a good BGA ball map, ground and power shielding is placed between signals or blind and buried vias are used in the dense area under the BGA to minimize crosstalk.”

 

Uber eyes an AI future

by Leave a Comment

Rick Merritt of the EE Times recently reported that Uber is using banks of Nvidia GPUs for deep learning applications. The transportation network company (TNC) is also testing FPGAs from multiple startups such as Eyeris, Graphcore and Wave Computing. In addition, Uber maintains a dedicated AI research team that oversees more than a dozen deep-learning models including recommendation engines for Uber Eats, fraud detection services and estimated driver arrival times.

Stock image of chip

“The algorithms span a half-dozen varieties, implemented across a laundry list of mainly open-source frameworks and libraries,” writes Merritt. “The underlying AI hardware today consumes as much as 40 kW in a rack of systems — twice the power that standard servers use — and can require flows of more than 100 petabytes of data.”

According to Merritt, Uber has begun designing its own compute and storage servers, as well as a variant of a 19-inch rack to house them along with vertically mounted switches. More specifically, Uber’s “super-hot” storage server utilizes two Intel Cascade Lake processors, with each 1U system packing more than 16 TB of NAND flash. Meanwhile, Uber’s warm storage system is built around Broadwell processors, with up to 70 8-TB SATA hard-disk drives in a 4U design that puts 6 petabytes in a rack.

“[Uber’s] systems have served more than 10 billion Uber rides to date at a rate of about 15 million a day across 600 cities in 65 countries,” he adds.

Commenting on the above, Steven Woo, Fellow and Distinguished Inventor in Rambus Labs, told Rambus Pressthat neural networks, machine learning and autonomous vehicle processors typically demand silicon with the highest memory bandwidths and power-efficiencies for inference and training. The latter, says Woo, can also require large memory capacities, especially for data center applications.

“Some companies use on-chip memory to power their silicon, while others deploy HBM2 or GDDR6,” he explains. “On-chip memory provides the highest bandwidths and power efficiencies but sacrifices capacity to achieve this.”

HBM2, says Woo, is being used in many high-end designs today, as the memory standard provides high bandwidth and is power efficient — although it comes with higher costs and is more complex to design with. As noted above, GDDR6 is also seeing interest, with the memory standard providing high bandwidth as well as high capacity. However, GDDR6 has higher power and challenging signal integrity constraints.

“As with other markets that use high-performance memories, the minimum expectation is to at least follow the historic trend of doubling memory bandwidth and improving power efficiency with each succeeding DRAM generation,” he adds.

Interested in learning more about AI and machine learning? You can check out our article archive on the subject here.

IDC White Paper Lays Out Future of Memories and Interfaces

by Leave a Comment

A recently published IDC white paper calls attention to the growing significance of chip connectivity.  Entitled, “Hidden Signals: The Memories and Interfaces Enabling IoT, 5G, and AI,” the paper states that data created by electronic systems will grow 10x to 103 zettabytes between 2014 to 2023.  Moreover, it says the average server will support over 670 gigabytes of DRAM in 2023.

IDC Report Blog Image

The underpinnings to this market growth are not only interfaces or chip connectivity and where data is stored in real-time, but also the connectivity between systems and processors computing the data, according to IDC.  Technologies critical in this scenario include GDDR6, HBM, and memory buffers. These are the components that move, accelerate, and store data and enable IoT, 5G, and AI.  These are the trends driving new markets for the next 10 years, according to the IDC white paper.

The 10-page paper provides the reader a comprehensive market analysis of various product classifications, a historical perspective and market growths per category, trends, and detailed definitions.

As for memory and interface technology and associated trends, IDC explains that interfacesbetween high-performance data processors have been evolving because the market has recognized that monolithic, homogeneous microprocessors are insufficient to process and secure the amount and disparity of data flooding into high-performance systems.

The paper adds that CPUs, GPUs, FPGAs, and ASICs share a need for external memories—such as CPUs for main memory and GPUs for frame buffers—that hold data close to the processor. These needs thus put specific pressure on the interfaces between these processors and their memories. Thus, interfaces and memories are inextricably intertwined and the two are critical to making the often-hidden data signals flow efficiently.

IDC also explains that systems like servers will need increasing amounts of DRAM. However, such increasing capacities means increasing the burden on the system microprocessor to manage that memory across the memory bus. Memory buffer chips reduce that burden by bringing local intelligence to the memory on the memory modules.

Buffer chips on Load-Reduced DIMMs (LR-DIMMs), for example, reduce the load on the microprocessor’s memory controller and across the memory bus by handling all data, command, and control signals sent to memory and handling all reads and writes to the DRAM chips.

All in all, the IDC paper paints a favorable landscape and market outlook for IoT, 54G, and AI based on today’s Rambus memory and interface technologies.  GDDR6 PHYs, 32 gigabit per second (Gbps) SerDes for 5G, 112Gbps ADC/DSP-based SerDes for next generation servers, and DDR5 buffer chips are leading the charge to further advance those product categories.

It’s definitely a good read for any and all who follow or want to learn more about these industry advances and the technologies driving them. Get a hold of a copy of this IDC White Paper right now.