Memory PHYs

In part one of this two-part blog series, Steven Woo, Rambus fellow and distinguished inventor, spoke with Ed Sperling of Semiconductor Engineering about utilizing various number formats to optimize memory bandwidth for artificial intelligence (AI) applications. In this blog post, we’ll be taking a closer look at the impact specific number formats have on performance and accuracy.

“At Hot Chips 29 (2017), Microsoft highlighted how the performance of certain neural network engines improved dramatically when they reduced the precision of the numbers they were working with,” Woo explains.

“In [one] particular case, Microsoft [went] from 16-bits down to 8-bits and saw improvements for both the Stratix 5 (3x) and Stratix 10 (7x).” Woo elaborates. “So now the question becomes: since I’m better able to use my memory bandwidth [by] packing more numbers into it, am I losing accuracy?”

According to Woo, when Microsoft trained with full 32-bit floating-point numbers, they reported a reference level of accuracy of 1.0. However, when Microsoft dropped all the way down to its fp9 format (9 bits), accuracy levels decreased slightly.

“[However], Microsoft showed that through retraining and changes to the algorithm, accuracy could be gained back – and in some cases, with even better accuracy than their reference 32-bit implementations,” says Woo. “[This example highlights how] memory bandwidth is considered so important and precious. [Indeed, system designers] are willing to change their algorithms to make the best use of what’s widely viewed as the scarcest resource for these types of applications.”
In addition to highlighting Microsoft’s fp9 format, Woo discusses an extreme scheme that allows for only one bit of fractional precision. In practical terms, this means only numbers that end in .0 or.5 can be encoded such as 0.0, 0.5, 1.0, and 1.5. However, the above-mentioned scheme is understandably limited.

“And so researchers developed an interesting idea called stochastic rounding where you probabilistically round up or down – depending on what values you can represent and what the value of your number is,” Woo elaborates.

“This example [above] shows the probability that I’m going to round my number down to zero is dependent on whatever the highest value is, the fraction that I can represent, the value I’m trying to encode, and then the range between the highest and lowest values that I can encode with my fractional precision. The probability that I round that number up is just one minus the probability I’m rounding down.”

To highlight how stochastic rounding works in practice, Woo provides an example based on one-bit fractional precision.

“With one-bit fractional precision my high value is 0.5 and my low value is 0.0 and so the value I really want to encode is 0.2. So, the probability that I’m rounding my number down is just the high minus the value which is 0.5 minus 0.2 divided by that range: 0.5 minus 0.0. So, it’s a 60% chance that I’m going to round my number down,” he states. “In the case of rounding up, it’s just one minus that fraction which is 40%. So, 40% of the time I would take my 0.2 and round it all the way up to 0.5. 60% of the time, I take my 0.2 and round it all the way down to zero. So, my expected value in the long run is just 60% x 0 (the number that I’m rounding down to) + 40% x 0.5 (the number that I’m rounding up to). The long-term average – my expected value – is at 0.2 which is exactly what we wanted to encode.”

As Woo notes, stochastic rounding can be an effective technique when there are multiple trials and sufficient data to avoid deviation.

“You have to make sure you’re able to achieve kind of the long-term averages for whatever calculations you’re doing. So, the more data [you have], the more accurate this becomes. The more opportunity you have to employ something like this, the better it’s going to work,” he adds.

Stochastic rounding, says Woo, can be leveraged by systems that require high levels of power efficiency.

“For example, inferencing on battery-operated endpoints for IoT type applications [requires system designers to delicately] balance performance against using the minimal amount of memory, storage and bandwidth,” he explains. “Being able to use very small numbers with very few bits of fractional precision can really help.”

However, Woo emphasizes that a holistic design approach is needed for AI systems and applications, as
stochastic rounding represents only one optimization technique.

“With [a] co-design [approach], the entire system should be thought of not only from the hardware side, but the software side as well, and how you use your data and solution. A whole host of techniques are being employed to [try and] compensate for the fact that you can’t get as much memory bandwidth as you want – and the techniques that are in use today aren’t necessarily as power efficient as people would like. So, techniques like stochastic rounding help you to get around these types of things. Our industry just needs to continue working hard and coming up with new methods to continue advancing AI,” he concludes.

Steven Woo, Rambus fellow and distinguished inventor, recently spoke with Ed Sperling of Semiconductor Engineering about how certain number formats can help system designers optimize memory bandwidth for artificial intelligence (AI) applications. As Woo observes, such optimization is necessary because the semiconductor industry is constantly struggling to keep up with an exponential increase of digital data.

“The amount of digital data we’re generating is growing dramatically – really far faster than almost any other technology can keep up. It’s become very difficult to keep all that data on chip and so external memory is [often used],” Woo explains.

“However, system designers would like to get even more bandwidth, although memory technology can only progress so quickly. This is why [a number of] interesting techniques have been developed to try and compensate for the difference between what the processing engines want and what the industry is able to provide.”

One example of an early optimization technique, says Woo, leveraged a 16-bit number format in an attempt to extend the capabilities of available memory bandwidth.

“Some years ago, the IEEE defined a popular format known as 32-bit floating-point numbers. These 32- bits are split up into three components: a sign bit, 8 bits for exponents which describes a range that the numbers cover, and a 23-bit mantissa which is the fractional part of the number,” he elaborates. “When you have a fixed amount of memory bandwidth, instead of using large numbers that are 32-bits long, you can actually go to smaller size numbers [such as] 16-bit numbers. This has the effect of allowing you to pack twice as many numbers into the available bandwidth.”

Although 16-bit floating point numbers still have a single sign bit, the number of bits in the exponent and the mantissa (the fractional part of the number) is reduced.

“This means we’re less precise with our numbers – and we can cover a smaller range compared to 32-bit floating-point numbers,” he adds. “[However], what people began to realize, especially when it came to AI applications, was that the range you needed to cover was very important. As well, it was [slightly] less important to have exactly the type of precision that was available in 32-bit floating-point numbers.”

Consequently, a new type of numeric format known as bfloat16 (Brain Floating Point) was developed. Essentially, Bfloat16 rebalances the 16 bits.

“You have one sign bit, but to match the range of 32-bit floating-point numbers, we went back to 8-bit of exponents. This means the fractional part of the number (the mantissa) is now down to seven bits,” Woo continues. “Bfloat16 [represents] a nice compromise between trying to get you the range that you really need – for many types of applications – while not compromising the fractional precision too much in order to make the networks not converge.”

As Woo points out, system designers have found that neural networks are extremely tolerant of small changes in precision.

“Rebalancing of the bits has actually been helpful in terms of allowing the memory bandwidth to be used more effectively while not compromising on the performance of neural networks,” he states.

Woo also notes that very large neural networks – across multiple and diverse verticals – tend to require optimized precision and range.

“[Similarly], when you’re training, you’d like to have very high precision. You might actually train using a much higher precision set of numbers,” he explains. “When you go to inference you might actually reduce down to smaller size numbers so that you don’t need to burn as much power or need to spend as much memory bandwidth to get the kind of performance that you’re training for.”

In addition, Woo says that various types of inputs can leverage different dynamic ranges and number types.

“[As well], even the training process itself and the inference process can use different precision numbers at different parts of the computation. For example, you might do accumulations at a much higher resolution – maybe 32-bits – while you might do the multiply at something smaller like 16 bits,” he adds.

Written by Steven Woo

Introduction

The ongoing transition from 4G to 5G is driving major infrastructure upgrades that include the integration of AI and machine learning capabilities at the edge. This is due to several major factors, the most important of which is the relentless growth in the amount of the world’s digital data. According to a recent Forbes article, approximately 2.5 quintillion bytes of data are created each day. By 2020, DOMO estimates that for every person on earth, 1.7 MB of data will be created every second.

Beyond the incredible rate of global data growth, carriers see 5G as a lucrative opportunity to generate new revenue streams and bolster the average revenue per user (ARPU). Neural networks and machine learning will continue playing prominent roles in supporting a range of low-latency, bandwidth-intensive applications at the edge including augmented reality, virtual reality, the IoT and Industry 4.0. As such, companies like Nvidia, Google, Intel and ARM are all shipping AI-optimized edge computing platforms.

Offloading Data Processing to Edge Nodes

The rapid growth of the world’s digital data, which will accelerate with the ongoing transition from 4G to 5G, is only increasing the already considerable strain placed on current network infrastructure. In addition to the services illustrated in the image above, IoT endpoints such as cameras, sensors, meters, and connected cars, are generating a significant amount of data today that is often transported to data centers for processing. Offloading certain types of data processing to edge nodes helps reduce transaction latencies, improve throughputs and takes the pressure off long-haul links that move data to more centralized data centers by sending along a smaller amount of higher-value data processed in these edge nodes.

Processing closer to data / endpoints improves latency bandwidth, energy use

Edge computing also improves energy consumption by minimizing data movement. Moreover, processing at edge nodes improves security, as data resides on infrastructure that is not as widely shared compared to more geographically centralized data centers.

Google’s Edge TPU & Nvidia’s Jetson

As we noted above, a number of companies are designing and shipping specialized silicon for AI at the edge. For example, Google’s Edge TPU leverages the company’s extensive data center TPU work, allowing AI models trained in data centers to run at the edge. The Edge TPU, a purpose-built ASIC designed to achieve high performance at low power and in a small footprint, includes an open-source TensorFlow Lite programming environment.

Deployed in this manner, the Edge TPU is intended to complement the data center TPU and associated services to form a complete, end-to-end AI solution with both hardware and software components.

Current iterations of Google’s Edge TPU include a Coral dev board and Coral accelerator. The Coral dev board – which runs Mendel Linux – is a single-board computer with a removable system-on-module (SOM) featuring the Edge TPU. Additional key specs include an NXP i.MX 8M SOC (quad Cortex-A53, Cortex-M4F), integrated GC7000 Lite Graphics, ML accelerator Google Edge TPU coprocessor, 1 GB LPDDR4 and 8 GB eMMC.

Meanwhile, Nvidia targets the edge with the Jetson TX2, a 7.5-watt supercomputer on a module that is built around the company’s Pascal-family GPU. The Jetson TX2 packs 8 GB of memory and 59.7 GB/s of memory bandwidth. The Jetson TX2i variant is aimed at high-performance edge computing devices such as industrial robots, machine vision cameras and portable medical equipment.

Conclusion

The industry transition from 4G to 5G is generating an explosion of data and serving as a catalyst for the integration of neural networks and machine learning capabilities at the edge. Offloading certain types of data processing to edge nodes minimizes transaction latencies, improves throughput, reduces energy consumption, bolsters security and takes the pressure off long-haul links.

Interested in learning more about AI and machine learning? You can check out our article archive here.

A Letter to Our Valued Customers: Taking the Next Step to Comprehensive Interface Solutions

Dear Customers,

We are thrilled to announce that today Rambus, a premier silicon IP and chip provider making data faster and safer, has closed the acquisition of Northwest Logic.

What does this mean for you?
On a day-to-day, it’s business as usual. But now, as a combined company, we have new resources and capabilities to continue driving value for you and your organization. We will have a comprehensive portfolio of physical and digital interface IP and remain committed delivering our solutions together or standalone to serve your needs in the best way possible.

What does this mean for us?
The entire staff of Northwest Logic is now a part of the Rambus family and the Hillsboro, Oregon office will remain in place. And we will continue to lead this dedicated and talented team under Rambus.

Our story continues…
While Northwest Logic’s innovative, best-in-class digital controllers complement Rambus’ proven strength in high-speed physical IP cores, together, Rambus will offer one of the most comprehensive high-performance interface IP solutions in the industry, leveraging our combined core strengths in semiconductors. Northwest Logic joining forces with a company as passionate about IP Cores as Rambus, is what takes us into the next chapter in our storybook. We’re energized to see where our innovation and growth takes us from here.

Thank you!
We want to take this opportunity to thank each and every one of our Northwest Logic customers for putting your trust in us over the last 25 years and bringing us to where we are today. We are delighted to come to work every day for you, our valued customers. Everyone’s passion is driven by our customers’ achievements and success. During this exciting time as Northwest Logic now becomes Rambus, this commitment is stronger than ever.

To read the full press release from Rambus, please click here or visit rambus.com/northwestlogic.