Rambus

At Rambus, we create cutting-edge semiconductor and IP products, spanning memory and interfaces to security, smart sensors and lighting.

Home > SerDes PHYs

SerDes PHYs

PCIe 5 Drill-Down with Rambus’ Suresh Andani: Part 1

by Leave a Comment

Semiconductor Engineering Editor in Chief Ed Sperling recently sat down with Suresh Andani, Senior Director, Product Marketing and Business Development at Rambus, to discuss the evolution of PCIe and its latest iteration: PCIe 5.

As Andani notes, PCIe 5 and subsequent iterations of the PCIe standard will continue to be one of the “key interfaces” that enable high-speed computing and processing in the data center.

“The base node in data centers are servers. In these servers, you will typically find the central processing unit (CPU), whether it is Xeon from Intel, EPYC from AMD or P9/P10 from IBM,” he explains.

“PCIe basically connects the SmartNIC or the NIC from the top of the rack switch to the CPU. It also connects the CPU to accelerators. In addition, more storage is moving away from SAS/SATA towards non-volatile memory express (NVMe) or PCIe. There are also a number of interfaces moving towards PCI Express and migrating to PCIe 5.”

According to Andani, with the amount of data exploding and high-performance computing (HPC) workloads increasing, PCIe 4 – even at x16 widths – is no longer sufficient, especially for applications that are running in the cloud.

“Networking in the cloud is also moving from 100GbE to 400GbE for which you need PCIe 5. The bandwidth enabled by PCIe 4 is simply not enough,” he adds.

As Andani observes, PCIe has steadily evolved from Gen 1 to Gen 5, with the Gen 1 specification –introduced in 2002 – running at 2.5 gigatransfers per second (GT/s).

“2.5 GT/s was once sufficient for the type of workloads that ran during earlier days. PCIe 5 runs at 32 GT/s and its successor, PCIe 6, was recently announced,” he explains. “It is important to note that between 2010 and 2017, SerDes I/O technology quadrupled, although the PCIe specifications stagnated at PCIe 3 for almost seven years. This made PCIe a system bottleneck.”

To avoid protracted refresh cycles and reduce system bottlenecks, PCI-SIG has detailed a steady two-year cadence at which the PCIe specs will be released in the future.

“PCIe 5 can take care of networking bandwidth all the way up to 400 gigabit Ethernet, solving a big problem that was there up until PCIe 4,” he states. “The PCIe 5 spec was finalized and released in May 2019 and the development of the PCIe 6 spec was announced in June 2019. It is expected that PCIe 6 will be finalized sometime in the 2021 timeframe. With PCIe moving at a two-year cadence, the interface] will no longer be a system bottleneck.”

Andani also notes that high-performance workloads, such as genomics, high-frequency trading and video transcoding, are all increasing in sophistication and demanding ever-more parallel processing.

“The parallel processing that’s needed for all these datasets requires heterogeneous computing. Specifically, it requires massively parallel architectures like GPUs or FPGAs, which is why a lot of these workloads are being offloaded from the main CPU to a co-processor (which some people refer to as an accelerator), whether they are GPUs, FPGAs, or even ASICs. In turn, heterogeneous computing with accelerators is pushing high bandwidth demand for PCIe, the key interface between the CPU and the accelerators,” he adds.

To better understand the evolution of PCIe, Andani presents a chart that compares the lane speeds of PCIe 4 and PCIe 5. PCIe 4 hit 16 GT/s per lane, while PCIe 5 doubles that speed to 32 GT/s per lane.

“If you look at the chart, you can see the full duplex aggregate bandwidth in gigabytes per second in x1, x2, x4, x8 and x16 configurations.

“So, the maximum aggregate bandwidth that PCIe 4 was able to provide was 64 gigabytes per second (GB/s) for duplex. PCIe 5 doubles that to 128 GB/s, a speed increase that is very much needed in hyperscale data center architectures.”

Indeed, sophisticated workloads continue to evolve along with an ever-increasing amount of data. This means bandwidth will need to double at a steady cadence, which is precisely why PCIe 6 will scale up to 64 GT/s. However, Andani emphasizes that the demand for bandwidth will always be insatiable.

“You will always need faster and faster interfaces as the workloads become more sophisticated,” he concludes.

PCI Express 5 vs. 4: What’s New? [Everything You Need to Know]

by Leave a Comment

Introduction

What’s new about PCI Express 5 (PCIe 5)? The latest PCI Express standard, PCIe 5, represents a doubling of speed over the PCIe 4.0 specifications.

We’re talking about 32 Gigatransfers per second (GT/s) vs. 16GT/s, with an aggregate x16 link bandwidth of almost 128 Gigabytes per second (GBps).

This speed boost is needed to support a new generation of artificial intelligence (AI) and machine learning (ML) applications as well as cloud-based workloads.

Want to know a little more about AI/ML applications and cloud-based workloads?

Both are significantly increasing network traffic. In turn, this is accelerating the implementation of higher speed networking protocols which are seeing a doubling in speed approximately every two years.

You can find much more about PCIe 5 in the article below.

Table of contents

1. PCI Express: Frequently Asked Questions (FAQ)
2. PCIe 5 – A New Era
3. PCIe 5 vs. PCIe 4 (+Comparison table included)
4. PCIe 5: Applications and Market Adoption
5. Complete PCIe 5 Interface Solutions from Rambus
6. Conclusion

 

PCI Express: Frequently Asked Questions (FAQ)

Let’s answer five frequently asked questions about PCI Express and PCIe 5.

a. What is PCI Express 5?

With the preliminary specification announced in 2017, PCIe 5 is a high-speed serial computer expansion bus standard that moves data at high bandwidth between multiple components. The PCIe 5.0 specification was formally released in May of 2019.

You might be wondering why a new PCI Express standard like PCIe 5 is needed. Well, PCIe 5 offers twice the data transfer rate of its PCIe 4 predecessor, delivering 32GT/s vs. 16GT/s. This speed increase is critical to support new AI/ML applications and cloud-centric computing.

b. Why both GT/s and GBps?

GT/s is a measure of raw speed – how many bits can we transfer in a second. The data rate, on the other hand, has to take into consideration the overhead for encoding the signal. Bandwidth is data rate times link width, so encoding overhead’s impact on the data rate translates directly to an impact on bandwidth.

Back in the days of PCIe 2, the encoding scheme was 8b/10b, so there was a hefty overhead penalty for encoding. With such a high overhead, it was particularly useful to have measures of transfer rate (x GT/s) and data rate (y Gbps), where “y” was only 80% of “x.”
With Gen 3 and continuing through to the present Gen 5, the PCI Express standard moved to a very efficient 128b/130b encoding scheme, so the overhead penalty is now less than 2%. As such, the link speed and the data rate are roughly the same.

For a PCI 5 x8 link, 32GT/s raw speed translates to 31.5GBps bandwidth (we chose a x8 link so we could go straight from bits to bytes). And since PCIe is a duplex link, total aggregate bandwidth rounds to 63GBps (32GT/s x 8 lanes / 8 bits-per-byte x 128/130 encoding x 2 for duplex).

c. What is a PCI Express lane?

So what’s a PCI Express lane? Well, a PCIe lane consists of four wires to support two differential signaling pairs. One pair transmits data (from A to B), while the other receives data (from B to A). Want to know the best part? Each PCIe lane is designed to function as a full-duplex transceiver which can simultaneously transfer 128-bit data packets in both directions.

d. What does PCIe x16 mean?

We’ve discussed lanes, but what do they have to do with x16? Well, the term “PCIe x16” is used to refer to a 16-lane link instantiated on a board or a card. Physical PCIe links may include 1, 2, 4, 8, 12, 16 or 32 lanes. The 32-lane link is a pretty rare beast, so in practical terms the x16 represents the top end of the PCI Express link options.

e. What is PCI Express used for?

We’ve talked a lot about PCIe 5, but what is PCI Express actually used for?

You can think of the PCIe interface as the system “backbone” that transfers data at high bandwidth between various compute nodes.What’s the bottom line? Put simply, PCIe 5 rapidly moves data between CPUs, GPUs, FPGAs and ASIC accelerators using links with various lane widths configured to meet the bandwidth requirements for the linked devices.

PCIe 5: A New Era

Now that we’ve explored some PCI Express basics, we can take a closer look at PCIe 5 which was formally released in May of 2019.

PCIe 5 offers long-awaited capabilities for system designers building new cloud-based and AI/ML applications.

You might be wondering how and why.

Well, it starts with PCIe 5 offering twice the bit transfer rate of its PCIe 4 predecessor at 32GT/s vs. 16GT/s. With a x16 link, that scales to ~128GBps aggregate bandwidth.

Here’s another important point: The higher transfer rates of PCIe 5 enables system designers to balance bandwidth requirements with simplicity by working with fewer lanes for certain applications such as 40GigE and SSDs. More demanding, high-bandwidth applications such as AI and ML can take full advantage of a x16 lane.

Want to know what else PCIe 5 technology offers system designers? PCIe 5 is expected to be the technological cornerstone of additional interconnects. For example, Intel recently announced that its upcoming Compute eXpress Link (CXL) cache coherent interconnect will use the PCIe 5 standard’s physical layer (PHY) specification.

Since going faster entails more signal integrity (SI) issues, PCIe 5 adds more capabilities under the hood for handling signal loss and noise than were built into PCIe 4.

PCIe 5 vs. PCIe 4

Here’s a handy by-the-numbers comparison of PCIe 5 vs. PCIe 4 with the actual aggregate bandwidth adjusted for the encoding overhead.

Comparison table: PCI express 5 vs PCIe 4
Comparison table: PCIe 5 vs PCIe 4

 

PCIe 5: Applications & Market Adoption

AI/ML and Cloud Computing

No surprise, PCIe 5 is the fastest PCI Express ever. While the speed upgrade makes the applications of today run faster, what’s particularly exciting is that PCIe 5 is enabling new applications in markets such as AI/ML and cloud computing.

AI applications generate, move and process massive amounts of data at real-time speeds. An example is a smart car which can generate as much as 4TB of data per day!

But that’s not all, the size of AI/ML training models are doubling every 3-4 months. The torrent of data, and the rapid growth in training models is putting tremendous stress on every aspect of the compute architecture, with interconnections between devices and systems being of critical importance. Also critical is fast access to memory as AI/ML workloads are extremely compute intensive.

But while AI/ML is one major megatrend, there are others. Data centers are changing, with enterprise workloads moving to the cloud at a rapid pace. Those applications mean moving more data, often with real-time speed and latency.

This shift to the cloud, along with ever-more sophisticated AI/ML applications, is accelerating the adoption of higher speed networking protocols that are experiencing a doubling in speed about every two years: 100GbE ->200GbE-> 400GbE->800GbE.

Now this is where PCI express 5 comes in. PCIe 5 delivers an aggregate link bandwidth of almost 128GBps in a x16 configuration. Put simply, PCI express 5 effectively addresses the demands of AI/ML and cloud computing by supporting higher speed networking protocols as well as higher speed interconnections between system devices..

Complete PCI Express 5 Interface Solutions from Rambus

Now that we’ve explained the important benefits of PCI express 5, where can you get a reliable PCIe 5 interface?

Rambus offers a comprehensive interface solution for PCI Express 5 (PCIe 5.0) that consists of a PCIe 5.0 PHY and a co-verified PCIe 5.0 digital controller (from recently acquired Northwest Logic).

Want to know the best part? Having a comprehensive solution with a co-verified PHY and controller means that integration complexity is significantly reduced for chip designers.

What if you already have a PCIe 5 PHY or controller from another vendor or have developed one in house?

Both the Rambus PCIe 5 PHY and controller can be paired with PIPE 5.2 – compliant 3rd-party solutions if so desired. In addition, both PHY and controller are backward compatible to PCIe 4, 3 and 2.

But that’s not all. The Rambus PCIe 5 PHY supports the Compute Express Link (CXL) connectivity standard, a new high-speed interconnect between CPUs and workload accelerators or other attached devices. This helps support accelerators used to complement CPUs in AI/ML applications.

Conclusion

In “PCI Express 5 vs. 4: What’s New?” we explain how PCI Express is the system backbone that transfers data at high bandwidth between CPUs, GPUs, FPGAs and ASIC accelerators using links of variable lane widths depending on the bandwidth needs of the linked devices.

We also detail how the latest PCI Express standard, PCIe 5, represents a doubling over PCIe 4 with a raw speed of 32GT/s vs. 16GT/s translating to aggregate bandwidth for a x16 link of ~128GB/s vs. ~64GB/s.

We then explored how the higher data rates of PCIe 5 are enabling system designers to support a new generation of cloud computing and AI/ML applications.

Lastly, we highlighted Rambus’ comprehensive interface solution for PCI Express 5 (PCIe 5.0) which comprises a PCIe 5.0 PHY and a co-verified digital controller. This comprehensive solution significantly reduces integration complexity for chip designers while providing state-of-the-art performance.

 

Complete Interface Solution for PCIe 5.0 Launched

by Leave a Comment

Rambus has announced a comprehensive interface solution for PCI Express 5 (PCIe 5.0) consisting of a new PCIe 5.0 PHY and a co-verified Northwest Logic Expresso 5.0 controller.

Have you read our primer? PCI Express 5 vs. 4: What’s New? [Everything You Need to Know]

“Our high-speed SerDes and memory interface solutions make possible amazing advancements in performance-intensive applications in AI, data center, HPC, storage and networking,” said Hemant Dhulla, VP and GM of IP Cores. “Now we’ve added PCIe 5 to our industry-leading portfolio of high-speed interface solutions giving chip makers another tool to unleash the power of their designs.”

PCIe 5.0 Interface Subsystem Example
PCI Express 5 Interface Subsystem Example

With co-verified PHY and controller, Rambus greatly reduces integration complexity for chip designers. Both PHY and controller support PCI Express 5 and are backward compatible to PCIe 4.0, 3.0 and 2.0. While together they are an ideal complete solution, both PHY and controller can be paired with PIPE 5.2 – compliant 3rd-party solutions if so desired.

In addition to PCI Express 5, the Rambus PHY supports the Compute Express Link (CXL) connectivity standard. CXL is a new high-speed interconnect between CPUs and workload accelerators or other attached devices, as well as CPUs and memory. It maintains memory coherency between the CPU and attached devices. This greatly benefits accelerators used to complement CPUs in applications such as artificial intelligence and machine learning.

This latest portfolio addition highlights Rambus leadership in high-speed SerDes PHY and controller IP, leveraging the company’s long tradition of signal and power integrity expertise coupled with the design talent and products from newly acquired Northwest Logic.

Keep on reading: all the details on the PCI Express 5 interface solution

Chiplets keep the scaling of integrated circuits (ICs) rolling

by Leave a Comment

As data grows at an accelerating pace, more compute power and bandwidth are required to process this data, driving the need for larger and more complex system on chips (SoCs). This is particularly true at a time when our old friend Moore’s Law has lost a step.

However, as the complexity of SOCs increases, so do the costs to manufacture in leading-edge FinFET geometries. As misery loves company, achieving first-time-right silicon has become more difficult as well. Additionally, there are greater challenges for power scaling and yield. In other words, everything gets harder.

Chip disaggregation, or chiplets, offers an alternative to the traditional monolithic SoC scaling approach. Aggregating multiple chiplets to perform the function of a single monolithic IC de-risks the overall system by reducing complexity and increasing yields.

For applications like artificial intelligence (AI), where there is a “Cambrian explosion” in the number of SoCs and architectural approaches under development, chiplets are an ideal solution. Greater experimentation, and faster time to market are possible when a designer can revise a single chiplet as opposed to having to re-spin an entire SoC.

A key enabling technology required to make chiplet performance on par with that of an SoC is the high-speed interconnects between the chiplets. These must run at extremely high data rates and at very low power.

The Optical Internetworking Forum (OIF) specified the Common Electrical I/O (CEI) 112G XSR (Extra Short Reach) interface to provide the interconnect between chiplets, and between chiplets and optical engines. It is designed for low complexity and very low power while offering extremely high throughput capabilities.

Today, we announced the Rambus 112G XSR SerDes on 7nm FinFET process is now available for licensing. It joins a portfolio of 7nm Rambus PHY solutions including GDDR6, HBM2 and 112G LR (Long Reach) tailored for advanced applications such as AI, advanced driver-assistance systems (ADAS) and hyperscale data centers. With these interface solutions, chip designers can satisfy their need for speed and unleash the power of new compute architectures.

Rambus 112G SerDes PHY Article in eeweb.com/EE Times

by Leave a Comment

Ken Dyer, Director, Engineering Architecture, is the author of a 112G Long Reach (LR) SerDes PHY article in eeweb.com/EE Times network.

The 112G is coming on to the market to comply with growing demands for greater bandwidth for next generation data centers as well as for advanced applications like artificial intelligence (AI) and machine learning, as well as 5G communications.

Dyer calls Rambus’ 112G LR SerDes PHY “the top contender to boost greater performance with acceptable power and area” for those next generation data center and network systems designs.  His article details the new advanced SerDes PHY. However, he explains to SoC and network system designers that they may want data in different ways.

He said, “They also may opt to have a forward error correction (FEC) on the output of the receiver to improve bit error rate (BER) of the received data.  FEC performs that operation by using redundancy.  It transmits slightly faster than the data required, hence extra information is transmitted.  By reviewing the extra information in the received signal, FEC determines if mistakes have occurred or not.”

Dyer calls special attention to today’s higher serial data rates and added that four-level pulse amplitude modulation or PAM-4 has made its entrance and explained how it plays into the 112Gbps SerDes PHY.

He said that “We first need to look at Nyquist loss for NRZ data for a generic legacy channel and for 112Gbps data transmission. Nyquist loss is the insertion loss of the input signal at half the symbol rate.  For NRZ at 112Gbps, its nominally 56GHz. NRZ is two-level data and it’s at a relatively low 70 decibels (dB) at 56 Gigahertz (GHz).

“In short, crosstalk has more power than the signal, hence signal-to-noise ratio (SNR) is negative.  This means error-free communication or signal recovery is not possible.  In this case, if a transmitter is operated at 56Gpbs, data would not be received at all. Therefore, PAM-4 becomes a more viable solution than the traditional NRZ (non-return-to-zero).”

Be looking for sequels to this first 112G LR article in eeweb.com/EE Times network.  Articles, like this one, are designed to help SoC and data center system designers to prepare their next generation designs.

 

112G LR SerDes PHY Ready for Next-Gen Networking Gear

by Leave a Comment

Rambus has officially announced the 112G Long Reach (LR) SerDes PHY. Hemant Dhulla, VP and GM of IP Cores, said, “By leveraging leading 7nm process technology, Rambus is enabling the next generation of communications and data center applications.”

He further stated, “We’re excited to continue to expand our IP portfolio and deliver our customers top-of-the-line performance and flexibility for today’s most challenging systems, including solutions like our 112G LR SerDes PHY.”

As the industry rapidly transitions to 400GB and 800GB wired communication applications, 112G is a key building block necessary to support the ever-growing demand for more bandwidth in data center and network applications, doubling the data rate of 56G SerDes.

Rambus is at the forefront of implementing 112G design to address the long-reach backplane requirements for next-generation data-intensive applications. Those applications include generation terabit switches, routers, optical transport networks (OTNs), and high-performance networking equipment.

The new Rambus 112G LR SerDes PHY provides the optimal combination of power efficiency, performance and area, adding to Rambus’ leading-edge large portfolio of silicon-proven intellectual property (IP), design tools and reference flows. With the introduction of 112G, this technology achieves higher performance to rapidly enable industry infrastructure for the 400GB and 800GB applications.

The 112G LR SerDes PHY uses both PAM4 and NRZ (PAM2) signaling to provide reliable operation across a broad range of data rates. As such, they support legacy protocols, and provide the scalability to achieve the highest levels of performance.

They are built on a configurable architecture, enabling power to be matched to the channel loss. An embedded micro-processor enables firmware-controlled PMA configuration, initialization and adaption for maximum flexibility with minimum SoC integration effort.

The 112G LR SerDes PHY also provides SoC and system designers two other key features.  Its DSP-based architecture hands them improved signal to noise (SNR) and extended reach.  Plus, the 112G is configurable to provide power, performance, and area (PPA) optimization for medium reach (MR) and long reach (LR) applications.

This latest portfolio addition highlights Rambus leadership in high-speed SerDes PHY IP, leveraging the company’s long tradition of signal and power integrity expertise — remaining at the forefront of innovation in interface technology.

For more on the Rambus 112G LR SerDes PHY, check out our eBook.

Download Rambus 112G XSR and LR SerDes PHYs

Upcoming Webinar: AI Requires Tailored DRAM Solutions

Register Today