SerDes PHYs

In part two of this three-part series, Semiconductor Engineering Editor in Chief Ed Sperling and Suresh Andani, Senior Director, Product Marketing and Business Development at Rambus, discussed early market adoption of PCIe 5, as well as the networking environment the specification will support in the data center. In this blog post, Sperling and Andani explore the compute and storage demands that necessitate PCIe 5.

“Once the data [is fed] into the CPU, it has to be processed. Workloads such as HPC and AI/ML require a lot of computing. As you can see [in the video below], the ToR switch to the NIC is going to 400G Ethernet – and the link between the NIC and CPU is PCIe 5,” he explains.

“Now the CPU has a linear architecture. With multi-threading, hyper-threading and adding more cores, you can do a bit more parallel processing. However, applications such as convolutional neural networks (CNNs) and deep neural networks (DNNs) used in AI/ML workloads such as video transcoding and image processing require a massively parallel compute architectures. This is where we are seeing more and more accelerators go into hyperscale servers.”

As Andani observes, there are multiple neural networks for various applications.

“For image recognition, you’re going to use a CNN or DNN. If you look at voice recognition or natural language processing, these are different types of AI workloads, so the algorithms needed to run these workloads are different,” he states. “That is why you see different types of neural networks, but they all work in parallel, depending on what kind of workload you’re running.”

Andani continues, pointing out that although the data is fed to the CPU (via PCIe 5), the CPU is assisted by accelerators which themselves will interface with the CPU via PCIe 5.

“These could be GPUs that you see a lot in AI training type of applications. You are also starting to see a lot of FPGAs, specifically for high-performance compute workloads like video transcoding, although FPGAs are also quite good from an AI inference perspective. In addition, some companies are building dedicated or custom ASICs,” he explains.

“With all accelerators, the way to connect to the CPU is via PCIe. With older workloads, which were not as sophisticated, it was sufficient to use PCIe 4 bandwidth. However, new sophisticated workloads like AI/ML require a lot of bandwidth over PCIe 5.”

“There is a distinction between certain workloads that are very latency sensitive, for example, real time AI. If you are running real time AI workload in a cloud, you cannot tolerate a lot of latency between various compute nodes. That is where some new cache coherent protocols like Compute Express Link (CXL) and CCIX are gaining more steam.”

On the storage side, says Andani, data centers that did not have a hyperconverged architecture typically maintained storage servers that were SAS/SATA based running over fiber channel. However, in the modern data centers these servers are becoming hyperconverged to save on TCO.

“It makes a lot of sense to have a storage drive like an SSD flash drive (which is quite fast) that can be plugged into the same compute server, so you don’t need extra storage servers. That is why you are seeing more and more NVMe-based SSD drives on the market,” he elaborates. “With more and more ultra-high definition videos being stored and accessed on demand, you cannot tolerate a lot of latency of this transfer, like video being transferred from the SSD drive to the CPU back to the network. So, you need a very fast and efficient interface between these SSD drives and the CPUs. Hence, these are basically now PCIe 5-based interfaces between NVMe SSDs and the CPU.”

According to Andani, this is typically an x4 link (between NVMe SSDs and the CPU), versus the SmartNIC and the accelerators which are typically x8 or x16. Because of the small form factor and limited space, SSDs are typically put on an x4 slot.

“The key point: PCIe is driving the speeds and feeds needed for very fast video or storage access to the NVMe drives between the CPU and the SSD controllers. From a storage perspective, the videos are [scaling to] higher and higher resolution, which means the interface between the controllers and the CPU must become faster and faster.”

As Andani points out, there are two ways of increasing the bandwidth between the SSD and the CPU.

“One, you can take the PCIe 3 or PCIe 4 that you have and go from an x4 form factor to an x8 or x16. However, you are very space-limited in a very crowded server and cannot afford to go to x8 or x16 for storage,” he says. “This is where the solution is to keep the x4 form factor, but to get the double bandwidth, you double the speed [by moving from] PCIe 4 to PCIe 5.”

In terms of thermal impact, Andani observes that the increase in data bandwidth, computation and  number of IOPS (Input/Output Operations Per Second), corresponds to an increase in heat.

“Your TDP per chip, whether it’s an accelerator or a storage drive or even a SmartNIC, is going higher and higher. Luckily, thermal cooling technologies are also evolving. We are seeing liquid cooling in more and more data centers, which is passing liquid over the hot chips. As well, we are also seeing racks immersed in liquid, which is quite sophisticated,” he explains. “The thermal cooling technology needs to keep up with [these trends]. PCIe 5 only solves the data bandwidth interface challenge, but there are many other problems that we need to solve collectively in a data center, thermal challenges being one of them.”

Andani concludes his conversation with Sperling by walking through a detailed summary chart.

“Here we see a server and an x16 slot where you have the NIC or the SmartNIC – this is the one that gets connected to the ToR. You have a couple of x4 slots on a server where you are connecting your storage and then there is another x16 slot for an accelerator like a GPU, ASIC or FPGA,” he states.

“This illustrates the different PCIe interfaces on a server like we [previously discussed]. So, between the CPU and the SmartNIC, you are running PCIe. Currently, there are 40G and 50G servers running PCIe 3 (x8). 100G servers (that are ramping now) are using PCIe 4 (x8). In the architecture phase now are the 200G and 400G servers that, as we did the bandwidth calculation a few moments ago, will require PCIe 5 (x16), especially for 400G ethernet.”

Andani continues, pointing out the PCIe link between the CPU and the accelerator.

“Depending on the workloads that were running in the past, PCIe 3 or PCIe 4 was enough. However, with modern workloads like AI/ML and HPC, you are going to need the PCIe 5 (x16) bandwidth – and that is where we are going to see PCIe 5 deployed. Finally, storage is moving towards a U.2 form factor (SSD) which is moving from PCIe 3 to PCIe 4 and PCIe 5 (x4),” he concludes.

Miss part one and two? Catch up on this series here:
Part 1
Part 2

In part one of this three-part series, Semiconductor Engineering Editor in Chief Ed Sperling and Suresh Andani, Senior Director, Product Marketing and Business Development at Rambus, discussed the evolving PCIe specification. In this blog post, Sperling and Andani explore early market adoption of PCIe 5, as well as the networking environment the specification will support in the data center.

According to Andani, the enterprise and cloud data centers will be early adopters of PCIe 5. However, PCIe 5 will ultimately be adopted at the edge to support an increasing number of low-latency and time-sensitive applications.

“I do expect to see PCIe 5 adoption very soon – about in a year or so – within the edge data centers as well,” he adds.

To illustrate the need for PCIe 5, Andani highlights various aspects of a typical hyperscale data center, pinpointing where the interface will be deployed.

“There are three main elements to a hyperscale data center: networking, compute and storage. What you see [shown in the network section of the image below] is a very typical cloud architecture, also known as the leaf-spine architecture of a data center. At the base node of this architecture you will see racks of servers and these racks are combined to form clusters,” he explains.

“The base computation, the processing, basically happens in these servers. As workloads become more sophisticated, there is a lot of east/west traffic. The servers, the applications, are running on multiple servers that are located within a rack – and traffic can also be exchanged between servers across racks and clusters. A lot of traffic flows in the east/west direction – from a rack to another rack.”

Workloads such as AI/ML training have grown so large, it needs to be performed over many servers. As Andani points out, this flow of data is enabled by the network switch fabric.

“At the base level, you see the servers, then the top of rack (ToR) switch which is responsible for data traffic exchange within a rack. Connecting these tops of the rack switches are the leaf switches that enable data traffic between racks within a cluster. When you go one layer up, there is a spine switch that enables traffic to flow between one cluster to the other clusters within a data center,” he continues.

“At the front panel of the top of the racks switch are the Ethernet QSFP nodes connecting to the standard two socket server with an Ethernet link – and there is a network interface card (NIC) sitting on the server that connects the two. So, this link between the ToR switch and the server, the NIC of the server, is Ethernet. However, the traffic that originates from the network has to be passed on to the CPU for processing and that happens over PCIe.”

Data centers, says Andani, are moving from 40G Ethernet server networking to 100G. The upgrade to 400G is just beginning and a transition to 800G will follow.

“If you look at 100G Ethernet servers, that is a total bandwidth of 200 gigabits per second (Gbps). If you convert that into gigabytes per second (GB/s), this translates to roughly 30 GB/s. Now 30 GB/s coming from the ToR switch to the NIC can easily be served with a PCIe 4 (x8), which is 32 GB/s (so that was not the bottleneck),” he states. “However, as we move from 100G ethernet networking to 400G networking between the ToR and the server, we can do the math again. 400G in full duplex is equal to 800 Gbps. If you divide by eight, that is about 100 GB/s.”

According to Andani, 100 GB/s traffic between the ToR and the server cannot be realistically served with PCIe 4. This is because the maximum bandwidth that PCIe 4 can support is 64 GB/s, which is well short of what is needed to support 400G Ethernet.

“This is where we need to get to PCIe 5 (x16). If you look at PCIe 5 (x16), it provides an aggregate duplex bandwidth of 128 GB/s, enough to service 100 GB/s (required by 400G Ethernet),” he adds.
As Andani confirms, this paradigm is the same for both an Exascale cloud operation and on-premises enterprise servers.

“On-premise data centers are looking more like hyperscale or cloud data centers because the type of processing that some big enterprises need to get done is at the same level. However, they do it on- premise because of security concerns. Nevertheless, at the end of the day, the workloads are just as intensive. For the cloud, you will observe slightly more intensive workloads, but the cloud will likely trailblaze PCIe 5 and then enterprise will soon follow,” he concludes.

Semiconductor Engineering Editor in Chief Ed Sperling recently sat down with Suresh Andani, Senior Director, Product Marketing and Business Development at Rambus, to discuss the evolution of PCIe and its latest iteration: PCIe 5.

As Andani notes, PCIe 5 and subsequent iterations of the PCIe standard will continue to be one of the “key interfaces” that enable high-speed computing and processing in the data center.

“The base node in data centers are servers. In these servers, you will typically find the central processing unit (CPU), whether it is Xeon from Intel, EPYC from AMD or P9/P10 from IBM,” he explains.

“PCIe basically connects the SmartNIC or the NIC from the top of the rack switch to the CPU. It also connects the CPU to accelerators. In addition, more storage is moving away from SAS/SATA towards non-volatile memory express (NVMe) or PCIe. There are also a number of interfaces moving towards PCI Express and migrating to PCIe 5.”

According to Andani, with the amount of data exploding and high-performance computing (HPC) workloads increasing, PCIe 4 – even at x16 widths – is no longer sufficient, especially for applications that are running in the cloud.

“Networking in the cloud is also moving from 100GbE to 400GbE for which you need PCIe 5. The bandwidth enabled by PCIe 4 is simply not enough,” he adds.

As Andani observes, PCIe has steadily evolved from Gen 1 to Gen 5, with the Gen 1 specification –introduced in 2002 – running at 2.5 gigatransfers per second (GT/s).

“2.5 GT/s was once sufficient for the type of workloads that ran during earlier days. PCIe 5 runs at 32 GT/s and its successor, PCIe 6, was recently announced,” he explains. “It is important to note that between 2010 and 2017, SerDes I/O technology quadrupled, although the PCIe specifications stagnated at PCIe 3 for almost seven years. This made PCIe a system bottleneck.”

To avoid protracted refresh cycles and reduce system bottlenecks, PCI-SIG has detailed a steady two-year cadence at which the PCIe specs will be released in the future.

“PCIe 5 can take care of networking bandwidth all the way up to 400 gigabit Ethernet, solving a big problem that was there up until PCIe 4,” he states. “The PCIe 5 spec was finalized and released in May 2019 and the development of the PCIe 6 spec was announced in June 2019. It is expected that PCIe 6 will be finalized sometime in the 2021 timeframe. With PCIe moving at a two-year cadence, the interface] will no longer be a system bottleneck.”

Andani also notes that high-performance workloads, such as genomics, high-frequency trading and video transcoding, are all increasing in sophistication and demanding ever-more parallel processing.

“The parallel processing that’s needed for all these datasets requires heterogeneous computing. Specifically, it requires massively parallel architectures like GPUs or FPGAs, which is why a lot of these workloads are being offloaded from the main CPU to a co-processor (which some people refer to as an accelerator), whether they are GPUs, FPGAs, or even ASICs. In turn, heterogeneous computing with accelerators is pushing high bandwidth demand for PCIe, the key interface between the CPU and the accelerators,” he adds.

To better understand the evolution of PCIe, Andani presents a chart that compares the lane speeds of PCIe 4 and PCIe 5. PCIe 4 hit 16 GT/s per lane, while PCIe 5 doubles that speed to 32 GT/s per lane.

“If you look at the chart, you can see the full duplex aggregate bandwidth in gigabytes per second in x1, x2, x4, x8 and x16 configurations.

“So, the maximum aggregate bandwidth that PCIe 4 was able to provide was 64 gigabytes per second (GB/s) for duplex. PCIe 5 doubles that to 128 GB/s, a speed increase that is very much needed in hyperscale data center architectures.”

Indeed, sophisticated workloads continue to evolve along with an ever-increasing amount of data. This means bandwidth will need to double at a steady cadence, which is precisely why PCIe 6 will scale up to 64 GT/s. However, Andani emphasizes that the demand for bandwidth will always be insatiable.

“You will always need faster and faster interfaces as the workloads become more sophisticated,” he concludes.

Rambus has announced a comprehensive interface solution for PCI Express 5 (PCIe 5.0) consisting of a new PCIe 5.0 PHY and a co-verified Northwest Logic Expresso 5.0 controller.

Have you read our primer? PCI Express 5 vs. 4: What’s New? [Everything You Need to Know]

“Our high-speed SerDes and memory interface solutions make possible amazing advancements in performance-intensive applications in AI, data center, HPC, storage and networking,” said Hemant Dhulla, VP and GM of IP Cores. “Now we’ve added PCIe 5 to our industry-leading portfolio of high-speed interface solutions giving chip makers another tool to unleash the power of their designs.”

With co-verified PHY and controller, Rambus greatly reduces integration complexity for chip designers. Both PHY and controller support PCI Express 5 and are backward compatible to PCIe 4.0, 3.0 and 2.0. While together they are an ideal complete solution, both PHY and controller can be paired with PIPE 5.2 – compliant 3^rd-party solutions if so desired.

In addition to PCI Express 5, the Rambus PHY supports the Compute Express Link (CXL) connectivity standard. CXL is a new high-speed interconnect between CPUs and workload accelerators or other attached devices, as well as CPUs and memory. It maintains memory coherency between the CPU and attached devices. This greatly benefits accelerators used to complement CPUs in applications such as artificial intelligence and machine learning.

This latest portfolio addition highlights Rambus leadership in high-speed SerDes PHY and controller IP, leveraging the company’s long tradition of signal and power integrity expertise coupled with the design talent and products from newly acquired Northwest Logic.

Keep on reading: all the details on the PCI Express 5 interface solution

As data grows at an accelerating pace, more compute power and bandwidth are required to process this data, driving the need for larger and more complex system on chips (SoCs). This is particularly true at a time when our old friend Moore’s Law has lost a step.

However, as the complexity of SOCs increases, so do the costs to manufacture in leading-edge FinFET geometries. As misery loves company, achieving first-time-right silicon has become more difficult as well. Additionally, there are greater challenges for power scaling and yield. In other words, everything gets harder.

Chip disaggregation, or chiplets, offers an alternative to the traditional monolithic SoC scaling approach. Aggregating multiple chiplets to perform the function of a single monolithic IC de-risks the overall system by reducing complexity and increasing yields.

For applications like artificial intelligence (AI), where there is a “Cambrian explosion” in the number of SoCs and architectural approaches under development, chiplets are an ideal solution. Greater experimentation, and faster time to market are possible when a designer can revise a single chiplet as opposed to having to re-spin an entire SoC.

A key enabling technology required to make chiplet performance on par with that of an SoC is the high-speed interconnects between the chiplets. These must run at extremely high data rates and at very low power.

The Optical Internetworking Forum (OIF) specified the Common Electrical I/O (CEI) 112G XSR (Extra Short Reach) interface to provide the interconnect between chiplets, and between chiplets and optical engines. It is designed for low complexity and very low power while offering extremely high throughput capabilities.

Today, we announced the Rambus 112G XSR SerDes on 7nm FinFET process is now available for licensing. It joins a portfolio of 7nm Rambus PHY solutions including GDDR6, HBM2 and 112G LR (Long Reach) tailored for advanced applications such as AI, advanced driver-assistance systems (ADAS) and hyperscale data centers. With these interface solutions, chip designers can satisfy their need for speed and unleash the power of new compute architectures.