SerDes PHYs

From 400 Gbps to 800 Gbps

A recently published report by the Dell’Oro Group states that 400 Gbps is expected to comprise 20 percent of data center switching revenue by 2020. According to the research group, higher speeds – 100 Gbps, 200 Gbps, 400 Gbps and 800 Gbps – are all forecast to drive significant growth over the next five years. Perhaps not surprisingly, cloud data centers are expected to play a key role in the upcoming speed jump for networks (from 100 Gbps to 400 Gbps).

Meanwhile, a market report released by the 650 Group confirms that 200 Gbps, 400 Gbps and 800 Gbps will all ship in the next five years – with the latter projected to ramp early next decade.

“The first wave of 200 Gbps and 400 Gbps will hit the market in early 2018 as the Ethernet switch market expands the number of port speeds just three years since the last major set of technology advances,” says Alan Weckel, Founder and Technology Analyst at 650 Group. “Both 200 Gbps and 400 Gbps will emerge off of 50 Gbps SerDes technology announced in 2017. The rapid pace of innovation is led not only by impressive technology improvements, but by Software Defined Networks (SDN) which is enabling the cloud to better utilize the compute and networking resources at their disposal.”

Testing 400 Gbps, eyeing 1.6 Tbps

According to Ronen Isaac of Military Embedded Systems, the industry is currently testing and ratifying technologies that will bring speeds up to 400 Gbps and beyond.

“Between 2018 and 2020, 50 Gbps and 200 Gbps will be tested and adopted. Thousands of 25GbE servers and eventually 50GbE servers in hyper-scale data centers, such as cloud service providers, will drive the need for 400GbE to the metropolitan area networks (MAN) and wide area networks (WANs),” he says. “In the not-too-distant future, testing will begin on 200 Gbps, 8000 Gbps, and astonishingly data rate speeds of 1 Tbps and 1.6 Tbps with expected testing and ratification of the standards by the year 2020.”

Conclusion

Ethernet is quickly moving from 40Gbps to 100Gbps to 400Gbps, thereby spurring a number of new SerDes initiatives and developments. Indeed, 25Gbps SerDes served as the key enabler for 100Gbps Ethernet, with the industry expected to initially leverage 50Gbps SerDes for 400Gbps Ethernet before moving on to 100Gbps SerDes technology.

Concurrently, SerDes technology is shifting from NRZ to PAM4 as it accelerates from 25Gbps to 50Gbps. This has prompted a number of architectural changes, including the replacement of traditional analog with ADC + DSP to help meet performance targets and maintain a similar power/area envelope.

Interested in learning more about SerDes technology? You can download our eBook below.

Earlier this month, EDN’s Martin Rowe wrote an article that explores various industry viewpoints – shared at a DesignCon 2018 panel – about the future of NRZ and PAM4 (NRZ: non-return-to-zero). According to Rowe, four-level pulse amplitude modulation (PAM4) has overtaken NRZ, save for the shortest distances over copper connections. Indeed, PAM4 has become the standard, especially for medium-haul and long-haul serial data links that run over fiber where each link transmits 50 to 56 Gbps per lane.

However, as Intel’s Mike Li told panel participants and attendees, NRZ is not dead.

“At 56 Gbps, NRZ is still good for ultra-short reach applications. You could use PAM4, but it’s expensive. NRZ circuits are much simpler,” he explained. “NRZ could work for extra-short reach (XSR) and very-short reach (VSR) applications.”

Li also noted that “we’ve solved 56 Gbps and now we’re on to 112 Gbps.”

According to Inphi’s Mark Marlett, digital-signal processing (DSP) will make 112 Gbps data rates possible.

“DSP is enabling PAM4 optical channels,” he said. “Reaches for 10 km, 40 km and 80 km are enabled with DSP. We can also combine as many as 80 wavelengths on a fiber today. NRZ is dead in fiber. PAM4 and FEC (forward error correction) are in.”

A closer look at NRZ

NRZ – which consists of 1’s and 0’s – is the preferred and standardized encoding scheme for 28 Gbps rates.  As noted above, NRZ could also be used for various XSR and VSR scenarios at 56 Gbps rates. Put simply, NRZ signaling transmits data bits serially one at a time. This means a signal can be a 1 or a 0 depending on the voltage level. The baud rate, or the speed at which a symbol can change, equals the bit rate for NRZ signals. The Nyquist frequency of the signals equals one-half the baud rate, so faster data rates can be achieved by transmitting the signal at higher fundamental Nyquist frequency.

NRZ technology can still pose significant design and reuse challenges for SerDes designers working on XSR and VSR applications – with refreshed standards from OIF and IEEE mandating increased receiver sensitivity (down to 35 mV, differential peak-2-peak). As semiconductor process nodes shrink from 28nm to deep sub-micron nodes such as 16/14nm and 10/7nm, the transistor threshold voltage (VT) is also scaling downwards. This combination of higher data rates and shrinking process nodes leaves very little margin for error.

With shorter unit intervals and closing eyes, triggering becomes ever more complex and requires enhanced receiver equalization such as continuous-time-linear equalization (CTLE) and decision feedback equalization (DFE) to correct (the CTLE and DFE work closely together to open the eye). In addition, channel loss and reflections (noise) at increased data rates and noise complicate forward error correction (FEC). Moreover, jitter budgets are stringent for 400GbE systems at 17ps UI. Last, but certainly not least, design of circuits such as capture latches, analog-to-digital converter (ADC) and transmit drivers are particularly challenging to implement at high data rates.

Understanding PAM4

With serial data rates hitting 56 Gbps per channel and beyond, signal impairments caused by increased bandwidth has prompted the high-speed serial data industry to adopt PAM4. For PAM4 signals, the baud rate equals one-half the bit rate and the Nyquist frequency equals one-fourth the bit rate. Compared to PAM2/NRZ, essentially, PAM4 cuts the bandwidth for a given data rate in half by transmitting two bits in each symbol. This allows engineers to double the bit rate in the channel without doubling the required bandwidth (using PAM4 signaling, a 56 Gbps bit rate is transmitted at 28 GBauds and has a Nyquist frequency of 14 GHz; with NRZ signaling, the 56 Gbps bit rate is transmitted at 56 GBauds and has a Nyquist frequency of 28 GHz).

According to David Maliniak of Teledyne LeCroy, PAM4 takes the L (Least Significant Bit) signal, divides it in half and adds it to the M (Most Significant Bit) signal. The result is four signal levels instead of two, with each signal level corresponding to a two-bit symbol. To be sure, PAM4 doubles the number of bits in serial data transmissions by increasing the number of levels of pulse-amplitude modulation – although it does so at the cost of noise susceptibility. In short, PAM4 increases the number of voltage levels from two to four, while reducing noise tolerance (33% of amplitude compared to NRZ).

Indeed, multiple symbol levels make PAM4 more sensitive to amplitude noise than PAM2/NRZ. According to Tektronix, PAM4’s resilience to ISI at a given data rate on lossy electrical channels like backplanes is the primary reason for switching from PAM2/NRZ. In general, PAM4 signaling is preferred if the channel loss at NRZ Nyquist frequency is larger than the loss at PAM4 Nyquist frequency by more than 9.6dB.6

As with NRZ, PAM4 signals are also affected by jitter, channel loss and inter-symbol interference. In addition, measurements for the three eyes are further complicated by new receiver behavior, such as three slicer thresholds, individual slicer timing skew, equalization and clock and data recovery. Moreover, moving to 56G PAM4 immediately causes a loss of 9.6 dB, although there is still a demand for 30 dB+ reach for these systems. Unsurprisingly, PAM4 signal analysis borrows a great deal from the techniques developed to analyze jitter and noise for PAM2/NRZ. A number of PAM2/NRZ techniques are also applicable to PAM4, including differential signaling, clock recovery and equalization for both the transmitter and receiver.

Interested in learning more about NRZ and PAM4 signal integrity challenges? You can download our eBook below.

Niraj Mathur, VP of high speed interface products at Rambus, recently penned an article for Semiconductor Engineering that explores the importance of PCI Express 4.0 in the data center.

Read our primer: Rambus launched PCI Express 5 »

“Modern CPUs rely on the following primary interconnect types: memory interconnects, primarily supported by DDR4 today; high speed chip-to-chip cache coherent interconnect, typically supported by proprietary standards; and low speed links such as USB and SATA for low-level management and configuration,” Mathur explained.

“For almost everything else, there is PCIe. Easily scalable via multi-lane links, PCIe has always been backwards compatible and extensively supported by all modern OSs, software and drivers. Not surprisingly, PCIe has been widely deployed throughout the data center, enterprise and client PC markets.”

According to Mathur, the advent of cloud computing has demanded that data centers continuously increase compute power by adding faster CPUs with ever increasing numbers of cores. Data centers have also adopted newer processing techniques with GPUs and accelerators to support emerging machine learning, artificial intelligence and deep learning workloads.

“These kinds of use-cases require higher performance processing [paired] with higher performance storage, all with minimal latency – a paradigm that demands interconnects to optimally feed processing capabilities,” he elaborated.

As Mathur notes, PCIe 1.0 supported a transfer rate of 2.5 Gbps per lane at launch. Subsequent upgrades released approximately every 3-4 years doubled bandwidth each time (PCIe 2.0 at 5 Gbps in 2006, and PCIe 3.0 at 8 Gpbs in 2010). However, this cadence was abandoned following the rollout of PCIe 3.0. In fact, there was a 7-year gap before speeds reached 16 Gpbs, with the PCIe 4.0 specification publically released in 2017.

“Multiple CPU, storage, accelerator and network adaptor vendors have already developed PCIe 4.0 compatible products in anticipation of broad deployment,” he continued. “This ramp is imminent with the public release of the specification and extensive ecosystem support.”

According to Mathur, there are two immediate technologies that will benefit from the proliferation of PCIe 4.0: the adoption of next-generation NVMe storage technologies and the deployment of GPU/FPGA accelerators in the data center.

NVMe is a non-volatile memory interface standard that utilizes PCIe interfaces to SSDs. Emerging NVMe storage technologies are saturating existing PCIe 3.0 interfaces and will only achieve optimal performance with the increased bandwidth offered by PCIe 4.0. Meanwhile, accelerator hardware such as GPUs and FPGAs, which are already resorting to proprietary interconnects to address bandwidth bottlenecks, will be able to leverage a high-performance industry standard interface resulting in seamless compatibility across platforms.

Moreover, complementary standard initiatives such as CCIX are purporting to add cache coherence capability, required for efficient accelerator offload, to the ubiquitous PCIe transport layer.

“PCI-SIG, the standards body that governs PCIe, is accelerating the development and release of the PCIe 5.0 specification, likely to address unabated market demand and the delay in ratifying PCIe 4.0,” Mathur concluded. “PCI-SIG has also developed a cabled technology called OCuLink to connect PCIe devices, thereby enabling new out-of-the-box compute and storage use-cases. Innovations such as these guarantee PCIe’s relevance to compute and storage infrastructure and cement its continuing role in defining the data center of the future.”

Rambus’ silicon-proven, high-speed SerDes solutions

Rambus is now offering a suite of silicon-proven, high-speed SerDes solutions developed for the GLOBALFOUNDRIES high-performance FX-14™ ASIC platform. The suite includes 16G MPSL (multi-protocol serial link), 30G C2C (chip-to-chip) and 30G VSR (very short reach) PHYs.

Rambus’ SerDes PHYs include a physical media attachment (PMA) hard macro and physical coding sub-layer with a built-in self-test (PCS-BIST) soft macro. The PHYs can be configured to multiple channel widths and packaging options, thereby simplifying integration and maximizing design flexibility.

According to Luc Seraphin, senior vice president and general manager of the Rambus Memory and Interfaces Division, the Rambus SerDes PHYs are built on GLOBALFOUNDRIES 14nm FinFET (14LPP) process technology and optimized for power and area at peak bandwidth. This allows the PHYs to support Ethernet speeds up to 100Gb and beyond for high-speed wireline, wireless, 5G network infrastructure, high-performance servers, storage, connectivity and compute applications.

“Data traffic and bandwidth demands have exploded, driving the insatiable need for highly-optimized, high-performance semiconductor solutions,” Seraphin explained. “Through our collaboration with GLOBALFOUNDRIES, we are delivering robust high-speed interface IP that enables innovative chips and systems designed specifically for the data center and communications markets – helping GLOBALFOUNDRIES deliver value to its customers through tested solutions.”

GLOBALFOUNDRIES 14nm FinFET (14LPP) process technology

According to GLOBALFOUNDRIES, the FX-14 ASIC platform is a comprehensive semiconductor design system targeted for wired and wireless networking, storage and cloud computing applications. FX-14 is designed to deliver more performance per watt and use fewer watts per GHz in less space. Compared to its predecessor, FX-14 offers significant power and area savings, including:

• Up to 50% less active power
• Up to 85% less leakage
• Up to 55% less area

This combination of performance, power and area advantages can help chip designers stay ahead of system-level demands driven by evolving network and data center architectures and the race to 5G solutions.

Rambus HBM2 PHY: Developed for GLOBALFOUNDRIES FX-14 ASIC platform

Earlier this year, Rambus announced the availability of its High Bandwidth Memory (HBM) Gen2 PHY, which was also developed for the GLOBALFOUNDRIES FX-14 ASIC Platform. Designed for systems that require low latency and high bandwidth memory, the Rambus HBM PHY, built on the GLOBALFOUNDRIES advanced 14nm Power Plus (LPP) process technology, targets networking and data center applications.

The PHY is fully compliant with the JEDEC HBM2 standard and supports data rates up to 2000 Mbps per data pin, enabling a total bandwidth of 256 Gigabytes per second. The Rambus HBM PHY delivers high performance with reduced power consumption when compared to other memory solutions. It combines 2.5D packaging with a wider interface (1024 bits) at a lower clock speed, which results in a higher overall throughput while remaining energy efficient for even the most high-performance computing applications. The Rambus HBM2 PHY is another key component in Rambus’ memory and SerDes high-speed interface IP portfolio for networking and data center applications.

Semiconductor Engineering’s Anna Steffora Mutschler has written an article about how peak power poses a serious design challenge for chips, electronic systems and data centers. Issues related to peak power, says Mutschler, have become more significant as process nodes shrink. This is because the dielectrics and wires are thinner – while the margins for noise are lower.

Peak power, elaborates Mutschler, is the maximum power generated at any point in time and often occurs when a device switches from an “off” or “sleep” state to an “on” state. Peak power can also occur in a power domain that is used to control certain blocks with memories, such as when one or more applications are being used for intensive computing.

According to Mutschler, peak power issues are a direct result of the growing complexity in designs, fixed or shrinking power budgets and the need to process more data more quickly. For example, in mobile devices, more features require more circuitry to be turned off or put into sleep mode. In the data center, the sheer volume of information that requires rapid processing is rapidly increasing. However, servers can only run so quickly due to rack thermal limits.

“This is all directly related to power dissipation and density,” Frank Ferro, senior director of product management at Rambus, told Semiconductor Engineering. “If you look at SerDes, the first question customers ask is, ‘What is the power, performance and area (PPA)?’ And they get there with memory, as well, as you go up in speed. These companies may budget a certain percentage for memory, SerDes, the processor and the PHY. If you look at some of the networking chips, they give off a lot of heat. If the peak power is not within expectations, they knock you out of the running early.”

As Ferro notes, scaling SerDes from 28nm processes to 14nm hasn’t resulted in significant performance improvement. However, says Ferro, this is changing due to a mix of digital with analog circuitry.

“You always need to look at the architecture for the digital side, because it’s easier to port, but it’s also a challenge to get the performance up,” he concluded. “But there’s also a challenge on the performance side. If you look at what’s been going on in Ethernet, it took seven years to move from 1 gigabit per second to 10 gigabits per second. Now, 28G is the workhorse and we’re seeing companies asking for 56 and 112. The rate of development is phenomenal.”

Interested in learning more? The full text of “Managing Peak Power” by Anna Steffora Mutschler is available here.

Semiconductor Engineering’s Ed Sperling recently penned an article about the challenges of IP reuse. The basic business proposition for third-party IP, says Sperling, is that it’s cheaper, faster and less problematic to buy rather than build.

[“However], things haven’t exactly worked out according to plan, either for companies that license IP or those that develop it,” he explained.

“For IP licensees, just keeping track of an endless series of updates is becoming unwieldy. Complex designs often include multiple builds of dozens of IP blocks, particularly at advanced process nodes.”

IP developers also face their own set of challenges, he emphasizes, as the demand for more customized IP transforms the business from “write-once, use everywhere,” to “write once, modify every time.” Even in cases where the IP is relatively fixed (such as processor cores), says Sperling, customers may require extensive system optimization tweaks.

To further complicate matters, every new rev of a process technology at advanced nodes requires changes to the IP. Designing at 10/7nm is particularly difficult because the process and the IP are in almost constant flux. Nevertheless, even at older nodes the advent of new processes to address new or existing markets have made it more difficult for IP vendors to stay current.

Commenting on the above, Frank Ferro, senior director of product management at Rambus, told Semiconductor Engineering that OEMs once shed ASIC teams in favor of buying commodity IP and chips.

“[This] made it a lot harder for these companies to differentiate themselves. Now companies are hiring back ASIC teams and doing a mixed model,” said Ferro. “They do IP development where it makes sense and add customization. But even with standard IP, they add a little differentiation to make it a little better.”

Nevertheless, says Ferro, the goal is still to build IP and license it to at least 10 customers.

“We’ve been in the custom business because of the nature of hard IP, which means we work with the foundries to develop that IP. But even if it’s off-the-shelf, there are little differences from one design to the next,” he continued. “So even if it’s a stable architecture like 28 [gigabits per second] SerDes, which is a known design, you might only be able to re-use a portion of that because it’s very process-dependent. For 28 SerDes, you have a version to use with PCIe, with PONs (passive optical networks) and with short-reach – and you do that because you don’t want to pay the power and area penalty for re-using the same IP. That happens with automotive IP, too.”

Interested in learning more about the challenges of IP reuse? The full text of “The Limits of IP Reuse” by Ed Sperling can be read here on Semiconductor Engineering.