Suresh Andani, senior director of product marketing at Rambus, has written an article for Semiconductor Engineering that takes a closer look at why data center scaling requires new interface architectures.
As Andani notes, global data traffic is growing at an exponential rate. More specifically, 5G networks are enabling billions of AI-powered IoT devices untethered from wired networks, while machine learning’s voracious appetite for enormous data sets is skyrocketing. Moreover, data intensive video streaming for both entertainment and business applications continues to accelerate, as ADAS and autonomous vehicles add yet another torrent of data.
“Nowhere is the impact of all this growth being felt more intensely than in the data centers at the heart of the global network,” he explains. “Ethernet networks interconnect the switch, server and storage devices of the data center, and are scaling rapidly to meet the tidal wave of data.”
According to Andani, Ethernet’s evolution kicked into high gear with the 25 Gigabit Ethernet Consortium’s introduction of 25G and 50G standards in July 2014. Flash forward to 2020, and the re-branded Ethernet Technology Consortium announced the 800G Ethernet standard in April. This represents a 16X increase in 6 years from the 50G milestone, or more than 2.5X increase every two years. As such, 1.6T Ethernet should be on pace to make its appearance in late 2022 or early 2023.
“At the silicon-level, scaling to smaller process nodes has enabled switch ASICs to advance from 12.8 Terabits per second (Tbps), to 25.6T, to the upcoming generation 51.2T,” writes Andani. “But there is a critical issue that arises with the end of Dennard scaling: power.”
Indeed, thanks to Dennard scaling, die shrinks that enabled a doubling of bandwidth, a doubling of transistor density, and a halving of per transistor power, allowed for power per area to remain constant. However, in a post-Dennard world, power consumption has become a dominant factor in system architectural design as power per area rises even with process node shrinks.
“12.8T switch ASICs used 25G and 50G medium reach (MR) and long reach (LR) SerDes for 25G, 50G and 100G Ethernet links. As switch ASICs migrate to 25.6T ASICs and 400G ports, 512 SerDes running at 50G are needed to move data on and off chip,” states Andani. “That number of MR and LR SerDes burn too much power to be practical. Architecturally, this motivates moving from the electrical domain to the optical domain to keep the power budget in check.”
In response, says Andani, the SerDes for 25.6T ASICs transition to 50G very short reach (VSR) interfaces to link the silicon with on-board or pluggable optical modules.
“8 50G lanes are aggregated for a 400G Ethernet connection. The power savings afforded with the simpler VSR architecture, eliminating much of the DFE circuitry required for an MR/LR class SerDes, is substantial,” he elaborates. “But with the move to 51.2T ASICS and 800G Ethernet, it most likely won’t be enough. [For] 51.2T ASICs, the SerDes need to scale to 100G to fit them all on chip. As everything has doubled, 512 SerDes running at 100G are needed for the bandwidth. These can be aggregated for 64 links of 800G Ethernet.”
As signal losses rise with data rate, the 100G VSR circuitry will be more complex and burn more power than the 50G variety. Thus, the architecture will need to evolve to one which moves the optics even closer to the silicon.
“Here’s where co-packaged optics (CPO) enters the picture. By integrating 800G optical engines in the same package as our switch ASIC, we can drop the size, complexity and power of the 100G links to extra short reach (XSR) requirements,” Andani explains. “Using 100G XSR links running at less than 1 picojoule per bit, we can reduce the I/O power by more than 80% and the switch ASIC thermal design power (TDP) by more than 25% compared to an MR/LR SerDes running at the same speed. This gives us the means to scale to the 51.2T ASICs needed for the next generation of network switches.”
With the end of Dennard Scaling, architectural innovation is the key to advancing the performance of network devices. Moving optics closer to the silicon, with optical modules and then with CPO, enables us to overcome the growing obstacle of power.
“112G XSR SerDes and CPO make possible network devices with 800G and then 1.6T Ethernet links to handle the growing flood of data traffic,” concludes Andani.