Improving I/O Performance and Reducing Costs with Single Root I/O Virtualization (SR-IOV)

While server virtualization is being widely deployed in an effort to reduce costs and optimize data center resource usage, an additional key area where virtualization has an opportunity to shine is in the area of I/O performance and its role in enabling more efficient application execution.

Single Root I/O Virtualization (SR-IOV) provides a step forward in making it easier to implement virtualization within PCIe systems, by enabling multiple Virtual Machines (VMs) to share PCI hardware resources.

We spoke with Philippe Legros, Senior Principal Engineer in the PCIe/CXL Interface IP unit at Rambus to discuss why designing systems that have been natively built on SR-IOV-enabled hardware may be the most cost-effective way to improve I/O performance and how to easily implement SR-IOV in PCIe devices

“Using virtualization provides several important benefits to system designers:

• It makes it possible to run a large number of virtual machines per server, which reduces the need for hardware and the resultant costs of space and power required by hardware devices
• It creates the ability to start or stop and add or remove servers independently, increasing flexibility and scalability
• It adds the ability to run different operating systems on the same host machine, again reducing the need for discreet hardware” 

What is a virtualized system exactly, Philippe?

“A virtualized system is a discreet system which contains:

• Several virtual machines (VM),
• A supervisor, also referred to as the Virtual Machine Manager (VMM), and
• PCI Express hierarchy

Within this virtualized system, the Supervisor plays a crucial role; it provides the interface between the hardware and the virtual machines, it is responsible for security, and it ensures that there is no possible interaction between virtual machines.”

How is a system virtualized?  

“Any system can be virtualized without specific SR-IOV technology. In a more traditional virtualization scenario, the Supervisor must emulate virtual devices and perform resource sharing on their behalf by instantiating a virtual Ethernet controller for each virtual machine. However, this creates an I/O bottleneck and often results in poor performance. In addition, it creates a tradeoff between the number of virtual machines a physical server can realistically support and the system’s I/O performance. Adding more VMs can significantly aggravate the bottleneck.”

Is there a better way?

“Designing systems with hardware that incorporates SR-IOV support allows virtual devices to be implemented in hardware and enables resource sharing to be handled by a PCI Express® device such as an Ethernet controller.”

“The benefit of using SR-IOV over more traditional network virtualization is that in SR-IOV virtualization, the VM is talking directly to the network adapter through Direct Memory Access (DMA). Using DMA allows the VM to bypass virtualization transports such as the VM Bus and avoids requiring any processing in the management partition. By avoiding the use of a switch, the best possible system performance is attained, providing close to “bare-metal” performance.”

Can you talk us through implementing SR-IOV in an Adapter Card?  

“The SR-IOV specification enables a hardware provider to modify their PCI card to define itself as independent devices to a VMM. To achieve this, the SR-IOV architecture distinguishes two types of functions: Physical Functions (PFs) and Virtual Functions (VFs).

• Physical Functions are full-featured PCIe functions; they are discovered, managed, and manipulated like any other PCIe device and have full configuration spaces. You can configure or control the PCIe device via the PF and in turn, the PF can move data in and out of the device. Each PCI Express device can have up to 256 physical PFs. Each PF is independent and is seen by software as a separate PCI Express device, which allows several devices to be present in the same chip and makes software development easier and less costly.
• Virtual Functions, on the other hand, are ‘lightweight’ PCIe functions designed solely to move data in and out of the device. Each VF is attached to an underlying PF and each PF can have from zero (0) to one (1) or more VFs. In addition, PFs within the same device can each have a different number of VFs. While VFs are similar to PFs, they intentionally have a reduced configuration space because they inherit most of their settings from their PF.

“In order to effectively implement SR-IOV, you must be able to access the VF configuration space. Traditional routing only allows 8 Physical Functions, and the device number is always “0” in PCIe. So we use alternate routing with the standard fields to extend the number of Physical Functions up to 256. SR-IOV extends this even further by allowing the use of several consecutive bus numbers for a single device, thus enabling many more than 256 functions.”

“Because VFs are light-weight function, their configuration space is significantly different from PFs.

• In VFs most registers are hardwired and are either set to “0” or “1” or take the same value as their PF. Status and command registers are implemented but only the master-enable and RWC status register bits. Base address/Expansion ROM are not implemented.
• PCI Express capability is implemented with the same settings as physical functions and MSI capability is implemented, with MSI-X capability optionally implemented, also with the same settings as physical functions
• Power management capability and extended capability is not implemented.
• Application-specific capabilities can be implemented, and configured, however.

In order to access the VF memory spaces, up to 6 VF BARs are implemented in the PF SR-IOV capability. They are similar to normal BAR registers except that VFs attached to a PF derive their own memory spaces from these VF BARs.

During initial SR-IOV set-up and initialization VFs are not enabled and are invisible. The Supervisor then detects the device and configures PFs. If the host system and device driver detect SR-IOV capability then they will :

• Configure the number of VFs,
• Assign addresses to VF BARs, and
• Enable VFs

Once set up, each VM can be assigned a virtual device and can access it directly via its VF driver. There must be at least one VF per VM, otherwise the Supervisor will need to perform some or all sharing management, reducing the benefits of utilizing SR-IOV. The market trend a few years ago was to have 64 VMs, but now we are seeing systems with 256 and sometimes even more VMs, so a SR-IOV capable device should support a similar number of VFs.

So where does Rambus come in?

“Many ASIC and FPGA designers are realizing that the IP they choose is crucial to the success of their SR-IOV implementation. By choosing PCIe IP vendors who understand and design for SR-IOV inherently, designers can ensure a more seamless integration with their host system.

Rambus, a long-time leader in PCIe IP innovation, designs its range of PCIe and CXL Controllers to provide native SR-IOV support and Rambus IPs deliver industry-leading specifications to fully enable SR-IOV functionality.

For example, our latest PCIe Controller for PCIe 6.0 provides SR-IOV support with up to 2048 Virtual Functions and optional MSI/MSI-X register remapping to memory for reduced gate count when SR-IOV is implemented. It is compliant with the PCI-SIG SR-IOV specification and provides an easy and cost-effective way to integrate SR-IOV support into PCIe devices.

In summary, the key benefits of using SR-IOV to achieve virtualization include:

• Enabling efficient sharing of PCIe devices, optimizing performance and capacity
• Creating hundreds of VFs associated with a single PF, extending the capacity of a device and lowering hardware costs
• Dynamic control by the PF through registers designed to turn on the SR-IOV capability, eliminating the need for time-intensive integration
• Increased performance via direct access to hardware from the virtual machine environment”

As the industry leader in PCI Express IPs, Rambus assures ease of integration and first-time right functionality. In addition, Rambus provides a comprehensive, SR-IOV Reference Design enabling quick implementation and reducing time-to-market and offers free evaluation to enable a hands-on trial of our PCIe and CXL IPs before purchase. For more information, visit Rambus at www.rambus.com.