PCI Express over optical cabling: Performance, simplicity, efficiency

This vendor-written tech primer has been edited by Network World to eliminate product promotion, but readers should note it will likely favor the submitter's approach.

Optical solutions can be a key enabler for network architects who see value in using PCI Express (PCIe) as an I/O technology for data center connectivity. Using PCIe to natively connect servers, switches and storage elements can lower overall system costs by reducing or eliminating the number of protocol conversions. Additionally, this increased system simplicity provides networks with latency, power and dollars-per-gigabit advantages.

The PCI Special Interest Group (PCI-SIG) in 2007 released an external cabling specification enabling interconnection of PCIe systems at 2.5Gbps (Gen1) and 5Gbps (Gen2), facilitating PCIe extension and expansion. Copper cabling solutions appeared in a variety of channel widths to provide connectivity, but were limited in link distance and became bulky in size and weight at higher channel counts.

IN THE WORKS: Tablets to get faster with PCI Express 4.0

While the copper solutions were able to meet the basic needs of PCIe Gen1 and Gen2 external expansion, it is hard for copper to meet the price, performance and size/weight needs for PCIe Gen3 and its 8Gbps speeds. Fiber optic technology provides an alternate for high-channel-count PCIe Gen3 interconnects, providing increased link distances, lower size/weight and higher performance. What's more, fiber is becoming increasingly price competitive. [Also see: "Intel's Thunderbolt with fiber optics years away"]

Collaborations by silicon vendors such as PLX Technology and Avago Technologies have yielded breakthroughs in this area, such as the first PCIe Gen3 end-to-end fiber optic link to deliver a full 64Gbps (128Gbps bidirectional) performance for PCIe applications. The PCIe bus is a high-speed serial I/O providing connectivity between system peripherals (graphics cards, memory/disk drives, external I/O cards) and the central processor unit (CPU). PCIe provides full-duplex interconnect (Transmit/Receive, or Tx/Rx) that can be scaled by adding multiple lanes such as x4, x8, x16, and possibly even x32 lane links.

A number of applications exist in which PCIe may be used to connect a CPU with devices outside the box. Even at 2.5 Gbps (Gen1) and 5Gbps (Gen2), physical connections are limited to a few meters in length using the available copper cables. Physical link distances decrease at higher data rates, so PCIe Gen3's 8Gbps will further reduce the usable distance of copper cables. System designers, therefore, have expressed interest in optical cabling for PCIe applications such as data centers and enterprise systems spread out over wider areas and requiring longer cabling distances. Optical fiber-based solutions, these designers have come to understand, not only allow connections over a much longer distance but are capable of providing better bit-error-rate performance, improved immunity to electromagnetic interference, and are thinner and lighter allowing for easier placement and routing.

By using optical cabling, nearly anything connected today using PCIe can now be connected remotely. This allows users to leverage the ubiquity of PCIe for several network-intensive uses, such as memory/disk system interconnects, high-end audio/video applications, high-performance computing, and multi-chassis system interconnects.

For the physical connection, PCIe Gen3 needs parallel solutions of at least four, and typically eight lanes. The most natural fit is using parallel optical Tx/Rx modules that use vertical cavity surface emitting laser arrays and provide up to 150 meters of connectivity. Twelve-channel parallel optic devices capable of operating at 8Gbps per lane with multiple mechanical form factors are available today.

In a recent demonstration a PCIe Gen3 x8 link was successfully implemented over 30 meters of OM3 multi-mode optical fiber. Eye quality plots created with a free PLX software-development tool give designers a visual read on signal integrity and how error-free data is recovered, even over 30 meters of fiber.

PCIe functionality supported with optics

The demonstration shows PCIe as an active, eight-lane aggregation port. In many such applications, this port remains active at all times and much of the PCIe Active State Power Management (ASPM) functions are not used. Natively, this demonstration does not support PCIe Active State Power Management or in-band synchronous resets (out-of-band independent reset only).

The reason for this the lack of ASPM support is that the standard PCIe protocol specifies two features that are typically difficult to implement when using re-drivers and optical links:

Receiver detection -- the mechanism by which a transmitting device can determine if there is proper loading of the transmission line. Where proper loading exists, the transmitter is triggered to operate in one of several modes based on what is observed at the device receiver. Optical links typically present a 50-ohm termination at all times. As such, optical links may not accurately indicate the presence of a valid PCIe receiver at the opposite end of the link.

Electrical idle -- a second key parameter of concern with optical links. Electrical idle indicates when a link has entered a temporarily inactive state (quiesed) and can herald entry/exit into low power states and link speed changes. To be PCIe-compliant, the link transmitter must both be in a static state and also hold the transmission line to a fixed common mode. Line chatter or improper bias can lead to false EIDLE \detection and/or exit from the EIDLE state. This methodology can bypass this signaling, yet manage standard PCIe link speed negotiation transparent to the user.

Spread Spectrum Clocking (SSC) -- Typically, when employing optical fiber, both ends of the link will not reside in the same enclosure. This means they will share neither the same Reset nor the same system clock. In the demonstration described above, there were no means to implement a synchronous reset or clock across the link. In this example, SSC was not active. However, because the interface is optical, there is a reduced need for electromechanical interference (EMI) suppression of the link, and SSC operation on the wire is not needed. In cases where system emissions are high, PCIe switches with SSC isolation, such as those from PLX, allow the system to operate with lower emissions, yet keep the optical link in a constant frequency mode.

Synchronous Reset -- The remote optical card can undergo a separate reset upon power-up and is ready for link training once the host box becomes active. Typically, the remote box is powered ahead of the system box (server/PC). Alternately, if the operating system (OS) is under full control of the management layer, the systems can be powered up in any order. Once both systems are confirmed to be powered (for example, the management software checks for link status) the OS can initiate standard system enumeration/programming methods.

Some of these functions, such as no clock sharing SSC operation and optically based "in-band" reset mechanisms, are under evaluation. As implemented, the link supports: asynchronous operation -- no native SSC, but SSC isolation provisions; L0 active state only (link enable/disable functional under a controlled OS); PCIe normal link speed negotiation; and the ability to configure a PCIe standard link width down training.

PCIe has established itself as the first-choice interconnect for network infrastructures. Now, designers can extend its usefulness even more efficiently and over longer distances through optical cabling, enabling servers, switches and storage systems to be integrated into robust data centers, while providing significant savings in latency, power and costs.

Reginald Conley is senior director of hardware applications at PLX Technology. He can be reached at rconley@plxtech.com.

Read more about data center in Network World's Data Center section.

Join the newsletter!

Error: Please check your email address.

More about EMIetworkIntelPCI Special Interest GroupPLX TechnologySASTechnology

Show Comments