PCI Express: Difference between revisions

This article has multiple issues. Please help improve it or discuss these issues on the talk page. (Learn how and when to remove these messages) No issues specified. Please specify issues, or remove this template. (November 2009) (Learn how and when to remove this message)

PCI Express (Peripheral Component Interconnect Express), officially abbreviated as PCIe (or PCI-E, as it is commonly called), is a computer expansion card standard designed to replace the older PCI, PCI-X, and AGP standards. PCIe 2.0 is the latest standard for expansion cards that is available on mainstream personal computers.^[1]

PCI Express is used in consumer, server, and industrial applications, as a motherboard-level interconnect (to link motherboard-mounted peripherals) and as an expansion card interface for add-in boards. A key difference between PCIe and earlier buses is a topology based on point-to-point serial links, rather than a shared parallel bus architecture.

The PCIe electrical interface is also used in a variety of other standards, most notably the ExpressCard laptop expansion card interface.

Conceptually, the PCIe bus can be thought of as a high-speed serial replacement of the older (parallel) PCI/PCI-X bus.^[2] At the software level, PCIe preserves compatibility with PCI; a PCIe device can be configured and used in legacy applications and operating systems which have no direct knowledge of PCIe's newer features. In terms of bus protocol, PCIe communication is encapsulated in packets. The work of packetizing and depacketizing data and status-message traffic is handled by the transaction layer of the PCIe port (described later). Radical differences in electrical signaling and bus protocol require the use of a different mechanical form factor and expansion connectors (and thus, new motherboards and new adapter boards).

PCIe, unlike previous PC expansion standards, is structured around point-to-point serial links, a pair of which (one in each direction) make up lanes; rather than a shared parallel bus. These lanes are routed by a hub on the main-board acting as a crossbar switch. This dynamic point-to-point behavior allows more than one pair of devices to communicate with each other at the same time. In contrast, older PC interfaces had all devices permanently wired to the same bus; therefore, only one device could send information at a time. This format also allows channel grouping, where multiple lanes are bonded to a single device pair in order to provide higher bandwidth.

The number of lanes is negotiated during power-up or explicitly during operation. By making the lane count flexible a single standard can provide for the needs of high-bandwidth cards (e.g., graphics cards, 10 Gigabit Ethernet cards and multiport Gigabit Ethernet cards) while also being economical for less demanding cards.

Unlike preceding PC expansion interface standards, PCIe is a network of point-to-point connections. This removes the need for arbitration of the bus or waiting for the bus to be free and allows for full duplex communications. This means that while standard PCI-X (133 MHz 64 bit) and PCIe ×4 have roughly the same data transfer rate, PCIe ×4 will give better performance if multiple device pairs are communicating simultaneously or if communication within a single device pair is bidirectional.

Specifications of the format are maintained and developed by the PCI-SIG (PCI Special Interest Group): a group of more than 900 companies that also maintain the Conventional PCI specifications.

PCIe devices communicate via a logical connection called an interconnect^[3] or link. A link is a point-to-point communication channel between 2 PCIe ports, allowing both to send/receive ordinary PCI-requests (configuration read/write, I/O read/write, memory read/write) and interrupts (INTx, MSI, MSI-X). At the physical level, a link is composed of 1 or more lanes;^[3] low-speed peripherals (such as an 802.11 Wi-Fi card) use only a single-lane (×1) link, while a graphics-adapter typically uses a much wider (and thus faster) 16-lane link.

A lane is composed of a transmit and a receive pair of differential lines. Each lane is composed of 4 wires, meaning that, conceptually, each lane is a full-duplex byte stream, transporting packets containing the data in 8 bit 'byte' format, between the two endpoints of a link, in both directions simultaneously.^[4] Physical PCIe slots may contain from one to thirty-two lanes, in powers of two (1, 2, 4, 8, 16 and 32).^[3] Lane counts are written with an × prefix (e.g., ×16 represents a sixteen-lane card or slot), with ×16 being the largest size in common use.^[5]

The bonded serial format was chosen over a traditional parallel format due to the phenomenon of timing skew. Timing skew is a direct result of the limitations imposed by the speed of an electrical signal traveling down a wire, which it does at a finite speed. Because different traces in an interface have different lengths, parallel signals transmitted simultaneously from a source arrive at their destinations at different times. When the interconnection clock rate rises to the point where the wavelength of a single bit is less than this difference in path length, the bits of a single word do not arrive at their destination simultaneously, making parallel recovery of the word difficult. Thus, the speed of the electrical signal, combined with the difference in length between the longest and shortest trace in a parallel interconnect, leads to a naturally imposed maximum bandwidth. Serial channel bonding avoids this issue by not requiring the bits to arrive simultaneously. PCIe is just one example of a general trend away from parallel buses to serial interconnects. Other examples include Serial ATA, USB, SAS, FireWire and RapidIO. The multichannel serial design also increases flexibility by allowing slow devices to be allocated fewer lanes than fast devices.

A PCIe card will fit into a slot of its physical size or bigger, but may not fit into a smaller PCIe slot. Some slots use open-ended sockets to permit physically longer cards and will negotiate the best available electrical connection. The number of lanes actually connected to a slot may also be less than the number supported by the physical slot size. An example is a x8 slot that actually only runs at ×1; these slots will allow any ×1, ×2, ×4 or ×8 card to be used, though only running at the ×1 speed. This type of socket is described as a ×8 (×1 mode) slot, meaning it physically accepts up to ×8 cards but only runs at ×1 speed. The advantage gained is that a larger range of PCIe cards can still be used without requiring the motherboard hardware to support the full transfer rate—in so doing keeping design and implementation costs down.

PCI Express Mini Card (also known as Mini PCI Express, Mini PCIe, and Mini PCI-E) is a replacement for the Mini PCI form factor based on PCI Express. It is developed by the PCI-SIG. The host device supports both PCI Express and USB 2.0 connectivity, and each card uses whichever the designer feels most appropriate to the task. Most laptop computers built after 2005 are based on PCI Express and can have several Mini Card slots.

PCI Express Mini Cards are 30×50.95 mm. There is a 52 pin edge connector, consisting of two staggered rows on a 0.8 mm pitch. Each row has 8 contacts, a AGP equivalent to 4 contacts, then a further 18 contacts. A half-length card is also specified 30×26.8 mm. Cards have a thickness of 1.0 mm (excluding components).

PCI Express Mini Card edge connector provide multiple connections and buses:

• PCIe ×1
• USB 2.0
• SMBus
• Wires to diagnostics LEDs for wireless network (i.e., WiFi) status on computer's chassis
• SIM card for GSM and WCDMA applications
• Future extension for another PCIe lane
• 1.5 and 3.3 volt power

Some netbooks (notably the Asus EeePC and Dell mini9) use a variant of the PCI Express Mini Card for flash or SSD. This variant uses the reserved and several non-reserved pins to implement SATA and IDE interface passthrough, keeping only USB, ground lines and maybe the core 1x bus intact.^[6] This makes the 'miniPCIe' flash and solid state drives sold for netbooks largely incompatible with true PCI Express Mini implementations. Also the typical Asus miniPCIe SSD is 71mm long, causing the Dell 51mm model often being (incorrectly) referred to as half length.

PCI Express External Cabling (also known as External PCI Express or Cabled PCI Express) specifications were released by the PCI-SIG in February 2007.^[7][8]

Standard cables and connectors have been defined for ×1, ×4, ×8, and ×16 link widths, with a transfer rate of 250 MB/s per lane. The PCI-SIG also expects the norm will evolve to reach the 500 MB/s as found in PCI Express 2.0. The maximum cable length hasn't been determined yet.

There are several other expansion card types derived from PCIe. These include:

• Low height card
• ExpressCard: successor to the PC card form factor (with ×1 PCIe and USB 2.0; hot-pluggable)
• PCI Express ExpressModule: a hot-pluggable modular form factor defined for servers and workstations
• XMC: similar to the CMC/PMC form factor (with ×4 PCIe or Serial RapidI/O)
• AdvancedTCA: a complement to CompactPCI for larger applications; supports serial based backplane topologies
• AMC: a complement to the AdvancedTCA specification; supports processor and I/O modules on ATCA boards (×1, ×2, ×4 or ×8 PCIe).

While in development, PCIe was initially referred to as HSI (for High Speed Interconnect), and underwent a name change to 3GIO (for 3rd Generation I/O) before finally settling on its PCI-SIG name PCI Express. It was first drawn up by a technical working group named the Arapaho Work Group (AWG) which for initial drafts consisted of an Intel only team of architects. Subsequently the AWG was expanded to include industry partners.

PCIe is a technology under constant development and improvement. The current PCI Express implementation is version 2.0, with version 3.0 already proposed. The following subsections briefly describe PCIe versions 1.0 through 3.0.

In 2004, Intel introduced PCIe 1.0, with a data rate of 250 MB/s and a transfer rate of 2.5 GT/s.

PCI-SIG announced the availability of the PCI Express Base 2.0 specification on 15 January 2007.^[9] The PCIe 2.0 standard doubles the per-lane throughput from the PCIe 1.0 standard's 250 MB/s to 500 MB/s. This means a 32-lane PCI connector (x32) can support throughput up to 16 GB/s aggregate. The PCIe 2.0 standard uses a base clock speed of 5.0 GHz, while the first version operates at 2.5 GHz.

PCIe 2.0 motherboard slots are fully backward compatible with PCIe v1.x cards. PCIe 2.0 cards are also generally backward compatible with PCIe 1.x motherboards, using the available bandwidth of PCI Express 1.1. Overall, graphic cards or motherboards designed for v 2.0 will be able to work with the other being v 1.1 or v 1.0.

The PCI-SIG also said that PCIe 2.0 features improvements to the point-to-point data transfer protocol and its software architecture.^[10]

In June 2007 Intel released the specification of the Intel P35 chipset which supports only PCIe 1.1, not PCIe 2.0.^[11] Some people may be confused by the P35 block diagram which states the Intel P35 has a PCIe x16 graphics link (8 GB/s) and 6 PCIe x1 links (500 MB/s each).^[12] For simple verification one can view the P965 block diagram which shows the same number of lanes and bandwidth but was released before PCIe 2.0 was finalized.^{[original research?]} Intel's first PCIe 2.0 capable chipset was the X38 and boards began to ship from various vendors (Abit, Asus, Gigabyte) as of October 21, 2007.^[13] AMD started supporting PCIe 2.0 with its RD700 chipset series and nVidia started with the MCP72.^[14] The specification of the Intel P45 chipset includes PCIe 2.0.

PCI Express 2.1 supports a large proportion of the management, support, and troubleshooting systems planned to be fully implemented in PCI Express 3.0. But, the speed is the same as PCI Express 2.0.

In August 2007, PCI-SIG announced that PCI Express 3.0 will carry a bit rate of 8 gigatransfers per second. The spec will be backwards-compatible with existing PCIe implementations and a final spec is due in the second quarter of 2010.^[15] New features for PCIe 3.0 specification include a number of optimizations for enhanced signaling and data integrity, including transmitter and receiver equalization, PLL improvements, clock data recovery, and channel enhancements for currently supported topologies.^[16]

Following a six-month technical analysis of the feasibility of scaling the PCIe interconnect bandwidth, PCI-SIG's analysis found out that 8 gigatransfers per second can be manufactured in mainstream silicon process technology, and can be deployed with existing low-cost materials and infrastructure, while maintaining full compatibility (with negligible impact) to the PCIe protocol stack.

PCIe 2.0 delivers 5 GT/s but employed an 8b/10b encoding scheme which took 20 percent overhead on the overall raw bit rate. By removing the requirement for the 8b/10b encoding scheme, and replacing it with a 128b/130b encoding scheme with only ~1.5 percent overhead,^[17] PCIe 3.0's 8 GT/s bit rate effectively delivers double PCIe 2.0 bandwidth. According to an official press release by PCI-SIG on 8 August 2007:

"The final PCIe 3.0 specifications, including form factor specification updates, may be available by late 2009, and could be seen in products starting in 2010 and beyond."^[18]

As of January 2010^[update], the release of the final specifications has been delayed until Q2 2010.^[19] PCI-SIG expects the PCIe 3.0 specifications to undergo rigorous technical vetting and validation before being released to the industry. This process, which was followed in the development of prior generations of the PCIe Base and various form factor specifications, includes the corroboration of the final electrical parameters with data derived from test silicon and other simulations conducted by multiple members of the PCI-SIG.

PCI Express has replaced AGP as the default interface for graphics cards on new systems. With a few exceptions, all graphics cards being released as of 2009 from ATI and NVIDIA use PCI Express. NVIDIA uses the high bandwidth data transfer of PCIe for its Scalable Link Interface (SLI) technology, which allows multiple graphics cards of the same chipset and model number to be run in tandem, allowing increased performance. ATI also has developed a multi-GPU system based on PCIe called CrossFire. Eventually AMD and NVIDIA released motherboard chipsets which support up to four PCIe ×16 slots, allowing tri-GPU and quad-GPU card configurations.

Uptake for other forms of PC expansion has been much slower and conventional PCI remains dominant. PCI Express is commonly used for disk array controllers, onboard gigabit Ethernet and wi-fi but add-in cards are still generally conventional PCI, particularly at the lower end of the market. Sound cards, modems, serial port cards and other cards with low-speed interfaces are still nearly all conventional PCI. For this reason most motherboards supporting PCI Express offer conventional PCI slots as well.

ExpressCard has been introduced on several mid- to high-range laptops such as Apple's MacBook Pro line. Unlike desktops, however, laptops frequently only have one expansion slot. Replacing the PC card slot with ExpressCard slot means a loss in compatibility with PC-card devices.

The PCIe link is built around dedicated unidirectional couples of serial (1-bit), point-to-point connections known as lanes. This is in sharp contrast to the conventional PCI connection, which is a bus-based system where all the devices share the same bidirectional, 32-bit (or 64-bit), parallel bus.

PCI Express is a layered protocol, consisting of a transaction layer, a data link layer, and a physical layer. The Data Link Layer is subdivided to include a media access control (MAC) sublayer. The Physical Layer is subdivided into logical and electrical sublayers. The Physical logical-sublayer contains a physical coding sublayer (PCS). (Terms borrowed from the IEEE 802 model of networking protocol.)

The PCIe Physical Layer (PHY, PCIEPHY , PCI Express PHY, or PCIe PHY) specification is divided into two sub-layers, corresponding to electrical and logical specifications. The logical sublayer is sometimes further divided into a MAC sublayer and a PCS, although this division is not formally part of the PCIe specification. A specification published by Intel, the PHY Interface for PCI Express (PIPE),^[20] defines the MAC/PCS functional partitioning and the interface between these two sub-layers. The PIPE specification also identifies the physical media attachment (PMA) layer, which includes the serializer/deserializer (SerDes) and other analog circuitry; however, since SerDes implementations vary greatly among ASIC vendors, PIPE does not specify an interface between the PCS and PMA.

At the electrical level, each lane consists of two unidirectional LVDS or PCML pairs at 2.525 Gbit/s. Transmit and receive are separate differential pairs, for a total of 4 data wires per lane.

A connection between any two PCIe devices is known as a link, and is built up from a collection of 1 or more lanes. All devices must minimally support single-lane (x1) link. Devices may optionally support wider links composed of 2, 4, 8, 12, 16, or 32 lanes. This allows for very good compatibility in two ways:

• a PCIe card will physically fit (and work correctly) in any slot that is at least as large as it is (e.g., an ×1 sized card will work in any sized slot);
• a slot of a large physical size (e.g., ×16) can be wired electrically with fewer lanes (e.g., ×1, ×4, or ×8) as long as it provides the power and ground connections required by the larger physical slot size.

In both cases, PCIe will negotiate the highest mutually supported number of lanes.

Even though the two would be signal-compatible, it is not usually possible to place a physically larger PCIe card (e.g., a ×16 sized card) into a smaller slot —though some motherboards have open-ended PCIe slots that will allow this.

The width of a PCIe connector is 8.8 mm, while the height is 11.25 mm, and the length is variable. The fixed section of the connector is 11.65 mm in length and contains 22 pins, while the length of the other section is variable depending on the number of lanes. The thickness of the card going into the connector is 1.8 mm.^[21][22]

PCIe sends all control messages, including interrupts, over the same links used for data. The serial protocol can never be blocked, so latency is still comparable to conventional PCI, which has dedicated interrupt lines.

Data transmitted on multiple-lane links is interleaved, meaning that each successive byte is sent down successive lanes. The PCIe specification refers to this interleaving as "data striping". While requiring significant hardware complexity to synchronize (or deskew) the incoming striped data, striping can significantly reduce the latency of the n^th byte on a link. Due to padding requirements, striping may not necessarily reduce the latency of small data packets on a link.

As with all high data rate serial transmission protocols, clocking information must be embedded in the signal. At the physical level, PCI Express utilizes the very common 8b/10b encoding scheme^[24] to ensure that strings of consecutive ones or consecutive zeros are limited in length. This is necessary to prevent the receiver from losing track of where the bit edges are. In this coding scheme every 8 (uncoded) payload bits of data are replaced with 10 (encoded) bits of transmit data, causing a 20% overhead in the electrical bandwidth. To improve the available bandwidth, PCI Express version 3.0 employs 128b/130b encoding instead: similar but with much lower overhead.

Many other protocols (such as SONET) use a different form of encoding known as scrambling to embed clock information into data streams. The PCIe specification also defines a scrambling algorithm, but it is used to reduce electromagnetic interference (EMI) by preventing repeating data patterns in the transmitted data stream.

The Data Link Layer implements the sequencing of the transaction layer packets (TLPs) that are generated by the transaction layer, data protection via a 32-bit cyclic redundancy check code (CRC, known in this context as LCRC) and an acknowledgement protocol (ACK and NAK signaling). TLPs that pass an LCRC check and a sequence number check result in an acknowledgement, or ACK, while those that fail these checks result in a negative acknowledgement, or NAK. TLPs that result in a NAK, or timeouts that occur while waiting for an ACK, result in the TLPs being replayed from a special buffer in the transmit data path of the data link layer. This guarantees delivery of TLPs in spite of electrical noise, barring any malfunction of the device or transmission medium.

ACK and NAK signals are communicated via a low-level packet known as a data link layer packet (DLLP). DLLPs are also used to communicate flow control information between the transaction layers of two connected devices, as well as some power management functions.

PCI Express implements split transactions (transactions with request and response separated by time), allowing the link to carry other traffic while the target device gathers data for the response.

PCI Express utilizes credit-based flow control. In this scheme, a device advertises an initial amount of credit for each of the receive buffers in its transaction layer. The device at the opposite end of the link, when sending transactions to this device, will count the number of credits consumed by each TLP from its account. The sending device may only transmit a TLP when doing so does not result in its consumed credit count exceeding its credit limit. When the receiving device finishes processing the TLP from its buffer, it signals a return of credits to the sending device, which then increases the credit limit by the restored amount. The credit counters are modular counters, and the comparison of consumed credits to credit limit requires modular arithmetic. The advantage of this scheme (compared to other methods such as wait states or handshake-based transfer protocols) is that the latency of credit return does not affect performance, provided that the credit limit is not encountered. This assumption is generally met if each device is designed with adequate buffer sizes.

PCIe 1.x is often quoted to support a data rate of 250 MB/s in each direction, per lane. This figure is a calculation from the physical signaling rate (2.5 Gbaud) divided by the encoding overhead (10 bits per byte.) This means a sixteen lane (×16) PCIe card would then be theoretically capable of 16×250 MB/s = 4 GB/s in each direction. While this is correct in terms of data bytes, more meaningful calculations will be based on the usable data payload rate, which depends on the profile of the traffic, which is a function of the high-level (software) application and intermediate protocol levels.

Like other high data rate serial interconnect systems, PCIe has a protocol and processing overhead due to the additional transfer robustness (CRC and acknowledgements). Long continuous unidirectional transfers (such as those typical in high-performance storage controllers) can approach >95% of PCIe's raw (lane) data rate. These transfers also benefit the most from increased number of lanes (×2, ×4, etc.) But in more typical applications (such as a USB or Ethernet controller), the traffic profile is characterized as short data packets with frequent enforced acknowledgements.^[25] This type of traffic reduces the efficiency of the link, due to overhead from packet parsing and forced interrupts (either in the device's host interface or the PC's CPU.) Being a protocol for devices connected to the same printed circuit board, it does not require the same tolerance for transmission errors as a protocol for communication over longer distances, and thus, this loss of efficiency is not particular to PCIe.

Theoretically, External PCIe could give a notebook the graphic power of a desktop, by connecting a notebook with any PCIe desktop video card (enclosed in its own external housing); however, only one finalized product and two concept products exist. All three deliver the power of the video card to external displays only, and all connect to a notebook through an ExpressCard interface, which limits the bandwidth from an inserted ×16 video card (4 GB/s in each direction), to just ×1 (250 MB/s in each direction).^{[26][27][28][29][30]}

IBM/Lenovo has also included a PCI-Express slot in their Advanced Docking Station 250310U. It provides a half sized slot with an x16 length slot, but only x1 connectivity.^[31] However, docking stations with expansion slots are becoming less common as the laptops are getting more advanced video cards and either DVI-D interfaces, or DVI-D pass through for port replicators and docking stations.

Additionally, Nvidia has developed Quadro Plex external PCIe Video Cards that can be used for advanced graphic applications. These video cards require a PCI Express ×8 or ×16 slot for the interconnection cable.^[32] AMD has recently announced the ATI XGP technology based on proprietary cabling solution which is compatible with PCIe signal transmissions.^[33]

There are now card hubs in development that one can connect to a laptop through an ExpressCard slot, though they are currently rare, obscure, or unavailable on the open market. These hubs can have full-sized cards placed in them, but they can support only an external monitor and not the laptop monitor.

Several communications standards have emerged based on high bandwidth serial architectures. These include but are not limited to InfiniBand, RapidIO, StarFabric and HyperTransport.

Essentially the differences are based on the tradeoffs between flexibility and extensibility vs latency and overhead. An example of such a tradeoff is adding complex header information to a transmitted packet to allow for complex routing (PCI Express is not capable of this). This additional overhead reduces the effective bandwidth of the interface and complicates bus discovery and initialization software. Also making the system hot-pluggable requires that software track network topology changes. Examples of buses suited for this purpose are InfiniBand and StarFabric.

Another example is making the packets shorter to decrease latency (as is required if a bus is to be operated as a memory interface). Smaller packets mean that the packet headers consume a higher percentage of the packet, thus decreasing the effective bandwidth. Examples of bus protocols designed for this purpose are RapidIO and HyperTransport.

PCI Express falls somewhere in the middle, targeted by design as a system interconnect (local bus) rather than a device interconnect or routed network protocol. Additionally, its design goal of software transparency constrains the protocol and raises its latency somewhat.

• Accelerated Graphics Port (AGP)
• AMR
• Extended Industry Standard Architecture (EISA)
• Industry Standard Architecture (ISA)
• List of computer standards
• List of device bandwidths (A useful listing of device bandwidths that includes PCI Express)
• Micro Channel architecture (MCA)
• NuBus
• PCI Configuration Space
• PCI Local Bus
• PCI-X
• VESA Local Bus (VLB)
• Zorro II and Zorro III (Amiga Autoconfig Bus)

• Introduction to PCI Protocol
• PCI Express Base Specification Revision 1.0, PCI-SIG, 29 April 2002
• PCI-SIG, the industry organization that maintains and develops the various PCI standards
• Intel Developer Network for PCI Express Architecture