Embedded system engineering magazine 2005.10

MULTI-HOST SYSTEMS are an effective solution enabling designers to meet targets for system processing in applications such as high-end graph-

ics, large-scale video systems and multimedia processing. Additionally to increases in processing requirements, telecom, networking and storage systems need to provide cost-effective redundancy to deliver five-9s availability. Since the PCI standards now dominate low-cost, highperformance interconnect including chip-to-chip and backplane switching fabrics, there is a clear requirement for an effective solution to successful multi-host design in the PCI environment.

Multiple processors running generic operating systems (OS) working on the same PCI fabric will each attempt to enumerate the entire system. This results in memory mapping conflicts, multiple attempts to service system requests, and system failure. Designers therefore need PCI and OS compatible solutions for isolating processor domains and preventing contentions whilst making provision for inter-domain transactions to sustain normal operation in addition to fail over support.

Non-transparent (NT) bridging is an industrystandard technique enabling separation of individual processor domains. However the functions of a non-transparent bridge encompass action as an endpoint as well as a switch/bridge so achieving a PCI-compatible NT system is a tough call. This has been achieved for PCI and PCI-X architectures, and suitable solutions for PCI-Express (PCIe) are now emerging.

Separating processor domains

Upon power-up of a PCI system (regardless of type) the system host OS attempts to discover all devices present, determine their memory and I/O space requirements and map these into memory. This enumeration process relies on a series of control and status registers (CSRs) to achieve this. The CSR of a conventional transparent bridge presents a “Type 1” header, which informs the processor to keep enumerating beyond this bridge, as additional devices lie downstream.

In contrast CSRs of endpoint devices, have “Type 0” headers to inform the OS that no additional devices lie downstream and so terminate

ID translation. It is this process that allows the NT port to meet the requirements PCI(e) places upon each device. The NT switch simply translates each request in order to route the result back to the requesting device.

Inter-domain transactions

In Transparent systems memory spaces are specified by the base and limit settings within the base address registers (BARs).

BARs in a non-transparent switch can perform direct address translation and look-up table-based translation. In direct address translation the addresses of all transactions are translated by an offset to the BAR in which the transaction landed. In look-up table-based address translation, the BAR uses a special look-up table for address translation of transactions that fall within its window. This approach provides more flexibility in mapping addresses from bus to bus.

For ID routed packets the IDs are based on bus, function and device numbers so in multi host systems there could be conflicts. The switches from PLX will translate these ID’s from one both sides of the NT port to avoid this conflict.

PCI Express went public in the summer of 2002 when the Arapahoe Work Group, (Compaq, Dell, Hewlettackard, IBM, Intel and Microsoft) passed

the specification to the PCI SIG. the first commercial products began to appear in 2003/4 and the standard is beginning to gather momentum now. As an indicator, the Intel developers forum in August saw Altera and Xilinx making PCIe announcements and Synopsis has recently added PCIe designs to its IP library.

Simply, PCI Express is an open, high-perform- ance serial I/O specification. The name “reflects the high-speed, point-to-point qualities of the new local bus architecture, and emphasizes that software compatibility is maintained between PCI and PCI Express.” In theory software written for PCI should run on PCI Express, making it an evolutionary step.

A fast serial architecture was chosen to overcome the limits inherent in parallel bus implementations. The specification was for the bandwidth and scalability to accommodate CPU speeds in excess of 10 GHz, faster memory speeds, higher-speed graphics, and the higher speeds of other interconnects.

It is designed for chip-to-chip and add-in card applications, to provide connectivity for adapter cards, as a graphics I/O attach point for increased graphics bandwidth, and as an attach point to other interconnects like 1394b, USB 2.0, InfiniBand Architecture and Ethernet.

Contents

Forthcoming guides:

November/December: IP for FPGAs

Dig deeper into PCIe at

www.esemagazine.co.uk

A PCI Express card can have one or more serial connections, called lanes. Each lane is a wire pair capable of a maximum transfer rate of 2.5 Gb/s in each direction, giving a 200 MB/ s/channel, already twice that of standard PCI. Even greater performance comes from combining several lanes. The specification is for 1, 2, 4, 8, 16 and 32 lanes, normally referred to as x1, x2, x4 etc. So far the majority of cards are at the lower end of the of the range.

Implementation

At the heart of a PCI Express implementation is a host bridge, connecting to the CPU and normally containing memory and IO bridges. The IO bridge will connect to a PCI Express switch, which in turn fans out to other PCI Express enabled devices. These may be, for example, bridges to other interconnects, such as gigabit Ethernet, as well as interfaces to classic IO tools.

Architecture

The architecture is built from five layers, physical, data link, transaction, software and configuration/OS. The physical layer manages the communication between two devices, disassembling and reassembling data to take advantage of the number of lanes available. The data link layer takes packets from the transaction layer, adds a sequence and CRC before passing to the physical layer. The transaction layer formats packets from the software layer, adding header data and assigning priority etc, before passing them to the transaction layer. The software layer maintains compatibility with PCI and communicates to operating system.

Testing

PCI-SIG has developed a range of compliance tests, covering Electrical testing, Configuration testing, Link protocol testing, Transaction protocol testing, and Platform BIOS testing. New devices are tested at a compliance workshop and successful devices added to the integrators list (www.pcisig.com/developers/compliance_program/integrators_list/pcie/pcie)

Additional specifications have been generated for, amongst others, PCI Express Card Electromechanical specification and PCI Express Mini Card Electromechanical specification, the PCI Express High-End Graphics specification, the PCI Express Server I/O Module, the PCI Express Wireless Form Factor specification and the PCI Express Cable specification.

Extension

Although the standard is only just beginning to become widely adopted, the PCI Express 2.0 Base Specification is currently out for review, with the main aim of more than doubling the throughput.

Although PCI Express began as a PC interconnect, its ability to bridge to other interconnect fabrics has seen it being used in servers and other specialist networking devices, and the test and measurement industry has created the LXI consortium (ESE September 2005, page 40).

This is only a brief summary. More information is available from PCI SIG and from those manufacturers listed in the buyers guide. <Ends>

www.pcisig.org