More Related Content
What's hot
What's hot (20)
Similar to Backplane Technology Overview for AdvancedTCA
Similar to Backplane Technology Overview for AdvancedTCA (20)
Recently uploaded
Recently uploaded (20)
Backplane Technology Overview for AdvancedTCA
- 1. Backplane Technology Overview B k l T h l O i for AdvancedTCA Hui Chen Theresienstr. 90 Supervisor: Rainer Ohlendorf 80290 Munich Germany July, 13th 2007 www.lis.ei.tum.de © Institute for Integrated Systems
- 2. Outline Introduction to AdvancedTCA Interconnect Protocols I t tP t l ① Ethernet ② InfiniBand ③ StarFabric ④ PCI Express ⑤ RapidIO Comparison of Interconnect Protocols ① Feature ② Protocol Efficiency ③ Effective Bandwidth ④ System Level Consideration Conclusion C l i Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 2 Hui Chen, July 13th 2007 Integrated Systems
- 3. What is AdvancedTCA ? • AdvancedTCA (Advanced Telecom Computing Architecture) Standard chassis/ backplane solution for high-availability communications equipment Adopted in 2003 • AdvancedTCA Chassis Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 3 Hui Chen, July 13th 2007 Integrated Systems
- 4. Base Spec of AdvancedTCA • Board dimension 1.2 inches in width 280 mm deep, ~322 mm high • Power supply & Cooling -48 V power feeds 200 Watts per board Board of AdvancedTCA • Backplane architecture Zone 1: P Z 1 Power, MManagement t Zone 2: Data Transport Interface Zone 3: User-defined I/O Backplane of AdvancedTCA Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 4 Hui Chen, July 13th 2007 Integrated Systems
- 5. Data Transport Interface • Base Interface: Dual Star Supports only Ethernet • Fabric Interface: Dual Star/ Full Mesh Supports various protocols Max. 4 channels (a) Dual Star configuration • Dual Star Redundant switch: eliminates single point of failure Effective for large redundant systems • F ll M h Full Mesh +) Higher data throughput -) Higher pins count & cost Effective for Eff ti f small systems ll t (b) Full Mesh configuration Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 5 Hui Chen, July 13th 2007 Integrated Systems
- 6. Outline Introduction to AdvancedTCA Interconnect Protocols ① Ethernet ② InfiniBand ③ StarFabric ④ PCI Express ⑤ RapidIO Comparison of Interconnect Protocols ① Feature ② Protocol Efficiency ③ Effective Bandwidth ④ System Level Consideration Conclusion Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 6 Hui Chen, July 13th 2007 Integrated Systems
- 7. Protocol ① ― Ethernet • Application: Initial in Local Area Network Successful in backplane but defective in congestion management • Congestion management PAUSE function: all data flows are paused when only one flow is congested → congestion spreading, when complex data p g p g, p paths • Topology P2P connections: Ethernet Switch ↔ Endpoints Ethernet Topology • Frame Header: large Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 7 Hui Chen, July 13th 2007 Integrated Systems
- 8. Protocol ② ― InfiniBand ⑴ • Application Processor Node Initial in System Area Network now also i b k l l in backplane, chassis-to-chassis h i t h i Supercomputer, Linux cluster • Topology InfiniBand Switches + Channel Adapters HCA: Host Channel Adapter in processor node TCA: Target Channel Adapter in I/O node I/O Node HCA talks TCAs to enable remote InfiniBand Topology storage and connectivity • Packet format Larger payload size than Ethernet Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 8 Hui Chen, July 13th 2007 Integrated Systems
- 9. Protocol ② ― InfiniBand ⑵ • Three link widths x1 Links x1 x4 x12 x4 Data Rate (Gb/s) 2 8 24 x12 • Supports Quality of Service (QoS) QoS: a method to prioritize network traffic Ensures most important data gets thru the network as quickly as possible Defines traffic to many streams, streams to differentiated classes; reserves bandwidth to streams and classes Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 9 Hui Chen, July 13th 2007 Integrated Systems
- 10. Protocol ② ― InfiniBand ⑶ • Supports Quality of Service Thru 16 Virtual Lanes (VLs) Multiplex independent data streams → the same physical link Represent a set of send & receive p buffers in a port Each port: one management VL (VL15) & up to 15 data VLs Priorities VL15 (highest) VL14 … VL0 Virtual Lanes Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 10 Hui Chen, July 13th 2007 Integrated Systems
- 11. Protocol ③ ― StarFabric ⑴ • Application Backplane, chassis-to-chassis Embedded distributed processing (e.g. multiprocessor systems) • Topology StarFabric Switch: six-port Connects other switches/ bridges Cascadable StarFabric Bridge StarFabric Topology Translates other protocols (e.g. PCI) into StarFabric traffic St F b i t ffi • Not scalable Operates only at 2 Gb/s O t l t Gb/ Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 11 Hui Chen, July 13th 2007 Integrated Systems
- 12. Protocol ③ ― StarFabric ⑵ • Packet format • Supports Quality of Service 7 classes of traffic asynchronous traffic (data) multicast … isochronous traffic (voice & video) • Backwards compatible to PCI Converting a serial fabric to parallel PCI Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 12 Hui Chen, July 13th 2007 Integrated Systems
- 13. Protocol ④ ― PCI Express ⑴ • Successor to PCI Backplane, chip-to-chip interconnect PC (e.g. Graphics cards) • Topology Switches Multiple Endpoints: I/O devices Host Bridge To create an I/O hierarchy PCI Express Topology • Six link widths: Links x1 x2 x4 x8 x16 x32 Data Rate (Gb/s) 2 4 8 16 32 64 Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 13 Hui Chen, July 13th 2007 Integrated Systems
- 14. Protocol ④ ― PCI Express ⑵ • Packet format • Supports Quality of Service Thru Thr 8 Traffic Classes (TCs) & 8 Virt al Channels (VCs) Virtual Each port : up to 8 VCs. VC0 (lowest priority), VC7 (highest) TCs: differentiated services Each E h packet a TC t determine which VC b ff t d k t TC: to d t i hi h buffer to drop packet i t k t into TC/VC mapping: ☺ TC[0:6]→VC0 TC7→VC[0:1] An Example of TC/VC mapping Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 14 Hui Chen, July 13th 2007 Integrated Systems
- 15. Protocol ⑤ ― RapidIO ⑴ • Application Backplane, chip-to-chip interconnect Embedded systems (e.g. DSPs) Networking (e.g. Carrier Cards) •T Topology l A Switch, multiple Endpoints Large systems: one / more host processors For system exploration, initialization exploration Monitors system-level activity RapidIO Topology • Two link widths (x1 x4) (x1, Links x1 x4 Data Rate (Gb/s) 1 2 2.5 4 8 10 Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 15 Hui Chen, July 13th 2007 Integrated Systems
- 16. Protocol ⑤ ― RapidIO ⑵ • Packet format Header: smaller than Ethernet: 1-byte source address ~ 28 Endpoints → sufficient for backplane applications Payload size: relatively small • Supports Quality of Service Six logical flows: as prioritized classes Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 16 Hui Chen, July 13th 2007 Integrated Systems
- 17. Outline Introduction to AdvancedTCA Interconnect Protocols ① Ethernet ② InfiniBand ③ StarFabric ④ PCI Express ⑤ RapidIO Comparison of Interconnect Protocols ① Feature ② Protocol Efficiency ③ Effective Bandwidth ④ System Level Consideration Conclusion Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 17 Hui Chen, July 13th 2007 Integrated Systems
- 18. Comparison – Feature Feature GigE InfiniBand StarFabric PCI Expr Expr. RapidIO Transport PCB, Copper & PCB, Copper & PCB & PCB, Copper & Media Fiber Fiber Optical PCB only CAT5 cable Scalable Link None Yes None Yes Yes Widths Encapsulated, Yes, Yes, Yes, PCI Migration None requires SW q transparent p transparent p requires SW Max Signal 1 Gbps 2 Gbps 2 Gbps 2 Gbps 2.5 Gbps Channel Rate y Max Payload 1500 B tes Bytes 4096 B t Bytes 128 B t Bytes 4096 B t Bytes 256 B tes Bytes Size Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 18 Hui Chen, July 13th 2007 Integrated Systems
- 19. Comparison – Protocol Efficiency • Protocol Efficiency = Payload / Packet size • Payload < ~500B : RapidIO, highest efficiency smallest overhead Payload > ~500B : PCI Express, highest Express Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 19 Hui Chen, July 13th 2007 Integrated Systems
- 20. Comparison – Effective Bandwidth • Effective Bandwidth = Protocol efficiency × Bandwidth • InfiniBand x4; PCIe x4; RapidIO 4x 2Gbps • O Overall: Gi E l ll GigE, lowest t Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 20 Hui Chen, July 13th 2007 Integrated Systems
- 21. Comparison – System Level • Quality of Service (QoS) GigE Gi E InfiniBand I fi iB d StarFabric St F b i PCI E Expr. RapidIO R idIO Prioritized Traffic － 16 7 8 6 Classes •P f Performance GigE InfiniBand StarFabric PCI Expr. RapidIO Max Bandwidth for M B d idth f 1 Gbps 8 Gbps 2 Gbps 8 Gbps 10 Gbps AdvancedTCA, x4 Latency High Medium Medium Low Low (~100 us) (1.29-2.6 us) (< 3 us) (0.3-0.7 us) (< 100 ns) Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 21 Hui Chen, July 13th 2007 Integrated Systems
- 22. Outline Introduction to AdvancedTCA Interconnect Protocols ① Ethernet ② InfiniBand ③ StarFabric ④ PCI Express ⑤ RapidIO Comparison of Interconnect Protocols ① Feature ② Protocol Efficiency ③ Effective Bandwidth ④ System Level Consideration Conclusion Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 22 Hui Chen, July 13th 2007 Integrated Systems
- 23. Conclusion • Low bandwidth backplane applications Ethernet +) well understood, low risk -) high overhead, hi h l ) hi h h d high latency -) not scalable • High bandwidth (tradeoffs of flexibility vs. latency, overhead) g ( y y, ) InfiniBand and StarFabric Complex header: +) for complex routing -) reduces effective bandwidth RapidIO Smaller packets: +) to lower latency -) overhead: a higher percentage of the packet -) reduces effective b d idth ) d ff ti bandwidth PCI Express Middle: +) design goal: software transparency -) constrains the protocol raises latency protocol, Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 23 Hui Chen, July 13th 2007 Integrated Systems
- 24. Thank you for your attention ! Any questions ? Special thanks to: Rainer Ohlendorf Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 24 Hui Chen, July 13th 2007 Integrated Systems