Downloaded 39 times
![Protocol ④ ― PCI Express ⑵
• Packet format
• Supports Quality of Service
Thru
Thr 8 Traffic Classes (TCs) & 8 Virt al Channels (VCs)
Virtual
Each port : up to 8 VCs. VC0 (lowest priority), VC7 (highest)
TCs: differentiated services
Each
E h packet a TC t determine which VC b ff t d
k t TC: to d t i hi h buffer to drop packet i t
k t into
TC/VC mapping: ☺ TC[0:6]→VC0 TC7→VC[0:1]
An Example of TC/VC mapping
Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for
14
Hui Chen, July 13th 2007 Integrated Systems](https://image.slidesharecdn.com/backplanetechadvancedtca-110129075934-phpapp01/85/Backplane-Technology-Overview-for-AdvancedTCA-14-320.jpg)
Backplane Technology Overview for AdvancedTCA
- 1. Backplane Technology Overview Bk l T h l O i for AdvancedTCA Hui Chen Theresienstr. 90 Supervisor: Rainer Ohlendorf 80290 Munich Germany July, 13th 2007 www.lis.ei.tum.de © Institute for Integrated Systems
- 2. Outline Introduction to AdvancedTCA InterconnectProtocols I t tP t l ① Ethernet ② InfiniBand ③ StarFabric ④ PCI Express ⑤ RapidIO Comparison of Interconnect Protocols ① Feature ② Protocol Efficiency ③ Effective Bandwidth ④ System Level Consideration Conclusion C l i Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 2 Hui Chen, July 13th 2007 Integrated Systems
- 3. What is AdvancedTCA ? • AdvancedTCA (AdvancedTelecom Computing Architecture) Standard chassis/ backplane solution for high-availability communications equipment Adopted in 2003 • AdvancedTCA Chassis Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 3 Hui Chen, July 13th 2007 Integrated Systems
- 4. Base Spec of AdvancedTCA • Board dimension 1.2 inches in width 280 mm deep, ~322 mm high • Power supply & Cooling -48 V power feeds 200 Watts per board Board of AdvancedTCA • Backplane architecture Zone 1: P Z 1 Power, MManagement t Zone 2: Data Transport Interface Zone 3: User-defined I/O Backplane of AdvancedTCA Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 4 Hui Chen, July 13th 2007 Integrated Systems
- 5. Data Transport Interface • Base Interface:Dual Star Supports only Ethernet • Fabric Interface: Dual Star/ Full Mesh Supports various protocols Max. 4 channels (a) Dual Star configuration • Dual Star Redundant switch: eliminates single point of failure Effective for large redundant systems • F ll M h Full Mesh +) Higher data throughput -) Higher pins count & cost Effective for Eff ti f small systems ll t (b) Full Mesh configuration Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 5 Hui Chen, July 13th 2007 Integrated Systems
- 6. Outline Introduction to AdvancedTCA InterconnectProtocols ① Ethernet ② InfiniBand ③ StarFabric ④ PCI Express ⑤ RapidIO Comparison of Interconnect Protocols ① Feature ② Protocol Efficiency ③ Effective Bandwidth ④ System Level Consideration Conclusion Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 6 Hui Chen, July 13th 2007 Integrated Systems
- 7. Protocol ① ― Ethernet •Application: Initial in Local Area Network Successful in backplane but defective in congestion management • Congestion management PAUSE function: all data flows are paused when only one flow is congested → congestion spreading, when complex data p g p g, p paths • Topology P2P connections: Ethernet Switch ↔ Endpoints Ethernet Topology • Frame Header: large Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 7 Hui Chen, July 13th 2007 Integrated Systems
- 8. Protocol ② ― InfiniBand ⑴ •Application Processor Node Initial in System Area Network now also i b k l l in backplane, chassis-to-chassis h i t h i Supercomputer, Linux cluster • Topology InfiniBand Switches + Channel Adapters HCA: Host Channel Adapter in processor node TCA: Target Channel Adapter in I/O node I/O Node HCA talks TCAs to enable remote InfiniBand Topology storage and connectivity • Packet format Larger payload size than Ethernet Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 8 Hui Chen, July 13th 2007 Integrated Systems
- 9. Protocol ② ― InfiniBand ⑵ •Three link widths x1 Links x1 x4 x12 x4 Data Rate (Gb/s) 2 8 24 x12 • Supports Quality of Service (QoS) QoS: a method to prioritize network traffic Ensures most important data gets thru the network as quickly as possible Defines traffic to many streams, streams to differentiated classes; reserves bandwidth to streams and classes Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 9 Hui Chen, July 13th 2007 Integrated Systems
- 10. Protocol ② ― InfiniBand ⑶ •Supports Quality of Service Thru 16 Virtual Lanes (VLs) Multiplex independent data streams → the same physical link Represent a set of send & receive p buffers in a port Each port: one management VL (VL15) & up to 15 data VLs Priorities VL15 (highest) VL14 … VL0 Virtual Lanes Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 10 Hui Chen, July 13th 2007 Integrated Systems
- 11. Protocol ③ ― StarFabric ⑴ •Application Backplane, chassis-to-chassis Embedded distributed processing (e.g. multiprocessor systems) • Topology StarFabric Switch: six-port Connects other switches/ bridges Cascadable StarFabric Bridge StarFabric Topology Translates other protocols (e.g. PCI) into StarFabric traffic St F b i t ffi • Not scalable Operates only at 2 Gb/s O t l t Gb/ Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 11 Hui Chen, July 13th 2007 Integrated Systems
- 12. Protocol ③ ― StarFabric ⑵ •Packet format • Supports Quality of Service 7 classes of traffic asynchronous traffic (data) multicast … isochronous traffic (voice & video) • Backwards compatible to PCI Converting a serial fabric to parallel PCI Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 12 Hui Chen, July 13th 2007 Integrated Systems
- 13. Protocol ④ ― PCI Express ⑴ •Successor to PCI Backplane, chip-to-chip interconnect PC (e.g. Graphics cards) • Topology Switches Multiple Endpoints: I/O devices Host Bridge To create an I/O hierarchy PCI Express Topology • Six link widths: Links x1 x2 x4 x8 x16 x32 Data Rate (Gb/s) 2 4 8 16 32 64 Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 13 Hui Chen, July 13th 2007 Integrated Systems
- 14. Protocol ④ ― PCI Express ⑵ •Packet format • Supports Quality of Service Thru Thr 8 Traffic Classes (TCs) & 8 Virt al Channels (VCs) Virtual Each port : up to 8 VCs. VC0 (lowest priority), VC7 (highest) TCs: differentiated services Each E h packet a TC t determine which VC b ff t d k t TC: to d t i hi h buffer to drop packet i t k t into TC/VC mapping: ☺ TC[0:6]→VC0 TC7→VC[0:1] An Example of TC/VC mapping Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 14 Hui Chen, July 13th 2007 Integrated Systems
- 15. Protocol ⑤ ― RapidIO ⑴ •Application Backplane, chip-to-chip interconnect Embedded systems (e.g. DSPs) Networking (e.g. Carrier Cards) •T Topology l A Switch, multiple Endpoints Large systems: one / more host processors For system exploration, initialization exploration Monitors system-level activity RapidIO Topology • Two link widths (x1 x4) (x1, Links x1 x4 Data Rate (Gb/s) 1 2 2.5 4 8 10 Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 15 Hui Chen, July 13th 2007 Integrated Systems
- 16. Protocol ⑤ ― RapidIO ⑵ •Packet format Header: smaller than Ethernet: 1-byte source address ~ 28 Endpoints → sufficient for backplane applications Payload size: relatively small • Supports Quality of Service Six logical flows: as prioritized classes Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 16 Hui Chen, July 13th 2007 Integrated Systems
- 17. Outline Introduction to AdvancedTCA InterconnectProtocols ① Ethernet ② InfiniBand ③ StarFabric ④ PCI Express ⑤ RapidIO Comparison of Interconnect Protocols ① Feature ② Protocol Efficiency ③ Effective Bandwidth ④ System Level Consideration Conclusion Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 17 Hui Chen, July 13th 2007 Integrated Systems
- 18. Comparison – Feature Feature GigE InfiniBand StarFabric PCI Expr Expr. RapidIO Transport PCB, Copper & PCB, Copper & PCB & PCB, Copper & Media Fiber Fiber Optical PCB only CAT5 cable Scalable Link None Yes None Yes Yes Widths Encapsulated, Yes, Yes, Yes, PCI Migration None requires SW q transparent p transparent p requires SW Max Signal 1 Gbps 2 Gbps 2 Gbps 2 Gbps 2.5 Gbps Channel Rate y Max Payload 1500 B tes Bytes 4096 B t Bytes 128 B t Bytes 4096 B t Bytes 256 B tes Bytes Size Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 18 Hui Chen, July 13th 2007 Integrated Systems
- 19. Comparison – Protocol Efficiency • ProtocolEfficiency = Payload / Packet size • Payload < ~500B : RapidIO, highest efficiency smallest overhead Payload > ~500B : PCI Express, highest Express Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 19 Hui Chen, July 13th 2007 Integrated Systems
- 20. Comparison – Effective Bandwidth • EffectiveBandwidth = Protocol efficiency × Bandwidth • InfiniBand x4; PCIe x4; RapidIO 4x 2Gbps • O Overall: Gi E l ll GigE, lowest t Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 20 Hui Chen, July 13th 2007 Integrated Systems
- 21. Comparison – System Level • Qualityof Service (QoS) GigE Gi E InfiniBand I fi iB d StarFabric St F b i PCI E Expr. RapidIO R idIO Prioritized Traffic － 16 7 8 6 Classes •P f Performance GigE InfiniBand StarFabric PCI Expr. RapidIO Max Bandwidth for M B d idth f 1 Gbps 8 Gbps 2 Gbps 8 Gbps 10 Gbps AdvancedTCA, x4 Latency High Medium Medium Low Low (~100 us) (1.29-2.6 us) (< 3 us) (0.3-0.7 us) (< 100 ns) Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 21 Hui Chen, July 13th 2007 Integrated Systems
- 22. Outline Introduction to AdvancedTCA InterconnectProtocols ① Ethernet ② InfiniBand ③ StarFabric ④ PCI Express ⑤ RapidIO Comparison of Interconnect Protocols ① Feature ② Protocol Efficiency ③ Effective Bandwidth ④ System Level Consideration Conclusion Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 22 Hui Chen, July 13th 2007 Integrated Systems
- 23. Conclusion • Low bandwidthbackplane applications Ethernet +) well understood, low risk -) high overhead, hi h l ) hi h h d high latency -) not scalable • High bandwidth (tradeoffs of flexibility vs. latency, overhead) g ( y y, ) InfiniBand and StarFabric Complex header: +) for complex routing -) reduces effective bandwidth RapidIO Smaller packets: +) to lower latency -) overhead: a higher percentage of the packet -) reduces effective b d idth ) d ff ti bandwidth PCI Express Middle: +) design goal: software transparency -) constrains the protocol raises latency protocol, Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 23 Hui Chen, July 13th 2007 Integrated Systems
- 24. Thank you foryour attention ! Any questions ? Special thanks to: Rainer Ohlendorf Hauptseminar am Lehrstuhl für Integrierte Systeme © Institute for 24 Hui Chen, July 13th 2007 Integrated Systems