Arnold Nipper - Building resilient and scalable Terabit Exchangepoints
•
0 likes•18 views
This document discusses the design and implementation of a new scalable infrastructure for DE-CIX internet exchange points. A star topology was chosen using DWDM technology for interconnects and optical switching for resilience. The infrastructure was built using core switches in existing points of presence connected with diverse dark fiber routes. Testing showed the infrastructure could failover traffic during outages. The design allows for scaling edge and core switches to increase bandwidth capacity up to over 40 terabits per second to support future growth.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Arnold Nipper - Building resilient and scalable Terabit Exchangepoints
Similar to Arnold Nipper - Building resilient and scalable Terabit Exchangepoints (20)
Recently uploaded
Recently uploaded (20)
Arnold Nipper - Building resilient and scalable Terabit Exchangepoints
- 1. PLNOG Mazurkas Conference Center, Warsaw, 2009-01-15 Building resilient and scalable Terabit Exchangepoints
- 2. Agenda • Motivation • Design of infrastructure • Implementation of infrastructure • Scaling the infrastructure • Summary
- 4. Topology until mid 2008 • DE-CIX1 and DE-CIX2 on the same campus, DE-CIX3 and DE-CIX4 in separate colocations • combined edge and distribution in resilient star • grown over time since 1995 from single switch, dual edge, STP resilience to fibre protection based resilience
- 5. • mix of edge and distribution • hard to plan for growth in backbone bandwith and ports for customers • outage of distribution switch also affects customers
- 6. Agenda • Motivation • Design of infrastructure • Implementation of infrastructure • Scaling the infrastructure • Summary
- 7. Design of new infrastructure • which topology? – star – ring • which technology for interconnects? – simple dark fibre – DWDM • which technology for resilience? – STP – MRP (Metro Ring Protocol) – Layer 0
- 8. Design of new infrastructure • star topology • DWDM technology for interconnects • Layer 0 technology for resilience • other considerations – scalable, simple and robust – cost-efficient – easy migration
- 9. Building blocks • core switches (star) • DWDM de/muxes (interconnect) • optical switches (resilience)
- 10. cores • where to locate – new POP? – existing POP? – which POP? • Take existing POP's • Hardware is Force10 networks E1200
- 11. DWDM / dark fibre • passive • up to 16 channels • diverse routes • one pair per edge • Cube Optics as a system partner
- 12. darkfiber routes
- 13. Resilience • using simple fibre switches • power is only needed to switch fibres • primary and backup path going via diverse fibres • master/slave • Lynx networks Lightleader • all optical switches synchronously switch over to standby core
- 15. Normal operation mode
- 16. Something went wrong
- 18. Problem fixed Normal operation mode
- 19. Agenda • Motivation • Design of infrastructure • Implementation of infrastructure • Scaling the infrastructure • Summary
- 20. Implementation • Order lines, DWDM transceivers (long leadtime), DWDM chassis, fibre switches and core switches • Install and test everything • connect core to existing infrastructure
- 21. Migration • replace existing connections with DWDM/LL connections • use old backup ports to interconnect new core • during off peak hour replace existing interconnects (etherchannels) with new interconnect to core • test failover scenario
- 22. Testing failover • simulating fibre cut by re/plugging a cable • doing a contolled failover • while testing failover backup connection of one edge did not come up • optical budget was not enough • removing attenuator solved the problem • failover will be done on a regular basis
- 23. Agenda • Motivation • Design of infrastructure • Implementation of infrastructure • Scaling the infrastructure • Summary
- 24. Edge • simply add another switch • more efficient to use bigger edges as well file:///C:/Dokumente und Einstellungen/nipper/Eigene Dateien/Visios/decix-scaling-edge.png #switches local traffic 1 100% 2 50% 3 33% 4 25% 5 20% 6 17% 7 14% 8 13% 9 11% 10 10%
- 25. Core • use bigger switches • will upgrade core to Foundry MLX32 soon • „stack“ core switches when you can't get bigger ones • scales up to 32 core members (~41TBps) file:///C:/Dokumente und Einstellungen/nipper/Eigene Dateien/Visios/decix-scaling-core.png
- 26. Summary • Building resilient, scalable highbandwidth IXP is possible • Scaling is possible up to >40 Tbps which will safely take us to when 100Gbps ports are widely available
- 27. Thanks! DE-CIX Management GmbH Lindleystr. 12 60314 Frankfurt a.M. info@de-cix.net +49 69 1730 902 0 DE-CIX office @ Frankfurt docklands