Modern Carrier Strategies for Traffic Engineering

1,005 views

Published on

This classic talk from 2002-03, captures some of the key traffic engineering and core network design strategies deployed by carriers from the early 1990's to early 2000's, and (now, in 2011!) provides a great historical perspective on how network cores have evolved. It will prove valuable for those looking to understand network evolution, and the operational principles and considerations behind it...

Published in: Technology, Business
  • Be the first to comment

Modern Carrier Strategies for Traffic Engineering

  1. 1. Metanoia, Inc. Critical Systems Thinking™Modern Carrier Strategies for Traffic Engineering Dr. Vishal Sharma Principal Consultant Metanoia, Inc. Voice: +1 408 394 6321 Email: v.sharma@ieee.org Web: http://www.metanoia- © Copyright 2002All Rights Reserved inc.com
  2. 2. Metanoia, Inc. Critical Systems Thinking™Basic Service Provider GoalsThe two fundamental tasks before any service provider: Deploy a physical topology that meets customers’ needs Map customer traffic flows on to the physical topology Earlier (1990s) the mapping task was uncontrolled!  By-product of shortest-path IGP routing  Often handled by over-provisioning©Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE 2
  3. 3. Metanoia, Inc.The Early Years (< 1994-95): Routed Critical Systems Thinking™Network Topology IP Router IP Router IP Router Network Cloud Router Interconnections IP Router©Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE 3
  4. 4. Metanoia, Inc.The Early Years (< 1994-95): A Critical Systems Thinking™Stacked View IP Routers Router -- Service creation -- Pkt. switching -- Stat muxing FDDI rings -- Connectivity (100 Mb/s) TDM over copper (T1/T3) MUX -- Speed match I/Fs DCS/DXC SDH/SONET -- TDM transport -- Fault isolation -- Restoration©Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE 4
  5. 5. Metanoia, Inc.TE in Carrier Networks: Traditional Critical Systems Thinking™Routed Core (pre 1994-95) Prior to advent of ATM ... ... IP metrics were the only means available to control traffic distribution through IP networks Approach was ad-hoc  Observe traffic flow through network  Adjust weight of links with load lower/higher than desired  Overprovision network as, needed©Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE 5
  6. 6. Metanoia, Inc. Critical Systems Thinking™Two Paths to TE in IP Networks With increase in traffic, emergence of ATM, and higher-speed SONET, two approaches emerged Use a Layer 2 (ATM) network Use only Layer 3 (IP) network  Build ATM backbone  Build SONET infrastructure  Deploy complete PVC mesh,  Rely on SONET for resilience bypass use of IP metrics  Run IP directly on SONET (POS)  TE at ATM layer  Use metrics (systematically) to  With time, evolve ATM to MPLS- control flow of traffic (more on based backbone this later)©Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE 6
  7. 7. Metanoia, Inc. Critical Systems Thinking™Genesis of the ATM-Core Growth in traffic ⇒ needed faster backbones (> T3/45 Mb/s) Denser backbones ⇒ metric manipulation impractical IP routers lagged: offered only DS3 I/Fs & s/w forwarding ATM emerged, was designed for WAN from start ⇒ In 1994-95 had OC-3, and later OC-12 I/Fs available ⇒ Allowed carriers to redesign their networks for high-speeds⇒ As an evolutionary step, SPs moved to a switched ATM core©Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE 7
  8. 8. Metanoia, Inc. Critical Systems Thinking™ATM-based Cores (mid to late 1990s) Router -- Service creation -- Pkt. switching -- Stat muxing -- Connectivity ATM -- Traffic engg. -- Hardware fwding MUX -- Speed match I/Fs SDH/SONET -- TDM transport -- Fault isolation -- Restoration©Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE 8
  9. 9. Metanoia, Inc. Critical Systems Thinking™Physical Topology with an ATM Core OC-3 Router POP 1 OC-3 POP i+1 POP j POP 2 OC-12 OC-3 ATM SAR I/F POP i ATM Switch POP N ATM Core©Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE 9
  10. 10. Metanoia, Inc. Critical Systems Thinking™Logical Topology with an ATM Core PVC 1 Primary PVC MeshA PVC 1 C A C PVC 4 PVC 4 PVC 2 PVC 3 B PVC 5 D PVC 3 B D PVC 5 Secondary PVC Mesh PVC 2 ATM PVC Layout L3 Logical Topology ATM-core (usually) fully owned by SP Dedicated (usually) to supporting IP backbone Utilized ATM UBR or ATM VBR-rt/CBR, depending on classes of traffic©Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE 10
  11. 11. Metanoia, Inc.Genesis of the IP-over-SONET/SDH Critical Systems Thinking™Approach Desire to minimize # of network layers  Easier management  Simpler operation  Potentially scalable Belief that high-speed SONET/SDH I/Fs would become available with advances in components (vindicated with time) Dictated partly by how (in time) a carrier’s network evolved©Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE 11
  12. 12. Metanoia, Inc.SONET/SDH-based Cores (mid-to- Critical Systems Thinking™late 1990s and beyond) Router -- Service creation -- Pkt. switching -- Stat muxing -- TE w/ metrics or MPLS MUX -- Speed match I/Fs SDH/SONET -- Framing -- Fault isolation -- Restoration (moving away) DWDM -- B/w on existing plant©Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE 12
  13. 13. Metanoia, Inc. Critical Systems Thinking™Physical Topology with SONET/SDH Core OC-3/12 or STM-1/4 Router POP 1 POP i+1 OC-48 BLSR/ POP j MS-SPRing POP 2 Point-to-point SONET/ SDH framed link OC-3 SONET/ STM-1 SDH I/F POP i OC-3/13 UPSR/ POP N DXC SNCP Ring SONET/SDH Core©Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE 13
  14. 14. Metanoia, Inc. Critical Systems Thinking™ Logical Topology with SONET/SDH Core Parallel Logial Links for Load Sharing Ckt. 1 A C A C Ckt. 5Ckt. 4 Ckt. 3 B D B D Ckt. 2 Each logical link provisioned for 2X the bandwidth SONET/SDH Circuit Layout L3 Logical Topology  SONET/SDH infrastructure (usually) owned by SP  Logical links between POP routers realized over a physical SONET/SDH circuit going over a fiber path  Parallel logical links (physically disjoint) provisioned b/w each router pair ©Copyright 2002, All Rights Reserved Modern Carrier Strategies for TE 14
  15. 15. Metanoia, Inc. Critical Systems Thinking™ Global Crossing IP Backbone Network Courtesy: Thomas Telkamp, GBLX100,000 route miles 200+ POPs 5 continents 27 countries 250 major cities ©Copyright 2002, All Rights Reserved Modern Carrier Strategies for TE 15
  16. 16. Metanoia, Inc. Critical Systems Thinking™Global Crossing (GBLX): A Bit of History First independent global fiber network  Launched operations -- March 1997  First segment turned on -- May 1998 Expanded network & svcs. by acquisitions & JVs  Frontier Telecommunications, Sept 1999  Racal Telecom, Nov 1999  Hutchison Global Crossing, Jan 2000  IXNET/IPC, June 2000 International network, worldwide reach  100,000 route miles, 27 countries, 250 major cities  195 POPs (mid 2001)©Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE 16
  17. 17. Metanoia, Inc. Critical Systems Thinking™Global Crossing IP Network OC-48c/STM-16c (2.5Gbps) IP backbone  Selected 10Gbps links operational (e.g. Atlantic) Edge Equipment Cisco 7500/7200 ESR, OSR Juniper M10/20/40 Core Equipment Cisco GSR 12000/12400 [12.0(17) SI] Services offered  Internet access & Transit services  IP VPNs -- Layer 3 and Layer 2  MPLS and DiffServ deployed globally©Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE 17
  18. 18. Metanoia, Inc.Global Crossing: Critical Systems Thinking™Network Design Philosophy Ensure there are no bottlenecks in normal state On handling congestion  Prevent via MPLS-TE  Manage via Diffserv Over-provisioning  Well traffic engineered network can handle all traffic  Can withstand failure of even the most critical link(s) Avoid excessive complexity & features  Makes the network unreliable/unstable©Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE 18
  19. 19. Metanoia, Inc. Critical Systems Thinking™Global Crossing’s Approach: Big Picture Modem Bank POP2 Web To other ISPs Server HR BR To Customers AR DR DR BR CR HR AR WR POP1 CR WR POP3 OC-3/OC-12AR = Access Router WR OC-12/OC-48BR = Border Router OC-48/OC-192 CRDR = DSL Aggregation DR ARCR = Core RouterHR = Hosting Router HR BRWR = WAN Router Ethernet Switch©Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE 19
  20. 20. Metanoia, Inc.TE in the US IP Network: Critical Systems Thinking™Deployment Strategy Decision to adopt MPLS for traffic engineering & VPNs  Y2000: 50+ POPs, 300 routers; Y2002: 200+ POPs Initially, hierarchical MPLS system  2 levels of LSPs Later, a flat MPLS LSP full mesh only between core routers Started w/ 9 regions -- 10-50 LSRs/region ⇒ 100-2500 LSPs/region  Within regions: Routers fully-meshed  Across regions: Core routers fully-meshed Intra-region traffic ~Mb/s to Gb/s, Inter-region traffic ~ Gb/s Source [Xiao00]©Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE 20
  21. 21. Metanoia, Inc. Critical Systems Thinking™Design Principles: Statistics Collection 25 Mb/s A 25 Mb/s C B Using packets, we do not know traffic individually to B & C A LSP1 = 15 Mb/s C LSP2 = 10 Mb/s B LSP3 = 10 Mb/s Statistics on individual LSPs can be used to build matrices©Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE 21
  22. 22. Metanoia, Inc. Design Principles: LSP Control & Critical Systems Thinking™ Management New LSP takes New Request longer path A to D = 2.2 Gb/s B Links utilization is Manually move traffic away from OC-192 more balanced A potential congestion via ERO OC-48 D 10% in use before new req. B D B Lowered load, greater OC-192 headroom to grow A LSPs split across alternate routes B 2 LSPs of 1.2 D A Gb/s each OC-48 Load splitting B D D ratio = 0.5 each Adding new LSPs with a Bconfigured load splitting ratio D ©Copyright 2002, All Rights Reserved Modern Carrier Strategies for TE 22
  23. 23. Metanoia, Inc.Global Crossing’s Current LSP Critical Systems Thinking™Layout and Traffic Routing Region 1 Region 2 Source POP3 POP1 Full LSP Mesh in Core POP4 Core LSP between WRs in POPs 1 & 5 POP2 POP5 Region 3 Destination Region 4©Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE 23
  24. 24. Metanoia, Inc. Critical Systems Thinking™Sprint (FON): A Bit of History A century of evolution ... 1899: Brown Telephone Co., Abilene, KS 1976: United Telephone, Kansas City, MS, 3.5M customers 1984: Began building first, all-digital, US-wide, fiber-optic network 1986: United + GTE merge LD subsidiaries  US Sprint 1992: United buys GTE’s stake, renaming co. to Sprint Corp. 2002: ~$23B revenue, 23M customers, 70 countries, 80,000 employees 110,000+ route miles in the long distance (LD) network  34,000+ in US, 78,000+ in rest of the world Transport infrastructure common to voice, ATM, & IP network  Provides considerable leverage, as we’ll see later©Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE 24
  25. 25. Metanoia, Inc. Critical Systems Thinking™ Sprint (FON): IP Network Timeline Service via native IP bbone OC-12 4F-BLSR deployed OC-192 TAT links Cisco GSR tested/deployed OC-192 in Europe All bbone routers GSR12416s All DS3 IP network DEC FDDI Gigaswitch POP Deploy OC-48 POS over DWDM Work with Cisco for next router for OC-3 backbone GigaPOP bboneFirst IXC w/ Internet GSR in POP Deploy GSR12016 Expand to Asia,svc. on T1 network OC-3 WAN South America 92 93 95 96 97 98 99 01 02 ©Copyright 2002, All Rights Reserved Modern Carrier Strategies for TE 25
  26. 26. Metanoia, Inc. Critical Systems Thinking™SprintLinkTM IP Backbone Network Represents connectivity only (not to scale) 110,000+ route miles (common with Sprint LD network) 400+ POPs 19+ countries 5 continents 30+ major intl. cities (reach S. America as well) Courtesy: Jeff Chaltas©Copyright 2002, Sprint Public Relations 26All Rights Reserved Modern Carrier Strategies for TE
  27. 27. Metanoia, Inc. Critical Systems Thinking™SprintLinkTM IP Network Tier-1 Internet backbone  Customers: corporations, Tier-2 ISPs, univs., ...  Native IP-over-DWDM using SONET framing  4F-BLSR infrastructure (425 SONET rings in network) Backbone  US: OC-48/STM-16 (2.5 Gb/s) links  Europe: OC-192/STM-64 (10 Gb/s) links  DWDM with 8-40 λ’s/fiber Equipment  Core: Cisco GSR 12000/12416 (bbone), 10720 metro edge router  Edge: Cisco 75xxx series  Optical: Ciena Sentry 4000, Ciena CoreDirector©Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE 27
  28. 28. Metanoia, Inc. Critical Systems Thinking™SprintLinkTM IP Design Philosophy Large networks exhibit arch., design & engg. (ADE) non-linearities not seen at smaller scales ⇒ Even small things can & do cause huge effects (amplification) ⇒ More simultaneous events mean greater likelihood of interaction (coupling)∴ Simplicity Principle: simple n/wks are easier to operate & scale ⇒ Complexity prohibits efficient scaling, driving up CAPEX and OPEX! Confine intelligence at edges No state in the network core/backbone Fastest forwarding of packets in core ⇒ Ensure packets encounter minimal queueing©Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE 28
  29. 29. Metanoia, Inc. Critical Systems Thinking™SprintLinkTM Deployment Strategy L2 failure detection triggers switchover before L3 converges Parallel links 50% utilization under normal state A 4 Z 1 3 2 SONET framing for error detection Line Line Card Card SONET IP Data Overhead©Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE 29
  30. 30. Metanoia, Inc. Critical Systems Thinking™SprintLinkTM Design Principles Great value on traffic measurement & monitoring Use it for  Design, operations, management  Dimensioning, provisioning  SLAs, pricing  Minimizing the extent of complex TE & QoS in the core©Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE 30
  31. 31. Metanoia, Inc. Critical Systems Thinking™Sprint’s Approach to Monitoring Backbone Links Backbone Peering Links Router Probe Access Access Access Router Router Router Analysis platform located at Sprint ATL, Burlingame, CA Customers Customers Customers Adapted from [Diot99]©Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE 31
  32. 32. Metanoia, Inc. Critical Systems Thinking™Sprint Approach to TEAim: Thoroughly understand backbone traffic dynamicsAnswer questions such as: Composition of traffic? Origin of traffic? Between any pair of POPs  What is the traffic demand?  Volume of traffic?  Traffic patterns? (In time? In space?)  How is this demand routed? How does one design traffic matrics optimally?©Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE 32
  33. 33. Metanoia, Inc. Critical Systems Thinking™ Obtaining Traffic Matrices between POPs DA SA POP1 B 1.1.1.1 POP2 AIP Packet Destination Subnet D POP3 POP4 1.1.1/24 C Traffic Exit DA Exit POP POP # pkts Protocol Matrices Build Table POP1 Combine data, 1.1.1.1 POP4 POP2 Obtain matrix POP3 By Protocol City A City B City C City D POP4 City ACity A City B City C City D By Time City A B City of Day City B C City City C D City ©Copyright 2002, City D All Rights Reserved Modern Carrier Strategies for TE 33
  34. 34. Metanoia, Inc. Critical Systems Thinking™A Peek at a Row of a Traffic Matrix To Backbone Peer 2 Peer 1 ISP Web 2 Web 1 Sprint POP under study Distribution of aggregate access traffic across other POPs in the Sprint backbone Adapted from [Bhattacharya02] Summary of Data Collected©Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE 34
  35. 35. Metanoia, Inc. Critical Systems Thinking™Applications of Traffic Matrices Traffic engineering Verify BGP peering Intra-domain routing SLA drafting Customer reports©Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE 35
  36. 36. Metanoia, Inc. Critical Systems Thinking™Acknowledgements  Thomas Telkamp, Global Crossing  Robert J. Rockell, Jeff Chaltas, Ananth Nagarajan, Sprint  Steve Gordon, Cable and Wireless  Jennifer Rexford, Albert Greenberg, Carsten Lund, AT&T Research  Wai-Sum Lai, AT&T  Fang Wu, NTT America  Arman Maghbouleh, Alan Gous, Cariden Technologies  Yufei Wang, VPI Systems©Copyright 2002,All Rights Reserved Modern Carrier Strategies for TE 36

×