Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Carrier Strategies for Backbone Traffic Engineering and QoS

1,974 views

Published on

A fundamental problem before carriers today is to optimize network cost
and performance by better resource allocation to traffic demands. This is especially
important with the packet infrastructure becoming a critical business resource.

The key to achieving this is traffic engineering (TE), the process of
systematically putting traffic where there is capacity, and backbone
capacity management, the process of ensuring that there is enough network
capacity to meet demand, even at peak times and under failure conditions,
without significant queue buildups.

In this talk, we first focus on the TE techniques and approaches used
in the networks of two large carriers: Global Crossing and
Sprint, which represent the two ends of the traffic engineering spectrum.
We do so by presenting a snapshot of their TE philosophy, deployment strategy,
and network design principles and operation.

We then present the results of an empirical study of backbone traffic
characteristics that suggests that Internet traffic is not self-similar at
timescales relevant to QoS. Our non-parametric approach requires minimal
assumptions (unlike much of the previous work), and allows
us to formulate a practical process for ensuring QoS using backbone
capacity management.

(This latter work is joint with Thomas Telkamp, Global Crossing Ltd. and Arman
Maghbouleh, Cariden Technologies, Inc.)

Published in: Business, Technology
  • Be the first to comment

Carrier Strategies for Backbone Traffic Engineering and QoS

  1. 1. Carrier Strategies for Backbone Traffic Engineering and QoS Dr. Vishal Sharma President & Principal Consultant Metanoia, Inc. Voice: +1 408 394 6321 Email: [email_address] Web: http://www.metanoia-inc.com Metanoia, Inc. Critical Systems Thinking™ © Copyright 2004 All Rights Reserved
  2. 2. Agenda <ul><li>Traffic engineering techniques & approaches </li></ul><ul><ul><li>Global Crossing </li></ul></ul><ul><ul><li>Sprint </li></ul></ul><ul><li>Backbone traffic characterization for QoS via capacity management </li></ul><ul><li>[Joint work with Thomas Telkamp (Global Crossing), Arman Maghbouleh (Cariden Technologies), Stephen Gordon (SAIC, former C&W)] </li></ul>
  3. 3. Basic Service Provider Goals <ul><li>The two fundamental tasks before any service provider: </li></ul><ul><li>Deploy a physical topology that meets customers’ needs </li></ul><ul><li>Map customer traffic flows on to the physical topology </li></ul><ul><li>Earlier (1990s) the mapping task was uncontrolled! </li></ul><ul><ul><li>By-product of shortest-path IGP routing </li></ul></ul><ul><ul><li>Often handled by over-provisioning </li></ul></ul>
  4. 4. Two Paths to TE in IP Networks <ul><li>With increase in traffic, emergence of ATM, and higher-speed SONET, two approaches emerged </li></ul><ul><li>Use a Layer 2 (ATM) network </li></ul><ul><li>Build ATM backbone </li></ul><ul><li>Deploy complete PVC mesh, bypass use of IP metrics </li></ul><ul><li>TE at ATM layer </li></ul><ul><li>With time, evolve ATM to MPLS-based backbone </li></ul><ul><li>Use only Layer 3 (IP) network </li></ul><ul><li>Build SONET infrastructure </li></ul><ul><li>Rely on SONET for resilience </li></ul><ul><li>Run IP directly on SONET (POS) </li></ul><ul><li>Use metrics (systematically) to control flow of traffic </li></ul>
  5. 5. Global Crossing IP Backbone Network 100,000 route miles 27 countries 250 major cities 5 continents 200+ POPs Courtesy: Thomas Telkamp, GBLX
  6. 6. Global Crossing IP Network <ul><li>OC-48c/STM-16c (2.5Gbps) IP backbone </li></ul><ul><ul><li>Selected 10Gbps links operational (e.g. Atlantic) </li></ul></ul><ul><li>Services offered </li></ul><ul><ul><li>Internet access & Transit services </li></ul></ul><ul><ul><li>IP VPNs -- Layer 3 and Layer 2 </li></ul></ul><ul><ul><li>MPLS and DiffServ deployed globally </li></ul></ul>
  7. 7. Global Crossing: Network Design Philosophy <ul><li>Ensure there are no bottlenecks in normal state </li></ul><ul><li>On handling congestion </li></ul><ul><ul><li>Prevent via MPLS-TE </li></ul></ul><ul><ul><li>Manage via Diffserv </li></ul></ul><ul><li>Over-provisioning </li></ul><ul><ul><li>Well traffic engineered network can handle all traffic </li></ul></ul><ul><ul><li>Can withstand failure of even the most critical link(s) </li></ul></ul><ul><li>Avoid excessive complexity & features </li></ul><ul><ul><li>Makes the network unreliable/unstable </li></ul></ul>
  8. 8. Global Crossing’s Approach: Big Picture
  9. 9. TE in the US IP Network: Deployment Strategy <ul><li>Decision to adopt MPLS for traffic engineering & VPNs </li></ul><ul><ul><li>Y2000: 50+ POPs, 300 routers; Y2002: 200+ POPs </li></ul></ul><ul><li>Initially, hierarchical MPLS system  2 levels of LSPs </li></ul><ul><li>Later, a flat MPLS LSP full mesh only between core routers </li></ul><ul><li>Started w/ 9 regions -- 10-50 LSRs/region  100-2500 LSPs/region </li></ul><ul><ul><li>Within regions: Routers fully-meshed </li></ul></ul><ul><ul><li>Across regions: Core routers fully-meshed </li></ul></ul><ul><li>Intra-region traffic ~Mb/s to Gb/s, Inter-region traffic ~ Gb/s </li></ul>Source [Xiao00]
  10. 10. Design Principles: Statistics Collection Statistics on individual LSPs can be used to build matrices Using packets, we do not know traffic individually to B & C
  11. 11. Design Principles: LSP Control & Management Manually move traffic away from potential congestion via ERO Adding new LSPs with a configured load splitting ratio
  12. 12. Global Crossing’s Current LSP Layout and Traffic Routing
  13. 13. Global Crossing: Advanced Network Technologies <ul><li>MPLS Fast Reroute (FRR) </li></ul><ul><ul><li>Localizes impact of failures </li></ul></ul><ul><ul><li>Local to router detecting failure </li></ul></ul><ul><ul><li>Head-end establishes new e2e LSP </li></ul></ul><ul><li>Per-class traffic engineering </li></ul><ul><ul><li>Diffserv-aware TE </li></ul></ul><ul><ul><li>Avoids concentrating real-time traffic on any one link </li></ul></ul><ul><ul><li>Limits the bandwidth used per class, useful during FRR </li></ul></ul><ul><li>IGP Convergence </li></ul><ul><ul><li>Tune network for fast IS-IS convergence, few seconds </li></ul></ul><ul><ul><li>Use L2 failure detection and timers to achieve goal </li></ul></ul>
  14. 14. SprintLink TM IP Backbone Network 19+ countries 30+ major intl. cities 5 continents (reach S. America as well) 400+ POPs Courtesy: Jeff Chaltas Sprint Public Relations Represents connectivity only (not to scale) 110,000+ route miles (common with Sprint LD network)
  15. 15. SprintLink TM IP Network <ul><li>Tier-1 Internet backbone </li></ul><ul><ul><li>Customers: corporations, Tier-2 ISPs, univs., ... </li></ul></ul><ul><ul><li>Native IP -over-DWDM using SONET framing </li></ul></ul><ul><ul><li>4F-BLSR infrastructure (425 SONET rings in network) </li></ul></ul><ul><li>Backbone </li></ul><ul><ul><li>US: OC-48/STM-16 (2.5 Gb/s) links </li></ul></ul><ul><ul><li>Europe: OC-192/STM-64 (10 Gb/s) links </li></ul></ul><ul><ul><li>DWDM with 8-40  ’s/fiber </li></ul></ul><ul><li>Equipment </li></ul><ul><ul><li>Core: Cisco GSR 12000/12416 (bbone), 10720 metro edge router </li></ul></ul><ul><ul><li>Edge: Cisco 75xxx series </li></ul></ul><ul><ul><li>Optical: Ciena Sentry 4000, Ciena CoreDirector </li></ul></ul>
  16. 16. SprintLink TM IP Design Philosophy <ul><li>Large networks exhibit arch., design & engg. (ADE) non-linearities not seen at smaller scales </li></ul><ul><ul><li>Even small things can & do cause huge effects ( amplification ) </li></ul></ul><ul><ul><li>More simultaneous events mean greater likelihood of interaction ( coupling ) </li></ul></ul><ul><li>Simplicity Principle: simple n/wks are easier to operate & scale </li></ul><ul><ul><li>Complexity prohibits efficient scaling, driving up CAPEX and OPEX! </li></ul></ul><ul><li>Confine intelligence at edges </li></ul><ul><li>No state in the network core/backbone </li></ul><ul><li>Fastest forwarding of packets in core </li></ul><ul><ul><li>Ensure packets encounter minimal queueing </li></ul></ul>
  17. 17. SprintLink TM Deployment Strategy
  18. 18. SprintLink TM Design Principles <ul><li>Great value on traffic measurement & monitoring </li></ul><ul><li>Use it for </li></ul><ul><ul><li>Design, operations, management </li></ul></ul><ul><ul><li>Dimensioning, provisioning </li></ul></ul><ul><ul><li>SLAs, pricing </li></ul></ul><ul><ul><li>Minimizing the extent of complex TE & QoS in the core </li></ul></ul>
  19. 19. Sprint’s Monitoring Methodology Adapted from [Diot99] Analysis platform located at Sprint ATL, Burlingame, CA
  20. 20. Sprint Approach to TE <ul><li>Aim: Thoroughly understand backbone traffic dynamics </li></ul><ul><li>Answer questions such as: </li></ul><ul><li>Composition of traffic? Origin of traffic? </li></ul><ul><li>Between any pair of POPs </li></ul><ul><ul><li>What is the traffic demand? </li></ul></ul><ul><ul><ul><li>Volume of traffic? </li></ul></ul></ul><ul><ul><ul><li>Traffic patterns? (In time? In space?) </li></ul></ul></ul><ul><ul><li>How is this demand routed? </li></ul></ul><ul><li>How does one design traffic matrics optimally? </li></ul>
  21. 21. Obtaining Traffic Matrices between POPs
  22. 22. A Peek at a Row of a Traffic Matrix Adapted from [Bhattacharya02] Summary of Data Collected Distribution of aggregate access traffic across other POPs in the Sprint backbone Peer 1 Peer 2 Web 2 Web 1 ISP To Backbone Sprint POP under study
  23. 23. Applications of Traffic Matrices <ul><li>Traffic engineering </li></ul><ul><li>Verify BGP peering </li></ul><ul><li>Intra-domain routing </li></ul><ul><li>SLA drafting </li></ul><ul><li>Customer reports </li></ul>
  24. 24. Routing of Demands in the Sprint Backbone <ul><li>Matrices provide insight into aggregate traffic behavior </li></ul><ul><ul><li>Do not show the paths demands follow over the backbone </li></ul></ul><ul><li>In reality </li></ul><ul><ul><li>IS-IS link weights hand-crafted by network ops. experts </li></ul></ul><ul><ul><li>Weights chosen to restrict traffic b/ween an ingress-egress POP pair to only a few paths through the backbone </li></ul></ul><ul><ul><li>Intra-POP link weights heavily influence backbone paths </li></ul></ul><ul><li>Result: Despite several alternate paths between POPs </li></ul><ul><ul><li>Many remain underutilized </li></ul></ul><ul><ul><li>Few have v. high utilization </li></ul></ul>
  25. 25. Link Utilization Across the Sprint IP Backbone Almost 50% of the links have utilization under 15%! 8% of the links are 60% utilized <ul><li>Observe </li></ul><ul><li>Extent of link underutilization </li></ul><ul><li>Disparity in utilization levels </li></ul><ul><li>Need better load balancing rules </li></ul><ul><li>Require a systematic, policy-based approach to do so </li></ul>Source [Bhattacharya02]
  26. 26. Techniques for Aggregate Load Balancing <ul><li>Effective load balancing across backbone ... </li></ul><ul><li>Knowing how to split traffic over multiple alternate paths </li></ul><ul><li>Criteria used depend on purpose </li></ul><ul><ul><li>Different service levels  use TOS byte or protocol field </li></ul></ul><ul><ul><li>Backbone routing  use destination address (DA) as basis </li></ul></ul><ul><li>Gather inter-POP traffic into streams per DA-based prefixes </li></ul><ul><ul><li>E.g. An N-bit prefix produces a pN stream </li></ul></ul><ul><li>Assign streams to different paths to balance network load </li></ul>
  27. 27. Observations on Aggregate Streams <ul><li>Examine traffic volume & stability of streams over interval for which load balancing is to be performed </li></ul><ul><li>Findings </li></ul><ul><li>Elephants and mice ... </li></ul><ul><ul><li>Few very high-vol. streams, many low-vol. streams </li></ul></ul><ul><li>Ranking of streams stable over large timescales </li></ul><ul><li>Phenomenon is recursive </li></ul><ul><ul><li>E.g. p8 elephant sub-divided into p16 streams also has elephants & mice! </li></ul></ul>Result Engineering network for elephants alone gives practically all of the benefits of TE! (good for scalability as well)
  28. 28. Actual Behavior of Streams in the Sprint Backbone Elephants retain a large share of the bandwidth & maintain their ordering Source [Bhattacharya02] Time of day variation of elephants & mice to a busy egress POP Elephants Mice Decreasing Traffic Volume Distribution of traffic from p8 streams of POP under study to 3 egress POPs Less than 10 of the largest streams account for up to 90% of the traffic
  29. 29. Agenda <ul><li>Traffic engineering techniques & approaches </li></ul><ul><ul><li>Global Crossing </li></ul></ul><ul><ul><li>Sprint </li></ul></ul><ul><li>Backbone traffic characterization for QoS via capacity management </li></ul><ul><li>[Joint work with Thomas Telkamp (Global Crossing), Arman Maghbouleh (Cariden Technologies), Stephen Gordon (SAIC, former C&W)] </li></ul>
  30. 30. QoS for Backbone IP Networks <ul><li>QoS – nature of packet delivery service realized in the network </li></ul><ul><li>Characterized by achieved : bandwidth, delay, jitter, loss </li></ul><ul><li>For backbone networks </li></ul><ul><ul><li>No link oversubscription  achieved b/w ~ desired b/w </li></ul></ul><ul><ul><li>Controlled O/P queue size  bounded packet delays </li></ul></ul><ul><ul><li>Bounded packet delays </li></ul></ul><ul><ul><ul><li> Bounded jitter </li></ul></ul></ul><ul><ul><ul><li> No packet loss </li></ul></ul></ul><ul><li> Backbone QoS  Latency characteristics of traffic </li></ul><ul><ul><ul><ul><li> (Packet delay and jitter) </li></ul></ul></ul></ul>
  31. 31. Relevant Timescales <ul><li>Long-term: > 5 minutes </li></ul><ul><li>Short-term: < 5 minutes </li></ul>100ms 1sec 1h 0 10sec 1min Aggregate Flows Intra-Flow Users/Applications TCP (RTT) Flow Sizes/Durations Diurnal variation Timescale Dynamics Characteristics
  32. 32. Timescales Critical for QoS <ul><li>Some of the most stringent QoS requirements for IP traffic arise when carrying voice (e.g. ITU G.114) </li></ul><ul><li>Requirements include: </li></ul><ul><ul><li>Packet delay (one-way) < 150 ms </li></ul></ul><ul><ul><li>End-to-end jitter < 20 ms (for toll-quality voice) </li></ul></ul><ul><li> Need resolution at millisecond timescales to understand </li></ul><ul><ul><li>Trajectory of individual packets </li></ul></ul><ul><ul><li>Queueing behavior in the core </li></ul></ul><ul><li>Good performance at ms extends naturally to larger time-scales </li></ul>
  33. 33. Short-term Traffic Characterization <ul><li>Investigate burstiness within 5-minute intervals </li></ul><ul><li>Measure at timescale critical for queueing </li></ul><ul><ul><li>E.g., 1 ms, 5 ms, or 10 ms </li></ul></ul><ul><li>Analyze statistical properties </li></ul><ul><ul><li>Variance, autocorrelation, … </li></ul></ul><ul><li>Done one-time at specific locations, as it involves </li></ul><ul><ul><li>Complex setup </li></ul></ul><ul><ul><li>Voluminous data collection </li></ul></ul>
  34. 34. Data Collection and Measurement <ul><li>12 traces, 30 seconds each </li></ul><ul><ul><li>Collected over a month </li></ul></ul><ul><ul><li>Different times and days </li></ul></ul><ul><li>Mean b/w 126 – 290 Mbps (<< 1 Gbps) </li></ul><ul><li> No queueing/shaping on O/P interface </li></ul><ul><li>Trace utilizations uniformly < 1Gbps over any 1 ms interval </li></ul>Shomiti Fiber Tap Tap Analyzer GbE backbone link Measurement PC
  35. 35. Raw Results 30 sec of data, 1ms scale <ul><li>Mean = 950 Mbps </li></ul><ul><li>Max. = 2033 Mbps </li></ul><ul><li>Min. = 509 Mbps </li></ul><ul><li>95-percentile: 1183 Mbps </li></ul><ul><li> 5-percentile: 737 Mbps </li></ul><ul><li>~250 packets per interval </li></ul>Mean rate over 30 sec Output queue rate (available link bandwidth)
  36. 36. Traffic Distribution Histogram (1ms scale) <ul><li>Fits normal probability distribution well (Std. dev. = 138 Mbps) </li></ul><ul><li>No heavy-tails </li></ul><ul><li>Suggests small over-provisioning factor </li></ul>
  37. 37. Autocorrelation Lag Plot (1ms scale) <ul><li>Scatter plot for consecutive samples of time-series </li></ul><ul><li>Are periods of high usage followed by other periods of high usage? </li></ul><ul><li>Autocorrelation at 1ms is 0.13 (=uncorrelated) </li></ul><ul><li> High bandwidth bursts do not line up to cause marked queueing </li></ul><ul><li>High autocorrelation </li></ul><ul><li>Points concentrated along 45 ° line </li></ul><ul><li>Clearly not the case here </li></ul>45 °
  38. 38. Poisson versus Self-Similar Traffic Scale Invariant! Refs. [Liljenstolpe01], [Lothberg01] Ref. [Tekinay99] Markovian Process Self-Similar Process
  39. 39. Internet Traffic: Variance versus Timescale <ul><li>Random variable X </li></ul><ul><ul><li>Var(X (m) ) = σ 2 m -1 </li></ul></ul><ul><li>Self-similar process, with Hurst parameter H </li></ul><ul><ul><li>Var(X (m) ) = σ 2 m 2H-2 </li></ul></ul><ul><li>Long range dependence (LRD) </li></ul><ul><li> 0.5 < H < 1 </li></ul><ul><li> Var(X (m) ) converges to zero slower than a rate m -1 </li></ul>150 ms Note: m = sample size, σ 2 = Var(X) Variance decreases in proportion to timescale Variance decreases slower  self-similarity Slope = -1  Poisson
  40. 40. Traffic: Summary <ul><li>Long-term well behaved traffic </li></ul><ul><li>Short-term uncorrelated traffic </li></ul>
  41. 41. IP Capacity Allocation <ul><li>Measurement data </li></ul><ul><ul><li>5-min average utilization </li></ul></ul><ul><li>Performance goals, e.g. </li></ul><ul><ul><li>Packet loss < 1% </li></ul></ul><ul><ul><li>Jitter < 10 ms </li></ul></ul><ul><ul><li>End-to-end delay < 20 ms </li></ul></ul><ul><li>But … we have no “Erlang formulas” for IP traffic … </li></ul>Model traffic, fit parameters, evaluate parametric solution Two approaches to a solution Empirically derive guidelines by characterizing observed traffic Approach in this work
  42. 42. Queuing Simulation: Methodology <ul><li>Feed multiplexed, sampled traffic into a FIFO queue </li></ul><ul><li>Measure amount of traffic that violates set delay bound </li></ul>FIFO Queue Sampled Traffic Fixed Service Rate Monitor Queuing Delay Sampled Traffic Sampled Traffic Output Link under study 622 Mbps 572 Mbps 126 Mbps 240 Mbps 206 Mbps Example: 92% Utilization
  43. 43. Queuing Simulation: Results 89% 93% + Simulation 622 Mbps + Simulation 1000 Mbps ---- M/M/1 622 Mbps ---- M/M/1 1000 Mbps
  44. 44. Multi-hop Queueing: 8 hops <ul><li>P99.9 delay: Hop 1 = 2 ms, Hop 8 = 5.2 ms (increase not linear!) </li></ul>P99.9 = 2ms P99.9 = 5.2ms
  45. 45. Queueing: Summary <ul><li>Queueing simulation </li></ul><ul><ul><li>Backbone link (GbE) </li></ul></ul><ul><ul><ul><li>Over-provisioning ~7.5% to bound delay/hop to under 2 ms </li></ul></ul></ul><ul><ul><li>Higher speeds (2.5G/10G) </li></ul></ul><ul><ul><ul><li>Over-provisioning factor becomes very small </li></ul></ul></ul><ul><ul><li>Lower speeds (< 0.622G) </li></ul></ul><ul><ul><ul><li>Over-provisioning factor is significant </li></ul></ul></ul><ul><li>P99.9 multi-hop delay/jitter is not additive </li></ul>
  46. 46. Applications to Network Planning <ul><li>QoS targets  “Headroom” (over-provisioning %) </li></ul><ul><ul><li>Derived experimentally by characterizing short-term traffic </li></ul></ul><ul><li>Traffic matrix </li></ul><ul><ul><li>Derivable from the stable, well-behaved, long-term traffic </li></ul></ul>Determine minimum capacity deployment required to meet objectives under normal and failure conditions How to use this for planning? <ul><li>Trending – study impact of growth over time </li></ul><ul><li>Failure analysis – impact of failures on loading </li></ul><ul><ul><li>Derived experimentally by characterizing short-term traffic </li></ul></ul><ul><li>Optimization – LSP routing, IGP metrics </li></ul>
  47. 47. Acknowledgements <ul><li>Thomas Telkamp, Global Crossing </li></ul><ul><li>Robert J. Rockell, Jeff Chaltas, Ananth Nagarajan, Sprint </li></ul><ul><li>Steve Gordon, SAIC (former C&W) </li></ul><ul><li>Jennifer Rexford, Albert Greenberg, Carsten Lund, AT&T Research </li></ul><ul><li>Wai-Sum Lai, AT&T </li></ul><ul><li>Fang Wu, NTT America </li></ul><ul><li>Arman Maghbouleh, Alan Gous, Cariden Technologies </li></ul><ul><li>Yufei Wang, VPI Systems </li></ul><ul><li>Susan Cole, OPNET Technologies </li></ul>
  48. 48. References <ul><li>[Bhattacharya02] S. Bhattacharya, et al, “POP-Level and Access-Link Level Traffic Dynamics in a Tier-1 POP,” Proc. ACM SIGCOMM Internet Measurement Workshop , November 2001. </li></ul><ul><li>  </li></ul><ul><li>[Diot99] C. Diot, “Tier-1 IP Backbone Network: Architecture and Performance,”Sprint Advanced Technology Labs., 1999. Available at: http://www. sprintlabs .com/Department/IP- Interworking /Monitor/ </li></ul><ul><li>[Liljenstolpe01] Chris Liljenstolpe, Design Issues in Next Generation Carrier Networks , Proc. MPLS 2001, Washington, D.C., 7-9 October, 2001. </li></ul><ul><li>[Lothberg01] Peter Lothberg, A View of the Future: The IP-Only Internet , NANOG 22, Scottsdale, AZ, 20-22 May 2001, http://www. nanog .org/ mtg -0105/ lothberg .html </li></ul>
  49. 49. References <ul><li>[Morris00] Robert Morris and Dong Lin, Variance of Aggregated WebTraffic , IEEE Infocom’00, Tel Aviv, Israel, March 2000, pp. 360-366. </li></ul><ul><li>[Tekinay99] Zafer Sahinoglu and Sirin Tekinay, On Multimedia Networks: Self-Similar Traffic and Network Performance , IEEE Commun. Mag., vol. 37, no. 1, January 1999, pp. 48-53. </li></ul><ul><li>[Xiao00] X. Xiao et al, “Traffic Engineering with MPLS in the Internet,” IEEE Network , March/April 2000, vol. 14, no. 2, pp. 28-33. </li></ul>

×