Modern Carrier Strategies for Traffic Engineering

Metanoia, Inc.
Critical Systems Thinking™

Modern Carrier Strategies for
Traffic Engineering
Dr. Vishal Sharma
Principal Consultant
Metanoia, Inc.
Voice: +1 408 394 6321
Email: v.sharma@ieee.org
Web: http://www.metanoia-
© Copyright 2002
All Rights Reserved inc.com

Metanoia, Inc.

Basic Service Provider Goals
The two fundamental tasks before any service provider:
 Deploy a physical topology that meets customers’ needs

 Map customer traffic flows on to the physical topology

 Earlier (1990s) the mapping task was uncontrolled!
 By-product of shortest-path IGP routing
 Often handled by over-provisioning

©Copyright 2002,
All Rights Reserved Modern Carrier Strategies for TE 2

Metanoia, Inc.
The Early Years (< 1994-95): Routed Critical Systems Thinking™

Network Topology
IP Router

IP Router
IP Router

Network Cloud
Router
Interconnections

IP Router

©Copyright 2002,

Metanoia, Inc.
The Early Years (< 1994-95): A Critical Systems Thinking™

Stacked View
IP Routers

Router
-- Service creation
-- Pkt. switching
-- Stat muxing
FDDI rings -- Connectivity
(100 Mb/s)

TDM over
copper (T1/T3) MUX
-- Speed match I/Fs

DCS/DXC

SDH/SONET
-- TDM transport
-- Fault isolation
-- Restoration

©Copyright 2002,

Metanoia, Inc.
TE in Carrier Networks: Traditional Critical Systems Thinking™

Routed Core (pre 1994-95)

 Prior to advent of ATM ...
... IP metrics were the only means available to control traffic
distribution through IP networks

 Approach was ad-hoc
 Observe traffic flow through network
 Adjust weight of links with load lower/higher than desired
 Overprovision network as, needed

©Copyright 2002,

Metanoia, Inc.

Two Paths to TE in IP Networks
 With increase in traffic, emergence of ATM, and higher-speed
SONET, two approaches emerged

Use a Layer 2 (ATM) network Use only Layer 3 (IP) network
 Build ATM backbone  Build SONET infrastructure
 Deploy complete PVC mesh,  Rely on SONET for resilience
bypass use of IP metrics  Run IP directly on SONET (POS)
 TE at ATM layer  Use metrics (systematically) to
 With time, evolve ATM to MPLS- control flow of traffic (more on
based backbone this later)

©Copyright 2002,

Metanoia, Inc.

Genesis of the ATM-Core
 Growth in traffic ⇒ needed faster backbones (> T3/45 Mb/s)

 Denser backbones ⇒ metric manipulation impractical

 IP routers lagged: offered only DS3 I/Fs & s/w forwarding

 ATM emerged, was designed for WAN from start
⇒ In 1994-95 had OC-3, and later OC-12 I/Fs available
⇒ Allowed carriers to redesign their networks for high-speeds

⇒ As an evolutionary step, SPs moved to a switched ATM core

©Copyright 2002,

Metanoia, Inc.

ATM-based Cores (mid to late 1990s)
Router
-- Service creation
-- Pkt. switching
-- Stat muxing
-- Connectivity

ATM
-- Traffic engg.
-- Hardware fwding

MUX
-- Speed match I/Fs

SDH/SONET
-- TDM transport
-- Fault isolation
-- Restoration

©Copyright 2002,

Metanoia, Inc.

Physical Topology with an ATM Core
OC-3
Router
POP 1 OC-3 POP i+1

POP j
POP 2 OC-12

OC-3 ATM
SAR I/F

POP i ATM Switch POP N

ATM Core

©Copyright 2002,

Metanoia, Inc.

Logical Topology with an ATM Core

PVC 1' Primary PVC Mesh
A PVC 1 C A C

PVC 4' PVC 4
PVC 2
PVC 3'

B PVC 5 D

PVC 3 B D
PVC 5'

Secondary PVC Mesh
PVC 2'

ATM PVC Layout L3 Logical Topology

 ATM-core (usually) fully owned by SP
 Dedicated (usually) to supporting IP backbone
 Utilized ATM UBR or ATM VBR-rt/CBR, depending on classes of traffic
©Copyright 2002,

Metanoia, Inc.
Genesis of the IP-over-SONET/SDH Critical Systems Thinking™

Approach
 Desire to minimize # of network layers
 Easier management
 Simpler operation
 Potentially scalable

 Belief that high-speed SONET/SDH I/Fs would become
available with advances in components (vindicated with time)

 Dictated partly by how (in time) a carrier’s network evolved

©Copyright 2002,

Metanoia, Inc.
SONET/SDH-based Cores (mid-to- Critical Systems Thinking™

late 1990s and beyond)

Router
-- Service creation
-- Pkt. switching
-- Stat muxing
-- TE w/ metrics or
MPLS

MUX
-- Speed match I/Fs

SDH/SONET
-- Framing
-- Fault isolation
-- Restoration
(moving away)

DWDM
-- B/w on existing
plant
©Copyright 2002,

Metanoia, Inc.

Physical Topology with SONET/SDH Core
OC-3/12 or
STM-1/4
Router
POP 1 POP i+1

OC-48 BLSR/ POP j
MS-SPRing
POP 2
Point-to-point SONET/
SDH framed link
OC-3 SONET/
STM-1 SDH I/F

POP i OC-3/13 UPSR/ POP N
DXC SNCP Ring

SONET/SDH Core

©Copyright 2002,

Metanoia, Inc.

Logical Topology with SONET/SDH Core
Parallel Logial Links for
Load Sharing
Ckt. 1
A C A C
Ckt. 5
Ckt. 4

Ckt. 3

B D B D
Ckt. 2
Each logical link provisioned
for 2X the bandwidth
SONET/SDH Circuit Layout
L3 Logical Topology

 SONET/SDH infrastructure (usually) owned by SP
 Logical links between POP routers realized over a physical SONET/SDH
circuit going over a fiber path
 Parallel logical links (physically disjoint) provisioned b/w each router pair
©Copyright 2002,

Metanoia, Inc.

Global Crossing IP Backbone Network

Courtesy: Thomas Telkamp, GBLX

100,000 route miles 200+ POPs 5 continents 27 countries 250 major cities
©Copyright 2002,

Metanoia, Inc.

Global Crossing (GBLX): A Bit of History
 First independent global fiber network
 Launched operations -- March 1997
 First segment turned on -- May 1998

 Expanded network & svcs. by acquisitions & JVs
 Frontier Telecommunications, Sept 1999
 Racal Telecom, Nov 1999
 Hutchison Global Crossing, Jan 2000
 IXNET/IPC, June 2000

 International network, worldwide reach
 100,000 route miles, 27 countries, 250 major cities
 195 POPs (mid 2001)
©Copyright 2002,

Metanoia, Inc.

Global Crossing IP Network
 OC-48c/STM-16c (2.5Gbps) IP backbone
 Selected 10Gbps links operational (e.g. Atlantic)
Edge Equipment

Cisco 7500/7200 ESR, OSR
Juniper M10/20/40

Core Equipment

Cisco GSR 12000/12400
[12.0(17) SI]
 Services offered
 Internet access & Transit services
 IP VPNs -- Layer 3 and Layer 2
 MPLS and DiffServ deployed globally

©Copyright 2002,

Metanoia, Inc.
Global Crossing: Critical Systems Thinking™

Network Design Philosophy
 Ensure there are no bottlenecks in normal state

 On handling congestion
 Prevent via MPLS-TE
 Manage via Diffserv

 Over-provisioning
 Well traffic engineered network can handle all traffic
 Can withstand failure of even the most critical link(s)

 Avoid excessive complexity & features
 Makes the network unreliable/unstable

©Copyright 2002,

Metanoia, Inc.

Global Crossing’s Approach: Big Picture
Modem
Bank POP2
Web To other ISPs
Server HR BR
To Customers AR
DR
DR
BR CR
HR AR
WR
POP1 CR

WR

POP3

OC-3/OC-12
AR = Access Router WR OC-12/OC-48
BR = Border Router OC-48/OC-192
CR
DR = DSL Aggregation
DR AR
CR = Core Router
HR = Hosting Router HR BR

WR = WAN Router Ethernet
Switch
©Copyright 2002,

Metanoia, Inc.
TE in the US IP Network: Critical Systems Thinking™

Deployment Strategy
 Decision to adopt MPLS for traffic engineering & VPNs
 Y2000: 50+ POPs, 300 routers; Y2002: 200+ POPs

 Initially, hierarchical MPLS system  2 levels of LSPs

 Later, a flat MPLS LSP full mesh only between core routers

 Started w/ 9 regions -- 10-50 LSRs/region ⇒ 100-2500 LSPs/region
 Within regions: Routers fully-meshed
 Across regions: Core routers fully-meshed

 Intra-region traffic ~Mb/s to Gb/s, Inter-region traffic ~ Gb/s
Source [Xiao00]
©Copyright 2002,

Metanoia, Inc.

Design Principles: Statistics Collection
25 Mb/s
A
25 Mb/s
C
B

Using packets, we do not know
traffic individually to B & C

A LSP1 = 15 Mb/s

C
LSP2 = 10 Mb/s
B
LSP3 = 10 Mb/s
Statistics on individual LSPs can
be used to build matrices
©Copyright 2002,

Metanoia, Inc.
Design Principles: LSP Control & Critical Systems Thinking™

Management
New LSP takes
New Request
longer path
A to D = 2.2 Gb/s

B Links utilization is
Manually move traffic away from OC-192 more balanced
A
potential congestion via ERO
OC-48
D
10% in use
before new req.

B
D

B
Lowered load, greater
OC-192 headroom to grow
A LSPs split across
alternate routes B

2 LSPs of 1.2 D A
Gb/s each
OC-48 Load splitting
B D
D ratio = 0.5 each

Adding new LSPs with a
B
configured load splitting ratio D

©Copyright 2002,

Metanoia, Inc.
Global Crossing’s Current LSP Critical Systems Thinking™

Layout and Traffic Routing
Region 1 Region 2

Source POP3
POP1 Full LSP Mesh
in Core
POP4

Core LSP between
WRs in POPs 1 & 5

POP2
POP5

Region 3 Destination

Region 4
©Copyright 2002,

Metanoia, Inc.

Sprint (FON): A Bit of History
 A century of evolution ...
1899: Brown Telephone Co., Abilene, KS
1976: United Telephone, Kansas City, MS, 3.5M customers
1984: Began building first, all-digital, US-wide, fiber-optic network
1986: United + GTE merge LD subsidiaries  US Sprint
1992: United buys GTE’s stake, renaming co. to Sprint Corp.
2002: ~$23B revenue, 23M customers, 70 countries, 80,000 employees

 110,000+ route miles in the long distance (LD) network
 34,000+ in US, 78,000+ in rest of the world

 Transport infrastructure common to voice, ATM, & IP network
 Provides considerable leverage, as we’ll see later

©Copyright 2002,

Metanoia, Inc.

Sprint (FON): IP Network Timeline

Service via native IP bbone
OC-12 4F-BLSR deployed OC-192 TAT links
Cisco GSR tested/deployed OC-192 in Europe
All bbone routers
GSR12416s
All DS3 IP network
DEC FDDI Gigaswitch POP Deploy OC-48
POS over DWDM

Work with Cisco for next
router for OC-3 backbone
GigaPOP bbone
First IXC w/ Internet GSR in POP Deploy GSR12016 Expand to Asia,
svc. on T1 network OC-3 WAN South America

'92 '93 '95 '96 '97 '98 '99 '01 '02

©Copyright 2002,

Metanoia, Inc.

SprintLinkTM IP Backbone Network

Represents connectivity
only (not to scale)

110,000+ route miles
(common with Sprint LD network)

400+ POPs 19+ countries
5 continents 30+ major intl. cities
(reach S. America as well) Courtesy: Jeff Chaltas
©Copyright 2002, Sprint Public Relations 26
All Rights Reserved Modern Carrier Strategies for TE

Metanoia, Inc.

SprintLinkTM IP Network
 Tier-1 Internet backbone
 Customers: corporations, Tier-2 ISPs, univs., ...
 Native IP-over-DWDM using SONET framing
 4F-BLSR infrastructure (425 SONET rings in network)

 Backbone
 US: OC-48/STM-16 (2.5 Gb/s) links
 Europe: OC-192/STM-64 (10 Gb/s) links
 DWDM with 8-40 λ’s/fiber

 Equipment
 Core: Cisco GSR 12000/12416 (bbone), 10720 metro edge router
 Edge: Cisco 75xxx series
 Optical: Ciena Sentry 4000, Ciena CoreDirector
©Copyright 2002,

Metanoia, Inc.

SprintLinkTM IP Design Philosophy
 Large networks exhibit arch., design & engg. (ADE) non-linearities not
seen at smaller scales
⇒ Even small things can & do cause huge effects (amplification)
⇒ More simultaneous events mean greater likelihood of interaction (coupling)

∴ Simplicity Principle: simple n/wks are easier to operate & scale
⇒ Complexity prohibits efficient scaling, driving up CAPEX and OPEX!

 Confine intelligence at edges

 No state in the network core/backbone

 Fastest forwarding of packets in core
⇒ Ensure packets encounter minimal queueing
©Copyright 2002,

Metanoia, Inc.

SprintLinkTM Deployment Strategy
L2 failure detection triggers
switchover before L3 converges

Parallel links 50% utilization
under normal state

A 4
Z
1

3

2 SONET framing for
error detection
Line Line
Card Card

SONET
IP Data
Overhead
©Copyright 2002,

Metanoia, Inc.

SprintLinkTM Design Principles
 Great value on traffic measurement & monitoring

 Use it for
 Design, operations, management
 Dimensioning, provisioning
 SLAs, pricing
 Minimizing the extent of complex TE & QoS in the core

©Copyright 2002,

Metanoia, Inc.

Sprint’s Approach to Monitoring

Backbone
Links

Backbone
Peering Links
Router
Probe

Access Access Access
Router Router Router

Analysis platform located at
Sprint ATL, Burlingame, CA Customers Customers Customers

Adapted from [Diot99]
©Copyright 2002,

Metanoia, Inc.

Sprint Approach to TE
Aim: Thoroughly understand backbone traffic dynamics

Answer questions such as:
 Composition of traffic? Origin of traffic?

 Between any pair of POPs
 What is the traffic demand?
 Volume of traffic?

 Traffic patterns? (In time? In space?)

 How is this demand routed?

 How does one design traffic matrics optimally?

©Copyright 2002,

Metanoia, Inc.

Obtaining Traffic Matrices between POPs
DA SA POP1 B
1.1.1.1 POP2
A
IP Packet Destination
Subnet
D

POP3 POP4 1.1.1/24
C

Traffic
Exit
DA Exit POP POP
# pkts Protocol Matrices
Build
Table POP1
Combine data,
1.1.1.1 POP4 POP2 Obtain matrix
POP3 By
Protocol City A City B City C City D
POP4
City ACity A City B City C City D

By Time City A B
City
of Day City B C
City
City C D
City

©Copyright 2002,
City D

Metanoia, Inc.

A Peek at a Row of a Traffic Matrix
To Backbone
Peer 2

Peer 1

ISP
Web 2

Web 1

Sprint POP under study

Distribution of aggregate access
traffic across other POPs in the Sprint
backbone

Adapted from [Bhattacharya02]
Summary of Data Collected
©Copyright 2002,

Metanoia, Inc.

Acknowledgements
 Thomas Telkamp, Global Crossing
 Robert J. Rockell, Jeff Chaltas, Ananth Nagarajan, Sprint
 Steve Gordon, Cable and Wireless
 Jennifer Rexford, Albert Greenberg, Carsten Lund, AT&T Research
 Wai-Sum Lai, AT&T
 Fang Wu, NTT America
 Arman Maghbouleh, Alan Gous, Cariden Technologies
 Yufei Wang, VPI Systems

©Copyright 2002,

Modern Carrier Strategies for Traffic Engineering

More Related Content

What's hot

Viewers also liked

Similar to Modern Carrier Strategies for Traffic Engineering

More from Vishal Sharma, Ph.D.

Recently uploaded

Modern Carrier Strategies for Traffic Engineering

Editor's Notes