Metanoia is a technology consultancy that provides expertise in telecommunications networks. The document outlines Metanoia's background and expertise. It then summarizes a workshop on Metro Ethernet technologies that Metanoia will present. The workshop will cover legacy networks, Metro Ethernet Forum standards, Ethernet over legacy networks, native Ethernet services, and technologies like MPLS, VPNs, and resilience.

1. Metanoia, Inc.
Critical Systems Thinking™
Metro Ethernet:
Understanding Key Underlying
Technologies
Metanoia, Inc.
consultants@metanoia-inc.com
+1-888-641-0082
http://www.metanoia-inc.com
© Copyright 2007
All Rights Reserved

2. Metanoia, Inc.
Critical Systems Thinking™
Who is Metanoia, Inc.?
 Specialty technology consultancy founded in mid-2001, with HQ in Mountain View, California
 Undertakes deep-dive technical consulting in telecom network, systems, software and chip
architecture and design for clients across the world
 Services have spanned 4 continents, with clients in: North America, Europe, Asia, and Australia.
 Principals provided services in technology strategies, architecture and design trade-offs, product
development, hardware/software architecture, and knowledge enhancement to organizations that
include large equipment manufacturers, international, national and regional ISPs, premier
metro/access systems startups, network planning tool vendors, established software and technology
houses and leading component and semiconductor vendors
 Principals are technologists at the forefront of new developments, as leaders, creators,
implementers, researchers, academics, strategists, and advisors in the US and abroad
 Expertise spans Layer 1 through Layer 4, and wireline (optical, Ethernet, IP/ATM, SONET/SDH)
through wireless (Wi-Fi, cross-layer design, Wi-Max, cellular data, 2.5-3G)
 125+ man years of technology design and development, and technology management experience,
having worked at leading global corporations, such as Apple, AOL Time Warner, BBN, Cisco, 3Com,
Fujitsu, LSI Logic, Motorola, Tellabs, Siemens, Nokia, Tibco, and Qualcomm, and having worked
at/consulted to corporates in the US and abroad for almost the last decade
 70+ patents collectively issued/pending
 Advanced graduate degrees from some of the most distinguished universities in the world – the
University of California, Stanford University, Iowa State University, the University of Texas, the
University of Waterloo, and the Indian Institute of Technology
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 2

3. Metanoia, Inc.
Critical Systems Thinking™
Workshop Outline
 Legacy networks & Ethernet over legacy networks
 Value propositions and business drivers
 Ethernet over SDH/SONET
 Metro Ethernet Forum (MEF)
 MEF architecture
 E-Line and E-LAN services
 Native Ethernet as Carrier-class transport
 Provider Bridges
 Provider Backbone Bridges (PBB), Provider Backbone Transport (PBT)
 MPLS – an enabler for Ethernet services
 Layer 2 VPNs: VPWS, VPLS, H-VPLS
 Advanced concepts: traffic engineering, QoS, OAM, resilience
 Conclusions
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 3

4. Metanoia, Inc.
Critical Systems Thinking™
Ethernet over
Legacy Networks

5. Metanoia, Inc.
Critical Systems Thinking™
Issues with Legacy Networks
 Low bandwidth
 No flexibility to scale
 High cost of installation
 Slow provisioning
 Bandwidth growth inflexible/non-linear
 Limited by multiplexing hierarchy
 TDM-based access: inefficient for converged data
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 5

6. Metanoia, Inc.
Critical Systems Thinking™
Next-Generation SDH
Customer
Network
Central NG-SDH
Office NG ADM
Switch t
Ck
M
TD Ethernet
Core NG-SDH
Network Customer
NG ADM
Network
STM/4/16
Cross TD
M Ring
Connect Ck
t
NG NG-SDH
ADM
Customer
Ethernet Network
Customer
Network
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 6

7. Metanoia, Inc.
Critical Systems Thinking™
Ethernet-over-SDH
 Framing protocol
 Encapsulates Ethernet frames in SDH payloads
 Mapping of SDH payload to SDH channels
 Virtual concat.: for allocation of non-contiguous VCs
 Flow control mechanism
 Avoids packet drops due to speed mismatch between SDH and
Ethernet
 Mechanism to increase/decrease allocated SDH bandwidth
 Add or remove VCs
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 7

8. Metanoia, Inc.
Critical Systems Thinking™
Ethernet-over-SDH (contd)
 Very popular in carriers with installed base of SDH rings
 E.g. BSNL in India
 Good deployment choice when traffic primarily circuit
switched
 Inefficient if major traffic is bursty packet-switched data
 Solution: Carrier-class Ethernet!
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 8

9. Metanoia, Inc.
Critical Systems Thinking™
Metro Ethernet Value Propositions
 Lower per-user provisioning costs
 Technically simple relative to TDM ckts.
 Due to large installed base
 Efficient and flexible transport
 Wide range of speeds: 128 Kbps--10 Gbps
 QoS capabilities
 Ease of inter-working
 Plug-and-play feature
 Ubiquitous adoption
 The technology of choice in enterprise networks
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 9

10. Metanoia, Inc.
Critical Systems Thinking™
Ethernet Business Drivers
 Business connectivity
 Storage networks
 Data centers
 Video conferencing
 Residential services
 Triple-play services (IPTV)
 On-line gaming
 High-speed Internet access
 Wireless backhaul
 Reduced cost, complexity for mobile operators
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 10

11. Metanoia, Inc.
Critical Systems Thinking™
Metro Ethernet Services

12. Metanoia, Inc.
Critical Systems Thinking™
Metro Ethernet Forum (MEF)
 Industry forum at forefront of Carrier Ethernet
standardization
 Carrier Ethernet architecture
 Ethernet services
 Founded in 2001. Currently approx. 120 members
 Technical Sub-committees
 Architecture
 Services
 Protocols and Transport
 Management
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 12

13. Metanoia, Inc.
Critical Systems Thinking™
MEN Architectural Components
T T
S S
End Customer Customer End
Network
MEN Network
User User
End user Interface End user Interface
UNI Reference Point UNI Reference Point
Ethernet Virtual Connection
End-to-End Ethernet Flow
 Ethernet Flow
 Unidirectional stream of Ethernet frames
 UNI
 Interface used to interconnect MEN subscriber to provider
 EVC
 Defines association between UNI for delivering Ethernet flow across MEN
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 13

14. Metanoia, Inc.
Critical Systems Thinking™
MEN Layer Model
Application Service
Layer
(IP, MPLS, PDH, E1/E3, SDH)
Ethernet Service
Layer
Transport Service
Layer
(802.1, SONET/SDH, MPLS)
MEN Layer Model
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 14

15. Metanoia, Inc.
Critical Systems Thinking™
MEF Services Definition Framework
 Service Type
 Construct used to create broad range of services
 Service Attributes
 Defines characteristics of a service type
 Attribute Parameters
 Set of parameters with various options
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 15

16. Metanoia, Inc.
Critical Systems Thinking™
Service Types
 E-Line
EVC1
 Point-to-point Ethernet Virtual
Circuit (EVC)
EVC2
 E-LAN
 Multipoint-to-multipoint
Ethernet Virtual Circuit
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 16

17. Metanoia, Inc.
Critical Systems Thinking™
Service Attributes
 Physical Interface
 Medium, speed, mode, MAC layer
 Traffic Parameters
 CIR, CBS, PIR, MBS
 QoS Parameters
 Availability, delay, jitter, loss
 Service Multiplexing
 Multiple instances of EVCs on a given physical I/F
 Bundling
 Multiple VLAN IDs (VID) mapped to single EVC at UNI
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 17

18. Metanoia, Inc.
Critical Systems Thinking™
Ethernet Services
 Ethernet Private Line (EPL)
 Uses E-Line
 Does not allow service multiplexing
 High degree of transparency
 Low delay, delay variation, and packet loss ratio
 Ethernet Virtual Private Line (EVPL)
 Uses E-Line
 Allows for service multiplexing
 Need not provide full transparency
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 18

19. Metanoia, Inc.
Critical Systems Thinking™
Service Types and Ethernet Services
Service Types
E-Line E-LAN
(p2p connectivity) (mp2mp connectivity)
Ethernet Private Ethernet Virtual Ethernet Private Ethernet Virtual Private
Line (E-line) Private Line (E-VPL) LAN (E-LAN) LAN (E-VPLAN)
Ethernet Services
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 19

20. Metanoia, Inc.
Critical Systems Thinking™
Native Ethernet as
Carrier-class Transport

21. Metanoia, Inc.
Critical Systems Thinking™
Requirements for Carrier-class Ethernet
 Scalability
 Network should support millions of subscribers
 Protection and restoration
 50ms resilience
 Quality-of-Service (QoS)
 Ability to offer differentiated levels of service
 Service Monitoring and Fault Management
 Support for TDM traffic
 Seamless integration with legacy networks
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 21

22. Metanoia, Inc.
Critical Systems Thinking™
Ethernet Ring
Ethernet
Switch
Ethernet Ethernet
Switch Switch
Core 1/10 Gigabit Ethernet
Customer
Network Ethernet Ring Network
Ethernet
Switch
Ethernet Customer
Network
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 22

23. Metanoia, Inc.
Critical Systems Thinking™
Native Ethernet in Metro Access
 How does one create the notion of a virtual circuit?
 VLAN tagging with point-to-point VLAN
 VLAN stacking
 Outer tag ↔ service instance; Inner tag ↔ individual customer
 802.1Q in 802.1Q (Q-in-Q) - IEEE 802.1ad
6bytes 6bytes 4bytes 4bytes 4bytes
C-DA C-SA S-TAG C-TAG Client data FCS
C-DA: Customer Destination MAC
C-SA: Customer Source MAC
C-TAG: IEEE 802.1q VLAN Tag
C-FCS: Customer FCS
S-TAG: IEEE 802.1ad S-VLAN Tag
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 23

24. Metanoia, Inc.
Provider Bridge (IEEE 802.1ad) Critical Systems Thinking™
Architecture
CE-B
CES
Customer
CE-A UNI-B Network
Customer
Network
CES
UNI-A
CES
Spanning tree
UNI-C
CE-C
CE: Customer Equipment
UNI: User-to-Network Interface
Customer
CES: Core Ethernet Switch/Bridge Network
P-VLAN: Provider VLAN
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 24

25. Metanoia, Inc.
Critical Systems Thinking™
Limitations of Provider Bridge Scalability
 Limited to 4096 service instances
 Core switches must all MAC addresses
 Broadcast storms ensue due to learning
 MAC address tables explode!
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 25

26. Metanoia, Inc.
Critical Systems Thinking™
Provider Backbone Bridging (802.1ah)
 Encapsulate customer MAC with provider MAC at edge
 Edge switch adds 24-bit service tag (I-SID), not VLAN tag
 Core switches need only learn edge switch MAC adds.
6bytes 6bytes 4bytes 5bytes 6bytes 6bytes 4bytes 4bytes
B-DA B-SA B-TAG I-TAG C-DA C-SA C-TAG Client data B-FCS
S-TAG: IEEE 802.1ad S-VLAN Tag
B-DA: IEEE 802.1ah Backbone Destination
B-SA: IEEE 802.1ah Backbone Source MAC
I-TAG: IEEE 802.1ah Service Tag
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 26

27. Metanoia, Inc.
Provider Backbone Bridging (PBB) Critical Systems Thinking™
Architecture
CPE B CPE A CPE B
CPE A CPE C CPE D
Provider backbone Provider backbone
network (802.1ad) 802.1ad network (802.1ad)
Provider backbone
network (802.1ah)
Provider backbone
network (802.1ad) Provider backbone
network (802.1ad)
802.1q
CPE C CPE B
CPE B CPE A CPE D
CPE C
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 27

28. Metanoia, Inc.
Critical Systems Thinking™
Benefits of PBB
 Scalability
 Addresses limitations of 4096 service instances
 Robustness
 Isolates provider network from broadcast storms
 Security
 Provider need switch frames only on provider addresses
 Simplicity
 Provider & customers can plan networks independently
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 28

29. Metanoia, Inc.
Critical Systems Thinking™
Traffic Engineering in PBB
 Via Multiple Spanning Tree Protocol (MSTP)
 Maps a VLAN to ST or multiple VLANs to ST
 Enables use of links that would otherwise be idle in ST
 Eliminates wasted bandwidth … but …
 Too slow for protection switching
 Not suitable for complex mesh topologies
 Difficult to predict QoS
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 29

30. Metanoia, Inc.
Challenges with an All-Ethernet Critical Systems Thinking™
Metro Service
 Restriction on # of customers – 4096 VLANs!
 Service monitoring
 Scaling of Layer 2 backbone
 Service provisioning
 Carrying a VLAN is not a simple task!
 Inter-working with legacy deployments
⇒ Need hybrid architectures …
Multiple L2 domains connected via IP/MPLS backbone
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 30

31. Metanoia, Inc.
Critical Systems Thinking™
What Solutions do we Have?
 Ethernet-based Architecture
 Provider Bridge (802.1ad) in edge
 Provider Backbone Transport (PBT) in Core
 Hybrid Architecture
 802.1ad in the edge
 Multiprotocol Label Switching (MPLS) in core
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 31

32. Metanoia, Inc.
Critical Systems Thinking™
Provider Backbone Transport (PBT)
 Connection-oriented, traffic-engineered Ethernet tunnels
 Replaces spanning tree control plane with either a:
 Management plane
 External control plane
 No learning !
 Forwarding info. provided by management plane
 Forwarding done on MAC + VID (60-bit) address
 VID is not network global; however, MAC + VID is
 B-MAC identifies destination
 B-VID identifies per-destination alternate paths
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 32

33. Metanoia, Inc.
Critical Systems Thinking™
PBT Architecture
Central TE Module
PE2
PE1
Customer
Customer
Network
Network
SA : PE1 SA : PE1
DA : PE2 DA : PE2
VLAN 22 VLAN 33
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 33

34. Metanoia, Inc.
Critical Systems Thinking™
Benefits of PBT
 No learning
 Eliminates undesirable broadcast storms
 Resolves MAC flooding problem
 Addresses scaling by forwarding on MAC + VID-highly scalable
 Protection
 Sets-up backup paths
 50ms restoration possible
 QoS support available
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 34

35. Metanoia, Inc.
Critical Systems Thinking™
MPLS – An Enabler for
Ethernet Services:
Fundamentals & Operations

36. Metanoia, Inc.
Critical Systems Thinking™
Basic Concept of MPLS
DA Next hop N/w DA Next hop N/w
router Int. router Int.
129.89.10.x 198.168.7.6 1 129.89.10.x 129.89.10.1 1 Routing Table
179.69.x.x 198.168.7.6 1 179.69.x.x 179.69.42.3 2
128.89.10.x
In Out Address Prefix N/w In Out Address Prefix N/w
label label
128.89.10.1
label Int. label Int.
X 3 1 3 5 1 Label Table 2
128.89.10.x 128.89.10.x
X 4 179.69.x.x 1 4 7 179.69.x.x 2 R3
Advertises binding
1 <5, 128.89.10.x>
R1 1 R2
2
198.168.7.6
Advertises bindings Advertises binding
<3, 128.89.10.x> <7, 179.69.x.x>
<4, 179.69.x.x>
179.69.x.x
 Routing fills routing table
R4
 Signaling fills label forwarding table 179.69.42.3
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 36

37. Metanoia, Inc.
Critical Systems Thinking™
Basic Concept of MPLS
Pop
label 5
In Out Address Prefix N/w In Out Address Prefix N/w Forward
label label Int. label label Int. packet
X 3 1 3 5
5 128.89.10.x 1 5
128.89.10.x 3 128.89.10.x
X 4 179.69.x.x 1 4 7 179.69.x.x 2 128.89.10.1
2
R3
Swap
Label 5
3
1
R1 1 R2
2
3 198.168.7.6
Push
Label
Packet arrives
DA=128.89.10.25
179.69.x.x
R3 R4
179.69.42.3
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 37

38. Metanoia, Inc.
So what about MPLS Control and Critical Systems Thinking™
Forwarding?
 Superset of conventional router control
Control  Distribute info. via n/w layer routing protocols (OSPF, BGP, etc.)
Component
 Algos. to convert routing info. into forwarding table:
Create binding from FEC  label
Assign & distribute labels to peer LSRs via signaling
 Label switching forwarding table (or label information base LIB)
Incoming Label First Subentry Second Subentry
Map (for multicast or load balancing)
Outgoing label Outgoing label
Incoming
Outgoing inf. Outgoing inf.
Label Next hop address Next hop address
Forwarding
Component Next hop label forwarding entry (NHFLE)
 Forwarding algo = label swapping, independent of control
component (implementable in optimized H/W or S/W)
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 38

39. Metanoia, Inc.
What does a Label Represent? The Critical Systems Thinking™
Issue of Label Granularity
 Packets form Forwarding Equivalence Class (FEC)
 Treated identically by participating routers
 Assigned the same label
 Membership in FEC must be determinable from IP header + other info. that
ingress router has about the packet
 Entities that may be grouped into an FEC are flexible. E.g. FEC could be:
 Connection between two IP ports on two hosts or between IP hosts
 Traffic headed for a particular network with same TOS bits
 All destination networks with a certain prefix
 Manually configured connection
 Traffic belonging to a customer or department VLAN
 Traffic of a given application – voice, video, plain data, management traffic
… and many others
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 39

40. Metanoia, Inc.
Critical Systems Thinking™
Let’s Recap: Elements of MPLS
 Label Forwarding
 Use data link addressing. E.g. ATM VPI/VCI, FR DLCI
 “Shim” header between data link and IP header
Data
Plane Variable 4 bytes 20 bytes
MPLS “shim” Higher Layers
L2 header header L3 IP header
1 bit
EXP/
Label S TTL
CoS
20 bits 3 bits 8 bits
 Label Creation and Binding
Control
Plane  Label Assignment and Distribution
 Ride piggyback on routing protocols, where possible (BGP)
 Separate label distribution protocol – RSVP, LDP
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 40

41. Metanoia, Inc.
Primary Label Assignment and Critical Systems Thinking™
Distribution Modes
1 Requests
Edge LSR
2
6
5 3
4
Downstream-on-demand
with Ordered Control
Assignments Edge LSR
1 Requests
Edge LSR
2
Assignments 2’
3’ 3
4
Downstream-on-demand
with Independent Control
Edge LSR
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 41

42. Metanoia, Inc.
Critical Systems Thinking™
Advantages of MPLS
Original justification
 Availability of fast, amortized, ATM hardware; emergence of H/W
forwarding engines has practically eliminated this
Current justifications
 Separates forwarding from control, allowing
 Routing functionality to evolve independently of forwarding algorithm
 MPLS to control non-packet technologies: SONET/SDH ckts., lightpaths
 Provides explicit, manageable IP routes
 Enables policy routing and traffic engineering
 Offers TE for Ethernet tunnels in metro-Ethernet environments
 Facilitates scalable hierarchical routing
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 42

43. Metanoia, Inc.
Critical Systems Thinking™
The Utility of Hierarchical Label Switching
Edge LSRs
Swap
Swap Core LSRs
and Push Pop
Concept is similar to VLAN stacking in PBT we saw earlier
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 43

44. Metanoia, Inc.
Critical Systems Thinking™
Hierarchical Label Stacking/Switching
 Inside a transit AS, each core router must keep track of all
networks that might be reached through it
 With hierarchical labels, only edge routers need know what
networks might eventually be reached through them
 All transit traffic can be made to tunnel through core routers
using LSPs with stacked labels
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 44

45. Metanoia, Inc.
Explicit Manageable Routes -- Policy Critical Systems Thinking™
routing, Traffic engineering
 Carriers want certain traffic to go over certain routes. Such
network engineering:
 Keeps network loads balanced
 Enhances network stability and reliability
 Enables better QoS and performance assurances
 Allows carriers to meet customer SLAs
 Constraint-based routing together with MPLS allows carriers to
 Bind Ethernet tunnels to an LSP,
 Place (or route) LSP over the desired sequence of LSRs in the n/w
 TE tunnels are helpful for VPLS-based carrier Ethernet n/ws
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 45

46. Metanoia, Inc.
Critical Systems Thinking™
IP/MPLS-based Layer 2 VPNs

47. Metanoia, Inc.
Critical Systems Thinking™
L2 VPN Components
VC LSP
A A
Emulated
PE1 LAN A PE2
B Routed B
backbone
AC
Emulated
LAN B
PE3
What does the P1-PE2
connection really look like?
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 47

48. Metanoia, Inc.
Critical Systems Thinking™
L2 VPN Component Details
6 PW Signaling
PE1 PE2
From CE
devices 5 PSN Tunnel
3 PWs
Routed backbone
1 ACs 2 From CE
with P routers
Bridge devices
Module Emulated LAN
4 Forwarder
Instance
Emulated LAN
Interface
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 48

49. Metanoia, Inc.
Critical Systems Thinking™
VPLS Network Overview
PW A
LAN Service (full mesh)
VSI VSI
VSI
CE L3/MPLS
VSI
Backbone
B
B
CE
AC
A VSI Tunnel LAN Service
(full mesh)
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 49

50. Metanoia, Inc.
Critical Systems Thinking™
VPLS Protocols Involved
Control Ethernet MP-iBGP (PW) + RSVP-TE /LDP (tunnel) Ethernet
Plane STP Targeted LDP (PW) + LDP (tunnel) STP
A
BGP/Targeted LDP
PE PE
CE LSP or PSN Tunnel
B
B
CE
Ethernet Ethernet/MPLS Ethernet
Data Ethernet or Ethernet or
Ethernet/IPSec
Plane Ethernet in IP/ Ethernet/GRE Ethernet in IP/
ATM/FR/SDH/ ATM/FR/SDH/
SONET SONET
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 50

51. Metanoia, Inc.
Critical Systems Thinking™
Operational Characteristics of VPLS
Operational Requirement Realized Via
MAC address learning and
- VSI Forwarder
switching, work with 802.1p/q
- Bridge Module
tags and VLANs
Flooding pkts. with unknowns
Frame replication on PWs
broadcast, or multicast address
Provider edge signaling – inform
- Targeted LDP
PE's to autoconfigure, and of
- BGP
membership, tunnelling
- BGP
VPLS membership discovery
- Configuration
Inter-provider connectivity Globally unique VPLS ID
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 51

52. Metanoia, Inc.
Data Plane: Flooding, Address Critical Systems Thinking™
Learning and Forwarding
Src. MAC = 09:10:01:45:00:AB
1 Dest. MAC = 08:00:69:02:01:FC
3
A
VSI 2 VSI
CE ?
VSI
PWs PE2
PE1
B 2
PE3 PE4 B
A
VSI
VSI
CE
3
 All address unknown frames (unicast, multicast, broadcast)
flooded over corresponding PWs to all relevant PEs only
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 52

53. Metanoia, Inc.
Critical Systems Thinking™
Address Learning
 Layer 2 reachability directly learned in data plane
 Use standard learning bridge functions for local MACs
 PW-based association for remote MACs
 Allow PE to determine from which physical port or LSP a given MAC
address came
 VSI FIB keeps mapping between Ethernet MAC ↔ PW to use
Qualified Learning Unqualified Learning
- Each customer VLAN is its own - All customer VLANs are part of
VPLS instance the same VPLS
- Has its own PW mesh and brdcast - One PW mesh and single brdcast
domain domain
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 53

54. Metanoia, Inc.
Critical Systems Thinking™
Address Learning Example
Src. MAC = 08:AA:FC:01:10:DE (S1)
2
Dest. MAC = FF:FF:FF:FF:FF:FF (D1)
(broadcast) 4
1 VSI A
Inbound
CE
VC LSP Label = 1002
i/f1 i/f2
i/f1
VSI PE1 PE2
3 Outbound
Local Learning VC LSP Label = 2001 Dest. VC
Tunnel Out I/F
MAC Label
S1 1002 - i/f1
PE3 Remote
Learning
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 54

55. Metanoia, Inc.
Critical Systems Thinking™
Forwarding and Encapsulation
Forwarding requires ability to
 Dynamically learn MAC addresses on
 Physical ports
 Pseudowire VCs (VC LSPs)
 Forward/replicate pkts. across physical ports and VC LSPs
Encapsulation
 PW header applied to Ethernet packet w/o preamble + FCS
 VLAN tag denoting customer’s VPLS instance can be stripped at
ingress, reapplied at egress
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 55

56. Metanoia, Inc.
Tunnel and PW Topology and Critical Systems Thinking™
Loop Freedom
Dest. MAC = 08:00:69:02:01:FC PW A
? (full mesh)
VSI
VSI
PE1 PE2
VSI
CE VSI
B
AC CE
A
Tunnel
(full mesh)
VSI PE3 PE4
 Full mesh of PW and tunnels deployed
 Tunnels
 Help transport the PW payload
 Aggregate traffic from multiple PWs
 Pseudowires – demultiplex the L2 traffic traversing tunnels
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 56

57. Metanoia, Inc.
Critical Systems Thinking™
Scaling VPLS: Hierarchical VPLS
 Base VPLS requires full mesh of VC LSPs between PE routers
 Adequate for PE routers in CO – multiple customers aggregated
 Inadequate for PE routers in MTU basements!
MTU MTU
PE PE
MTU MTU
PE PE
LSP explosion
Operational nightmare!
PE
MTU
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 57

58. Metanoia, Inc.
Critical Systems Thinking™
Hierarchical VPLS Advantages
MTU MTU
PE PE
Hub PE
MTU Core VC MTU
LSP mesh
PE Spoke PE
VCs
(VLL or Q-in-Q)
Benefits
 Simplifies signaling
 Reduces pkt. replication
PE
MTU  Simplifies MTU
 Scalable inter-domain VPLS
 Simplifies new site addition
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 58

59. Metanoia, Inc.
Hierarchical VPLS: Case Study for Critical Systems Thinking™
a Metro Region
100 MTUs; 10 customers/MTU; 2 VPLS/cust.; 100 stations/VPLS
VPLSs/MTU = 10x2 = 20
MTU100 MTU91
MACs/MTU = 20x100 = 2000 CE CE
MTU1 MTU 100
PE PE MTU1
Hub PE MTU90
CE
CE
MTU2 MTU99 MTU10 PE PE MTU81
PE PE CE
CE
PE
PE PE CE CE
MTU3 MTU40 MTU31 MTU40
No hierarchy ⇒ PE supports Hierarchy (10 MTU/PE) ⇒ PE
2000 MACs supports
LDP/BGP sessions = (100x99)/2 x 2000 x 10 = 20,000 MACs
20 = 245,000 LDP/BGP sessions = (10x9)/2 x 200 = 9000
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved
# of spoke VLLs = 10 x 20 = 200
2007, Bangalore, India 59

60. Metanoia, Inc.
Critical Systems Thinking™
Benefits of IP/MPLS-based L2 VPNs
 Separation of administrative responsibilities
 Migration from traditional L2 VPNs: seamless transport of Ethernet
services
 Privacy of routing
 Layer 3 independence
 Less operational overhead
 Ease of configuration (?)
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 60

61. Metanoia, Inc.
Critical Systems Thinking™
Advanced Features:
Traffic Engineering,
Resilience, OAM, QoS

62. Metanoia, Inc.
Critical Systems Thinking™
Traffic Engineering Concepts
© Copyright 2006
All Rights Reserved

63. Metanoia, Inc.
Critical Systems Thinking™
Constraint Based Routing
 A class of routing systems that computes routes through a
network subject to a set of constraints and requirements
QoS-based Routing Policy-based Routing
 Path of flows determined by  Path/routing decision based
 Knowledge of resource on administrative policy
availability in network
 QoS requirements of flows
 Can be on-line or off-line
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 63

64. Metanoia, Inc.
Critical Systems Thinking™
CB Routing System
 Inputs
Resources
 Flow/path attributes:
required b/w, hop count, ...
 Resource attributes: Attributes
Topology
properties of nodes/links
 Network topology & state
Constraint-Based
Routing Process
 Outputs
 Computed feasible path Feasible Path
ERO {1,3,4,5}
 Explicit route of the path 3
5
1
4
2
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 64

65. Metanoia, Inc.
Critical Systems Thinking™
MPLS-based Resilience for the Metro
© Copyright 2006
All Rights Reserved

66. Metanoia, Inc.
Critical Systems Thinking™
Fundamental Characteristics of RSVP
 Allows apps. to signal QoS requests to n/w, and n/w to respond
with success or failure
 Designed to transport
 Classification info. (Sender_Template)
 Allows flows with specific QoS reqs. to be recognized
 Traffic specs of source/sender (Tspec)
 QoS needs of receivers (Rspec)
 Soft-state protocol
 Path/Resv transmitted periodically to refresh reservation
 Refresh Reduction [RFC2961] has practically eliminated original
scalability concerns with use of soft state
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 66

67. Metanoia, Inc.
Critical Systems Thinking™
Basic Operation of RSVP-TE
Path (Label_Req) Path (Label_Req)
A B C D E
Resv Resv Resv Resv
Label=21 Label=49 Label=7 Label=5
Path Message Resv Message
RSVP Header RSVP Header
Application for which RSVP
SESSION Same as that in Path Msg.
reservation is to be made SESSION
Identifies pkts. of the sender Specifies senders that may
SENDER_TEMPLATE STYLE use the reserved resources
SENDER_TSPEC Defines traffic output by sender LABEL Label assigned to this hop
LABEL_REQUEST Request for label on this hop RRO Record route taken by Path
Specific path to which flow is
ERO/RRO RSpec QoS desired by receiver
to be bound
SESSION_ATTRIBUTE Flow for which QoS is
LSP attributes for this sender SENDER_TEMPLATE
desired
PHOP IP address of I/F that NHOP IP address of I/F originating
transmitted Path Msg. the Resv msg.
Flow Descriptor
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 67

68. Metanoia, Inc.
Fast Re-Route (FRR) using Critical Systems Thinking™
RSVP-TE
 Rerouting is done when
 A better path is available Originates LSPs
with IDs 1 and 2
 Upon failure along LSP Src
Here they are treated as different
LSPs within the same Session
 Use SESSION Obj. & SE style Rcvr
Tunnel ID in
LSP ID = L1 Session Obj
 Tunnel uniquely identified by
 Destination IP address
 Tunnel ID
 Ingress IP address
 Tunnel ingress made to appear LSP ID = L2 On these links the
as 2 different senders to the LSPs share resources
RSVP session (via LSP ID) LSPs 1 and 2 have a common SESSION Obj, but
a new LSP ID in the SENDER_TEMPLATE and a
different ERO (with possibly common hops)
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 68

69. Metanoia, Inc.
TE with Constraint-based Routing Critical Systems Thinking™
in a Nutshell
Operator Input Route Computation Resource Enhanced IGP
(Flow or LSP Process TED Attributes Process
Attributes) (on-line (CSPF) or offline) (OSPF-TE)
Network
Output Topology + State
Routing Table
Computed
(RIB)
Demand or Traffic driven feasible path
(ERO) Control driven route computation
LSP path selection
and LSP path selection
Link State
Signaling Process Database Standard IGP
(RSVP-TE) (LSDB) Process (OSPF)
CONTROL PLANE
DATA PLANE
LSP
Establishment Link Attribute
Modification
MPLS LSPs
(Label Info. Base) Forwarding
Info. Base (FIB)
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 69

70. Metanoia, Inc.
Critical Systems Thinking™
How it All Fits Together
Last-mile Ethernet
PBB clouds
CE3
LSP Tunnels
CE1
PE1 PE3 CE4
Pseudo-wires
PE2
IP/MPLS Core
CE2
Attachment circuits
-- Physical (PDH/SDN)
-- Logical (FR, ATM, VLANs, tunnels)
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 70

71. Metanoia, Inc.
Critical Systems Thinking™
OAM: The Traditional Achilles Heel of
Ethernet
© Copyright 2006
All Rights Reserved

72. Metanoia, Inc.
Critical Systems Thinking™
Why Ethernet OAM?
 Current management protocols lack per-customer
granularity to handle Ethernet services
 Most management protocols operate are point-to-point
 Ethernet OAM can exploit multipoint capability
 Link management required for last-mile connection
 Similar to link mgt. in FR and ATM
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 72

73. Metanoia, Inc.
Critical Systems Thinking™
Ethernet OAM Types
 Service OAM
 e2e connectivity and fault mgt. per service instance
 Part of IEEE 802.1ag, CFM project
 Link OAM
 Monitoring & fault mgt of individual Ethernet link (physical/emulated)
 Part of IEEE 802.3, Clause 57 (formerly 802.3ah (not to be confused
with 802.1ah))
 Ethernet Local Mgt. Interface (E-LMI)
 Configuration & operational provisioning of customer edge device
 Part of MEF Standard MEF-16
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 73

74. Metanoia, Inc.
Critical Systems Thinking™
Service OAM
 Works on per-EVC basis
 Independent of underlying transport technology
 CFM messages
 Continuity Check Message
 Detects loss of service connectivity
 Link Trace Message
 Traces the path hop-by-hop (like IP traceroute)
 Loopback Message
 Detects whether target point is reachable (like ICMP Ping)
 AIS (Alarm Indication Signal) Message
 Asynchronous notification to indicate fault
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 74

75. Metanoia, Inc.
Critical Systems Thinking™
Link OAM
 Discovery
 Identifies devices at both ends of the link
 Link Monitoring
 Detects link faults
 Statistics of packet errors
 Remote Failure Indication
 Conveys loss-of-signal indication to peers, due to poor SNR, power
failure, or other critical events
 Remote Loopback
 Determines quality of link during installation and troubleshooting
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 75

76. Metanoia, Inc.
Critical Systems Thinking™
E-LMI
 Provides local configuration & operational parameters to
customer edge
 VLAN-EVC mapping
 QoS profiles of EVC
 Reduces configuration errors, improves performance
 Dynamic EVC management
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 76

77. Metanoia, Inc.
Critical Systems Thinking™
Quality-of-Service: Ah! that elusive QoS
© Copyright 2006
All Rights Reserved

78. Metanoia, Inc.
MPLS and Quality-of-Service for Critical Systems Thinking™
Ethernet Services
 MPLS supports (not extends) a packet-based QoS model
 MPLS does not run in hosts (only in metro/core routers)
 QoS, however, is an end-to-end mechanism
 MPLS helps carriers offer QoS-enabled services efficiently
 Can support MEF QoS model via DiffServ QoS framework
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 78

79. Metanoia, Inc.
Critical Systems Thinking™
Differentiated Services Framework
 Traffic flows aggregated into small # of classes
Drop Precedence Class Priority DSCP
EF 101110
 Per-flow state is not required 001xx0
AF1x
AF2x 01xx10
AF3x 11xx10
 More scalable than IntServ
AF4x 1xxx10
3 2 1
BE
 Class encoded in IP header via
 Best Effort (BE)
DiffServ Code Point (DSCP)
 Expedited Forwarding (EF)
 Minimal delay & loss
 Edge router …
 Assured Forwarding (AF)
 Classifies packets to DifServ classes
 4 classes
 3 drop precedence’s each
 DSCP identifies Per Hop Behavior (PHB) Workshop, 17
Next-Generation Systems & Networks th
 12 possibilities total
July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 79

80. Metanoia, Inc.
Critical Systems Thinking™
Differentiated Services Architecture
Diffserv Domain
Core Functions
Edge Functions
EF
Traffic Conditioning
Colored packet Strict
Meter (marked DSCP) Priority
Aggregate
AF PHBs
Classifier Marker Shaper Scheduling
BE
WFQ
Queueing
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 80

81. Metanoia, Inc.
MPLS Support of DiffServ: Critical Systems Thinking™
Mapping DSCPs to LSPs (or labels)
 Map DSCP  EXP bits in MPLS “shim” header
 6 DS bits (64 PHBs) and only 3 EXP bits (8 classes)!
 Complete mapping is infeasible
 For many practical cases, 8 PHBs may suffice
IP Header MPLS “shim” header
6 bits
DSCP
DSCP Label EXP S TTL
DS byte 3 bits
Results in an LSP called an E-LSP
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 81

82. Metanoia, Inc.
MPLS Support of DiffServ: Critical Systems Thinking™
Mapping DSCPs to LSPs (or labels)
 Map {PHB, FEC}  MPLS Label
 That is, provide the info. in the label itself!
 Requires enhancing the label distribution protocols
 Use EXP bits for drop precedence
 That is to determine different PHBs of a PHB scheduling class
DS class drop
precedence
6 bits DS class: EF, AFx
DSCP
DSCP Label EXP S TTL
DS byte 3 bits
IP Header MPLS “shim” header
Results in an LSP called an L-LSP
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 82

83. Metanoia, Inc.
Critical Systems Thinking™
Conclusions and Discussion

84. Metanoia, Inc.
Critical Systems Thinking™
Conclusions
 Ethernet poised to be dominant choice in metro networks
 Reduces capex and opex for providers
 Enables new revenue generating services
 802.1ad provider bridge with OAM of 802.1ag …
 … a choice at the edge
 Two architectures emerging for Ethernet in the metro core
 Provider Backbone Transport (PBT)
 IP/MPLS-based L2 VPNs
Next-Generation Systems & Networks Workshop, 17th July.
©Copyright 2007
All Rights Reserved 2007, Bangalore, India 84

85. Metanoia, Inc.
Critical Systems Thinking™
Thank You!
Questions?