PLNOG 9: Emil Gągała - Fast Service Restoration

Fast Service Restoration
High Availability for 2547 VPN Service
Emil Gągała
JNCIE
PLNOG, Kraków, 21.10.2012

ACKLOWLEDGEMENTS
Many thanks to Yakov Rekhter, Hannes Gredler for their
contributions to the development of this technology
Special thanks to Yimin Shen, Minto Jeyananth & Wen Lin who
are driving the technical details in JNPR and protocol drafts in
2 Copyright © 2011 Juniper Networks, Inc. www.juniper.net
are driving the technical details in JNPR and protocol drafts in
IETF.

Agenda
Background
Improving L3VPN convergence
Tail-end protection
Solution details
CLI example
Summary

High Availability Quiz
99.999% availability means:
A)15.36 minutes downtime in year
B) 5.26 minutes downtime in year
C) 2.53 minutes downtime in year
Non Stop Routing needs support on neighboring routers
Non Stop Routing needs support on neighboring routers
True or false?
It is possible to achieve with LDP FRR behavior
True or false?

MPLS BACKGROUND

The Purple Line – MPLS as a transport for all services
VoIP
Internet
(search, e-
commerce,
advertising,
video, IM,
“over-the-top”
…)
Ethernet,ATM,FR
PWs(VPLS/VPWS)
VoIPPeering
IPVPNs
IPTV/VoD
DTV
IMS
(services
delivered
to IP-
enabled
mobile
handsets)
Privateservices
LeasedLines,Frame
RelayATM,POTS
IP Services Plane
MPLS Data Plane (P2P, P2MP, MP2P, MP2MP)
Ethernet Framing
DWDM
Fiber
VoIP
Infrastructure Control Plane
Ethernet,ATM,FR
PWs(VPLS/VPWS)
VoIPPeering
IPVPNs
IPTV/VoD
DTV
OTN SW
Privateservices
LeasedLines,Frame
RelayATM,POTS
OTN Muxing (G.709, FEC, OAM)
SERVICES
TRANSPORT

MPLS AS A TRANSPORT
• Unified transport plane for services
• Well tested fast restoration (FRR, LFA)
• Ease of service placement (with Seamless MPLS)
• Nice scaling characteristics

MPLS FOR SERVICES
• Purple line is moving up
• MPLS is a transport layer for services
• And a lot of services are MPLS-based
• Virtual networks using BGP VPNs
• Circuit transport using BGP PWs and LDP PWs
• Mobile backhaul using PWs
• IPTV using MPLS Multicast
• But the service layer is fragile…
• Failure restoration of MPLS services is still not 50ms
• Service layer needs to be robust to move the purple line

Securing the Edges
• Protecting L3VPN services
• Protecting LDP PW services
• Protecting BGP PW services
• Protecting VPLS
• Protecting Hosts
• Summary
PE1 PE2
L3VPN Cloud
• Summary
PE2
PE4
PE3
PE1
VPN A/Site 2
VPN A/Site 1
PLR
Host-BHost-A

IMPROVING L3VPN CONVERGENCE

Goals:
High service availability
2547 VPN as the service
both IPv4 and IPv6 2547 VPN service
Service disruption time less than 50 msec
in the presence of failures within the service provider infrastructure

Core failures (e.g., PE-P link, P-P link, P router)
Existing MPLS FRR link/node local protection mechanisms allows to provide
sub-50msec connectivity recovery
Ingress PE router failure, ingress CE-PE link failure
CE detects primary PE router failure (or CE-PE link failure)
Could be accomplished using L2 OAM or BFD between CE and (ingress) PE router
CE re-routes traffic towards the backup (ingress) PE router – local protection
Allows to provide sub-50msec connectivity recovery
2547 VPN Service Failures Decomposition
Egress PE-CE link failure
Egress PE detects PE-CE link failure
Could be accomplished using L2 OAM or BFD between (egress) PE and CE
Egress PE re-routes traffic towards the backup (egress) PE router – local
protection
Allows to provide sub-50msec connectivity recovery
Egress PE router failure
Not covered by the existing MPLS FRR local protection schemes
Sub-50msec connectivity recovery using local protection is the focus
of this presentation

Digression: global vs local protection for egress PE
failure (1)
P router adjacent to (egress) PE detects PE
failure, and advertises it into IGP (ISIS/OSPF)
IGP (ISIS/OSPF) is used to propagate failure
notification to other (ingress) PEs
Using OSPF/ISIS flooding procedures
Other (ingress) PEs adjust their forwarding
tables, once they receive the failure
notification via ISIS/OSPF
P router adjacent to (egress) PE detects PE
failure
P router adjacent to PE adjusts its forwarding
table
P router becomes Point of Local Repair (PLR)
At this point connectivity is restored
Connectivity recovery does not depend on
propagating failure notification in ISIS/OSPF
Global Protection
(using IGP to propagate failure notification)
Local Protection
At this point connectivity is restored
Connectivity recovery depends on
Connectivity recovery time can not be less
than the time it takes to propagate and
process failure notification in ISIS/OSPF
Propagation time involves ISIS/OSPF control
plane processing delay on all the intermediate
nodes (several control plane hops)
Several 100s of msec
Connectivity recovery time is dependent of
(OSPF/ISIS) routing convergence speed
Connectivity recovery time does not depend
on ISIS/OSPF propagating and processing
failure notification all the way to the ingress
PEs
Connectivity recovery time can be comparable
to the time it takes for PLR to detect PE failure
50 msec
Connectivity recovery time is independent
of routing convergence speed

Digression: global vs local protection for egress PE
failure (2)
Local protection is the fastest and the most scalable way to
provide connectivity recovery
Restoring connectivity does not require propagating any control
plane information from PLR to other nodes
Connectivity recovery time is independent of routing convergence
speed
Actions/changes required to restore connectivity upon failure
detection are fully localized to the router closest to the failure
detection are fully localized to the router closest to the failure
The router that detects the failure becomes Point of Local Repair
(PLR)
Enables connectivity recovery time under 50 msec
That is precisely why we focus on local protection as a
way to achieve high service availability

Local vs. Global repair
link break, local-repair start
Local-repair complements Global-repair
Local-repair keeps traffic flowing while
Global-repair gets things right
Variation of “Make before break”
local repair stop
global repair stop
20 - 40ms
150 – 800 ms
global repair start

2547 VPN Operations (Background)
VPN A/Site 1
VPN B/Site 1
PE2
PE4
PE1
VPN B/Site 2
(multi-homed to
PE1 and PE2)
VPN B/Site 3
10.2/16
P1
BGP: RD1, 10.2/16, RT-B, Next-Hop=PE1, Label 70
BGP: RD2, 10.2/16, RT-B, Next-Hop=PE2, Label 50
10.2.2.210.2.2.2
P2
P3
•• P routers maintain no VPN state (VPN
state is present only on PEs)
• P routers maintain state only for inter-
PE tunnels/LSPs (e.g., T1, T2) 10.2.2.2
IBGPIBGP
IBGPIBGP
T2|50|10.2.2.2T2|50|10.2.2.2
10.2.1.1
10.2.1.110.2.1.1
10.2.2.210.2.2.2
•• PEs connected to a multi-homed site of a given VPN use different RDs (but the same RT)
when originating VPN-IP routes for the destinations within the site
• Results in several VPN-IP routes with the same IP prefix, same RTs, but different RDs and
Next-Hop.
• PEs connected to other sites of that VPN import all these routes, creating Equal Cost
Multi-Path (ECMP) for the destinations within the multi-homed site
VPN B/Site 1
PE5 VRF-B::
PE3
PE1
PE5
10.2/16
VPN A/Site 2
(multi-homed to PE1
and PE3)
Dest: 10.2/16, Tunnel T1 (PE1), Label 70
Dest: 10.2/16, Tunnel T2 (PE2), Label 50
10.2.1.110.2.1.1
10.2.2.210.2.2.2
RouteRoute
ReflectorReflector
IBGPIBGP
T1|70|10.2.1.1T1|70|10.2.1.1

TAIL-END PROTECTION

“Big picture” (1)
VPN A/Site 2
(multi-homed to
PE1 and PE3)
PE2
PE4
PE3
PE1
VPN A/Site 1
VPN B/Site 1
VPN B/Site 2
(multi-homed to
PE1 and PE2)
VPN B/Site 3
10.2/16
10.2/16
10.2.1.1
10.2.2.3
PLR
InterInter--PE LSPPE LSP
Goal: In the presence of PE1 failure provide
50 msec connectivity recovery time for traffic
from VPN B/Site 1 to VPN B/Site 2 that used
to go via PE1
How: by using local protection - penultimate
hop P router acts as PLR and re-routes this
traffic via PE2
Goal: In the presence of PE1 failure provide
50 msec connectivity recovery time for traffic
from VPN A/Site 1 to VPN A/Site 2 that used
to go via PE1
How: by using local protection - penultimate
hop P router acts as PLR and re-routes this
traffic via PE3
PLR can not accomplish this on its own, as doing this would require VPN-related
state on PLR, yet PLR (being P router) does not maintain any VPN-related state
PE1 and PE3)10.2/16

PROBLEMS TO BE SOLVED
• PLR has label state only for
transport LSP
• PLR needs to divert the transport
#1 Point of Local Repair (PLR) has no
label state for service routes
• The backup node has to know all
incoming-label -> FEC mappings
advertised by the protected node
#2 The backup node has to correctly
interpret labels used by the service
LSPs
• PLR needs to divert the transport
LSP to some other node
• As a result, all Service LSPs carried
over the outer LSP will be re-routed
to that other node as well
advertised by the protected node
for all the service LSPs
• The backup node should use this
mapping for the forwarding of
service LSPs

PE2
PE4
PE3
PE1
VPN A/Site 1
VPN B/Site 2
VPN B/Site 3
10.2/16
10.2.1.1
PLR
Protector
“Big picture” (2) – introducing Protector
Make PLR re-route to Protector the traffic that used to go via PE1
Make Protector maintain VPN routes for VPN A and VPN B
This way Protector will re-route via PE3 traffic from VPN A/Site 1 to VPN A/Site 2,
and via PE2 traffic from VPN B/Site 1 to VPN B/Site 2
PE3
VPN A/Site 2
VPN B/Site 1
10.2/16
10.2.2.3

PE2
PE4
PE3
PE1
VPN A/Site 1
VPN B/Site 2
VPN B/Site 3
10.2/16
10.2.1.1
PLR
Protector
STEP 1: PLR DETECTS (EGRESS) PE FAILURE
Step 1: PLR detects (egress) PE failureStep 1: PLR detects (egress) PE failure
E.g., PLR detects PE1 failure
Could be accomplished using L2 OAM or BFD between PLR and (egress) PE
Further details are outside the scope of this presentation
PE3
VPN A/Site 2
VPN B/Site 1
10.2/16
10.2.2.3

PE2
PE4
PE3
PE1
VPN A/Site 1
VPN B/Site 2
VPN B/Site 3
10.2/16
10.2.1.1
PLR
Protector
STEP 2: PLR REDIRECTS TRAFFIC TO PROTECTOR
Step 2: PLR redirects to Protector the traffic that used to goStep 2: PLR redirects to Protector the traffic that used to go
(via PLR) to the (failed) PE(via PLR) to the (failed) PE
E.g., PLR sends to Protector traffic that used to go via PLR to PE1:
from VPN A/Site 1 to VPN A/Site 2, from VPN B/Site 1 to VPN B/Site 2
More details later…
PE3
VPN A/Site 2
VPN B/Site 1
10.2/16
10.2.2.3

PE2
PE4
PE3
PE1
VPN A/Site 1
VPN B/Site 2
VPN B/Site 3
10.2/16
10.2.1.1
PLR
Protector
STEP 3: PROTECTOR FORWARDS TRAFFIC TO
APPROPRIATE (EGRESS) PE
Step 3: Protector forwards the traffic received from PLR to theStep 3: Protector forwards the traffic received from PLR to the
appropriate other (egress) PEsappropriate other (egress) PEs
E.g., Protector sends via PE3 traffic from VPN A/Site 1 to VPN A/Site 2
E.g., Protector sends via PE2 traffic from VPN B/Site 1 to VPN B/Site 2
More details later…
PE3
VPN A/Site 2
VPN B/Site 1
10.2/16
10.2.2.3

SOLUTION DETAILS

Step 2: PLR redirecting traffic to Protector – How ? (1)
VPN A/Site 2
PE2
PE4
PE3
PE1
VPN A/Site 1
VPN B/Site 1
VPN B/Site 2
(multi-homed to
PE1 and PE2)
10.2/16
10.2.1.1
PLR
to 10.0.0.1to 10.0.0.1
BGP: RD3, 10.2/16, RT-A, Next-Hop=10.0.0.1, Label 60
BGP: RD2, 10.2/16, RT-B, Next-Hop=10.0.0.1 Label 70
Context Identifier 10.0.0.1
On Protected PE (PE1):
Configure (additional) IP address – identifies forwarding context (PE) that
has to be protected – “Context Identifier”“Context Identifier”
Advertise this Context Identifier into OSPF/IS-IS with small metric (e.g., 1)
Use this Context Identifier as BGP Next-Hop for VPN-IP routes originated by
Protected PE (PE1)
Creates association between Context Identifier and a set of routes to be protected
Inter-PE transport LSP used by these routes is associated with Context Identifier
VPN A/Site 2
(multi-homed to
PE1 and PE3)
PE3VPN B/Site 1
10.2/16
10.2.2.3
to 10.0.0.1to 10.0.0.1
OSPF: 10.0.0.1
metric 1

PE2
PE4
PE1
VPN A/Site 1 VPN B/Site 2
(multi-homed to
PE1 and PE2)
10.2/16
10.2.1.1
Protector
OSPF: 10.0.0.1
metric 2^24
On Protector:
Configure IP address that is used as Context Identifier on Protected PE
Creates coupling between Protected PE and Protector
Advertise this address into OSPF/IS-IS with large metric (e.g., 2^24)
VPN A/Site 2
(multi-homed to
PE1 and PE3)
PE3VPN B/Site 1
10.2/16
10.2.2.3
PLR
to 10.0.0.1to 10.0.0.1
OSPF: 10.0.0.1
metric 1

PE2
PE4
PE1
(multi-homed to
PE1 and PE2)
10.2/16
10.2.1.1
PLR
Protector
OSPF: 10.0.0.1
metric 2^24
Bypass LSP
to 10.0.0.1
On PLR use MPLS FRR procedures to create a Bypass LSP from PLR to Protector
Bypass LSP terminates on Protector
Basic LFA FRR may not be sufficient (except for particular network topology cases)
setting up Bypass LSP is likely to require RSVP-TE
Direct consequence of the inability of basic LFA FRR to provide full coverage use
RSVP-TE LSP to extend coverage
VPN A/Site 2
(multi-homed to
PE1 and PE3)
PE3VPN B/Site 1
10.2/16
10.2.2.3
PLR
to 10.0.0.1to 10.0.0.1
OSPF: 10.0.0.1
metric 1

PE2
PE4
PE1
(multi-homed to
PE1 and PE2)
10.2/16
10.2.1.1
Protector
OSPF: 10.0.0.1
metric 2^24
Bypass LSP to
10.0.0.1 (T-P)
Stitching inter-PE LSP
When PLR detects PE1 failure, PLR “stitches” inter-PE LSP and Bypass LSP
Using MPLS FRR procedures
Results in sending to Protector the traffic that used to go via PLR to PE1:
E.g., from VPN A/Site 1 to VPN A/Site 2
E.g., from VPN B/Site 1 to VPN B/Site 2
VPN A/Site 2
(multi-homed to
PE1 and PE3)
PE3VPN B/Site 1
10.2/16
10.2.2.3
PLR
to 10.0.0.1 (T1)to 10.0.0.1 (T1)
OSPF: 10.0.0.1
metric 1
Stitching inter-PE LSP
and Bypass LSP

Step 3: Protector Forwarding – How ?
PE2
PE4
PE1
VPN A/Site 1
VPN B/Site 2
(multi-homed to
PE1 and PE2)
10.2/16
10.2.1.1
Protector
(protects PE1)
BGP: RD1, 10.2/16, RT-B, Next-Hop=10.0.0.2, Label 50
BGP: RD2, 10.2/16, RT-B, Next-Hop=10.0.0.1, Label 70
bgp.l3vpn:
From PE1 (Protected Route):
RD2 10.2/16, RT-B, Label 70, N-H 10.0.0.1
From PE2 (Backup Route):
RD1 10.2/16, RT-B, Label 50, N-H 10.0.0.2
T-P|70|10.2.1.1
T2|50|10.2.1.1
_10.0.0.1_.mpls.0:
70 swap to 50, push T2
VPN A/Site 2
(multi-homed to
PE1 and PE3)
PE3
VPN A/Site 1
VPN B/Site 1
10.2/16
10.2.2.3
PLRT1|70|10.2.1.1
Put L3VPN routes whose BGP Next Hop matches the context identifier for which we are protector into bgp.l3vpn
E.g., RD2 10.2/16, RT-B, Label 70, N-H 10.0.0.1
Identify matching backup routes and put them into bgp.l3vpn:
Exact matching Route Target
Exact matching IP Prefix part of VPN-IP NLRI (not RD, as RDs may be different)
E.g., RD1 10.2/16, RT-B, Label 50, N-H 10.0.0.2 is backup for RD2 10.2/16, RT-B, Label 70, N-H 10.0.0.1
Splice MPLS label information from bgp.l3vpn matching routes into LFIB ( __context__.mpls.0)
On Protector:On Protector:

CLI EXAMPLE

PREPARING PROTECTION FOR PRIMARY PE
Primary PE cli changes
[edit protocols BGP]
protocols {
replace:
bgp {
group INTERNAL {
type internal;
local-address 192.168.53.102;
family inet-vpn {
Egress-protection stanza allows BGP to
rewrite protocol nh to 10.10.10.10 for family
inet-vpn (afi1/safi128).
This feature may be used for many future
capabilities, e.g. iso-vpn, vpls, labeled-unicast
for interprovider-vpn’s.
Hence the need to define the egress-
protection per bgp-family.
Once configured under [edit protocols bgp
group <> family inet-vpn unicast], it acts
as GLOBAL setting for all VRF’s and gets
family inet-vpn {
unicast {
egress-protection {
context-identifier {
10.10.10.10;
}
}
}
}
neighbor 192.168.53.104;
}
}
}
as GLOBAL setting for all VRF’s and gets
inherited to the local configured VRF’s. This
can be overwritten, see next slides [edit
routing-instances C1 egress-protection]
stanza
If egress-protection is configured on the
vrf-level, then it is NOT required to have it
configured under [edit protocols bgp group <>
family inet-vpn unicast]

PRIMARY PE MUST ADVERTISE CONTEXT-ID INTO LDP/IGP
TO ALLOW OTHER PE’S RESOLVE THE PROTOCOL NH
protocols {
replace:
mpls {
interface all;
interface fxp0.0 {
disable;
}
egress-protection {
context-identifier 10.10.10.10 {
primary;
}
As the primary PE will have all its
mpBGP-updates send with a protocol
nh of 10.10.10.10, this context-identifier
must be reachable by other PE’s.
The egress-protection knob under
[edit protocols mpls] enforces
advertisement of 10.10.10.10 into IGP
}
}
advertisement of 10.10.10.10 into IGP
and LDP (same is needed on the
protector)

PROTECTOR – ATTRACTING TRAFFIC
protocols {
replace:
mpls {
interface all;
interface fxp0.0 {
disable;
}
egress-protection {
protector;
}
The protector can protect multiple
context-id’s
This stanza lets the protector advertise
metric 2000;
protector;
}
}
}
}
This stanza lets the protector advertise
the context-identifier into LDP and ISIS
with a default of max-metric -1 to attract
traffic in case the primary PE fails.
Metric is configurable, see snippet.
See next slide for results

PROTECTOR
LABEL-MIRRORING AND SWAP-TABLES
Defining the route-targets to listen on. Any route-updates for the given
VRF are now being processed by the Protector.
- Protector learns the prefixes (and VPN-lables) advertised by Primary PE
and the backup PE
- Protector learns if any backup-PE exists offering the same prefixes with different RD.
- Protector learns as well VPN-labels as advertised from Backup PE
- Protector learns as well VPN-labels as advertised from Backup PE
- As result of learning VPN labels from backup PE and primary PE, the protector can now
populate the mpls swap tables
- Config next slide

CREATING THE PROTECTOR
protocols {
bgp {
group internal {
replace:
family inet-vpn {
unicast {
replace:
policy-options {
policy-statement LB {
term 1 {
then {
load-balance per-packet;
}
}
Enabling the VPN-label mirroring for given
vrf-targets
unicast {
egress-protection {
keep-import PROTECTOR-COMMUNITY;
}
}
}
}
}
}
}
}
policy-statement PROTECTOR-COMMUNITY {
term a {
from community [ COMM_1 COMM_2 ];
then accept;
}
}
community COMM_1 members target:100:1;
community COMM_2 members target:100:2;
}

HOW DOES PROTECTOR IDENTIFY & FORWARD
TRAFFIC TO CORRECT BACKUP PE?
mpls.0
_10.10.10.10_.mpls.0
Incoming packet
arriving on bypass
from PLR
Pop bypass
label
Swap primary PE
VPN label with
backup PE VPN label
Push Backup PE
tunnel label
Outgoing
packetParticular Forwarding
Protector
Context within Protector
Bypass LSP identifies the PE being protected
Based on bypass LSP label, protector PE knows to lookup packet in a special MPLS table
MPLS context table identifies the egress PE the protector is protecting
Based on VPN label, protector identifies VPN
Protector subsequently sends packet to backup PE, using VPN label advertised by backup PE

ROUTE DETAILS ON PROTECTOR
protector> show route table mpls.0 label 300144
300144(S=0) *[MPLS/0] 00:34:50 > to table __10.10.10.10__.mpls.0
⇒ LSP label 300144 points to context table, identifying Primary PE
protector> show route table __10.10.10.10__.mpls.0 label 45
45 *[Egress-Protection/170] 2d 08:09:19 > Swap 80, Push 300500
⇒Primary PE VPN label (45) being swapped with Backup PE VPN label (80)
⇒Traffic tunneled over transport LSP (label 300500) to backup PE
⇒Traffic tunneled over transport LSP (label 300500) to backup PE

SUMMARY

SUMMARY END-POINT PROTECTION PE FAILURE
Junos tail-end-protection allows FRR/LFA rerouting in case a primary
egress PE fails
<50ms recovery time
As recovery is quick, there is no urgent need to speedup global
convergence
Tailend-protection simply reroutes to a protector.
Protector swaps VPN labels and forwards to a applicable backup-PE
Protector swaps VPN labels and forwards to a applicable backup-PE

STATUS
Standardization
draft-minto-2547-egress-node-fast-protection

Implications on the overall connectivity recovery time
for 2547 VPN service: system-wide perspective
Ingress PE failure, ingress CE-PE link failure – connectivity
recovery time could be under 50 msec (local protection)
PE-P link, P-P link, P node failure – connectivity recovery
time could be under 50 msec (local protection)
Egress PE-CE link failure – connectivity recovery time
could be under 50 msec (local protection)
Egress PE failure with global
protection - connectivity recovery
time is several 100s of msec
Egress PE failure with local protection
(as described in this presentation) -
connectivity recovery time could be under
50 msec
Overall connectivity
recovery time several
100s of msec
Overall connectivity
recovery time could
be under 50 msec
“Your chain is as strong as your weakest link”

In conclusion…
This presentation outlines a scheme that provides local protection
against egress PE router failure
Without imposing any constrains on network topology
Applicable to both IPv4 and IPv6 2547 VPN service
Similar approach can be applied to provide local protection in the
presence of ASBR failures
Useful for supporting 2547 VPN inter-AS option (b) and (c)
Useful for supporting 2547 VPN inter-AS option (b) and (c)
When BGP is used as an inter-area routing and label distribution
protocol (“seamless MPLS”) similar approach can be applied to provide
local protection in the presence of ABR failures
The scheme outlined in this presentation fills a crucial missing piece
required to provide high availability 2547 VPN service

PLNOG 9: Emil Gągała - Fast Service Restoration

PLNOG 9: Emil Gągała - Fast Service Restoration

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to PLNOG 9: Emil Gągała - Fast Service Restoration

Similar to PLNOG 9: Emil Gągała - Fast Service Restoration (20)

Recently uploaded

Recently uploaded (20)

PLNOG 9: Emil Gągała - Fast Service Restoration