Describing how Segment Routing is laying the foundations for controller based TE in Google's RSVP based B2 network. Solving network hyperscale challenges one step at a time.

1. Confidential + ProprietaryConfidential + Proprietary
Introducing Segment Routing TE
Alex Bogdanov

2. Confidential + Proprietary
Asia Pacific Americas Europe, Middle East, & Africa
Google Network
Regions, PoPs, Network
HK-G (HK, GU) 2019
Havfrue (US, IE, DK) 2019
Curie (CL, US) 2019
JGA (AU, GU, JP) 2019
Network
SJC (JP, HK, SG) 2013
Unity (US, JP) 2010
Faster (US, JP, TW) 2016
PLCN (HK, LA) 2019
Monet (US, BR) 2017
Junior (Rio, Santos) 2018
Tannat (BR, UY, AR) 2018
Indigo (SG, ID, AU) 2019
Dunant (US, FR) 2020
Edge points of
presence 134
3London
3
Netherlands
3
3 Finland
3
Belgium
Los Angeles
4Iowa
3 N. Virginia
3
S. Carolina
3 Montreal
3
São Paulo
3
Tokyo
3Osaka
3
Taiwan3
Mumbai
3
Sydney
3
Singapore
3Oregon
Current region and
number of zones
Future region and
number of zones
Frankfurt
Zurich
3
3
3
Edge points of
presence 134
3
Hong Kong
3Jakarta
Presented in April 2019 at MPLS/SDN Conference

3. Confidential + Proprietary
What is B2?
B2 is the Google backbone that carries all user facing traffic
● MPLS network which uses BGP, ISIS, RSVP-TE
● Multi-Vendor platform architecture
● Provides external connectivity to Google data centres
● Peering edge consists of vendor and SDN elements (Espresso)
Presented in April 2019 at MPLS/SDN Conference

4. Confidential + Proprietary
Google B2 Backbone Growth
B2 Growth (~7x growth in last 5 years)
Presented in April 2019 at MPLS/SDN Conference

5. Confidential + Proprietary
Managing Transit RSVP State
LSP state is expected to continue growing
#ofactiveRSVPReservations
Quarter
17q1 17q3 18q1 18q3 19q1 19q3 20q1 20q3 21q1 21q3
Presented in April 2019 at MPLS/SDN Conference

6. Confidential + Proprietary
Managing RSVP HeadEndsAches
Date (month)
#ofclearLSPevents
LSPs are cleared often to speed up re-optimization
Presented in April 2019 at MPLS/SDN Conference

7. Confidential + Proprietary
Shortest Path Placement
>98% of HIPRI LSPs remain on the shortest path
Presented in April 2019 at MPLS/SDN Conference

8. Confidential + Proprietary
Path Diversity
LSP pathing does not offer enough diversity
50% chance of avoiding
a common device
0% chance of avoiding a
common device
● 60-80% of LSPs may end up
on same device in metro
● ~7% of HIPRI LSPs never had
any diversity for time
observed
● ~99% of HIPRI LSPs failed to
have at least 50% of device
diversity for 23.55h day
● Higher probability of diversity
with larger number of LSPs =
more state required
Presented in April 2019 at MPLS/SDN Conference

9. Confidential + Proprietary
Segment Routing Traffic Steering:
● Stateless paths
● Pathing granularity and isolation for labeled and unlabeled traffic
● ECMP/WECMP not limited by number of LSPs (as is the case for RSVP HE)
RSVP Co-Existence:
● Shortest path placement
● Distributed bandwidth management
Streaming Telemetry:
● Event driven low latency data
Building Blocks of a Solution
Addressing RSVP Scale and Pathing Challenges Presented in April 2019 at MPLS/SDN Conference

10. Confidential + Proprietary
ISIS-SR
● Block/Node/Adjacency label steering
Segment Routing Traffic Steering
Network

11. Confidential + Proprietary
Network
SR Policy: Labeled and Unlabeled Traffic Steering
● SR labels can be used to steer traffic at HeadEnd(s)
○ Static policies can be invoked in addition to BGP
○ Advertising Segment Routing Policies in BGP https://tools.ietf.org/html/draft-ietf-idr-segment-routing-te-policy-xx
○ Segment Routing Policy Architecture https://tools.ietf.org/html/draft-ietf-spring-segment-routing-policy-xx
Segment Routing Traffic Steering
Labeled Path
BGP
Static
SR label stack for Path1
Policy label
IP
SR label stack for Path n
IP
IP
IP

12. Confidential + Proprietary
Network
Segment Routing Traffic Steering
SR Policy: The Endpoint conundrum
● Scaling the Edge with color-only policies
○ https://tools.ietf.org/html/draft-ietf-spring-segment-routing-policy-xx
A single <HEADEND, COLOR, NULL> (color only) policy can meet the following requirements:
● Steer both IPv4 and IPv6 traffic
● Steer based on traffic-class to devices behind a destination block
1
n
DestinationBlock
EdgeDevices

13. Confidential + Proprietary
RSVP Co-Existence
SR Shortcuts
● SR traffic can avoid RSVP LSP(s) at HE devices
Share our similarities, celebrate our differences
M Scott Peck
Network

14. Confidential + Proprietary
RSVP Co-Existence
Dark Bandwidth
● SR traffic can alter RSVP Maximum-Reservable-Bandwidth
○ Recommendations for RSVP-TE and Segment Routing (SR) Label Switched Path (LSP) Coexistence
https://tools.ietf.org/html/rfc8426
max-reservable-bw
RSVP
Dark
IP
InterfaceTraffic
RSVP
Reserved
InterfaceBandwidth
RSVP
Reserved
Dark
InterfaceBandwidth
max-reservable-bw
IP IP
RSVP
Unreserved
(0-7) Unreserved(0-7)
Network
Share our similarities, celebrate our differences
M Scott Peck

15. Confidential + Proprietary
Streaming Telemetry: Open Sourced Data
● Source-timestamped
○ Accurate rates
● Event-driven
○ Push as soon as the data changes (low latency)
● Subscription-based
○ Subscriptions are provided up front by the collectors
● gRPC Network Management Interface (gNMI)
○ http://github.com/openconfig/gnmi
gNMI
Streaming Telemetry
State Collector
gNMI Collector
Cache
Client
Client
Client
Network

16. Confidential + Proprietary
Current Feature Status
● Multi-Vendor Technology Adoption (shipping code)
○ ISIS-SR
○ Dark Bandwidth, SR Shortcuts
○ Forwarding State Programming (SR Policy)
● Multi-Platform Scale Improvements (shipping code)
○ Feature Rich Platforms
○ Lean Platforms (ongoing BRCM FEC optimizations work)
● Scaling Telemetry State (shipping code)
○ Ongoing event driven work
Continue Optimizing Scale and Cost

17. Confidential + Proprietary
Thank You
Presented in April 2019 at MPLS/SDN Conference