Routing to SDN Era

1 Copyright © 2018 Juniper Networks, Inc. www.juniper.net
ROUTING TO SDN ERA
SHAOWEN MA, APAC CHIEF ARCHITECT, MASHAO@JUNIPER.NET
FEB 26TH, 2018

ACKNOWLEDGEMENTS
• This slides contain public Information on OTT Network from
Internet/Youtube/Sigcomm White Paper/Github, etc.
• Thanks a lot for all the contributors
• Sorry not include all reference information on those slides.

Juniper Confidential
Path to SDN
Routing in Google(Firepath)/Facebook(Open/R)
Routing in the Fat Tree
Cloud SDN Controller
Summary
AGENDA

PRIVATE INTERNET & DARK FIBER
June 2016
2018
Nov 2016
May 2018

SDN CHANGING NETWORK ARCHITECT
DR
DC fabric
DR
DC fabric
DCI(private)
DCCAMPUS
DCCAMPUS
public facing
backbone
(private infra)
PR PR
off-net
caching
metro area
caching
metro area
caching
off-net
caching
off-net
caching
off-net
caching
Internet

SDN CHANGING NETWORK ARCHITECT
DR
DC fabric
DR
DC fabric
DCI(private)
DCCAMPUS
DCCAMPUS
public facing
backbone
(private infra)
PR PR
off-net
caching
metro area
caching
metro area
caching
off-net
caching
off-net
caching
off-net
caching
SDN
SDN
Internet
SDN

Software
Defined
Data Center
3
Espresso
4
2017
TCP BBR
2016
Software
Defined
WAN
Software
Defined
Virtualization
Software
Defined
Edge Peering
Maglev
*Public information from youtube

E2E INTERNET SLA NEVER DONE BEFORE BEFORE OTT SDWAN
Carousel
Premium
Standard
With GCP direct Connect, E2E internet
bandwidth Guarantee is possible.
Which non SP/ISP can archive before!
Features Premium Tier Standard Tier
Plain VM instance Yes - Global Yes - Regional
HTTP(S) Load Balancing
(LB)
Yes - Global Yes - Regional
TCP/SSL Proxy LB (non-
HTTP traffic)
Yes - Global Yes - Regional
Network / Internal LB
Yes - Regional VIP (+
Client can be anywhere)
Yes - Regional VIP (+
Client needs to be in
same region)
Google Cloud Storage,
Google Container
Engine
Yes
Yes - Regional but only
via LB
Cloud CDN Yes No
Cloud VPN/Cloud
Router
Yes No

CONTRAIL/NORTHSTAR VS OTHER SDN CONTROLLERS
Cloud Focus
Software Overlay
WAN Focus
Underlay
DC Switch Focus
Hardware Overlay
CONTRAIL NORTHSTAR
APIC
• Virtual Router/vSwitch Focus
• VPN provisioning by BGP/XMPP
• NSX use OVSDB, L2 Focus
• 10,000+ Servers
VMVM
IP CLOS
vPE vPE
VMVM
IP CLOS
VTEP VTEP
• Switch Focus
• Openflow/VTEP Tunnel
• “Static Route” on every Hops
• 1,000+ Switch
• WAN Focus
• Segment Routing/RSVP-TE Tunnels
• PCEP, Openflow etc
• 1,00+ Nodes
CONTRAIL

!
WSG – Architecture and Strategy
All!of!the!above!are!properties!that!tend!to!simplify!the!design.!!What!makes!things!interesting!are!the!next!two!
requirements:!
· There!are!very!large!volumes!of!traffic!between!changing!pairs!of!end>points!(depending!on!the!requirements!
of!the!applications!at!a!given!time),!necessitating!pro>active!bandwidth!provisioning!driven!by!the!requirements!
of!the!applications.!
· Bandwidth!must!be!utilized!as!close!to!100%!as!possible!
The!figure!below!shows!the!footprint!of!the!Google!back>end!network,!based!on!some!public!announcements!they!have!
made.!!
Figure$3:$The$Google$G<Scale$(back<end)$network$
Originally!Google’s!back>end!network!was!implemented!similarly!to!the!front!network!–!using!general>purpose!routers!
GOOGLE B4 WAN NETWORK
SPECIALHYBRIDSDN
Site A
Data Center
OFA
Switch
OFA
Switch
OFA
Switch
OFA
Switch
Data
Center
Site B
Servers
RAP TE-AGENT
OFC
paxosQuaggaQuagga
Quagga Paxos Site B
Controllers
Servers
Switch
Hardware
iBGP, ISIS
eBGP
GatewayGateway
Central TE
Servers
Central TE
Servers
Global
TE
94.00%
95.00%
96.00%
97.00%
98.00%
99.00%
100.00%
101.00%
Q1 Q2 Q3 Q4
Floor Avg Peak
*Public information from youtube

GOOGLE EARLY DAYS OF ROUTE CONTROLLER, NOT FOR JUPITER
Customized ISIS SPF.
Master( server) compose Central LSD, Client(Leaf) Distributed SPF Calculation
*Public information from whitepaper

GOOGLE FIREPATH WITH OPENFLOW
tunnel use LPM and ACL(openflow) result to loadbalance(TE)
Figure :M ultipath W A N Forw arding Exam ple.
(a) (b)
Figure :System transition from one path assignm ent(a)to another(b
Figure :M ultipath W A N Forw arding Exam ple.
Figure :Layering tra c engineering on top ofshortestpath for-
w arding in an encap sw itch.
the packetbased on a table entry pre-con gured by TE.A er de-
capsulation,thesw itch forw ardsto thedestination based on thein-
nerpacketheader,using LongestPre x M atch (LPM )entries(from
c
c
5
O
g
p
n
i
m
G
r
o
*Public information from whitepaper

How to Select Which Peer to send
▪ Controller/RR may morning the BGP Peer Link
▪ Controller/RR find a tunnel from Ingress to ASBR
▪ Controller/RR based on certain rules to select ASBR
BGP EPE/IPE DESIGN PHILOSOPHY
How ASBR identify a Peer
▪ Per Peer /32 address per label
▪ Install the MPLS Label POP for every Peer
▪ When ASBR received different label and
send traffic to specific Peer
How Ingress mapping traffic to ASBR/Peer
▪ Ingress push tunnel label to ASBR
▪ Ingress push BGP-LU label

GOOGLE ESPRESSO BGP EPE CONTROLLER
4th Pillar of Google SDN: { Jupiter, B4, Andromeda, Espresso}.
· First, it allows us to dynamically choose from where to serve individual users based on measurements of how end-to-end network
connections are performing in real time.
· Second, we separate the logic and control of traffic management from the confines of individual router “boxes.” Rather than relying
on thousands of individual routers to manage and learn from packet streams, we push the functionality to a distributed system that
extracts the aggregate information

ROUTING ALGORITHMS
Distance-Vector
▪ Vectors of destination and distance sent to neighbors
▪ “Tell your neighbors about the rest of the network”
▪ Destination in terms of a network prefix
▪ Distance in terms of a metric: hop count, delay, bandwidth
▪ Use Distributed Bellman-Ford path selection algorithm
▪ Popular protocol: Routing Information Protocol (RIP)
Link-State
▪ Flood description of your links (link state)
▪ “Tell the rest of the network about your neighbors”
▪ Links described by
▪ End-point routers of subnet in internet
▪ Cost of subnet: delay, bandwidth
▪ Use Dijkstra path selection algorithm
▪ Popular protocol: Open Shortest Path First (OSPF)
Path-Vector
▪ Routes advertised as full-paths
▪ Paths described by sequence of ASs
▪ Popular protocol is Border Gateway Routing Protocol (BGP)
FIB PATH

DISTANCE-VECTOR/LINK-STATE/PATH-VECTOR
ROUTING FOUNDATION IS GOOD, NEED SDN UPGRADE
2
4
3
65
1
2
4
3
65
1
Path-Vector
BGP
Link-State
OSPF/ISIS
Distance-Vector
RIP
Router Announced LSDB, Dijkstra
“Tell rest of the network your neighbors”
Full-paths announced in BGP.
Paths described by sequence of ASs
Vectors of destination and distance
“Tell your neighbors rest of the network”
2
4
3
65
1
2
4
3
65
1
2
4
3
51 6

BUT HOW TO EXCHANGE INFORMATION IS IMPORTANT
• QUIC/THRIFT For RIFT
• THRIFT for Open/R
IP Port 89 for OSPF
Not
TCP/UDP
IP Port 89
TCP Port
179
TCP Port 179 for BGP

GET RID OF UNNECESSARY COMPLEX
• Hello
• Database Description (DBD)
• Link State Request (LSR)
• Link State Update (LSU)
• Link State Acknowledgment (LSAck)
Adjacency state machine
Router Attribute
Protocol Message
• Down
• Attempt
• Init
• 2-Way
• ExStart
• Exchange
• Loading
• Full
• Designated router
• Backup designated router
LSA Description
1 Router-LSA
2 Network-LSA
3 Inter-Area-Prefix-LSA
4 Inter-Area-Router-LSA
5 AS-External-LSA
6 MOSPF-LSA
7 NSSA-LSA
8 Link-LSA
9 Intra-Area-Prefix-LSA
All kind of LSA
✔

SPF EXAMPLE, FUNDAMENTAL NO NEED TO CHANGE

OPEN/R: OPEN ROUTING ARCHITECTURE
OPENSOURCE IN NOV 2017
Wedge 100
(32*100GE)
BackPack(100GE)
FBOSS
OPEN/R
MX/PTX
*Public information from WWW

BUILDING EXPRESS BACKBONE:
FACEBOOK NEW LONGHAUL NETWORK(B4), MAY 2017
Independent and Identical parallel forwarding plane
• 4 way Active-Active Redundancy, Incremental
change and canary, Being able to conduct A/B
testing between the planes
• Centralized (and highly redundant) ensemble of
BGP-based route injectors to move traffic on/off
the network
• sFlow collector, based on collecting samples, used
to feed in active demands into the controller
• Traffic engineering controller, which computes
and programs optimum routes based on the
current demand set.
• Open/R agents running on network devices to
provide IGP and messaging functionality.
• LSP agents, also running on network devices to
interface with the device forwarding tables on
behalf of the central controller.
*Public information from WWW

HOW FAR YOU CAN GO?

KV STORE AND MESSAGE BUS
Two important Tools for Modern Protocols Design
SDN Controller
In memory
Distributed DB
KvStore
CRDT
RIB
L-RIB
L-RIB L-RIB
L-RIB
L-RIB
ZeroMQ
SDN Controller

OPEN/R: OPEN ROUTING ARCHITECTURE
https://code.facebook.com/posts/1142111519143652/introducing-open-r-a-new-modular-routing-platform

FACEBOOK OPEN/R COMPONENTS
KvStore - Store and Sync
• KvStore provides a self-contained, in-memory key-value data store which is eventually consistent. Underlying
implementation is based on conflict-free replicated data type (CRDT).
• The stores are interconnected in a mesh, and synchronize their contents in an eventually consistent fashion.
This store is used to disseminate a set of key-value pairs to all nodes in the network/cluster. For example, a
node may post information to its local store about its adjacent neighbors under a key adj:myRouteName and
this information will propagate to all other stores in the network, under the same key name.
• PUB/SUB Channel All incremental changes in local KvStore are published as thrift::Publication messages One
prominent feature is that all values are opaquely encoded as Thrift objects using client's choice of protocol
Decision
• This module is responsible for computing the local routing table from the Adjacency and Prefix databases
advertised by every node in network (read from KvStore)

JUNIPER SUPPORT OPEN/R IN 2017
Linux
JUNOS
RPDOpen/R
TCP/IP
Linux
PFE
RIB APIs
FIB
RELC
Socket RPC
server
Netproxy-server
LKM
Socktun.ko
Interface notifications PRPD Components:
• RIB APIs
• Interface Notification API
Netproxy Components:
• Socket API interceptor
• Socktun kernel module
• Netproxy client
• Netproxy server

ROUTING FOR CLOS TOPOLOGIES
• Clos Offers Well-Understood non-Blocking
Probabilities, Work Done at AT&T (Bell
Systems) in 1950s
• Fully Connected Clos is Dense and Expensive.
Data Centers Today Tend to Be Variations of
“Folded Fat-Tree”
S1 S2 S3
S1 S2 S3
TOR
AGGR
SPINE
Folded Fat-TreeFat-Tree
CLOS

NEW DRAFT TO OPTIMIZEDISIS/OSPF
IETF Data Center Routing Group just Chartered
TOR
AGGR
SPINE
Controller
SDN Controller
DIS

REQUIREMENTS BREAKDOWN (RFC7938+)
FOR A “MINIMAL OPEX FABRIC”
Problem / Attempted Solution BGP modified for DC
(all kind of “mods”)
ISIS modified for DC
(RFC7356 + “mods”)
RIFT
Native DC
Peer Discovery/Automatic Forming of Trees/Preventing Cabling
Violations
⚠️ ⚠️
Minimal Amount of Routes/Information on ToRs
High Degree of ECMP (BGP needs lots knobs, memory, own-AS-path
violations) and ideally NEC and LFA
⚠️
Traffic Engineering by Next-Hops, Prefix Modifications
See All Links in Topology to Support PCE/SR ⚠️
Carry Opaque Configuration Data (Key-Value) Efficiently ⚠️
Take a Node out of Production Quickly and Without Disruption
Automatic Disaggregation on Failures to Prevent Black-Holing and Back-
Hauling
Minimal Blast Radius on Failures (On Failure Smallest Possible Part of
the Network “Shakes”)
Fastest Possible Convergence on Failures
Simplest Initial Implementation

ROUTING FOR CLOS TOPOLOGIES
Topology sort
Link States flooding to Up
Distance vector Down
One layer bounced
2
2
3
3
1
3
4
4
4
W E
N
S
Level1Level2Level0

AUTOMATIC TOPOLOGY CONSTRAINTS
• Automatic Rejection of Adjacencies
Based on Minimum Configuration,
POD/Levels
• Automatically keep information within
levels.
• Protocol Will Work as Well If Level 0 is
Allowed to Connect to Level 2 but
Optimal Routing Would Need Larger FIBs
on Leafs
• 10K FEET VIEW: “AUTOMATIC CABLING
AND MINIMAL TOPOLOGY EXCHANGE”
Level1Level2Level0
POD 1 POD 2

AUTOMATIC DE-AGGREGATION
4
Level1Level2Level0
• Representation of the L2 Spine
is Reflected by the L1 Layer
• Lower L2 Spine Sees that
Upper Node has No Adjacency
to the Only available Next-Hop
to P1
• Lower L2 Spine Node
Disaggregates P1
P1
P1
P1
P1
P1P1
0/0
0/0
0/0

SUMMARY OF RIFT ADVANTAGES FOR IP FABRIC
Advantages of Link-State and Distance
Vector
• Fastest Possible Convergence
• Automatic Detection of Topology
• Minimal Routes on TORs
• High Degree of ECMP
• Minimal Blast Radius on Failures
• Fast De-comissioning of Nodes
• Maximum Propagation Speed with Flexible
# Prefixes in an Update
No Disadvantages of Link-State or Distance
Vector
• Reduced Flooding
• Automatic Neighbor Detection
And Some Neither Can Do
• Automatic Disaggregation on Failures
• Minimal blast radius on failure
• Key-Value Store
• Horizontal Links Used for Protection Only
• Can utilized all path for ECMP without
loop
But maybe over engineering??? ☺

RIFT IMPLEMENTATION AND DEMO
On MacPro Laptop (Low Power I7 with 4 Real Cores)
21 Nodes, 60 links, 600 Prefixes
Convergence From Cold Start
• ~ 4x Faster Than Flat IGP (~ 300 Millisecs)
• ~ 3x Less Flooding Than Flat IGP
Single Link Flap at Super-Spine
• ~ 2x Faster Than Flat IGP
• < 50 Millisecs Convergence (35 avg, 70 Max)
On Shaowen’s
MacBook Pro

MORE AND MORE MODERN PROTOCOL ON JUNOS
Linux
RPD
(BGP/SR)
SDN
Controller
Linux
PFE
RELC
…Open/R RIFT

JUNIPER DC ARCHITECTURES
MC-LAG
Flexibility |
1GbE/10GbE/ 40GbE
| ISSU |
Hyper
Scalability
Junos: One common operating system
Business Critical IT & Private Cloud SaaS, Web Services
Multi-Fabric LAG |
Multi-version support |
Open
Virtual Chassis
NFV use cases
Hyper Scalability and
Segregation
L2 Scale-out
MC-LAG
Virtual
Chassis
Fabric
IP Fabric
(BGP/RIFT/
OpenR?)
Ethernet Fabric
Junos
Fusion
L2 Scale-out
Virtual Chassis
Overlays
(EVPN/VXLAN
and MPLS)
Multi-Tier Ethernet
QFX5x00 Leaf SwitchQFX10000 Spine Switch
Common
Building Blocks
IT/Cloud OpsNetwork Ops DevOps
YANG
OpenConfig

GOOGLE DC CONTROLLER
ANDROMEDA: VIRTUALIZE PHYSICAL NETWORK
ToR
10.1.1/24
10.1.2/24
ToR
10.1.3/24
ToR
10.1.4/24
ToR
VNET:5.4/16
VNET:192.168.32/24
VNET:10.1.124
Load
Balancing
DoS
ACLs
VPN
N FV
InternalNetw ork
Androm eda:Google’s Network Virtualization Controller
13
H ostV M M
Androm eda
Controller
G oogle In frastru ctu re Services V M V M
Andromeda: Similar Contrail+ Openstack/K8S
Tunnel: GRE/VXLAN

GOOGLE DC CONTROLLER
ANDROMEDA: VIRTUALIZE PHYSICAL NETWORK
MPLSoGRE or VXLAN
A ndrom eda datapath goals & techniques
Ourgoals:
● near-nativethroughputandlatency(uSecs matter!)
● highCPU efficiency
Leadtoengineeringdecisions,suchas:
● Datapathpipelined
● &replicatedmultipletimes ononeserver,as VMresources scaleout
● Placefunctionalityin-network,whenit’s efficienttodoso
V M
TX
R ate
Lim iting
Firew all/
S ecurity
B illing
R outing
P hys
TX
E ncap
27

SDN NETWORK AS A ROUTER
VM VM
VM VM
VM VM
VM VM
Physical underlay network
Forward encapsulated packets from server to server
Virtual overlay network
Implemented using overlay tunnels
Contrail Controller or NSX
OpenStack or VMware
Contrail vRouter
or VMware vSwitch
Neutron Plugin
PHYSICAL GRAPH
Control
Plane
DATA PATH GRAPH
OVERLAY GRAPH
Virtual
Fabric
CLOS FABRIC
L3 IP FABRIC
3rd Switch
Virtual
LineCard
Vmware vSwitch
Contrail vRouter
SDN Router

Control Plane
OVERLAY PROTOCOLS CHOSEN BY CONTRAIL
Contrail SDN Controller
OpenStack
Configuration Analytics
Control
Virtualized Server
VM VM VM
Virtualized Server
VM VM VM
Quantum
XMPP
for vRouters
MP-BGP + NETCONF
for GW routers/TOR
Data Plane
MPLS over GRE
MPLS over UDP
VXLAN

CONTRAIL - BASED ON MPLS EVPN TECHNOLOGY
VM
Hypervisor with vRouter
Server
Tenant VRF
Encapsulation Tunnel
XMPP (BGP)
Datacenter
Route
Reflector
BGP
Provider Network
L3 VPNs for Inter-Site Connectivity
Traffic segmentation in the WAN
MPLS over MPLS label encapsulation tunnels
BGP route signaling
Contrail Virtual Networks in Datacenters
Traffic segmentation in the LAN
MPLS over GRE or VXLAN label encapsulation tunnels
XMPP (with BGP payload) route signaling
Protocols,
Architecture
Customer Site
CE Router
PE Router
Customer VRF
Encapsulation Tunnel
OpenStack
Cloud Manager
Contrail
Controller

CONTRAIL REFERENCED DRAFT
draft-ietf-l3vpn-end-system
BGP: Gateway Router
Route-Server: Control Node
End-Syste: vRouter
Example update notification message
from Route Server to VPN Forwarder:

CONTRAIL ROUTE DISTRIBUTION:L3VPN
Compute 1
VRF (Dynamic Tunnel Encapsulation)
Compute 2
VRF
IP Network
Control Node
Configuration Node REST API
70.10.10.1 151.10.10.1
Control Plane (XMPP) IF-MAP
vRouter
Agent
vRouter
Agent
10.1.1.1: NH = 70.10.10.1; LBL = 39
10.1.1.1: NH = 70.10.10.1; LBL = 39
10.1.1.1: NH = 70.10.10.1; LBL = 39
VM-A
10.1.1.1
VM-B
10.1.1.2
10.1.1.2: NH = 151.10.10.1; LBL = 17
10.1.1.2: NH = 151.10.10.1; LBL = 17
10.1.1.2: NH = 151.10.10.1; LBL = 17
10.1.1.2 10.1.1.1 PAYLOAD
PriDstIP PriSrcIP
150.10.10.1 70.10.10.1 GRE LBL=17 10.1.1.2 10.1.1.1 PAYLOAD
PubDstIP PubSrcIP PriDstIP PriSrcIP
10.1.1.2 10.1.1.1 PAYLOAD
PriDstIP PriSrcIP
Outer MAC headers left
out to reduce clutter
(Dynamic Tunnel Encapsulation)
Control Plane (XMPP)

CONTRAIL – SOFTWARE NETWORKING SYSTEM
Config Plane: Bi-directional real-time
message bus using XMPP
Scale-out Multi-vendor VNFs can
run on the same platform
Interoperates with different
Orchestration systems
Integrates with
▪ different Linux Hosts,
▪ multiple hypervisors, and
▪ multi-vendor X86 servers
Multi-vendor SDN Gateway (any router that can talk BGP and
the dynamic tunneling protocols)
Data Plane: Overlay Tunnels
(MPLSoGRE, MPLSoUDP, VXLAN)
Control Plane: BGP Control Plane
(logically centralized, physically
distributed Controller elements)
Automation: REST APIs to integrate
with different Orchestration Systems
Control /Config Plane: for Bare Metal
support - OVSDB or EVPN + Netconf
Multi-vendor TOR support to connect
Bare Metal Servers, using standard
control plane & config plane protocols

CONTRAIL – LEVERAGE RABBITMQ AND CASSANDRA DB
…
Config Node
Rest API
Server
Schema
transformer
RabbitMQ
REST API calls from
Orchestrator
Control Node …
Config Node
…
Distributed
Sync
Zookeeper
Svc-
monitor
Zookeeper
Device
manag
er
Control Node
Cassandra Cassandra

4 CONTROLLER(DC/HOST/WAN/PEERING) IN GOOGLE’S NETWORK
23
ControlSystem s in Google’s Network
OFA OFA OFA OFA OFA
OFA OFA OFA OFA OFA
OFA OFA OFA OFA OFA
OFA OFA OFA OFA OFA
External External
iBGP |ISIS
eBGP
OF
BwE Controller
Fabric
Controller(s)
Androm eda
Controller
Bw E Bw E
Fabric
Controller(s)
Androm eda
Controller
TE Controller
B4 Controller B4 Controller
NORTHSTAR
EPE Peering
Controller
eBGP
CONTRAIL
Espresso
BGP Peering
Controller
NORTHSTAR
eBGP

TELEMETRY
-STREAMING ANALYTICS, OVERLAY-UNDERLAY CORRELATION
BIG DATA ANALYTICS & MACHING LEARNING
- CONSTANT MONITORING, FEEDBACK-LOOP
- PREDICTIVE & ADAPTIVE NETWORKS
ANALYTICS & MACHINE LEARNING

ROAD TO SELF DRIVEN NETWORK
SDN/NFV
Controller
Network
Telemetry
Network

WHAT HAPPENS WHEN THE ROBOTS GET ALL THE JOBS?

Routing to SDN Era

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Routing to SDN Era

Similar to Routing to SDN Era (20)

More from APNIC

More from APNIC (20)

Recently uploaded

Recently uploaded (20)

Routing to SDN Era