Technical Forum
Scaling the Software Driven Cloud Network
Autumn 2015
Technical Forum
VXLAN - L2	over	Layer3
Hosts Hosts
Goals of Network Virtualization
Apps deployed based on
resource utilization not location
Optimisation of the
resource pool
40%
VM
Any to any connectivity
Remove islands of
service connectivity
VM
Relieves management burdens
Operational Efficiency
Automation of the virtual and
the physical
Decrease & automate
Deployment time
VM VM
Technical Forum
EOS Network Virtualization -VXLAN
§ Ethernet in IP overlay
§ Abstract Virtual from physical
§ Reduces size of failure domain
§ VMWare NSX Integration
§ Enables service stitching
• Orchestration inetgration
• Macro-segmentation
§ Scale >4K VLANs: 16M VNIs
§ Does not extend fault domain
§ Consistent physical and virtual
workloads
§ Seamless scale-out
Any-to-any Layer2 across Layer3 Automated VXLAN without controller
Between Data Centers
Within a Data Center
Technical Forum
At scale manual configuration of HER flood-list can be arduous,potential for excessive traffic flooding
during learning processes
VXLAN Control-Plane – Unicast Replication
Host 4
VTEP 4
VNI
5000
VTEP 1
Host 1 Host 2
VTEP 3
Host 3
VTEP flood list on VTEP 1
VNI 5000 à VTEP 3
VNI 5000 à VTEP 4
VTEP flood list on VTEP 3
VNI 5000 à VTEP 1
VNI 5000 à VTEP 4
VTEP flood list on VTEP 4
VNI 5000 à VTEP 1
VNI 5000 à VTEP 31
2
3
5 5
4 4
1. VTEP flood-list - manually configured on
each VTEP for each VNI
2. BUM traffic received from a locally attached
node on VTEP-1
3. VTEP-1 replicates the BUM traffic for each
VTEP in the flood-list of the associated VNI
4. Individual unicasts frames are sent on the
wire to each VTEP in theVNI
5. RemoteVTEPs receive BUM traffic
6. RemoteVTEP’s learn inner source MAC and
map it to the outer SRC IP (remoteVTEP of
origin)
Leaf 3 Leaf 4Leaf 1
Leaf 2
VTEP 2
Leaf 2
Technical Forum
EOS Scaling Intra-Switch
§ Single binary architecture for all
platforms
§ Abstracts platform hardware specifics
§ Presents multiple open interfaces
upstream
§ Delivers decoupled state sharing
architecture (Sysdb)
§ Provides highly stable platform with
great feature velocity
§ Enables agility in hardware choice
CLI eAPI OVSDB SDK
Common binary/APIs for all platforms
Low Inter-Process Communication
Arista EOS Architecture
Efficient Subscribe/Publish
Linear Cloud Scaling
Publish
Notify
PIM
SNMP BGP
MLAG
STP eAPI
IGMPSysdbDriver
Technical Forum
Introducing CloudVision eXchange (CVX)
EOS EOS EOS
Single Interface to EOS devices
EOS
publish/subsc
ribe
model
§ Part of EOS CloudVision Framework
§ Abstracts the physical switch infrastructure
• MLAG support VTEP resilience
§ Provides a single access point for real-time
provisioning
• Orchestration and integration with multi-vendor controllers
§ Distributed EOS state: CVX mounts state from
all switches (sysDBs) in the network
§ No new protocol needed – uses EOS framework,
same openness
§ Management plane, not data plane
§ CVX runs asVM
EOS
CVX
CLI eAPI OVSDB SDK
Open Standards-based
northbound APIs
State publish/subscribe
CloudVision
eXchange
MLAG resilience is
abstracted by CVX
Technical Forum
Leaf 2
CVX – simplified provision and learning
Automated flood-list configuration and MAC address distribution
VXLAN Control-Plane – CVX
1. MAC learnt locally onVTEP 1 From
generated host traffic
2. Local VXLAN states are mounted by
CVX
3. CVX has a global view of each VTEP
- local VXLAN MAC address tables, VNI
configured on each VTEP
4. Remote MACs for locally configured
VNI Written to local VXLAN table
5. Remote MAC added to local
VXLAN hardware tableHost 4, MAC D
VTEP 4
VNI 5000
VTEP 1
Host1, MAC A Host 2
VTEP 2 VTEP 3
Host 3
1
2
5 5
4
Network Database
VTEP 1: VNI 5000:MAC A
VTEP 4: VNI 5000:MAC D
VXLAN table
VNI 5000 MAC A VTEP 1
VNI 5000 MAC D VTEP 4
CloudVision
eXchange
3
Leaf 2Leaf 1 Leaf 3 Leaf 4
Technical Forum
VXLAN Deployment Solutions
VTEP-1
Openstack
NSX, Nuage, …
Automated VXLAN without
3rd party controller
Automation and integration
with 3rd party controller
Small Scale DC and DCI solution
Head Replication (HER)
• Manually configured VTEP-flood
list
• Traffic flooded via the defined
flood-list.
• Flow-based MAC learning
• No need for Multicast in the IP
fabric
• Suitable for DCI solutions and
small scale intra-DC solution due
to manual config
CVX standalone
• CVX provides centralized database
of all VXLAN state.
• MAC address learning via the
CVX, flow-based learning optional
• HER flood-list automatically
populated by the CVX
• No need for Multicast in the IP
fabric
• Scalable for intra-DC solutions
where a level of automation is
required
CVX + 3rd party integration
• Centralized database of CVX
shared with third-party controller
(NSX, OpenStack, Nuage, etc)
• Distributed MAC address learning
between Software and hardware
VTEPs.
• VNI provisioning via centralized
controller
• Solution for scalable DCs with
HW to SW VTEP automation
CloudVision
eXchange
CloudVision
eXchange
Technical Forum
INTERFACES AND APIsTO CVX
Technical Forum
3rd Party Integration with CVX
§ Third-party controllers can consume the
state and share state using their
preferredAPI
§ Provides the third-partycontroller with
single automation point for the
physical infrastructure
§ Provides the third-partycontroller with
visibility of the control-plane from the
physical infrastructure
CVX has a global view of the physical hardware infrastructure, with multiple
programmable open APIs
Network wide view of the physical network
Global VXLAN Database
CLI/SNMP
State SyncState Sync State Sync
OVSDBeAPI SDK others
CloudVision
eXchange
SW-1: VNI-X: MAC-
A,
VNI-Y: MACB
SW-2: VNI-Y:MAC-D
VXLAN VNI, VTEP,
MAC, LLDP
States example:
VXLAN DB example:
Technical Forum
CVXWorkload Service #1: Full Physical Topology
§ Leaf switch builds their local topology table using standard LLDP
§ Contains directly attached compute nodes,which will host the virtual machines
§ CVX mounts the local LLDP tables,providing a network wide view
§ CVX knows the physical location (switch and interface) each compute node is attached
eAPI
cvs-switch#show network physical-topology neighbors
Interface Neighbor Intf Neighbor Host
------------------ ------------------ --------------
Ethernet1 Ethernet1 atf-spine1
Ethernet2 Ethernet1 atf-spine2
Ethernet3 eth1 atf-oshost1
Ethernet4 eth1 atf-oshost2
Network wide
Topology Table
cvs-switch#show network physical-topology hosts
Unique Id Hostname
--------------------- ---------------------
0050.5686.ba66 atf-host1
0050.5686.4711 atf-host2
0050.5686.1184 atf-host3 Compute Nodes
Network wide topology visible from CVX eAPI to consume the info northbound
LLDP
LLDP
compute compute
et2
Network Topology
Database
LLDP State
et1
LLDP
LLDP
compute compute
et2
LLDP State
et1
CloudVision
eXchange
Technical Forum
EOS EOS EOS
EOS
CVXCloudVision
eXchange
CVXWorkload Service #2: VXLAN
• VXLAN Control Services (VCS)
• Control plane that integrates
with theVXLAN data plane
• VCS is used to distribute
• VXLAN-related configuration
• Network reachability state across
the network
• Simplifies Unicast Head-End
Replication (HER)
Propagates VXLAN
Network State
VCS
VM	mobility	
Across	Layer	3	subnets
Technical Forum
Same publish/subscribe
model
‘BYOC’
Single Interface
to EOS devices
CVXWorkload Service #3: Third Party Integration
• Integration and provisioning point for :
• 3rd party controllers / services
• orchestration systems
• Controller-Agnostic approach
• Network-wide visibility via single access
point
• Collective state presented to controllers
• Integration via OVSDB, eAPI, EOS SDK
• Most efficient Physical-to-Virtual (P-to-V)
synchronization
EOS EOS EOS
EOS
CVXCloudVision
eXchange
MLAG resilience is
abstracted by CVX
Technical Forum
Native	
Openstack
Native	vSphere	3.x,	4.x,	5.x,	6.x
Universal Integration with CloudVision
Multiple controllers and virtualisation platforms can coexist and integrate with CVX
CloudVision
eXchange
Technical Forum
Platform forAutomation andVisibility across the Network
Network-Wide Database
Aggregation of Network
wide‘Sysdb’
Abstraction of the
physical network
Single integration point to
the network
More scalable controller
integration
Provisioning the logical
topology
State-sync
Technical Forum
Interacting with Controllers via OVSDB
vSwitch Driver Vendor A Driver
Networking Layer
Vendor Z Driver
asdasdasd
pSwitch asdasdasd
pSwitchasdasdasd
vSwitchasdasdasd
vSwitchasdasdasd
vSwitchasdasdasd
vSwitchasdasdasd
vSwitch
asdasdasd
pSwitchasdasdasd
pSwitchasdasdasd
pSwitchasdasdasd
pSwitch
asdasdasd
pSwitchasdasdasd
pSwitchasdasdasd
pSwitchasdasdasd
pSwitch
§ Controller controls all HW and SW
endpoints individually
§ Must scale to combined host & network
• Provisioning,learning, monitoring
§ Must understand topology logic for HW
infrastructure
§ Must implement network layer
redundancy logic for every vendor
§ Every controller vendor must implement
support for each vendor independently
§ Introduces version- and vendor-
dependencies
Virtualisation
Controller
Technical Forum
X
X
X
X
X
Introducing the Hardware Switch Controller
§ HSC is a vendor provided abstraction
layer
§ Removes the need for detailed
knowledge of the Physical network
§ Enables multi-vendor version and
platform independence
§ Reduces time to onboard technology
§ Decouple infrastructure from controller
asdasdasd
pSwitch asdasdasd
pSwitchasdasdasd
vSwitchasdasdasd
vSwitchasdasdasd
vSwitchasdasdasd
vSwitchasdasdasd
vSwitch
asdasdasd
pSwitchasdasdasd
pSwitchasdasdasd
pSwitchasdasdasd
pSwitch
asdasdasd
pSwitchasdasdasd
pSwitchasdasdasd
pSwitchasdasdasd
pSwitch
vSwitch Driver
Vendor A HSC
Networking Layer
Vendor Z HSC
X
Virtualisation
Controller
Technical Forum
Overlay
Controller
Scaling Controller Integration
18
OVSDB
Overlay
Controller
Network Layer
Controller Layer
10x
Improvement
OVSDB Sysdb
State Sync
Topology/Device	
Dependent
Topology/Device	
Abstraction
Traditional
Approach
CloudVision
Approach
Technical Forum
ThankYou

Atf 3 q15-4 - scaling the the software driven cloud network

  • 1.
    Technical Forum Scaling theSoftware Driven Cloud Network Autumn 2015
  • 2.
    Technical Forum VXLAN -L2 over Layer3 Hosts Hosts Goals of Network Virtualization Apps deployed based on resource utilization not location Optimisation of the resource pool 40% VM Any to any connectivity Remove islands of service connectivity VM Relieves management burdens Operational Efficiency Automation of the virtual and the physical Decrease & automate Deployment time VM VM
  • 3.
    Technical Forum EOS NetworkVirtualization -VXLAN § Ethernet in IP overlay § Abstract Virtual from physical § Reduces size of failure domain § VMWare NSX Integration § Enables service stitching • Orchestration inetgration • Macro-segmentation § Scale >4K VLANs: 16M VNIs § Does not extend fault domain § Consistent physical and virtual workloads § Seamless scale-out Any-to-any Layer2 across Layer3 Automated VXLAN without controller Between Data Centers Within a Data Center
  • 4.
    Technical Forum At scalemanual configuration of HER flood-list can be arduous,potential for excessive traffic flooding during learning processes VXLAN Control-Plane – Unicast Replication Host 4 VTEP 4 VNI 5000 VTEP 1 Host 1 Host 2 VTEP 3 Host 3 VTEP flood list on VTEP 1 VNI 5000 à VTEP 3 VNI 5000 à VTEP 4 VTEP flood list on VTEP 3 VNI 5000 à VTEP 1 VNI 5000 à VTEP 4 VTEP flood list on VTEP 4 VNI 5000 à VTEP 1 VNI 5000 à VTEP 31 2 3 5 5 4 4 1. VTEP flood-list - manually configured on each VTEP for each VNI 2. BUM traffic received from a locally attached node on VTEP-1 3. VTEP-1 replicates the BUM traffic for each VTEP in the flood-list of the associated VNI 4. Individual unicasts frames are sent on the wire to each VTEP in theVNI 5. RemoteVTEPs receive BUM traffic 6. RemoteVTEP’s learn inner source MAC and map it to the outer SRC IP (remoteVTEP of origin) Leaf 3 Leaf 4Leaf 1 Leaf 2 VTEP 2 Leaf 2
  • 5.
    Technical Forum EOS ScalingIntra-Switch § Single binary architecture for all platforms § Abstracts platform hardware specifics § Presents multiple open interfaces upstream § Delivers decoupled state sharing architecture (Sysdb) § Provides highly stable platform with great feature velocity § Enables agility in hardware choice CLI eAPI OVSDB SDK Common binary/APIs for all platforms Low Inter-Process Communication Arista EOS Architecture Efficient Subscribe/Publish Linear Cloud Scaling Publish Notify PIM SNMP BGP MLAG STP eAPI IGMPSysdbDriver
  • 6.
    Technical Forum Introducing CloudVisioneXchange (CVX) EOS EOS EOS Single Interface to EOS devices EOS publish/subsc ribe model § Part of EOS CloudVision Framework § Abstracts the physical switch infrastructure • MLAG support VTEP resilience § Provides a single access point for real-time provisioning • Orchestration and integration with multi-vendor controllers § Distributed EOS state: CVX mounts state from all switches (sysDBs) in the network § No new protocol needed – uses EOS framework, same openness § Management plane, not data plane § CVX runs asVM EOS CVX CLI eAPI OVSDB SDK Open Standards-based northbound APIs State publish/subscribe CloudVision eXchange MLAG resilience is abstracted by CVX
  • 7.
    Technical Forum Leaf 2 CVX– simplified provision and learning Automated flood-list configuration and MAC address distribution VXLAN Control-Plane – CVX 1. MAC learnt locally onVTEP 1 From generated host traffic 2. Local VXLAN states are mounted by CVX 3. CVX has a global view of each VTEP - local VXLAN MAC address tables, VNI configured on each VTEP 4. Remote MACs for locally configured VNI Written to local VXLAN table 5. Remote MAC added to local VXLAN hardware tableHost 4, MAC D VTEP 4 VNI 5000 VTEP 1 Host1, MAC A Host 2 VTEP 2 VTEP 3 Host 3 1 2 5 5 4 Network Database VTEP 1: VNI 5000:MAC A VTEP 4: VNI 5000:MAC D VXLAN table VNI 5000 MAC A VTEP 1 VNI 5000 MAC D VTEP 4 CloudVision eXchange 3 Leaf 2Leaf 1 Leaf 3 Leaf 4
  • 8.
    Technical Forum VXLAN DeploymentSolutions VTEP-1 Openstack NSX, Nuage, … Automated VXLAN without 3rd party controller Automation and integration with 3rd party controller Small Scale DC and DCI solution Head Replication (HER) • Manually configured VTEP-flood list • Traffic flooded via the defined flood-list. • Flow-based MAC learning • No need for Multicast in the IP fabric • Suitable for DCI solutions and small scale intra-DC solution due to manual config CVX standalone • CVX provides centralized database of all VXLAN state. • MAC address learning via the CVX, flow-based learning optional • HER flood-list automatically populated by the CVX • No need for Multicast in the IP fabric • Scalable for intra-DC solutions where a level of automation is required CVX + 3rd party integration • Centralized database of CVX shared with third-party controller (NSX, OpenStack, Nuage, etc) • Distributed MAC address learning between Software and hardware VTEPs. • VNI provisioning via centralized controller • Solution for scalable DCs with HW to SW VTEP automation CloudVision eXchange CloudVision eXchange
  • 9.
  • 10.
    Technical Forum 3rd PartyIntegration with CVX § Third-party controllers can consume the state and share state using their preferredAPI § Provides the third-partycontroller with single automation point for the physical infrastructure § Provides the third-partycontroller with visibility of the control-plane from the physical infrastructure CVX has a global view of the physical hardware infrastructure, with multiple programmable open APIs Network wide view of the physical network Global VXLAN Database CLI/SNMP State SyncState Sync State Sync OVSDBeAPI SDK others CloudVision eXchange SW-1: VNI-X: MAC- A, VNI-Y: MACB SW-2: VNI-Y:MAC-D VXLAN VNI, VTEP, MAC, LLDP States example: VXLAN DB example:
  • 11.
    Technical Forum CVXWorkload Service#1: Full Physical Topology § Leaf switch builds their local topology table using standard LLDP § Contains directly attached compute nodes,which will host the virtual machines § CVX mounts the local LLDP tables,providing a network wide view § CVX knows the physical location (switch and interface) each compute node is attached eAPI cvs-switch#show network physical-topology neighbors Interface Neighbor Intf Neighbor Host ------------------ ------------------ -------------- Ethernet1 Ethernet1 atf-spine1 Ethernet2 Ethernet1 atf-spine2 Ethernet3 eth1 atf-oshost1 Ethernet4 eth1 atf-oshost2 Network wide Topology Table cvs-switch#show network physical-topology hosts Unique Id Hostname --------------------- --------------------- 0050.5686.ba66 atf-host1 0050.5686.4711 atf-host2 0050.5686.1184 atf-host3 Compute Nodes Network wide topology visible from CVX eAPI to consume the info northbound LLDP LLDP compute compute et2 Network Topology Database LLDP State et1 LLDP LLDP compute compute et2 LLDP State et1 CloudVision eXchange
  • 12.
    Technical Forum EOS EOSEOS EOS CVXCloudVision eXchange CVXWorkload Service #2: VXLAN • VXLAN Control Services (VCS) • Control plane that integrates with theVXLAN data plane • VCS is used to distribute • VXLAN-related configuration • Network reachability state across the network • Simplifies Unicast Head-End Replication (HER) Propagates VXLAN Network State VCS VM mobility Across Layer 3 subnets
  • 13.
    Technical Forum Same publish/subscribe model ‘BYOC’ SingleInterface to EOS devices CVXWorkload Service #3: Third Party Integration • Integration and provisioning point for : • 3rd party controllers / services • orchestration systems • Controller-Agnostic approach • Network-wide visibility via single access point • Collective state presented to controllers • Integration via OVSDB, eAPI, EOS SDK • Most efficient Physical-to-Virtual (P-to-V) synchronization EOS EOS EOS EOS CVXCloudVision eXchange MLAG resilience is abstracted by CVX
  • 14.
    Technical Forum Native Openstack Native vSphere 3.x, 4.x, 5.x, 6.x Universal Integrationwith CloudVision Multiple controllers and virtualisation platforms can coexist and integrate with CVX CloudVision eXchange
  • 15.
    Technical Forum Platform forAutomationandVisibility across the Network Network-Wide Database Aggregation of Network wide‘Sysdb’ Abstraction of the physical network Single integration point to the network More scalable controller integration Provisioning the logical topology State-sync
  • 16.
    Technical Forum Interacting withControllers via OVSDB vSwitch Driver Vendor A Driver Networking Layer Vendor Z Driver asdasdasd pSwitch asdasdasd pSwitchasdasdasd vSwitchasdasdasd vSwitchasdasdasd vSwitchasdasdasd vSwitchasdasdasd vSwitch asdasdasd pSwitchasdasdasd pSwitchasdasdasd pSwitchasdasdasd pSwitch asdasdasd pSwitchasdasdasd pSwitchasdasdasd pSwitchasdasdasd pSwitch § Controller controls all HW and SW endpoints individually § Must scale to combined host & network • Provisioning,learning, monitoring § Must understand topology logic for HW infrastructure § Must implement network layer redundancy logic for every vendor § Every controller vendor must implement support for each vendor independently § Introduces version- and vendor- dependencies Virtualisation Controller
  • 17.
    Technical Forum X X X X X Introducing theHardware Switch Controller § HSC is a vendor provided abstraction layer § Removes the need for detailed knowledge of the Physical network § Enables multi-vendor version and platform independence § Reduces time to onboard technology § Decouple infrastructure from controller asdasdasd pSwitch asdasdasd pSwitchasdasdasd vSwitchasdasdasd vSwitchasdasdasd vSwitchasdasdasd vSwitchasdasdasd vSwitch asdasdasd pSwitchasdasdasd pSwitchasdasdasd pSwitchasdasdasd pSwitch asdasdasd pSwitchasdasdasd pSwitchasdasdasd pSwitchasdasdasd pSwitch vSwitch Driver Vendor A HSC Networking Layer Vendor Z HSC X Virtualisation Controller
  • 18.
    Technical Forum Overlay Controller Scaling ControllerIntegration 18 OVSDB Overlay Controller Network Layer Controller Layer 10x Improvement OVSDB Sysdb State Sync Topology/Device Dependent Topology/Device Abstraction Traditional Approach CloudVision Approach
  • 19.