Session: The Data Center Network Evolution: Journey to the Programmable Fabric Presenter: Robert Zalobinski, Technical Solutions Architect Date: October 6, 2015

1. Journey to the Programmable Fabric
The Data Center Network Evolution
Robert Zalobinski
Technical Solutions Architect
rzalobin@cisco.com
In partnership with:

2. Cisco Confidential 2© 2013-2014 Cisco and/or its affiliates. All rights reserved.
•Basics of SDN and Overlay Networks
•Application Centric Infrastructure (ACI)
•Virtual Topology System (VTS 2.0)
•Unified Open NX-OS

3. Cisco Confidential 3© 2013-2014 Cisco and/or its affiliates. All rights reserved.
What is SDN
Software defined networking (SDN) is an approach to building computer
networks that separates and abstracts elements of these systems
In other words…
In the SDN paradigm, not all processing happens
inside the same device

4. Cisco Confidential 4© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Current Network Architecture
Tightly coupled Control and Data Planes
• One Control Plane per Device
• Each Device Managed Individually
• All Command Line Managed
Routing protocols (i.e. OSPF, IS-IS, BGP),
Spanning Tree, SYSLOG, AAA
(Authentication Authorization Accounting),
NDE (Netflow Data Export), CLI
(Command Line interface), SNMP
Layer 2 switching, Layer 3 (IPv4 | IPv6)
switching, MPLS forwarding, VRF
Forwarding, QOS (Quality of Service)
Marking, Classification, Policing, Netflow
flow collection, Security Access Control
Lists
cpu
asic

5. Cisco Confidential 5© 2013-2014 Cisco and/or its affiliates. All rights reserved.
The Promise of SDN
Overlay Protocol
Physical Network
10001101000110101
Control & Data Plane
Decoupled
Network
Virtualization
Direct
Programmability
Centralized Management
Simplification
Agility Programmatically Configured
Dynamic
Automated

6. Cisco Confidential 6© 2013-2014 Cisco and/or its affiliates. All rights reserved.
SDN Programming Models
Applications Control
Data
API
ControllerAPI API
Data
ControllerAPI API
Control
Data
vSwitchAPI Overlay
Protocol
Control
DataOverlay
Controller

7. Cisco Confidential 8© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Types of Overlay Edge Devices
• Virtual end-points only
• Single admin domain
• VXLAN, NVGRE, STT
• Physical and Virtual
• Resiliency + Scale
• Cross-organizations/Federation
• Open Standards
Network Overlays Integrated OverlaysHost Overlays
• Router/switch end-points
• Protocols for resiliency/loops
• Traditional VPNs
• OTV, VXLAN, VPLS, LISP
App
OS
App
OS
Virtual Physical
Fabric DB
VM
OS
VM
OS
Virtual Virtual
VM
OS
VM
OS
Physical Physical

8. Cisco Confidential 9© 2013-2014 Cisco and/or its affiliates. All rights reserved.
VXLAN Overview
Outer
MAC
DA
Outer
MAC
SA
Outer
802.1Q
Outer
IP DA
Outer
IP SA
Outer
UDP
VXLAN
ID
(24 bits)
Inner
MAC
DA
Inner
MAC
SA
Optional
Inner
802.1Q
Original
Ethernet
Payload
CRC
VXLAN Encapsulation Original Ethernet Frame
CRC
Data
Plane
16 M Segments
Control
Information
Tunnel Endpoints Discovery
Host Reachability Information
• Mac Address
• IP address
Draft
Data Plane Multicast based flood and learn

9. Cisco Confidential 10© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Eth Eth Eth
vEth vEth vEth vEth vEth vEth
Overlay Network Communications - VTEP
VXLAN utilizes a VTEP:
• Virtual Tunnel End Point
• IP address assigned
• Layer-3 Transportable
• IP/UDP Packets
10.10.10.101 10.10.10.211 172.18.22.12
Inter VXLAN communications
VTEP VTEP VTEP

10. Cisco Confidential 11© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Eth Eth Eth
vEth vEth vEth vEth vEth vEth
Overlay Network Communications
VXLAN 55110
VXLAN 45235
16m VXLANs
VXLAN requires a network gateway function:
• VXLAN to VLAN Bridge
• VXLAN to VLAN Router
• VXLAN to VXLAN Router
VLANs

11. Cisco Confidential 12© 2013-2014 Cisco and/or its affiliates. All rights reserved.
VXLAN54210
VXLAN Gateway Functions
VXLAN55110
VXLAN45235 VLAN 235
VLAN 110
VXLAN55110 VXLAN45235
VXLAN55110 VLAN 235
VXLAN to VLAN Bridging (L2 Gateway)
VXLAN-to-VXLAN Routing (L3 Gateway)
VXLAN-to-VLAN Routing (L3 Gateway)
VXLAN45235

12. Cisco Confidential 13© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Programmable NetworkProgrammable FabricApplication Centric
Infrastructure
DB DB
Web Web App Web App
VxLAN-BGP EVPN
standard-based
3rd party controller support
Modern NX-OS with enhanced
NX-APIs
Automation Ecosystem
(Puppet, Chef, Ansible etc.)
Common NX-API
across N2K-N9K
Turnkey integrated solution with
security, centralized management,
compliance and scale
Automated application centric-policy
model with embedded security
Broad and deep ecosystem
Cisco SDN: Providing Choice in Automation and Programmability
Mass Market
(commercial, enterprises, public sector)
Service Providers Mega Scale Datacenters
VTS for software overlay
provisioning and management
across N2K-N9K

13. Application Centric Infrastructure (ACI)

14. Cisco Confidential 15© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Two Types of Languages
Infrastructure Applications
Human
Translator
• Application Tier Policy and
Dependencies
• Security Requirements
• Service Level Agreement
• Application Performance
• Compliance
• Geo Dependencies
• VLAN
• IP Address
• Subnets
• Firewalls
• Quality of Service
• Load Balancer
• Access Lists

15. Cisco Confidential 16©2014 Cisco and/or its affiliates. All rights reserved.
Introducing: Application Centric Infrastructure
Apps + Infrastructure
Physical + VirtualOpen + Secure
On-Premises + Cloud
Application Oriented Policy = Operational Simplicity

16. Cisco Confidential 17© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Application Centric Infrastructure Components
Fabric
Centralized Policy Management
Open APIs, Open Source,
Open Standards
Policy Controller
Application Network Profile APIC
End Points
Physical
Networking
Nexus 2K
Nexus 7K
Hypervisors and
Virtual Networking
Compute L4–L7
Services
Storage Multi DC
WAN and Cloud
Integrated
WAN Edge
Virtual
Physical
Northbound
Management
Integration
Partner
Ecosystem
Automation
OVM
Hypervisor
Management Monitoring
Systems
Management
Orchestration
Framework

17. Cisco Confidential 18© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Typical Three Tier Application
Web
Servers
Firewall
Server Load Balancer
Access Switch
Server
vSwitch
Firewall
Access Switch
App
Servers
Database
Server
Application Requirements tightly coupled to the Network
Port Group, VLAN, IP Address, IP Mask
Interface, Trunk, VLAN,
IP Subnets
Interface, Trunk, VLAN,
IP Subnets
• Network Connectivity
• Security Policies
• Quality of Service
• Layer 4 – 7 Application
Services
• Storage Policies
• Compute Policies
• Hypervisor Policies

18. Cisco Confidential 19© 2013-2014 Cisco and/or its affiliates. All rights reserved.
The Policy-based Datacenter
IP Fabric
• Single APIC Controller:
• End-to-end Application
Profile
• ACI IP Fabric encompasses
o Infrastructure
o Physical
o Virtual
o Services
• ANP Profile pushed to all
components
• Full Workload Mobility, Replication
and Instantiation Application Network Profile
Web Servers App Servers Database Server

19. Cisco Confidential 20© 2013-2014 Cisco and/or its affiliates. All rights reserved.
ACI Benefit: Deep Telemetry — Application and Tenant
APIC
APP
TENANT
Tenant
Tenant 1 Tenant 2
Tenant 3 Tenant 4

20. Cisco Confidential 21© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Application Approach To Networking
F/W DB DBDecouple Policy from Infrastructure
Simple & Scalable Stateless Infrastructure
Optimized Forwarding & Mobility
Abstracted Policies for definition of
Applications & Connectivity
Open REST APIs
Centralized Management
Open Source
APIC
Application Network Profile
F/W F/W F/W
STORAGE STORAGE
WEB DBAPP
Highest Performance & Reliability
Lowest Power Consumption

21. Virtual Topology System (VTS)

22. Cisco Confidential 23© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Programmable Fabric
NX-API, VXLAN BGP EVPN Fabric, and Virtual Topology System (VTS)
Operations /
Programmability
& Automation
Automated
DCI / WAN
VM
OS
VM
OS
NX-API
Physical Virtual DCI/WAN
Bare Metal Virtualized
BGP-EVPN VXLAN Fabric
VTS
VTS for overlay provisioning and management across Nexus 2000 – Nexus 9000 (2H 2015)

23. Cisco Confidential 24© 2013-2014 Cisco and/or its affiliates. All rights reserved.
vCenter
REST API
VTS
GUI
Across Nexus Portfolio
Nexus 2K – 9K
Programmable Fabric
Automated
Seamless integration with Orchestrators
Overlay provisioning and DCI/WAN integration
Scalable VXLAN Management
MP-BGP EVPN control plane
High performance virtual forwarding
Open and Programmable
REST Northbound APIs
Multi-protocol and Multi-hypervisor support
Virtual Topology System (VTS)
Overlay Provisioning & Management System
Flexible Overlays
Physical and virtual overlays
Bare-metal and Virtualized workloads

24. Cisco Confidential 25© 2013-2014 Cisco and/or its affiliates. All rights reserved.
VXLAN as Data Center Overlay technology
VTEP
Local LAN Local LAN Local LAN Local LAN
IP Transport
Network
VTEP VTEP VTEP
VXLAN VNI
LAN Segment
Underlay Network:
• IP routing – proven, stable, scalable
• ECMP – utilize all available network paths
Overlay Network:
• Standards-based overlay
• Layer-2 extensibility and mobility
• Expanded Layer-2 name space
• Scalable network domain
• Multi-Tenancy
Modes of Operation:
• Multicast based flood and learn (No control plane)
• BGP EVPN (BGP control plane with MP-BGP Extensions)

25. Cisco Confidential 26© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Advantages with EVPN Control Plane
Industry standard protocol for multi-vendor support
Built-in Multi tenancy support
Truly scalable with protocol-driven control plane architecture
Fast convergence upon network failures and host movements
Minimize flooding through ARP suppression
Security through VTEP peer-authentication
AdvantagesofEVPN
ControlPlane

26. Cisco Confidential 28© 2013-2014 Cisco and/or its affiliates. All rights reserved.
VTF
Cisco Network Services
Orchestrator
VMware vCenter
GUI
DVS
Unified Information Model (REST API)
YANG CLI NX-API BGP-EVPN
Virtual Topology System
Service and Infrastructure Policy
Inventory
Database
Resource Management
PolicyPlaneControl
Plane
IOS XRv
Device Management
Control Plane Federation
MP-BGP
Cisco Nexus 2000, 3000,
5000, and 7000 Series
Cisco Nexus 9000 Series Cisco ASR 9000 Series
Virtual Compute Environment
VTS Architecture

27. Cisco Confidential 29© 2013-2014 Cisco and/or its affiliates. All rights reserved.
VTS Architecture
Cisco VTS
ToR ToR
Spine Spine
ToR
Hypervisor
VM
x86 Server
Hypervisor
VM
x86 Server
Hypervisor
VMVM
x86 Server
REST API
DCI
NX-API,
CLI, YANG
VTEP
VTEP VTEP
Border Leaf VTEP
VMware vCenter
Virtual Topology System
Service and Infrastructure Policy
Inventory
Database
Resource Management
PolicyPlaneControl
Plane
IOS XRv
Device Management

28. Cisco Confidential 30© 2013-2014 Cisco and/or its affiliates. All rights reserved.
VTS Architecture – Hardware Switches
ToR ToR
Spine Spine
ToRVTEP VTEP
Cisco VTS
Hypervisor
VM
x86 Server
Hypervisor
VM
x86 Server
Hypervisor
VMVM
x86 Server
REST API
DCI
NX-API,
CLI, YANG
VTEP
Border Leaf VTEP
VMware vCenter

29. Cisco Confidential 31© 2013-2014 Cisco and/or its affiliates. All rights reserved.
VTS Architecture - VTF
Cisco VTS
ToR ToR
Spine Spine
ToR
Hypervisor
VM
x86 Server
Hypervisor
VMVM
x86 Server
REST API
DCI
NX-API,
CLI, YANG
VTEP VTEP
Border Leaf VTEP
VMware vCenter
Hypervisor
VM
x86 Server
VTEP
vSwitch vSwitch
VTF
(VM)
Tenant VM Tenant VM
vSwitch
VTF
(VM)
Tenant VM Tenant VM
KVMESXi
NIC NIC
User space, Multi-tenant, line rate packet forwarder
Uses Vector Packet Processing technology
Fully integrated with Intel DPDK
Supports VXLAN, can be extended to support MPLSoGRE,
L2TPv3, MPLSoUDP, native MPLS and SR
Programmed by VTS using Restconf/YANG

30. Cisco Confidential 32© 2013-2014 Cisco and/or its affiliates. All rights reserved.
VTS Functionality
• Discover ToRs, Servers and interconnections
• Manage switch and network topology status
• Topology information via API or GUI
Discovery
• VXLAN Provisioning (BGP EVPN & Flood/Learn)
• VXLAN Overlay management (Add/Modify/Delete)
• Multi-tenancy support
• Track and Update VNIDs as VM moves
• Network facing resource management
Provisioning
• Tenant to VNID mappings and VNID status
• VNID to VTEP mappings
• VTEP to VLAN and end host mapping
• Trace VMs connected to VTEP
• VTEP status within a VNID
• VXLAN and fabric statistics
Overlay Visibility

31. Unified Open NX-OS

32. Cisco Confidential 34© 2013-2014 Cisco and/or its affiliates. All rights reserved.
ExtensibilityAuto Deployment
Options
Open Application
Integration
Programmability
Tool Choice
DevOps
Enabling
POAP NXAPI
Yocto
SDK
Standard Open
Interfaces
Open
Interfaces
Automation
and Visibility
Adaptable
NXOS
Adaptable
SDK
ProgrammableBootStrap and
Provisioning
BootStrap/
Provisioning
Package and
Application
Management
Native
Application
Integration
PXE
Data
Models
Server
Management
Tools
OPEN NX-OS - Extensible, Open, Programmable
34

33. Cisco Confidential 35© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Open NX-OS: Infrastructure Layer Enhancements
35
OPEN BOOTLOADERS & PROVISIONING
OPEN PACKAGE/APPLICATION INTEGRATION
OPEN INTERFACES
OPEN OBJECT BASED API’s (NX-API, Model Driven)
Open NX-OS consistent across
both ToR and Modular
Open NXOS

34. Cisco Confidential 36© 2013-2014 Cisco and/or its affiliates. All rights reserved.
VTEP VTEP VTEP VTEP
• Leverage existing compute deployment
infrastructure (PXE/iPXE) for
operationalizing NX-OS
• Deploy NX-OS from a web server via
HTTPS or TFTP server with support for
both IPv4 and IPv6
• NX-OS CLI option added to select boot
option either <bootflash(default) > or
<pxe>
Boot Server(DHCP &
HTTP/TFTP)
NX-OS Image Repository
DHCP
DISCOVER(v4/v6)
IP Address &
File/Image URL
TFTP GET
FILE/HTTP
URL
http://n9k-
dk9….bin..
Validate
Image
Checksum &
Boot
Open NX-OS Bootloaders & Provisioning
iPXE
36

35. Cisco Confidential 37© 2013-2014 Cisco and/or its affiliates. All rights reserved.
• Ability to third party packages in
Secure Guestshell or natively in NX-
OS kernel
• Install all third party applications
(Puppet/Chef, etc) as RPMs
• Daemon managed via standard Linux
interfaces
• Built-in support for YUM package
manager
• Patching and upgrade using standard
rpm/yum workflows
• NX-OS processes(BGP) can be
upgraded/patched via “yum update”
37
Package as RPM
C app with
standard Linux
constructs
Open Embedded
64 bit Build
Environment
Cisco/Local
Repository
RPM local
repository
RPM upload
YUM Install
Linux Daemon
Linux Kernel
• Raw Socket
• Netdevs
• Libpcap
init.d
Monitoring
server
ASIC
Build Server Target Switch
Open NX-OS Package Management via YUM/RPM
LXC and Native Daemons

36. Cisco Confidential 38© 2013-2014 Cisco and/or its affiliates. All rights reserved.
• Leverage Linux command toolkit for monitoring
configuration and troubleshooting
• # tcpdump -w file.pcap -i eth1-1
• Use ethtool to display detailed interface
statistics:
• #ethtool –S eth2-1
• Use ifconfig to change mtu for an interface to
jumbo MTU:
• #ifconfig eth2-1 mtu 9000
• Use ip route to add a static route for a given
interface:
• #ip route add 203.0.113.0/24 via
198.51.100.2 dev eth2-1
• Leverage bash for NX-OS scripting automation
• vsh –c “show interface brief” | grep
up | awk/sed
38
Open NX-OS Linux Interfaces
Bash Access

37. Cisco Confidential 39© 2013-2014 Cisco and/or its affiliates. All rights reserved. 39
• Tool provides a convenient way
for network engineers to get up
to speed with scripting and
automation via web browser
interface
• Available on all Nexus
platforms.
• CLI commands embedded in
structured input and output
(JSON/XML) via HTTP/HTTPS
• Use “feature nxapi” to enable
access on the platform
Open NX-OS Programmability
NX-API Developer Sandbox

38. Cisco Confidential 40© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Updates on Nexus Portfolio Offerings
Programmable Network
Application Centric
Infrastructure
NEW! Unified Open NX-OS Release for
Nexus 3000 and Nexus 9000 (Q3 2015)
• Enhancements to NX-API – object store
and model driven
• Native 3rd party RPM applications
integration (tcollector, Nagios, Ganglia,
Puppet / Chef etc.)
• Linux utilities support for seamless tool
integration across compute and network
• SDK for custom application integration
NEW! ACI Release for Nexus 9000
(Shipping June 2015 )
• Microsoft Azure and System Center
Integration
• Programmability examples: vCenter
plug-in, ACI toolkit etc.
• Simplified operations
• Stretched fabric, multiple destinations
from 30KMs to 150KMs
• Group-based policy on Openstack
• New ACI ecosystem partners (CliQr)
DB
DB
Web Web App Web App
NEW! Common NX-API across N2K-
N9K (2H 2015)
Programmable Fabric
NEW! Virtual Topology System
(VTS) for software overlay
provisioning and management
across for Nexus 2K-9K (2H
2015)
• Standards-based fabric
support on Nexus 5600/7x00
with VXLAN BGP EVPN
(shipping with Nexus 9000
today)
VTS

39. Cisco Confidential 41© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Nexus 9000® Series
Your Deployment, This Makes it Happen!
Cisco Nexus 9300
Platform Fixed Switches
NX-OS and ACI
Choice of Fabric Architectures
Feature Consistency with Silicon Innovations
Cisco Nexus 9500
Platform Modular Switches
Nexus 9516 – Best of Interop Data Center 2014
APIC – Best of Interop SDN 2015

40. In partnership with: