Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Operationalizing VRF in the Data Center

3,535 views

Published on

This presentation gives an overview of why we need VRF in the data center and operational workflow of VRF

Published in: Technology

Operationalizing VRF in the Data Center

  1. 1. v Operationalizing VRF in the Data Center Ivan Pepelnjak, David Ahern, Dinesh G Dutt June 23, 2016
  2. 2. Agenda Why Do We Need VRF in the DC VRF in the Linux Kernel Operational Workflow of VRF June 29, 2016 cumulusnetworks.com 2
  3. 3. Key Takeaways VRF is a first class citizen in Linux Uniform model across hosts and routers  Pushing routing all the way to the edge Cumulus Linux 3.0 release supports all the familiar constructs around VRF usage June 29, 2016 cumulsnetworks.com 3
  4. 4. Agenda Why Do We Need VRF in the DC VRF in the Linux Kernel Operational Workflow of VRF June 29, 2016 cumulusnetworks.com 4
  5. 5. 5 © ipSpace.net 2016 VRF 101 What Is a Virtual Routing and Forwarding (VRF) Table? VRF = independent routing instance • Routing protocols, routing and forwarding table • Management plane is common • Exposure to DoS is shared • Inter-device connectivity with trunking interfaces, overlays or MPLS VRF-capable router Management Trunking interface or MPLS OSPF topology BGP table OSPF process BGP process IP routing table IP forwarding table
  6. 6. 6 © ipSpace.net 2016 VRF 101 Typical Use Case: Management VRF • Management plane processes attached to management VRF • User VRF does not have a route to management IP address • End-users cannot reach management plane even though they have IP connectivity to the switch • Simpler than protecting every switch port with ACLs VRF-capable switch Management plane Fabric Management VLAN
  7. 7. 7 © ipSpace.net 2016 VRF 101 Layer-3 Path Isolation (VRFs) Can Replace VLANs Facts: • VLANs are used in data center networks as a security zone mechanism • Layer-3 path isolation (L3VPN) provides the same (or better) isolation • L3VPN scales better than VLANs Use cases: • Retain separation while connecting L2 pods over L3 core (DC or WAN) • Combine ACLs with firewalls • Complex traffic forwarding policies
  8. 8. 8 © ipSpace.net 2016 VRF 101 Example: Multi-Tier Application with VRFs Each segment needs a different default gateway • Layer-2-only solution or multiple routing domains You might need routing within a segment (example: divert backup traffic directly to backup server) • Multiple layer-3 forwarding domains are the only option Implementations: • MPLS/VPN • VRF Lite (or Cisco EVN) • EVPN with VXLAN Outside Web servers App servers DB servers Default gateway Default gateway Default gateway
  9. 9. 9 © ipSpace.net 2016 VRF 101 Layer-3 Path Isolation in DCI Environment Replace stretched VLANs with L3VPN • Retain separation of security zones • Cannot solve hot VM mobility or stretched L2 clusters • Terminate L2 domains within data center • Use parallel VRFs, MPLS/VPN or EVPN (over MPLS or VXLAN) Parallel VRFs also known as: VRF-Lite, Multi VRF, EVN (Cisco), MCE (HP) DWDM, ME, IP Data Center A DC core Data Center B DC core
  10. 10. Agenda Why Do We Need VRF in the DC VRF in the Linux Kernel Operational Workflow of VRF June 29, 2016 cumulusnetworks.com 10
  11. 11. cumulusnetworks.com Linux Implementation for VRF Linux Implementation  Kernel v4.3 and forward Developed by Cumulus Networks for Linux  Consistent API across all Linux Devices - switches and hosts
  12. 12. cumulusnetworks.com 12 Distribution Kernel Version VRF status Capabilities Cumulus Linux 3.0 4.1 enabled All + Mgmt VRF Debian - stretch 4.6 enabled IPv4, IPv6 global, “VRF All” TCP sockets Ubuntu 16.04 4.4 enabled IPv4, IPv6 global Fedora 24 (May beta) 4.5 disabled none Fedora 23 4.5 disabled none Linux OS Distributions (June 2016)
  13. 13. Feature by Linux Kernel Version 13 Version Feature 4.3 Basic IPv4 support, FIB tracepoint 4.4 Basic IPv6 support (global addresses only) 4.5 “VRF-all” sockets, FIB6 tracepoint 4.6 Bug fixes, IPv6 global addresses on admin down 4.7 Ingress device for IP_PKTINFO, IP6_PKTINFO 4.8 IPv6 linklocal and multicast, local VRF traffic, FIB rule cumulusnetworks.com v4.8 == Cumulus Linux 3.1
  14. 14. cumulusnetworks.com What about Network Namespaces? Wrong Model for VRFs  VRF is a Layer 3 separation  Network Namespace is a device layer separation 14 Default namespace mgmt namespace swp4 swp5 swp6 “red” namespace neigh swp2 swp3swp1 eth0 FIBsockets neigh FIBsockets neigh FIB sockets lldpd ipquaggaptmd collectd
  15. 15. cumulusnetworks.com Overview of VRF Design for Linux VRF represented as a virtual (pseudo) network device  VRF device correlates to routing table  FIB rules for VRF device direct lookups to VRF table  Network interfaces are associated with VRF device  Host and connected routes moved to VRF table L3-only concept  Impacts only IPv4 and IPv6 route and socket lookups
  16. 16. cumulusnetworks.com Advantages of Device Model Network interface-to-VRF association similar to bridges  familiar paradigm for Linux networking 16
  17. 17. cumulusnetworks.com Advantages of Device Model Network interface-to-VRF association similar to bridges netdevice is a fundamental construct in Linux networking stack  Network addresses on VRF device VRF-local loopback (127.0.0.1 and unnumbered deployments)  netfilter, tc, tcpdump on VRF device 17
  18. 18. cumulusnetworks.com Advantages of Device Model Network interface-to-VRF association similar to bridges netdevice is a fundamental construct in Linux networking stack VRFs can exist inside network namespaces  Follows existing paradigms for network interfaces and namespaces 18
  19. 19. cumulusnetworks.com Advantages of Device Model Network interface-to-VRF association similar to bridges netdevice is a fundamental construct in Linux networking stack VRFs can exist inside network namespaces Applications use existing socket APIs  SO_BINDTODEVICE, IP_PKTINFO 19
  20. 20. cumulusnetworks.com Advantages of Device Model Network interface-to-VRF association similar to bridges netdevice is a fundamental construct in Linux networking stack VRFs can exist inside network namespaces Applications use existing socket APIs Existing frameworks for monitoring, serviceability  iproute2 commands, netlink 20
  21. 21. Application Behavior With VRF  Applications by default run in the default VRF context  Applications that support binding to a device, can run within a specified VRF  Examples are ping, traceroute  Applications that don’t support binding to a device need a helper to set context to correct VRF  Cumulus Linux has “vrf exec” to do this, in the process of upstreaming  LD_PRELOAD is another option  Applications that work in multiple VRFs concurrently need to be modified  Example is BGP June 29, 2016 Cumulus Networks Confidential 21
  22. 22. Specifying VRF in Applications Use existing POSIX APIs  VRF context by binding sockets to device or cmsg setsockopt(SO_BINDTODEVICE), IP_PKTINFO 22cumulusnetworks.com
  23. 23. Agenda Why Do We Need VRF in the DC VRF in the Linux Kernel Operational Workflow of VRF  Interface configuration  Management VRF in Cumulus Linux  Routing June 29, 2016 cumulusnetworks.com 23
  24. 24. cumulusnetworks.com Cumulus Linux 3.0 Workflow: VRF Creation & Slave Port Attach Define configuration in /etc/network/interfaces 24 red FIB table 1001 swp1 swp2 OR auto all iface red address 127.0.0.1/8 vrf-table auto iface swp1 vrf red iface swp2 vrf red auto all iface red address 127.0.0.1/8 vrf-table auto %for i in range(1,4): iface swp${i} vrf red %endfor
  25. 25. cumulusnetworks.com Simplified Linux Workflow (Ubuntu 16.04) 25 Create VRF device with table id Setup FIB rules pointing to device Add default route for VRF Enslave interfaces ip link add red type vrf table 123 ip link set red up ip rule add iif red table 123 ip rule add oif red table 123 ip -6 rule add iif red table 123 ip -6 rule add oif red table 123 ip route add table 123 unreachable default metric 8192 ip -6 route add table 123 unreachable default metric 8192 ip link set eth1 master red Ip link set eth2 master red
  26. 26. VRF Implementation in Cumulus Linux VRF name is purely an operator's choice.  Exception is Management VRF; must be “mgmt” Table id’s limited to 1001-1255  Hardware considerations + future growth in number of VRFs Table-id to name mapping is saved  allows for name-based reference in iproute2 commands June 29, 2016 cumulusnetworks.com 26
  27. 27. Agenda Why Do We Need VRF in the DC VRF in the Linux Kernel Operational Workflow of VRF  Interface configuration  Management VRF in Cumulus Linux  Routing June 29, 2016 cumulusnetworks.com 27
  28. 28. Management VRF June 29, 2016 cumulusnetworks.com 28 Management VRF  Uses non-main table  Name must be “mgmt”  Loopback address on mgmt device Front panel ports default to “Default VRF” (main table) auto all iface mgmt address 127.0.0.1/8 vrf-table auto iface eth0 inet dhcp vrf mgmt
  29. 29. Management VRF and Commands Commands need to specify VRF to use – in general  ping and traceroute wrappers default these commands to front panel ports and default VRF • Use ‘ping –I <vrf> …’ and ‘traceroute –i <vrf> …’ to specify VRF of interest  Helper for commands that do not support bind to device or interface options: vrf exec <vrf> <command> • Remember, affects IPv4 and IPv6 sockets only! June 29, 2016 cumulusnetworks.com 29
  30. 30. Management VRF: Default VRF “Context” Login context defaults to Management VRF  IPv4/IPv6 sockets opened by processes automatically bound to Management VRF device  Do not need to specify VRF for commands unless a different VRF is wanted (except ping and traceroute because of the wrappers)  Default PS1 bash prompt provides VRF context cumulus@cel-redxp-99:mgmt-vrf:~$ June 29, 2016 cumulusnetworks.com 30
  31. 31. Enabling Services in VRFs systemd based services can be run in a VRF context  systemctl <action> <service>@<vrf> • systemctl enable ntp@mgmt. • systemctl start ntp@mgmt  Possible collision with port if service is running in default VRF • Process in default VRF with wildcard bind owns port across all VRFs Loopback address on Management VRF allows ntpq to work Cumulus Linux specific, upstream WIP June 29, 2016 cumulusnetworks.com 31
  32. 32. Agenda Why Do We Need VRF in the DC VRF in the Linux Kernel Operational Workflow of VRF  Interface configuration  Management VRF in Cumulus Linux  Routing June 29, 2016 cumulusnetworks.com 32
  33. 33. Routing Support for VRF Routing protocol constructs (routes, nexthops, router-id etc.) and processing are VRF-aware.  Cumulus Quagga routing protocol suite Static routes (IPv4 and IPv6) supported in a VRF June 29, 2016 cumulusnetworks.com 33
  34. 34. BGP Support for VRF Per-VRF neighbors  EBGP/IBGP  Numbered & Unnumbered Per-VRF parameters (address-families, redistribution etc.) BFD support (single / multihop) June 29, 2016 cumulusnetworks.com 34
  35. 35. VRF provisioning - Quagga ‘bgp multiple-instance’ is ON by default 3/13/16 cumulusnetworks.com 35 router bgp 64900 vrf red bgp router-id 6.0.3.7 network 20.7.2.0/24 network 20.7.3.0/24 neighbor 10.1.1.2 remote-as 20 neighbor 20.1.1.2 remote-as 30 ! ! router bgp 64900 vrf blue bgp router-id 18.0.4.7 neighbor CUSTPEER peer-group neighbor CUSTPEER remote-as external neighbor swp1.10 peer-group CUSTPEER ! address-family ipv4 unicast redistribute connected exit-address-family !
  36. 36. VRF Verification and Troubleshooting – Quagga Various existing commands are VRF-aware  show ip route [vrf <all | name>]  show ip bgp [vrf <name>] summary  show interface [vrf <name>] Couple of new commands  show vrf  show bgp vrfs Existing Quagga ‘debug’ commands cover VRF too June 29, 2016 cumulusnetworks.com 36
  37. 37. VRF Hardware Acceleration Supported on all switching silicon chipsets:  Broadcom chipsets – T+, T2, Tomahawk  Mellanox Spectrum Kernel table id to VRF id mapping  VRF_id = table_id – 1000  Except for default VRF which uses VRF 0 June 29, 2016 cumulusnetworks.com 37
  38. 38. Scale Numbers in Cumulus Linux 3.0 Maximum VRF supported is 64 No per-VRF limit on interfaces, neighbors, routes etc. June 29, 2016 cumulusnetworks.com 38
  39. 39. Routing Protocol Caveats in CL 3.0 Only BGP supported  OSPFv2 and OSPFv3 are not VRF-aware Each VRF table in the kernel needs a default route  keeps lookups from dropping to the next table  high metric allows overrides June 29, 2016 cumulusnetworks.com 39
  40. 40. Demo Topology June 29, 2016 cumulusnetworks.com 40  Show ping and traceroute  Traffic takes the shortest path  Node L1 is real switch  Rest are VMs  VRF Loopbacks are assigned as:  10.253.<vlan>.<nodeid>  Non-VRF loopbacks are assigned as:  10.252.<>.<> S1 S2 S3 L1 L2 NL1 NL2 FW1 FW2 H1 H2 H3 H4 10.7.x.3 10.7.x.4 10.8.x.5 10.8.x.6 10.252.0.210.252.0.1 10.253.2.9 10.253.3.9 10.253.4.9 10.253.2.10 10.253.3.10 10.253.4.10 10.253.2.11 10.253.3.11 10.253.4.11 10.253.2.13 10.253.3.13 10.253.4.13 10.253.2.7 10.253.3.7 10.253.4.7 10.253.2.8 10.253.3.8 10.253.4.8
  41. 41. cumulusnetworks.com References ifupdown2 support for Debian/Ubuntu  ifupdown2 with VRF support to be released soon  https://github.com/CumulusNetworks/ifupdown2  https://github.com/CumulusNetworks/ifupdown2/blob/master/a ddons/vrf.py 41
  42. 42. Summary VRF is a first class citizen in Linux now  For example, distributions such as Ubuntu 16.04 support it VRF design is such that it takes advantage of existing socket APIs Routing support is via BGP and static routes  Familiar configuration model June 29, 2016 Cumulus Networks Confidential 42
  43. 43. June 29, 2016 cumulusnetworks.com 43 •Routing on the Host: Concepts & Customer Case Studies •Guest Speaker: Ivan Pepelnjak •When: July 21 Next Month’s Webinar
  44. 44. © 2016 Cumulus Networks. Cumulus Networks, the Cumulus Networks Logo, and Cumulus Linux are trademarks or registered trademarks of Cumulus Networks, Inc. or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners. The registered trademark Linux® is used pursuant to a sublicense from LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide basis. ThankYou! cumulusnetworks.com 44June 29, 2016

×