Networking in the Cloud Age!         With references to Apache CloudStack!                      !              December 11...
Agenda!•    Why virtual networks?!•    Basic principles of Cloud Networking!•    Service insertion in virtual networks!•  ...
Apache CloudStack!                              •  Secure, multi-tenant cloud                                 orchestratio...
Apache CloudStack!                              •  Open Source!                                 •  Apache License!        ...
Networking in the Cloud Age!DRIVERS!
Drivers!                                                                          New-style!IAAS!                         ...
Traditional Style!                                                                          Traditional!IAAS!             ...
Traditional infra can be IAAS!          IAAS!                  Agility!                                             Gold-p...
Traditional!                         Cloud!                                                     •  10x more               ...
Defining Cloud Computing (IAAS)!•  Agility!   –  Re-provision complex infrastructure topologies in      minutes, not days• ...
Defining Cloud Computing (IAAS)!•  Scalability!   –  Ability to consume resources limited by budget, not      by infrastruc...
Cloud Networking              Requirements!•  Agile!   –  Complex networking topologies created by non-      network engin...
Cloud Networking              Requirements!•  Scalability!   –  Usually means L3 in the physical infrastructure•  Elastici...
Cloud-Style Workloads!•  Low cost!   –  Standardized, cookie cutter infrastructure   –  Highly automated and efficient•  L3...
Scale!“At scale, everything breaks”!            -­‐	  Urs	  Hölzle,	  Google!  " " "!                                     ...
Redundancy helps a little!                                           • Bugs in failover      40%!                         ...
Reliability Strategies!                         Cloud workloads!    Traditional-Style!                      New (“Amazon”)...
Reliability Styles!    Traditional workload!                   Cloud workload! Link aggregation!                   VM back...
Traditional Enterprise network!                   Backbone/                    Internet!                                Co...
Enterprise networks!•  Hierarchical tree structure!   –  Assumes N-S traffic predominant•  L2 domains!   –  Susceptible to ...
Scaled out network!                                         Backbone/                                          Internet!  ...
Scaled out network!•  L3 (routed) network!  –  ECMP for increased bandwidth/redundancy•  No oversubscription!  –  Uniform ...
Networking in the Cloud Age!VIRTUAL NETWORKINGPRINCIPLES!
The illusion of isolated networks on top ofshared physical infrastructure!
Usually requires!•  Hypervisors!   –  To share the same host with multiple tenants•  Virtual (software) switches!   –  Por...
Virtual-to-Physical Mapping!•  Option 1: VLAN!  –  1 virtual network = 1 VLAN in physical infra  !•  Option 2: Tunnels!  –...
Virtual-to-Physical Mapping!•  Option 3: IP address re-write!  –  1 tenant address mapped to 1 different provider     addr...
Virtual Switches!•  Linux bridge!  –  KVM, XenServer, XCP, Oracle VM•  Open vSwitch (OVS)!  –  KVM, XenServer, XCP•  VMwar...
Virtual Switches!    Hypervisor Host!         VM A1!        VM A2!    VM B1!      VM C1!                                  ...
Egress Traffic from VM!    Ethernet frame from VM A1 to vswitch (untagged)                                                 ...
Ingress Traffic to VM! From physical nic to vswitch( tagged)                                                             Pa...
VLAN networking!Trunks!              Trunks!                         Trunks!                               User	          ...
12 bits tag          =4094 virtual networks
VLANs – other problems!•  Configuration complexity!  –  Need to program switches carefully•  Large L2 domains!  –  Broadcas...
Tunnels!•  Map VM address (Tenant Address) to Physical   address (PA) of Hypervisor!  –  Software IPv4 tunnels between hyp...
GRE tunnel example!                                 GRE	  Key	  1	                GRE	  Key	  2	                         O...
GRE Example!Hypervisor 1!                                                                                        Hyperviso...
Layer 3 cloud networking!            Web	                               DB	                                Web	           ...
L3 isolation with distributed firewalls!                                                                                   ...
Networking in the Cloud Age!SERVICES!
Virtual Network Services!•  Provide L2-L7 network services that   applications expect:!  –  Load balancing, firewall, IDS, ...
Virtual Network Appliances!Network services are often provided by virtual appliances.!These are either commercial applianc...
Service insertion example!                                                           Tenant 1 Virtual Network 10.1.1.0/24 ...
Service insertion with VLAN !Trunks!                                Trunks!                                           Trun...
Network Services!    Network    Services!•  L2   connectivity!•  IPAM!•  DNS!•  Routing!•  ACL!•  Firewall!•  NAT!•  VPN!•...
Network Services!    Network           Service    Services!        Providers!•  L2              ü  Virtual   connectivity...
Network Services!    Network           Service             Network    Services!        Providers!           Isolation!•  L...
Service Catalog!•  Cloud users are not exposed to the nature of   the service provider!•  Cloud operator designs a service...
End-user experience!•  Deploy a VM in a network!  –  VM Template = Windows 2008 with Joomla     on VMWare!  –  Service off...
End-user experience!•  Deploy a VM in a network!  –  VM Template = Windows 2008 with Joomla on VMWare  –  Service offering...
End-user experience!•  CloudStack orchestration:!  –  Pick a free VLAN, pick a free public IP, free private IP  –  Pick hy...
Network services with VLANs!                                Tenant 1 Virtual Network 10.1.1.0/24                          ...
Network virtualization with VLANs!                                                      Tenant 1 Virtual Network 10.1.1.0/...
Network virtualization with VLANs!                                                         Tenant 1 Virtual Network 10.1.1...
Service insertion with VLANs!                                                               Tenant 1 Virtual Network 10.1....
Scaling services with VLANs!  Scale out edge services using virtual appliances!                                 10.1.1.0/2...
Scaling services with VLANs!  Scale out edge services using virtual appliances!              Scale up using hardware devic...
Multi-tier virtual networking!                            Internet!                                                      !...
Multi-tier virtual networking!                             Internet!                                                      ...
Multi-tier virtual networking!                             Internet!                                                      ...
Multi-tier networking with                   Overlay!                            Internet!                                ...
Multi-tier networking with            Overlay!                 Internet!Loadbalancer                                vswitc...
Networking in the Cloud Age!LAYER 3 ISOLATION!
Layer 3 cloud networking!          Web                             DB                           Web          VM!          ...
L3 isolation with distributed firewalls!                                                                                   ...
L3 isolation with distributed firewalls!                                                                                   ...
L3 isolation with distributed firewalls!                                                                                   ...
1 Firewall per Virtual Machine
A Million Firewalls?!VM!       VM!     VM!…!        …!                  VM!        VM!                  …!          …!VM! ...
Networking in the Cloud Age!SOFTWARE DEFINEDNETWORKING!
Definition!•  Separation of Control Plane from the hardware  performing the forwarding function!•  Control plane is logical...
SDN Advantages!•  Centralized control makes it easier to   configure, troubleshoot and maintain•  Eliminates ‘box’ mode of ...
Related to SDN!•  API layer over a collection of ‘boxes’!   –  API layer communicates with boxes using box-level      APIs...
Centralized control plane!Admin/User	  API	                  Controller	  Cluster	                          MySQL/NoSQL	  ...
SDN problems!•  Discovery of virtual address -> physical   address mapping!  –  VxLAN = multicast  –  GRE = programmed by ...
SDN problems!•  State maintenance!  –  Large number of endpoints + flows  –  High arrival rate of new flows  –  Needs fast a...
CloudStack and SDN!                                                                                                       ...
Virtual Networking "!NETWORKING IN APACHECLOUDSTACK!
Regions and Zones!•  A cluster of CloudStack management servers   manage the physical resources of a region   –  Single AP...
Region “West”             Region “East”                     Geographic                      separation                    ...
Region “West”                        Zone “West-Beta”   Zone “West-Alpha”                                   High Speed Bac...
Inside a zone!   Admin/User	  API	                                                       End	  users	                     ...
Orchestration!•  Orchestration describes the automated   arrangement, coordination, and management of   complex computer s...
CloudStack Architecture!                                               Hypervisor	                                        ...
CloudStack Architecture!                                                                 •   XenServer	                   ...
CloudStack Architecture!                                                                                                  ...
Problem:Manage Configuration of!  1000s of virtual appliances (or VRF)  Dozens of HW appliancesSolution:Database-driven sta...
Problem:Manage Configuration of!  1000s of virtual appliances (or VRF)  Dozens of HW appliancesSolution:Database-driven sta...
Problem:!Single-tenant HW appliances!Solution:!CloudStack API layers multi-tenancy, providesabstraction!   No direct acces...
Problem:Hardware appliances with no APIs   CLI only   Limited concurrent login sessionsSolution:  Recommend appliances wit...
Problem:Manage the configuration of 100s of thousands of firewallsSolution:Well-known software scaling techniques•  Message ...
Problem:Firewall (iptables) rules explosion on the host firewall!    Allow Security Group {Web} on TCP port 3060 !    !-A F...
Problem: Firewall (iptables) rules explosion on the host firewall     ! Solution:! Use ipsets: !ipset   –N   web_sg iptreem...
Apache CloudStack!•  Apache CloudStack!  –  http://www.cloudstack.org/!  –  Download it!  –  Use it!  –  Contribute to it!...
Upcoming SlideShare
Loading in...5
×

Networking in the Cloud Age (LISA 2012 Tutorial)

10,214

Published on

I (along with David Nalley - ke4qqq) presented a tutorial on Cloud Networking for cloud operators (not users) at LISA 2012

Published in: Technology
0 Comments
16 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
10,214
On Slideshare
0
From Embeds
0
Number of Embeds
8
Actions
Shares
0
Downloads
0
Comments
0
Likes
16
Embeds 0
No embeds

No notes for slide

Transcript of "Networking in the Cloud Age (LISA 2012 Tutorial)"

  1. 1. Networking in the Cloud Age! With references to Apache CloudStack! ! December 11 2012! ! Chiradeep Vittal! @chiradeep! David Nalley! @ke4qqq!
  2. 2. Agenda!•  Why virtual networks?!•  Basic principles of Cloud Networking!•  Service insertion in virtual networks!•  Virtual Networking using L3 isolation!•  Networking in Apache CloudStack!•  Software Defined Networking!•  Wrap-up!
  3. 3. Apache CloudStack! •  Secure, multi-tenant cloud orchestration platform! –  Turnkey platform for delivering IaaS clouds! –  Over 150 commercial Build your cloud the way deployments: private and public!the world’s most successful –  Full featured GUI, end-user API clouds are built! and admin API!
  4. 4. Apache CloudStack! •  Open Source! •  Apache License! •  Incubating in the Apache Software Foundation since April 2012! Build your cloud the waythe world’s most successful •  Open Source since May clouds are built! 2010! •  In production since 2009!
  5. 5. Networking in the Cloud Age!DRIVERS!
  6. 6. Drivers! New-style!IAAS! Workload! Agility! Application owns availability! Virtualization! API! High bandwidth! Self-service! Elasticity! Scale! Low cost! Distributed! L3! Cookie cutter! Multi-tenancy!These classes of drivers (IAAS and new-style workloads) are highlycomplementary and therefore most new-style applications operate on IAAS!
  7. 7. Traditional Style! Traditional!IAAS! Workload! Agility! Infra owns availability! Virtualization! API! Complex Packet Filters! Elasticity! Scale! High cost! Self-service! Gold-plated! Multi-tenancy! Infra! L2!It is possible to realize some of the benefits of IAAS for traditional workloads !
  8. 8. Traditional infra can be IAAS! IAAS! Agility! Gold-plated! Virtualization! Infra! API! Infra owns availability! Elasticity! Scale! High cost! Self-service! Multi-tenancy! L2! Complex Packet Filters!It is possible to realize some of the benefits of IAAS for traditional infrastructure!
  9. 9. Traditional! Cloud! •  10x more scaleable! •  2-5x lower cost! •  100% more open!Built for traditional Designed around big data,enterprise apps & client- massive scale & next-genserver compute! apps!•  Enterprise arch for 100s of •  Cloud architecture for 1000s hosts! of hosts!•  Scale-up (server clusters) ! •  Scale-out (multi-site server•  Apps assume reliability! farms)!•  IT Mgmt-centric [1:Dozens]! •  Apps assume failure!•  Proprietary vendor stack! •  Autonomic [1:1,000’s]! •  Open, value-added stack!
  10. 10. Defining Cloud Computing (IAAS)!•  Agility! –  Re-provision complex infrastructure topologies in minutes, not days•  API! –  Automate complex infrastructure tasks•  Virtualization! –  Enables workload mobility and load sharing•  Multi-tenancy! –  Share resources and costs
  11. 11. Defining Cloud Computing (IAAS)!•  Scalability! –  Ability to consume resources limited by budget, not by infrastructure•  Elasticity! –  Scale up and down on demand –  Reduce need to engineer for peak load•  Self-service! –  No IT assistance!
  12. 12. Cloud Networking Requirements!•  Agile! –  Complex networking topologies created by non- network engineers•  API! –  Language to talk with the network infrastructure layer (not CLI)•  Virtualization! –  Hypervisor-level switches work together with physical infrastructure
  13. 13. Cloud Networking Requirements!•  Scalability! –  Usually means L3 in the physical infrastructure•  Elasticity! –  Release resources when not in use –  Introduce new resources on demand•  Self-service! –  Novices deploying, maintaining, troubleshooting virtual networks
  14. 14. Cloud-Style Workloads!•  Low cost! –  Standardized, cookie cutter infrastructure –  Highly automated and efficient•  L3! –  Applications do not need persistent ip/mac –  L2 adjacency not required•  Application owns availability! –  At scale everything breaks –  Focus on MTTR instead of MTBF
  15. 15. Scale!“At scale, everything breaks”! -­‐  Urs  Hölzle,  Google! " " "! Server failure comes from:! ᵒ  70% - hard disk! 8%   ᵒ  6% - RAID controller! ᵒ  5% - memory! ᵒ  18% - other factors! Application can still fail for Annual  Failure  Rate  of  servers   other reasons:! ᵒ  Network failure!Kashi  Venkatesh  Vishwanath  and  Nachiappan  Nagappan,  Characterizing   ᵒ  Software bugs!Cloud  Compu3ng  Hardware  Reliability,   ᵒ  Human admin error!SoCC’10  
  16. 16. Redundancy helps a little! • Bugs in failover 40%! mechanism! • Incorrect configuration! • Protocol issues such as TCP back-off, timeouts, and Effectiveness of network redundancy in reducing spanning tree failures! reconfiguration! Phillipa Gill, Navendu Jain & Nachiappan Nagappan, Understanding Network Failures in Data Centers: Measurement, Analysis and Implications, SIGCOMM 2011 !16!
  17. 17. Reliability Strategies! Cloud workloads! Traditional-Style! New (“Amazon”) Style! Reliable hardware, backup Tell users to expect failure. entire cloud, and restore for Users to build apps that can users when failure happens! withstand infrastructure failure!Both styles of workloads must run reliably in the cloud!
  18. 18. Reliability Styles! Traditional workload! Cloud workload! Link aggregation! VM backup/snapshots ! Storage multi-pathing! Ephemeral resources! VM HA, fault tolerance! Chaos monkey! VM live migration! Multi-site redundancy!Expect reliability. Back-up entire Expect failure. Design app for failure. cloud. Admin controlled failure Self-service failure handling! handling! Think Amazon Web Services! Think Server Virtualization!
  19. 19. Traditional Enterprise network! Backbone/ Internet! Core Routers!N-S traffic! …! Access Routers! Packet Filters! Aggregation Switches! Load Balancers! …! Top of Rack Switches! Servers!
  20. 20. Enterprise networks!•  Hierarchical tree structure! –  Assumes N-S traffic predominant•  L2 domains! –  Susceptible to flooding –  Wasted capacity due to STP•  Services provided by redundant HW appliances! –  Firewall, IDS, ACL, Loadbalancer –  Often need L2 adjacency!•  Complex engineering, limited scale!
  21. 21. Scaled out network! Backbone/ Internet! Spine Routers! Leaf Routers! …! Servers!Host-based! Server Load Balancing!firewalls and ACL!
  22. 22. Scaled out network!•  L3 (routed) network! –  ECMP for increased bandwidth/redundancy•  No oversubscription! –  Uniform access to bandwidth•  Predominantly east-west traffic!•  Commodity hardware!•  Services provided at the host / vm level! –  Firewall, IDS, load balancing.
  23. 23. Networking in the Cloud Age!VIRTUAL NETWORKINGPRINCIPLES!
  24. 24. The illusion of isolated networks on top ofshared physical infrastructure!
  25. 25. Usually requires!•  Hypervisors! –  To share the same host with multiple tenants•  Virtual (software) switches! –  Port-level control to provide isolation•  Services provided in software / virtual contexts! –  Loadbalancer / firewall virtual appliances –  Host-based firewalls
  26. 26. Virtual-to-Physical Mapping!•  Option 1: VLAN! –  1 virtual network = 1 VLAN in physical infra !•  Option 2: Tunnels! –  VxLAN! –  (NV) GRE! –  STT! –  Others: MAC-in-MAC, NVO3, MPLS!
  27. 27. Virtual-to-Physical Mapping!•  Option 3: IP address re-write! –  1 tenant address mapped to 1 different provider address –  Hyper-V only (possible with KVM/Xen)•  Option 4: No mapping ! –  Tenant address is present on physical network –  Tenants isolated from each other and physical network using packet filters in hypervisor –  L3 isolation is CloudStack’s term for this mode –  Also called “Basic Networking”.
  28. 28. Virtual Switches!•  Linux bridge! –  KVM, XenServer, XCP, Oracle VM•  Open vSwitch (OVS)! –  KVM, XenServer, XCP•  VMware options! –  vSphere –  Distributed vSwitch (DVS) –  Cisco Nexus 1000v
  29. 29. Virtual Switches! Hypervisor Host! VM A1! VM A2! VM B1! VM C1! untagged (usually)!Virtual Nics! vswitch! vswitch! vswitch! Physical ! Nics! 192.168.1.0/24! VLAN TRUNK! VLAN 10! 192.168.1.0/24! VLAN 20! 10.1.1.0/24! VLAN 30!
  30. 30. Egress Traffic from VM! Ethernet frame from VM A1 to vswitch (untagged) Payload (IP Packet) 06:00:01:AA:BB:CC 06:02:12:1D:1E 0x800 46-1500 octets Dest, addr Src, addr Type Ethernet frame from vswitch to physical nic( tagged) Payload (IP Packet) 06:00:01:AA:BB:CC 06:02:12:1D:1E 0x8100 0xA 0x800 46-1500 octets Dest, addr Src, addr 802.1Q Tag Type*not all fields shown for clarity!
  31. 31. Ingress Traffic to VM! From physical nic to vswitch( tagged) Payload (IP Packet)06:02:12:1D:1E:1F 06:00:01:AA:BB:CC0x8100 0xA 0x800 46-1500 octets From vswitch to VM A1 (untagged) Payload (IP Packet)06:02:12:1D:1E:1F 06:00:01:AA:BB:CC 0x800 46-1500 octets
  32. 32. VLAN networking!Trunks! Trunks! Trunks! User   A   User   A   User   A   User   User   B   A   User   B   …  
  33. 33. 12 bits tag =4094 virtual networks
  34. 34. VLANs – other problems!•  Configuration complexity! –  Need to program switches carefully•  Large L2 domains! –  Broadcast in one VLAN can cause unintended load on unrelated hypervisors•  Live migration limited to a single VLAN!•  Limited mac table sizes in L2 switches! –  100s of vms per hypervisor –  1000s of mac addresses on uplink port
  35. 35. Tunnels!•  Map VM address (Tenant Address) to Physical address (PA) of Hypervisor! –  Software IPv4 tunnels between hypervisors –  Tunnel endpoints are PA of hypervisor –  Discriminator in tunnel header identifies tenant/ network •  GRE key in (NV) GRE tunnels (24-32 bits) •  VxLAN Network Identifier (VNI) in VxLAN (24 bits) •  Context ID in STT (64 bits)
  36. 36. GRE tunnel example! GRE  Key  1   GRE  Key  2   OVS   User   1  OVS   User   1   User   1   OVS   User  OVS   User   OVS   2   1   User   2  …   …   …  
  37. 37. GRE Example!Hypervisor 1! Hypervisor 2! VM   VM   VM   VM   A1   B1   A2   B2   192.168.10.55! 192.168.20.88! 192.168.10.5! 192.168.20.8! vswitch   vswitch   10.10.10.5! 10.20.20.9! 10.20.20.9! 10.10.10.5! GRE key=10! MAC A2! MAC A1! 192.168.10.5! 192.168.10.55!A1->A2!B1->B2! 10.20.20.9! 10.10.10.5! GRE key=20! MAC B2! MAC B1! 192.168.20.8! 192.168.20.88! Physical Address! Tenant L2 header! Tenant L3 header! Wire format!
  38. 38. Layer 3 cloud networking! Web   DB   Web   VM   VM   VM   Web   DB     Security   Security   Group   Group   Web   Web   DB   VM   VM   VM   …   …   …   Web   Web   VM   VM  
  39. 39. L3 isolation with distributed firewalls! ! Tenant 10.1.0.2Public Public IP 1 VM 1Internet address 65.37.141.11! 65.37.141.24! 10.1.0.1 ! Pod 1 Tenant 10.1.0.3 65.37.141.36! ! Leaf 2 VM 1 65.37.141.80! Switch ! ! Tenant 10.1.0.4 1 VM 2 L3 Core ! Pod 2 10.1.8.1 …! ! Leaf Switch 10.1.16.1 ! Load Pod 3 ! Balancer Leaf Switch … !
  40. 40. Networking in the Cloud Age!SERVICES!
  41. 41. Virtual Network Services!•  Provide L2-L7 network services that applications expect:! –  Load balancing, firewall, IDS, VPN, NAT, etc.•  Services are inserted in the virtual network topology! –  usually in the path to the public network•  Services are on-demand (api-driven), scalable, elastic!
  42. 42. Virtual Network Appliances!Network services are often provided by virtual appliances.!These are either commercial appliances in the virtual form factor orLinux-based networking appliances! Virtual Router! Public Network Nic! Virtual Network Nic! Control Network Nic!
  43. 43. Service insertion example! Tenant 1 Virtual Network 10.1.1.0/24 Public Public IP ! Tenant 10.1.1.2 Network address Gateway 1 VM 1 65.37.141.11! address 10.1.1.1 65.37.141.36 ! Tenant 1 ! Tenant 10.1.1.3 Edge 1 VM 2 Services ! Appliance(s) NAT!Internet! ! Tenant 10.1.1.4 DHCP! 1 VM 3 FW ! Tenant 10.1.1.5 1 VM 4
  44. 44. Service insertion with VLAN !Trunks! Trunks! Trunks! Tena nt  1   Tena nt  1   Tena nt  1   Tena Tena nt  2   Rout nt  1   er   VM  1   …   Public VLAN! Public VLAN! Public VLAN! Rout er   VM  2  
  45. 45. Network Services! Network Services!•  L2 connectivity!•  IPAM!•  DNS!•  Routing!•  ACL!•  Firewall!•  NAT!•  VPN!•  LB!•  IDS!•  IPS!!
  46. 46. Network Services! Network Service Services! Providers!•  L2 ü  Virtual connectivity! appliances!•  IPAM! ü  Hardware•  DNS! firewalls!•  Routing! ü  LB•  ACL! appliances!•  Firewall! ü  SDN•  NAT! controllers!•  VPN! ü  IDS /IPS•  LB! appliances!•  IDS! ü  VRF!•  IPS! ü  Hypervisor!!
  47. 47. Network Services! Network Service Network Services! Providers! Isolation!•  L2 ü  Virtual •  No connectivity! appliances!•  IPAM! isolation! ü  Hardware•  DNS! firewalls! •  VLAN•  Routing! ü  LB isolation!•  ACL! appliances!•  Firewall! ü  SDN •  Overlays!•  NAT! controllers! •  L3•  VPN! ü  IDS /IPS isolation!•  LB! appliances!•  IDS! ü  VRF!•  IPS! ü  Hypervisor!!
  48. 48. Service Catalog!•  Cloud users are not exposed to the nature of the service provider!•  Cloud operator designs a service catalog and offers them to end users.! –  Gold = {LB + FW, using virtual appliances} –  Platinum = {LB + FW + VPN, using hardware appliances} –  Silver = {FW using virtual appliances, 10Mbps}
  49. 49. End-user experience!•  Deploy a VM in a network! –  VM Template = Windows 2008 with Joomla on VMWare! –  Service offering {m1.large} = 2 x CPU x 2.0Ghz, 8 GB RAM! –  Disk Offering {Super fast}! –  Network Offering {Gold} = Source NAT + LB+ FW + 20 Mbps Internet access!
  50. 50. End-user experience!•  Deploy a VM in a network! –  VM Template = Windows 2008 with Joomla on VMWare –  Service offering {m1.large} = 2 x CPU x 2.0Ghz, 8 GB RAM –  Disk Offering {Super fast} –  Network Offering {Gold} = Source NAT + LB+ FW + 20 Mbps Internet access•  Network Offering Gold is realized by! –  VLAN isolation –  Source NAT & FW on Juniper SRX –  LB on F5 BigIp –  DHCP, DNS on virtual appliance
  51. 51. End-user experience!•  CloudStack orchestration:! –  Pick a free VLAN, pick a free public IP, free private IP –  Pick hypervisor with spare capacity –  Pick primary storage of SSD type accessible in hypervisor cluster –  Pick a Juniper SRX and F5 with spare capacity –  Spin up a new virtual appliance if necessary that runs DHCP and DNS service •  Pick hypervisor, call hypervisor APIs to provision virtual appliance on selected VLAN –  Call hypervisor APIs to provision VM on selected VLAN –  Call SRX and F5 APIs to place their internal interfaces on the VLAN, public interfaces on public VLAN –  Call SRX API to provision source NAT, default FW rules
  52. 52. Network services with VLANs! Tenant 1 Virtual Network 10.1.1.0/24 ! Tenant 10.1.1.2 Gateway 1 VM 1 address 10.1.1.1 ! Tenant 10.1.1.3 1 VM 2Internet! ! Tenant 10.1.1.4 1 VM 3 ! Tenant 10.1.1.5 1 VM 4
  53. 53. Network virtualization with VLANs! Tenant 1 Virtual Network 10.1.1.0/24 Public Public IP ! Tenant 10.1.1.2 Network address Gateway 1 VM 1 65.37.141.11! address 10.1.1.1 65.37.141.36 ! Tenant 1 ! Tenant 10.1.1.3 Edge 1 VM 2 ServicesInterne ! Appliance(s) NAT! ! Tenant t! DHCP! 1 VM 3 10.1.1.4 FW ! Tenant 10.1.1.5 1 VM 4
  54. 54. Network virtualization with VLANs! Tenant 1 Virtual Network 10.1.1.0/24 Public Public IP ! Tenant 10.1.1.2 Network address Gateway 1 VM 1 65.37.141.11! address 10.1.1.1 65.37.141.36 ! Tenant 1 ! Tenant 10.1.1.3 Edge 1 ! Tenant 1 VM 2 Edge Services Services Appliance(s) NAT! ! ! Appliance(s)Internet! ! Tenant 10.1.1.4 DHCP! 1 VM 3 FW Load Balancing! ! VPN Tenant 10.1.1.5 1 VM 4
  55. 55. Service insertion with VLANs! Tenant 1 Virtual Network 10.1.1.0/24 Public Public IP ! Tenant 10.1.1.2 Network address Gateway 1 VM 1 65.37.141.11! address 10.1.1.1 65.37.141.36 ! Tenant 1 ! Tenant 10.1.1.3 Edge 1 ! Tenant 1 VM 2 Edge Services Services Appliance(s) NAT! ! !Internet! Appliance(s) ! Tenant 10.1.1.4 DHCP! 1 VM 3 FW Load Balancing! ! Tenant 10.1.1.5 1 VM 4 Tenant 2 Virtual Network 10.1.1.0/24 Public IP address 65.37.141.24! Gateway address Tenant 2 VM 1 ! 10.1.1.2 65.37.141.80 10.1.1.1 ! Tenant 2 ! Tenant 10.1.1.3 Edge 2 VM 2 ! Services Appliance VPN! NAT! DHCP Tenant 2 VM 3 ! 10.1.1.4
  56. 56. Scaling services with VLANs! Scale out edge services using virtual appliances! 10.1.1.0/24! VLAN 100 VM 1! 10.1.1. 265.37.141.1 10.1.1.111! CS!65.37.141.1 Virtual VM 2!12 Router! 10.1.1. 3 DHCP, DNS! NAT! Load 10.1.1.4 VM 3! Balancing! VPN VM 4! 10.1.1.5
  57. 57. Scaling services with VLANs! Scale out edge services using virtual appliances! Scale up using hardware devices! 10.1.1.0/24! 10.1.1.0/24! VLAN 100 VLAN 100 VM 1! 65.37.141.11 10.1.1.1 10.1.1.2 VM 1! 10.1.1. 1 Juniper 2 SRX!65.37.141.1 10.1.1.111! CS! Firewall! NAT,65.37.141.1 Virtual VM 2! VPN! VM 2! 10.1.1. 10.1.1.312 Router! 3 65.37.141.11 10.1.1.112 DHCP, DNS! 2 Netscaler! NAT! Load Load 10.1.1.4 VM 3! VM 3! Balancer! 10.1.1.4 Balancing! VPN VM 4! VM 4! 10.1.1.5 10.1.1. 5 CS! DHCP, Virtual Router! DNS!
  58. 58. Multi-tier virtual networking! Internet! ! Loadbalancer Virtual appliance/! Hardware Devices! (virtual or HW)!Network Services!•  IPAM!•  DNS! Web VM 1!•  LB [intra]!•  S-2-S VPN!•  Static Routes! Web VM•  ACLs! 2!•  NAT, PF!•  FW [ingress & egress]! Web VM 3! Web VM 4! Web subnet ! 10.1.1.0/24! VLAN 101
  59. 59. Multi-tier virtual networking! Internet! ! Loadbalancer Virtual appliance/! Hardware Devices! (virtual or HW)!Network Services! App VM•  IPAM! 1!•  DNS! Web VM 1!•  LB [intra]!•  S-2-S VPN! App VM•  Static Routes! Web VM 2! VLAN 2724•  ACLs! 2!•  NAT, PF!•  FW [ingress & egress]! VLAN 353 Web VM DB VM•  BGP! 3! 1! Web VM 4! Web subnet ! App subnet DB Subnet! 10.1.1.0/24! VLAN 101 10.1.2.0/24! 10.1.3.0/24!
  60. 60. Multi-tier virtual networking! Internet! IPSec or SSL site-to-site VPN! ! Custome Loadbalancer Virtual appliance/! r! Hardware Devices! (virtual or HW)! Premises! MPLS VLAN!Network Services! App VM•  IPAM! 1!•  DNS! Web VM 1!•  LB [intra]!•  S-2-S VPN! App VM•  Static Routes! Web VM 2! VLAN 2724•  ACLs! 2!•  NAT, PF!•  FW [ingress & egress]! VLAN 353 Web VM DB VM•  BGP! 3! 1! Web VM 4! Web subnet ! App subnet DB Subnet! 10.1.1.0/24! VLAN 101 10.1.2.0/24! 10.1.3.0/24!
  61. 61. Multi-tier networking with Overlay! Internet! IPSec or SSL site-to-site VPN! Loadbalancer ! Custome (virtual Virtual Router! r! Premises! appliance)! MPLS VLAN!Network Services! App VM•  IPAM! Web VM 1!•  DNS! 1!•  LB [intra]! App VM•  S-2-S VPN! 2!•  Static Routes! Web VM GRE Key 2724 2!•  ACLs!•  NAT, PF!•  FW [ingress & egress]! Web VM GRE Key 353 DB VM•  BGP! 3! 1! Web VM 4! Web subnet ! App subnet DB Subnet! 10.1.1.0/24! GRE Key 101 10.1.2.0/24! 10.1.3.0/24!
  62. 62. Multi-tier networking with Overlay! Internet!Loadbalancer vswitches! (virtual appliance)! App VM 1! Web VM 1! App VM Web VM 2! GRE Key 2724 2! Web VM GRE Key 353 DB VM 3! 1! Web VM 4! Web subnet ! App subnet DB Subnet! 10.1.1.0/24! GRE Key 101 10.1.2.0/24! 10.1.3.0/24!
  63. 63. Networking in the Cloud Age!LAYER 3 ISOLATION!
  64. 64. Layer 3 cloud networking! Web DB Web VM! VM! VM! Web! DB ! Security Security Group! Group! Web Web DB VM! VM! VM! …! …! …! Web Web VM! VM!Ingress Rule: Allow VMs in Web Security Group access to VMs in DB Security Group on Port 33
  65. 65. L3 isolation with distributed firewalls! ! Tenant 10.1.0.2Public Public IP 1 VM 1Internet address 65.37.141.11! 65.37.141.24! 10.1.0.1 ! Pod 1 Tenant 10.1.0.3 65.37.141.36! ! Leaf 2 VM 1 65.37.141.80! Switch ! ! Tenant 10.1.0.4 1 VM 2 L3 Core ! Pod 2 10.1.8.1 …! ! Leaf Switch 10.1.16.1 ! Load Pod 3 ! Balancer Leaf Switch … !
  66. 66. L3 isolation with distributed firewalls! ! Tenant 10.1.0.2Public Public IP 1 VM 1Internet address 65.37.141.11! 65.37.141.24! 10.1.0.1 ! Pod 1 Tenant 10.1.0.3 65.37.141.36! ! Leaf 2 VM 1 65.37.141.80! Switch ! ! Tenant 10.1.0.4 1 VM 2 L3 Core ! Pod 2 10.1.8.1 …! ! Leaf Switch 10.1.16.1 ! Load Pod 3 ! Balancer Leaf Switch … ! Tenant 1 VM 3 ! 10.1.16.47 ! Tenant 10.1.16.85 1 VM 4
  67. 67. L3 isolation with distributed firewalls! ! Tenant 10.1.0.2Public Public IP 1 VM 1Internet address 65.37.141.11! 65.37.141.24! 10.1.0.1 ! Pod 1 Tenant 10.1.0.3 65.37.141.36! ! Leaf 2 VM 1 65.37.141.80! Switch ! ! Tenant 10.1.0.4 1 VM 2 L3 Core ! Pod 2 10.1.8.1 …! ! Leaf Switch ! Tenant 10.1.16.12 10.1.16.1 2 VM 2 ! Load Pod 3 ! Balancer Leaf ! Switch Tenant 2 VM 3 10.1.16.21 … ! Tenant 1 VM 3 ! 10.1.16.47 ! Tenant 10.1.16.85 1 VM 4
  68. 68. 1 Firewall per Virtual Machine
  69. 69. A Million Firewalls?!VM! VM! VM!…! …! VM! VM! …! …!VM! VM! …! VM! VM!VM! VM! VM! VM! VM! VM!VM! VM! VM!…! …! VM! VM! …! …!VM! VM! …! VM! VM!VM! VM! VM! VM! VM! VM!VM! VM! VM!…! …! VM! VM! …! …!VM! VM! …! VM! VM!VM! VM! VM! VM! VM! VM!VM! VM! VM!…! …! VM! VM! …! …!VM! VM! …! VM! VM!VM! VM! VM! VM! VM! VM!VM! VM! VM!…! …! VM! VM! …! …!VM! VM! …! VM! VM!VM! VM! VM! VM! VM! VM!VM!…!VM! VM! …! VM! VM! …! VM! VM! …! …! VM! …! VM! VM!VM! VM! VM! VM! VM!VM! VM! VM!…! …! VM! VM! …! …!VM! VM! …! VM! VM!VM! VM! VM! VM! VM! VM!VM! VM! VM!…! …! VM! VM! …! …!VM! VM! …! VM! VM!VM! VM! VM! VM! VM! VM!
  70. 70. Networking in the Cloud Age!SOFTWARE DEFINEDNETWORKING!
  71. 71. Definition!•  Separation of Control Plane from the hardware performing the forwarding function!•  Control plane is logically centralized!
  72. 72. SDN Advantages!•  Centralized control makes it easier to configure, troubleshoot and maintain•  Eliminates ‘box’ mode of configuration•  Enables control at a high level
  73. 73. Related to SDN!•  API layer over a collection of ‘boxes’! –  API layer communicates with boxes using box-level APIs / ssh / telnet•  OpenFlow! –  Standard protocol for the centralized control plane to talk to the forwarding elements.•  Tunnels / overlays! –  SDN is valuable for virtual topologies –  Initial target of SDN implementation
  74. 74. Centralized control plane!Admin/User  API   Controller  Cluster   MySQL/NoSQL   Openflow/ssh/netconf/other! Boxes!
  75. 75. SDN problems!•  Discovery of virtual address -> physical address mapping! –  VxLAN = multicast –  GRE = programmed by control plane –  L3 isolation = no mapping, no discovery
  76. 76. SDN problems!•  State maintenance! –  Large number of endpoints + flows –  High arrival rate of new flows –  Needs fast and scalable storage and processing
  77. 77. CloudStack and SDN! Hypervisor   Hypervisor   Resource   5 4 Resource   Hyperviso Hyperviso r  Plugins   r  Plugins   Plugin   Framew 6 ork   Network   API   7 SDN   Resource   Network     API   Network   controller   OrchestraSon  Engine   Plugins  1   API   Plugins     2 8 Allocator   9 3 Storage   Plugins   Plugins   Storage   Storage   Resource   Resource   Allocator   Allocator   Plugins   Plugins   Physical Resources ! Network plugin is the glue that understands the SDN controller’s API!
  78. 78. Virtual Networking "!NETWORKING IN APACHECLOUDSTACK!
  79. 79. Regions and Zones!•  A cluster of CloudStack management servers manage the physical resources of a region –  Single API endpoint per region•  Each region consists of zones•  Zones are physically proximate, but provide distinct failure domains (e.g., flood, earthquake, power)•  Zones are interconnected with high speed low latency links
  80. 80. Region “West” Region “East” Geographic separation InternetLow Latency Region “South”
  81. 81. Region “West” Zone “West-Beta” Zone “West-Alpha” High Speed Backbone (e.g., SONET ring) Zone “West-Delta” Zone “West-Gamma”
  82. 82. Inside a zone! Admin/User  API   End  users   CloudStack  Cluster   DC  Edge   MySQL   L2/L3  core   Leaf  Sw  Hypervisor  (Xen  /VMWare/KVM)   Secondary  Storage  Primary  Storage  NFS/ISCSI/FC   Pod   Pod   Pod   Pod   Pod  
  83. 83. Orchestration!•  Orchestration describes the automated arrangement, coordination, and management of complex computer systems, middleware and services –  Wikipedia!
  84. 84. CloudStack Architecture! Hypervisor   Hypervisor   Plugins   Plugins   Plugin   Framework   Network  Plugins   OrchestraSon  Engine   Network  Plugins   Allocator   Allocator   Plugins   Plugins   Storage  Plugins  
  85. 85. CloudStack Architecture! •  XenServer   • VMWare   • KVM   • OracleVM   Hypervisor     Hypervisor   Plugins   Plugins   Plugin   Framework   Nicira   •  • Netscaler   • Brocade   Network  Plugins   OrchestraSon  Engine   Network  Plugins   idoNet   • M   Allocator   •  Random   Allocator   • User-­‐ Plugins   Plugins   concentrated   • Intel  TXT   • Affinity    
  86. 86. CloudStack Architecture! Hypervisor   Hypervisor   Resource   5 4 Resource   Hyperviso Hyperviso r  Plugins   r  Plugins   Plugin   Framew 6 ork   Network   API   7 Network   Resource   Network     API   Network   Resource   OrchestraSon  Engine   Plugins  1   API   Plugins     2 8 Allocator   9 3 Storage   Plugins   Plugins   Storage   Storage   Resource   Resource   Allocator   Allocator   Plugins   Plugins   Physical Resources ! Orchestration steps can be executed in parallel or in sequence!
  87. 87. Problem:Manage Configuration of! 1000s of virtual appliances (or VRF) Dozens of HW appliancesSolution:Database-driven state management ofappliances! Message queues + Retry Logic Idempotent updates, Recreatable virtual appliances
  88. 88. Problem:Manage Configuration of! 1000s of virtual appliances (or VRF) Dozens of HW appliancesSolution:Database-driven state management ofappliances! Message queues + Retry Logic! Idempotent updates,! Recreatable virtual appliances! !
  89. 89. Problem:!Single-tenant HW appliances!Solution:!CloudStack API layers multi-tenancy, providesabstraction! No direct access to devices!
  90. 90. Problem:Hardware appliances with no APIs CLI only Limited concurrent login sessionsSolution: Recommend appliances with APIs Integrate with Network Orchestrators !
  91. 91. Problem:Manage the configuration of 100s of thousands of firewallsSolution:Well-known software scaling techniques•  Message queues•  Consistency tradeoffs•  Idempotent configuration & retriesCloudStack uses •  special purpose queues•  optimized for large security groups•  eventual consistency for rule updates
  92. 92. Problem:Firewall (iptables) rules explosion on the host firewall! Allow Security Group {Web} on TCP port 3060 ! !-A FORWARD -m tcp –p tcp –dport 3060 –src 10.1.16.31 – j ACCEPT-A FORWARD -m tcp –p tcp –dport 3060 –src 10.1.45.112 – j ACCEPT-A FORWARD -m tcp –p tcp –dport 3060 –src 10.1.189.5 – j ACCEPT …!-A FORWARD -m tcp –p tcp –dport 3060 –src 10.21.9.77 – j ACCEPT For large security groups, performance suffers
  93. 93. Problem: Firewall (iptables) rules explosion on the host firewall ! Solution:! Use ipsets: !ipset –N web_sg iptreemapipset –A web_sg 10.1.16.31ipset –A web_sg 10.1.16.112ipset –A …! web_sg 10.1.189.5ipset –A web_sg 10.21.9.77-A FORWARD –p tcp –m tcp –dport 3060 –m set –match-set web_sg src -j ACCEPT
  94. 94. Apache CloudStack!•  Apache CloudStack! –  http://www.cloudstack.org/! –  Download it! –  Use it! –  Contribute to it!•  Citrix CloudPlatform! –  Based on Apache CloudStack! –  Commercial support!

×