CloudStack	
  Networking	
  Deep	
  
Dive	
  
@chiradeep	
  
CloudStack	
  Collabora9on	
  Conference	
  	
  
Amsterdam	
 ...
Overview	
  
• 
• 
• 
• 
• 

10,000	
  foot	
  overview	
  of	
  CloudStack	
  Networking	
  
SoDware	
  architecture	
  
...
Virtual	
  Networking	
  
•  The	
  illusion	
  of	
  isolated	
  networks	
  on	
  top	
  of	
  
shared	
  physical	
  in...
Virtual	
  Network	
  Services	
  
•  Provide	
  L2-­‐L7	
  network	
  services	
  that	
  applica9ons	
  
expect:	
  
–  ...
Virtual	
  Network	
  Appliances	
  
Network	
  services	
  are	
  oDen	
  provided	
  by	
  virtual	
  appliances.	
  
Th...
Network	
  Services	
  
Network	
  
Services	
  
•  L2	
  
connec9vity	
  
•  IPAM	
  
•  DNS	
  
•  Rou9ng	
  
•  ACL	
  ...
Network	
  Services	
  
Network	
  
Services	
  
•  L2	
  
connec9vity	
  
•  IPAM	
  
•  DNS	
  
•  Rou9ng	
  
•  ACL	
  ...
Network	
  Services	
  
Network	
  
Services	
  
•  L2	
  
connec9vity	
  
•  IPAM	
  
•  DNS	
  
•  Rou9ng	
  
•  ACL	
  ...
Network	
  Offerings	
  
•  Cloud	
  users	
  are	
  not	
  exposed	
  to	
  the	
  nature	
  of	
  the	
  
service	
  prov...
Mul9-­‐9er	
  virtual	
  networking	
  
Internet!

IPSec VPN!
!

Customer!
Premises!

VR!
Loadbalancer	
  
(HW	
  or	
  
V...
Virtual	
  networking	
  with	
  overlays	
  
Internet!

IPSec VPN!
!

Customer!
Premises!

VR + vSwitches!
App
VM 1!

Web...
CLOUDSTACK	
  ARCHITECTURE	
  
CloudStack	
  Architecture	
  
5

4

1

API	
  
	
   API	
  
	
   API	
  
	
  

Plugin	
  
Framew
ork	
  

2

Orchestra9on...
Plugin	
  interac9on	
  

1

API	
  
	
   API	
  
	
   API	
  
	
  

Async	
  
Job	
  
Mgr	
  

2

Plugin	
  
Frame
4
work...
Plugin	
  Interac9on	
  Details	
  
•  Resource	
  calls	
  are	
  expected	
  to	
  be	
  idempotent	
  

–  It	
  is	
  ...
USE	
  CASES	
  
Isolated	
  vs.	
  VPC	
  
•  VPC	
  should	
  be	
  the	
  default	
  choice	
  
–  Isolated	
  =	
  VPC	
  with	
  a	
  ...
Real-­‐life	
  Use	
  cases	
  
• 
• 
• 
• 
• 
• 
• 
• 
• 

Monitoring-­‐as-­‐a-­‐Service	
  
Managed	
  Applica9ons	
  
C...
Monitoring	
  /	
  Backup	
  Service	
  
•  Shared	
  VLANs	
  for	
  monitoring/	
  
backup	
  
•  Mul9-­‐tenancy	
  enfo...
Monitoring	
  Using	
  Private	
  Gateway	
  
Monitoring	
  Server	
  

Private	
  Gateway	
  

!

VR!

Backup	
  Server	
...
Managed	
  Applica9ons	
  
Shared	
  App	
  Manager	
  	
  
(not	
  CloudStack	
  Managed)	
  

VR	
  
Hosted	
  App	
  
(...
S3	
  /	
  Object	
  Store	
  
VR	
  
VPC	
  for	
  
Tenant	
  A	
  

VM	
  

VM	
  

VM	
  

VM	
  

VM	
  

VM	
  

•  T...
MPLS	
  VPN	
  

Tenant	
  A	
  DC	
  

Private	
  GW	
  

VR	
  

WAN	
  
Tenant	
  A	
  

VM	
  

VM	
  

VM	
  

VM	
  ...
Cloud	
  Burs9ng	
  
RFC	
  1918	
  Public	
  IP	
  Range	
  reserved	
  for	
  Tenant	
  A	
  

VR	
  
Internet	
  
VM	
 ...
Internal	
  Load	
  Balancer	
  
!

VR!
Web
VM 1!

Web
VM 2!

Web
VM 3!

LB VM
1!

•  LB	
  VM	
  lifecycle	
  is	
  
mana...
Interoperate	
  with	
  legacy	
  appliances	
  /
Baremetal	
  DB	
  
Use	
  reserved	
  IP	
  
range	
  

VM	
  

HW	
  
...
Conundrum:	
  Inter-­‐AZ	
  private	
  traffic	
  

Tenant	
  A	
  

Tenant	
  A	
  

Tenant	
  B	
  

Zone	
  1	
  

Tenant...
Inter-­‐AZ	
  private	
  traffic	
  
•  Solu9on	
  1:	
  Use	
  VR-­‐VR	
  IPSec	
  VPN	
  
–  Coming	
  in	
  4.3	
  
–  Co...
PERFORMANCE	
  ISSUES	
  
VR	
  performance	
  
•  FAQ:	
  what	
  is	
  the	
  performance	
  of	
  the	
  VR?	
  
–  #	
  of	
  concurrent	
  conn...
Factors	
  affec9ng	
  performance	
  
•  Physical	
  network	
  

–  Public	
  network	
  capacity	
  (ports/buffers/etc.)	...
Topology	
  for	
  tes9ng	
  
Hypervisor	
  H1	
  

VR	
  

Web	
  
VM	
  2	
  
Hypervisor	
  H2	
  

Tool	
  Host	
  
Web...
Do	
  your	
  own	
  tes9ng	
  
•  Basic	
  Topology 	
  	
  

–  Intra	
  hypervisor,	
  guest	
  VM	
  -­‐>	
  guest	
  ...
VLAN	
  Issues	
  
•  En9re	
  VLAN	
  range	
  trunked	
  to	
  all	
  hypervisors	
  
–  Unlimited	
  broadcast	
  domai...
Upcoming SlideShare
Loading in …5
×

CloudStack Networking Deepdive CCCEU13

2,404 views

Published on

Networking in CloudStack is full-featured, full of bells and whistles and by necessity complicated. This session will take cloud operators through the ins-and-outs of CloudStack Networking. Attendees will learn the motivations behind how CloudStack networking is architected, solutions to common networking requirements, gotchas, troubleshooting CloudStack networking and finally some future directions for theses features.
It is assumed that the audience will have some experience administering CloudStack clouds.

Published in: Technology
0 Comments
11 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,404
On SlideShare
0
From Embeds
0
Number of Embeds
122
Actions
Shares
0
Downloads
0
Comments
0
Likes
11
Embeds 0
No embeds

No notes for slide

CloudStack Networking Deepdive CCCEU13

  1. 1. CloudStack  Networking  Deep   Dive   @chiradeep   CloudStack  Collabora9on  Conference     Amsterdam   Nov  22,  2013  
  2. 2. Overview   •  •  •  •  •  10,000  foot  overview  of  CloudStack  Networking   SoDware  architecture   Complex  use  cases   Performance  issues   Focus  on     –  Advanced  zone   –  VLAN  isola9on    
  3. 3. Virtual  Networking   •  The  illusion  of  isolated  networks  on  top  of   shared  physical  infrastructure   •  Requires  orchestra9on  of     –  Hypervisors   –  SoDware  switches   •  Services  are  provided  in  virtual  contexts   –  E.g.,  LB,  firewalls    
  4. 4. Virtual  Network  Services   •  Provide  L2-­‐L7  network  services  that  applica9ons   expect:   –  Load balancing, firewall, IDS, VPN, NAT, etc. •  Services  are  inserted  in  the  virtual  network  topology   –  usually in the path to the public network •  Services  are  on-­‐demand  (api-­‐driven),  scalable,  elas9c    
  5. 5. Virtual  Network  Appliances   Network  services  are  oDen  provided  by  virtual  appliances.   These  are  either  commercial  appliances  in  the  virtual  form  factor  or  Linux-­‐ based  networking  appliances   Virtual Router! Public Network Nic! Control Network Nic! Virtual Network Nic!
  6. 6. Network  Services   Network   Services   •  L2   connec9vity   •  IPAM   •  DNS   •  Rou9ng   •  ACL   •  Firewall   •  NAT   •  VPN   •  LB   •  IDS   •  IPS    
  7. 7. Network  Services   Network   Services   •  L2   connec9vity   •  IPAM   •  DNS   •  Rou9ng   •  ACL   •  Firewall   •  NAT   •  VPN   •  LB   •  IDS   •  IPS     Service Providers! ü  Virtual appliances! ü  Hardware firewalls! ü  LB appliances! ü  SDN controllers! ü  VRF! ü  Hypervisor!
  8. 8. Network  Services   Network   Services   •  L2   connec9vity   •  IPAM   •  DNS   •  Rou9ng   •  ACL   •  Firewall   •  NAT   •  VPN   •  LB   •  IDS   •  IPS     Service Providers! ü  Virtual appliances! ü  Hardware firewalls! ü  LB appliances! ü  SDN controllers! ü  VRF! ü  Hypervisor! Network   Isola1on   •  No  isola9on   •  VLAN   isola9on   •  Overlays   •  L3  isola9on  
  9. 9. Network  Offerings   •  Cloud  users  are  not  exposed  to  the  nature  of  the   service  provider   •  Cloud  operator  designs  a  service  catalog  and  offers   them  to  end  users.   –  Gold = {LB + FW, using virtual appliances} –  Platinum = {LB + FW + VPN, using hardware appliances} –  Silver = {FW using virtual appliances, 10Mbps}
  10. 10. Mul9-­‐9er  virtual  networking   Internet! IPSec VPN! ! Customer! Premises! VR! Loadbalancer   (HW  or   Virtual)   Private Gateway! Web VM 3! VLAN 398 VLAN 101 Web VM 2! App VM 2! VLAN 2724 App VM 1! Web VM 1! DB VM 1! Network Services! •  IPAM! •  DNS! •  LB [intra]! •  S-2-S VPN! •  Static Routes! •  ACLs! •  NAT, PF! •  FW [ingress & egress]!
  11. 11. Virtual  networking  with  overlays   Internet! IPSec VPN! ! Customer! Premises! VR + vSwitches! App VM 1! Web VM 2! Web VM 3! GRE KEY 398 GRE KEY 101 Web VM 1! App VM 2! GRE KEY 2724 Private Gateway! Loadbalancer   (Virtual)   DB VM 1! Network Services! •  IPAM! •  DNS! •  LB [intra]! •  S-2-S VPN! •  Static Routes! •  ACLs! •  NAT, PF! •  FW [ingress & egress]!
  12. 12. CLOUDSTACK  ARCHITECTURE  
  13. 13. CloudStack  Architecture   5 4 1 API     API     API     Plugin   Framew ork   2 Orchestra9on  Engine   Hyperviso Hyperviso r  Plugins   r  Plugins   6 8 3 7 Network   Network   Plugins   Plugins   Allocator   Storage   Plugins   Plugins   9 Hypervisor   Hypervisor   Resource   Resource   Network   Network   Resource   Resource   Storage   Storage   Resource   Resource   Allocator   Allocator   Plugins   Plugins   Physical Resources ! Orchestration steps usually executed in sequence!
  14. 14. Plugin  interac9on   1 API     API     API     Async   Job   Mgr   2 Plugin   Frame 4 work   Orchestra9on   Engine   Idempotent   5 Network   Network   Plugins   Plugins   6 Desired  State   3 CloudSt ack  DB   Idempotent   7 Network   Network   Resource   Resource   8 Desired  State   Opera9onal  State   Desired  State   Plugin  should  not  update   CloudStack  objects  
  15. 15. Plugin  Interac9on  Details   •  Resource  calls  are  expected  to  be  idempotent   –  It  is  the  job  of  the  plugin  to  ensure  this   –  apply(apply(config)) == apply(config) •  Plugins  should  not  update  CloudStack  resources     –  networks,  rules,  ip  addresses,  etc.   –   This  is  the  desired  state  as  expressed  by  the  API   •  Plugins  can  have  their  own  tables  inside  the  CloudStack  DB   •  Usually,  there  is  no  retry  from  the  CloudStack  orchestra9on  engine   upon  failure     –  API  reports  failure.   –  Security  groups  mechanism  will  con9nuously  retry  (eventual   consistency)   –  Refactor  proposed  to  provide  eventual  consistency  for  all  network   plugins  
  16. 16. USE  CASES  
  17. 17. Isolated  vs.  VPC   •  VPC  should  be  the  default  choice   –  Isolated  =  VPC  with  a  single  9er   •  Differences   –  Isolated:  Remote  Access  VPN,  Firewall  Rules  on  Public  IP   –  VPC:  Site-­‐to-­‐Site  IPSec  VPN,  ACL   •  RA  VPN  coming  very  soon   •  FW  on  public  IP  may  come  later   –  Use  ACL  for  equivalent  func9on   •  Use  Isolated  when  firewall  is     –  SRX   –  Cisco  vASA1000    
  18. 18. Real-­‐life  Use  cases   •  •  •  •  •  •  •  •  •  Monitoring-­‐as-­‐a-­‐Service   Managed  Applica9ons   Cloud  burs9ng   Mix  bare-­‐metal/virtual  workloads   Load  balance  applica9on  9er   MPLS  VPN   Mul9-­‐zone  failover   Mul9ple  IP  usecases   Metering  with  hardware  appliances  
  19. 19. Monitoring  /  Backup  Service   •  Shared  VLANs  for  monitoring/   backup   •  Mul9-­‐tenancy  enforced  by  using   PVLAN  (Xen/VMW/KVM)   ! VR! App VM 2! VLAN 2724 Web VM 3! VLAN 398 VLAN 101 Web VM 2! DB VM 1! Backup PVLAN (shared)! App VM 1! Web VM 1! Cloud  Operator  Resources  (aaS)     Monitoring  Server   Backup  Server   Patching PVLAN (shared)! Patching  Server   Monitoring PVLAN (shared)! Tenant  VPC    
  20. 20. Monitoring  Using  Private  Gateway   Monitoring  Server   Private  Gateway   ! VR! Backup  Server   NAT   Web VM 1! Web VM 2! App VM 2! Patching  Server   App VM 1! Web VM 3! Tenant  VPC     DB VM 1! Cloud  Operator     Resources   •  -­‐  Addi9onal  traffic  through  VR   •  -­‐  Performance  impact  on  VR   •  +  Usable  for  SDN  topologies  
  21. 21. Managed  Applica9ons   Shared  App  Manager     (not  CloudStack  Managed)   VR   Hosted  App   (CRM/VDI/ CMS/etc)  for   Tenant  A   VM   VM   VM   VM   VM   NAT   VM   VR   Hosted  App   for  Tenant  B   Cloud   Operator   Resources   VM   VM   VM   VM   VM   NAT   VM   Public  or   Private  GW   •  Mul9ple  tenants  share  the   applica9on  manager/ orchestrator   •  Tenant  will  get  charged  if  using   public  route.  
  22. 22. S3  /  Object  Store   VR   VPC  for   Tenant  A   VM   VM   VM   VM   VM   VM   •  Tenant  will  get  charged  for   network  bandwidth  between   object  store  and  compute  even   if  object  store  is  co-­‐located   VR   VPC  for   Tenant  B   VM   VM   VM   VM   VM   VM   Public  Network  
  23. 23. MPLS  VPN   Tenant  A  DC   Private  GW   VR   WAN   Tenant  A   VM   VM   VM   VM   VM   VM   Tenant  B  DC   VR   Tenant  B   Private  GW   VM   VM   VM   VM   VM   VM   •  S9tching  VLAN  to  MPLS  label  is   done  outside  CloudStack   •  Alterna9vely:  use  solu9on  like   Contrail  SDN   •  No  VLANs,  use  overlays   •  VR  func9on  replaced  by   logical  distributed  router  
  24. 24. Cloud  Burs9ng   RFC  1918  Public  IP  Range  reserved  for  Tenant  A   VR   Internet   VM   VM   VM   VM   VM   VR   Tenant  B   VM   VM   VM   VM   VM   VM   Tenant  A  DC   Public  VLAN  1   VM   Public  VLAN  2   Tenant  A   •  Clients  in  Tenant  DC  need  to  use   services  inside  VPC  in  the  cloud   •  Tenant  DC  is  already  networked   with  Cloud  (e.g.,  with  MPLS  VPN)   •  Public  VLAN  can  be  reserved  for  a   tenant  (use  RFC1918  address   range  if  needed).   •  Cloud  operator  can  choose  not  to   bill  for  ‘public’  data  traffic  
  25. 25. Internal  Load  Balancer   ! VR! Web VM 1! Web VM 2! Web VM 3! LB VM 1! •  LB  VM  lifecycle  is   managed  by  CloudStack   App 1 VM 1! App 1 VM 2! App 1 VM 3! LB VM 2! App 2 VM 1! App 2 VM 2! DB VM 1!
  26. 26. Interoperate  with  legacy  appliances  / Baremetal  DB   Use  reserved  IP   range   VM   HW   appliances   not   managed  by   CloudStack   VPN   VM   VM   VM   CS  VM  but   not   configured   by  CS   Shared  VLAN  200   Can  be   RFC1918  or   Public  IP   Not   CloudStack-­‐ Managed   Baremetal   DB   Shared  VLAN  100   •  Managed  hos9ng   primarily   •  Baremetal   advanced  zone   support   forthcoming    
  27. 27. Conundrum:  Inter-­‐AZ  private  traffic   Tenant  A   Tenant  A   Tenant  B   Zone  1   Tenant  B   High  speed  Private   Interconnect   Zone  2  
  28. 28. Inter-­‐AZ  private  traffic   •  Solu9on  1:  Use  VR-­‐VR  IPSec  VPN   –  Coming  in  4.3   –  Cons:     •  ipsec  overhead,     •  tricky  usage  accoun9ng,   •  Uses  public  internet   •  Solu9on  2:  overlay   –  GRE  tunnel  between  VRs   –  Not  yet  a  proposal,  just  a  possibility   •  Solu9on  3:  SDN  overlay   –  Stretch  overlay  subnets  between  DC   –  Cons:   •  Needs  change  to  CloudStack  model  (network  belongs  to  zone)   •  SDN  controllers  inherently  are  single-­‐DC  
  29. 29. PERFORMANCE  ISSUES  
  30. 30. VR  performance   •  FAQ:  what  is  the  performance  of  the  VR?   –  #  of  concurrent  connec9ons?   –  #  of  connec9ons  per  second?   –  Throughput?   –  Latency?   •  Answer:   –  “It  depends”  
  31. 31. Factors  affec9ng  performance   •  Physical  network   –  Public  network  capacity  (ports/buffers/etc.)   –  Top  of  rack  switch  capacity     •  Hypervisor   –  CPU  model  (L1/L2  cache,  clock  speed)   –  NICs  (1  GigE/10GigE/link  aggrega9on  mode/Jumbo   Frames)   –  RAM  and  CPU  allocated  to  dom0   –  Bridge  vs  OVS   •  Virtual  Router  specs   –  CPU  speed   –  RAM   –  (4.2.1)  32-­‐bit  vs  64-­‐bit  
  32. 32. Topology  for  tes9ng   Hypervisor  H1   VR   Web   VM  2   Hypervisor  H2   Tool  Host   Web   VM  1   Web   VM  3   Web   VM  4   Hypervisor  H3   Public   Network   Guest   Network  
  33. 33. Do  your  own  tes9ng   •  Basic  Topology     –  Intra  hypervisor,  guest  VM  -­‐>  guest  VM   •  VM3  -­‐>  VM4   –  Between  hypervisors,  guest  VM  -­‐>  guest  VM  on  same   VLAN   •  VM1  -­‐>  VM3   •  Routed  topology   –  VR  between  traffic  source  and  des9na9on   •  VR  on  same  hypervisor  as  des9na9on   –  Tools  Host  -­‐>VR  -­‐>    VM2  (port  forward)   •  VR  on  different  hypervisor  as  des9na9on   –  Tools  Host  -­‐>  VM1  (port  forward)   •  LB  test   –  Tools  Host  -­‐>  {VM1,  VM2,  VM3,  VM4}  
  34. 34. VLAN  Issues   •  En9re  VLAN  range  trunked  to  all  hypervisors   –  Unlimited  broadcast  domain   –  Per-­‐port  VLAN  limits  in  certain  switches   •  No  good  solu9ons:   –  VTP  (Vlan  Pruning)   –  Arista  VMTracer  (vSphere  only)   •  Overlays  (aka  SDN)  is  probably  the  way  to  go  

×