Your SlideShare is downloading. ×
The Evolution of                               CloudStack                             The roads taken and not taken       ...
We remember the most painful and pleasurable                      parts of an experience, not the durationSaturday, Decemb...
In the BeginningSaturday, December 1, 2012                      3
Circa                             2008Saturday, December 1, 2012           4
Circa                                        2008                             !"#$%&(Saturday, December 1, 2012           ...
Circa                                        2008                             !"#$%&(       !"#$%&(Saturday, December 1, 2...
Circa                                        2008                             !"#$%&(       !"#$%&(   !"#$%&(Saturday, Dec...
Circa                                               2008                             Open Source Xen                      ...
Circa                                               2008                             Open Source Xen                      ...
Circa                                               2008                             Open Source Xen                      ...
Circa                                               2008                             Open Source Xen                      ...
Circa                                               2008                                    Proprietary Orchestration Serv...
Circa                                               2008                             EC2                             API  ...
Circa                                                 2008                             EC2           EBS                  ...
Circa                                                 2008                             EC2           EBS                  ...
Circa                                                 2008                             EC2           EBS                  ...
The	  Virtual	  Datacenter	  OS	  allows	  businesses	  to	                  efficiently	  pool	  all	  types	  of	  hardwar...
The	  Virtual	  Datacenter	  OS	  allows	  businesses	  to	                  efficiently	  pool	  all	  types	  of	  hardwar...
2.0          AWS             2.2       3.0 Quality     4.X                                 Prototype   1.0 GA             ...
Circa                             3/2009Saturday, December 1, 2012            7
Circa                                        3/2009                             !"#$%&(Saturday, December 1, 2012         ...
Circa                                                   3/2009                             !"#$%&(   !"#$%&(Saturday, Dece...
Circa                                                   3/2009                             !"#$%&(   !"#$%&(     !"#$%&(Sa...
Circa                                                          3/2009                             Open Source Xen         ...
Circa                                                                  3/2009                             Open Source Xen ...
Circa                                                                  3/2009                             Open Source Xen ...
Circa                                                                     3/2009                                      Prop...
Circa                                                                      3/2009                             Proprietary ...
Circa                                                                      3/2009                                         ...
Circa                                                                      3/2009                                         ...
Circa                                                                      3/2009                                         ...
Circa                                                                      3/2009                                         ...
Circa                                                                      3/2009                                         ...
Circa                                                                      3/2009                                         ...
Circa                                                                            3/2009                                   ...
Circa                                                                            3/2009                                   ...
Circa                                                                            3/2009                                   ...
Availability Zone ArchitectureSaturday, December 1, 2012                           9
Availability Zone Architecture                                              L3 coreSaturday, December 1, 2012             ...
Availability Zone Architecture                                                End users                                   ...
Availability Zone Architecture                                                End users                                   ...
Availability Zone Architecture                                                End users                                   ...
Availability Zone Architecture                                                 End users                                  ...
Availability Zone Architecture                                                       End users                            ...
Availability Zone Architecture                                                       End users                            ...
Availability Zone Architecture                                                       End users                            ...
Availability Zone Architecture                                                          End users        Admin/User API   ...
Availability Zone Architecture                                                          End users        Admin/User API   ...
Multiple zones                             CloudStack                                 MySQL                               ...
Multiple zones                             CloudStack                                 MySQL                               ...
Network VirtualizationSaturday, December 1, 2012                            11
Network Virtualization                                                   Tenant	  	  1	  Virtual	  Network	  10.1.1.0/24  ...
Network Virtualization                                                                 Tenant	  	  1	  Virtual	  Network	 ...
Network Virtualization                                                                                                    ...
Network Virtualization                                                                                                    ...
Virtual Machine Placement                                                           End users                             ...
Network Virtualization - why?                    AWS-style security groups “unfamiliar”                    Adopt tradition...
Network Virtualization                    UDP encapsulation of Ethernet Frames                             similar to VXLA...
ZFS (Volume Service)                             ZFS for highly scalable, reliable storage                               R...
OrchestrationSaturday, December 1, 2012                   16
Orchestration                                             JVMSaturday, December 1, 2012                         16
Orchestration                                             JVM                                              MySQLSaturday, ...
Orchestration                                                   JVM                                     Message Bus       ...
Orchestration                                                            JVM                             Front-           ...
Orchestration                                                              JVM                                        VM  ...
Orchestration                                                                   JVM                                       ...
Orchestration                                                                     JVM                                     ...
Orchestration                                                                                JVM                          ...
Orchestration                                                                                    JVM                      ...
Orchestration                                                                                       JVM                   ...
Orchestration                                                                                       JVM                   ...
Orchestration                                                                                           JVM               ...
Reject Pile (2009)                             Local disks of hypervisors presented as                             network...
Reject Pile (2009)                             Security Groups                               Not well understood          ...
Reject Pile (early 2009)                             KVM                              Unstable                            ...
April 2009Saturday, December 1, 2012                20
April 2009                                          “5 guys in                                          a garage”Saturday,...
Ship It! ( Beta 6/2009)                             ISO/CD with Xen 3.3 / CentOs customized                             IS...
Road to 1.0                             ISCSI - ZFS zvols -> ∞ snapshots                             Performance tuning (U...
More rejects (2009)                             Network RAID of zvols over ISCSI/md                               Performa...
1.0 - Ship It (Nov 2009)                             6 engineers + 3 QA + 1 sales                             3 beta sites...
Orchestration + Usage server                                                      JVM                              Front- ...
Orchestration + Usage server                                                      JVM   MySQL                             ...
Orchestration + Usage server     Browser           LoadBalan                                                      JVM   My...
Orchestration + Usage server     Browser           LoadBalan                                                              ...
Orchestration + Usage server     Browser           LoadBalan                                                              ...
Orchestration + Usage server     Browser           LoadBalan                                                              ...
Orchestration + Usage server     Browser           LoadBalan                                                              ...
1.0 Issues                             Multicast scaling                             Oracle acquisition of OpenSolaris    ...
Multicast Scaling                             Broadcast (ARP) / Multicast from VM                             translates t...
Road Ahead                             Multi-hypervisor, SAN, and VLAN support                               Use commercia...
Road Ahead                             Throw out custom hypervisor and                             OpenSolaris            ...
Circa                                                                      3/2010                                         ...
Circa                                                                       3/2010                                        ...
Circa                                                                           3/2010                                    ...
Circa                                                                           3/2010                                    ...
Circa                                                                           3/2010                                    ...
Circa                                                                         3/2010                                      ...
Circa                                                                          3/2010                                     ...
Circa                                                                          3/2010                                     ...
Orchestration + Usage server     Browser           LoadBalan                                                              ...
Availability Zone Architecture                                                          End users        Admin/User API   ...
Availability Zone Architecture                                                          End users        Admin/User API   ...
Availability Zone Architecture                                                          End users        Admin/User API   ...
Availability Zone Architecture                                                          End users        Admin/User API   ...
System VM                             Data Path problem                               Need to move bits between storage ty...
System VMs                             Virtual Routers, Console Proxy VMs,                             Secondary Storage V...
2.0 6/2010                             Shipping just java binaries + system vm                             image          ...
Citrix Cloud Portal                             Business and operations support platform for                             S...
Reject Pile (2010)                             “Zone” and “Pod” VLANs to go beyond 4k                             limit   ...
Flexibility and Scale                                  (2010-2011)                             Network, hypervisor and vm ...
Network Flexibility (2.2                          refactor)                             Hardware appliances instead of vir...
Network Flexibility          Scale	  out	  edge	  services	  using	  virtual	  appliances                                 ...
Network Flexibility          Scale	  out	  edge	  services	  using	  virtual	  appliances                           Scale	...
VM Placement                             Customize vm placement                               Pick hosts and storage accor...
Multi-hypervisor                             XenServer, KVM and VMWare in the same                             zone       ...
Plugin Architecture (2.2)                             Orchestration                                EngineSaturday, Decembe...
Plugin Architecture (2.2)                                               Plugin                                            ...
Plugin Architecture (2.2)                                                         Hypervisor                              ...
Plugin Architecture (2.2)                                                         Hypervisor                              ...
Plugin Architecture (2.2)                                                         Hypervisor                              ...
Plugin Architecture (2.2)                                                         Hypervisor                              ...
Plugin Architecture                                                          Hypervisor                                   ...
Plugin Architecture                       •XenServer                                                                      ...
Plugin Architecture                       •XenServer                                                                      ...
Plugin Architecture                       •XenServer                                                                      ...
Security Groups                                          Web	                                      DB	                    ...
Security Groups                                          Web	                                      DB	                    ...
Security Groups                                          Web	                                      DB	                    ...
Evolution of CloudStack Architecture (Collab 2012)
Evolution of CloudStack Architecture (Collab 2012)
Evolution of CloudStack Architecture (Collab 2012)
Evolution of CloudStack Architecture (Collab 2012)
Evolution of CloudStack Architecture (Collab 2012)
Evolution of CloudStack Architecture (Collab 2012)
Evolution of CloudStack Architecture (Collab 2012)
Evolution of CloudStack Architecture (Collab 2012)
Evolution of CloudStack Architecture (Collab 2012)
Evolution of CloudStack Architecture (Collab 2012)
Evolution of CloudStack Architecture (Collab 2012)
Evolution of CloudStack Architecture (Collab 2012)
Evolution of CloudStack Architecture (Collab 2012)
Evolution of CloudStack Architecture (Collab 2012)
Evolution of CloudStack Architecture (Collab 2012)
Evolution of CloudStack Architecture (Collab 2012)
Evolution of CloudStack Architecture (Collab 2012)
Evolution of CloudStack Architecture (Collab 2012)
Evolution of CloudStack Architecture (Collab 2012)
Evolution of CloudStack Architecture (Collab 2012)
Upcoming SlideShare
Loading in...5
×

Evolution of CloudStack Architecture (Collab 2012)

7,628

Published on

Published in: Technology
0 Comments
10 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
7,628
On Slideshare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
0
Comments
0
Likes
10
Embeds 0
No embeds

No notes for slide

Transcript of "Evolution of CloudStack Architecture (Collab 2012)"

  1. 1. The Evolution of CloudStack The roads taken and not taken @chiradeep Collab 2012Saturday, December 1, 2012 1
  2. 2. We remember the most painful and pleasurable parts of an experience, not the durationSaturday, December 1, 2012 2
  3. 3. In the BeginningSaturday, December 1, 2012 3
  4. 4. Circa 2008Saturday, December 1, 2012 4
  5. 5. Circa 2008 !"#$%&(Saturday, December 1, 2012 4
  6. 6. Circa 2008 !"#$%&( !"#$%&(Saturday, December 1, 2012 4
  7. 7. Circa 2008 !"#$%&( !"#$%&( !"#$%&(Saturday, December 1, 2012 4
  8. 8. Circa 2008 Open Source Xen (custom) !"#$%&( !"#$%&( !"#$%&(Saturday, December 1, 2012 4
  9. 9. Circa 2008 Open Source Xen Security Groups (custom) !"#$%&( !"#$%&( !"#$%&(Saturday, December 1, 2012 4
  10. 10. Circa 2008 Open Source Xen (custom) Security Groups EBS !"#$%&( !"#$%&( !"#$%&(Saturday, December 1, 2012 4
  11. 11. Circa 2008 Open Source Xen (custom) Security Groups EBS S3 !"#$%&( !"#$%&( !"#$%&(Saturday, December 1, 2012 4
  12. 12. Circa 2008 Proprietary Orchestration Services Open Source Xen (custom) Security Groups EBS S3 !"#$%&( !"#$%&( !"#$%&(Saturday, December 1, 2012 4
  13. 13. Circa 2008 EC2 API Proprietary Orchestration Services Open Source Xen (custom) Security Groups EBS S3 !"#$%&( !"#$%&( !"#$%&(Saturday, December 1, 2012 4
  14. 14. Circa 2008 EC2 EBS API API Proprietary Orchestration Services Open Source Xen (custom) Security Groups EBS S3 !"#$%&( !"#$%&( !"#$%&(Saturday, December 1, 2012 4
  15. 15. Circa 2008 EC2 EBS S3 API API API Proprietary Orchestration Services Open Source Xen (custom) Security Groups EBS S3 !"#$%&( !"#$%&( !"#$%&(Saturday, December 1, 2012 4
  16. 16. Circa 2008 EC2 EBS S3 API API API Proprietary Orchestration Services Secret Sauce Open Source Xen (custom) Security Groups EBS S3 !"#$%&( !"#$%&( !"#$%&(Saturday, December 1, 2012 4
  17. 17. The  Virtual  Datacenter  OS  allows  businesses  to   efficiently  pool  all  types  of  hardware  resources  -­‐   servers,  storage  and  network  –  into  an  aggregated   on-­‐premise  cloud -­‐  VMWare  Press  Release  9/2008Saturday, December 1, 2012 5
  18. 18. The  Virtual  Datacenter  OS  allows  businesses  to   efficiently  pool  all  types  of  hardware  resources  -­‐   servers,  storage  and  network  –  into  an  aggregated   on-­‐premise  cloud -­‐  VMWare  Press  Release  9/2008 Eucalyptus  is  the  only  cloud  architecture  to  support   the  same  applicaKon  programming  interfaces  (APIs)   as  public  clouds,  and  today  Eucalyptus  is  fully   compaKble  with  the  Amazon  AWS  public  cloud   infrastructure.   Eucalyptus  Systems  Press  Release  4/2009Saturday, December 1, 2012 5
  19. 19. 2.0 AWS 2.2 3.0 Quality 4.X Prototype 1.0 GA Refactor Compatibility Refactor Improvements Refactor 2008 2009 2010 2011 2012 Sept 2008: Nov 2009: May 2010: July 2011: April 2012: VMOps CloudStack Cloud.com Citrix Apache Founded 1.0 GA Launch & Acquires CloudStack CloudStack Cloud.com 2.0 GASaturday, December 1, 2012 6
  20. 20. Circa 3/2009Saturday, December 1, 2012 7
  21. 21. Circa 3/2009 !"#$%&(Saturday, December 1, 2012 7
  22. 22. Circa 3/2009 !"#$%&( !"#$%&(Saturday, December 1, 2012 7
  23. 23. Circa 3/2009 !"#$%&( !"#$%&( !"#$%&(Saturday, December 1, 2012 7
  24. 24. Circa 3/2009 Open Source Xen (custom) !"#$%&( !"#$%&( !"#$%&(Saturday, December 1, 2012 7
  25. 25. Circa 3/2009 Open Source Xen Virtual Networks (custom) !"#$%&( !"#$%&( !"#$%&(Saturday, December 1, 2012 7
  26. 26. Circa 3/2009 Open Source Xen (custom) Virtual Networks NFS !"#$%&( !"#$%&( !"#$%&(Saturday, December 1, 2012 7
  27. 27. Circa 3/2009 Proprietary Control Plane Open Source Xen (custom) Virtual Networks NFS !"#$%&( !"#$%&( !"#$%&(Saturday, December 1, 2012 7
  28. 28. Circa 3/2009 Proprietary Orchestration Services Proprietary Control Plane Open Source Xen (custom) Virtual Networks NFS !"#$%&( !"#$%&( !"#$%&(Saturday, December 1, 2012 7
  29. 29. Circa 3/2009 GUI Proprietary Orchestration Services Proprietary Control Plane Open Source Xen (custom) Virtual Networks NFS !"#$%&( !"#$%&( !"#$%&(Saturday, December 1, 2012 7
  30. 30. Circa 3/2009 GUI Proprietary Orchestration Services Proprietary Control Plane Open Source Xen (custom) Virtual Networks NFS !"#$%&( !"#$%&( !"#$%&(Saturday, December 1, 2012 8
  31. 31. Circa 3/2009 GWT GUI Proprietary Orchestration Services Proprietary Control Plane Open Source Xen (custom) Virtual Networks NFS !"#$%&( !"#$%&( !"#$%&(Saturday, December 1, 2012 8
  32. 32. Circa 3/2009 GWT GUI Java App Server Proprietary Orchestration Services Proprietary Control Plane Open Source Xen (custom) Virtual Networks NFS !"#$%&( !"#$%&( !"#$%&(Saturday, December 1, 2012 8
  33. 33. Circa 3/2009 GWT GUI Java App Server Proprietary Orchestration Services JSON over TCP Proprietary Control Plane Open Source Xen (custom) Virtual Networks NFS !"#$%&( !"#$%&( !"#$%&(Saturday, December 1, 2012 8
  34. 34. Circa 3/2009 GWT GUI Java App Server Proprietary Orchestration Services JSON over TCP Proprietary Control Plane NFS (ZFS on Open Source Xen OpenSolaris) (custom) Virtual Networks NFS !"#$%&( !"#$%&( !"#$%&(Saturday, December 1, 2012 8
  35. 35. Circa 3/2009 GWT GUI Java App Server Proprietary Orchestration Services JSON over TCP Proprietary Control Plane UDP Tunnels NFS (ZFS on Open Source Xen OpenSolaris) (custom) Virtual Networks NFS !"#$%&( !"#$%&( !"#$%&(Saturday, December 1, 2012 8
  36. 36. Circa 3/2009 GWT GUI Java App Server Proprietary Orchestration Services JSON over TCP Proprietary Control Plane UDP Tunnels NFS (ZFS on Open Source Xen OpenSolaris) Agent (custom) Virtual Networks NFS !"#$%&( !"#$%&( !"#$%&(Saturday, December 1, 2012 8
  37. 37. Circa 3/2009 GWT GUI Java App Server Proprietary Orchestration Services JSON over TCP Proprietary Control Plane UDP Tunnels NFS (ZFS on Open Source Xen Agent OpenSolaris) Agent (custom) Virtual Networks NFS !"#$%&( !"#$%&( !"#$%&(Saturday, December 1, 2012 8
  38. 38. Availability Zone ArchitectureSaturday, December 1, 2012 9
  39. 39. Availability Zone Architecture L3 coreSaturday, December 1, 2012 9
  40. 40. Availability Zone Architecture End users DC Edge L3 coreSaturday, December 1, 2012 9
  41. 41. Availability Zone Architecture End users DC Edge L3 core Access Sw Hypervisor (Xen) Primary Storage ZFS PodSaturday, December 1, 2012 9
  42. 42. Availability Zone Architecture End users DC Edge L3 core Access Sw Hypervisor (Xen) Primary Storage ZFS Pod PodSaturday, December 1, 2012 9
  43. 43. Availability Zone Architecture End users DC Edge L3 core Access Sw Hypervisor (Xen) Primary Storage ZFS Pod Pod PodSaturday, December 1, 2012 9
  44. 44. Availability Zone Architecture End users DC Edge L3 core Access Sw Hypervisor (Xen) Primary Storage ZFS Pod Pod Pod PodSaturday, December 1, 2012 9
  45. 45. Availability Zone Architecture End users DC Edge L3 core Access Sw Hypervisor (Xen) Primary Storage ZFS Pod Pod Pod PodSaturday, December 1, 2012 9
  46. 46. Availability Zone Architecture End users DC Edge L3 core Access Sw Hypervisor (Xen) Primary Storage ZFS Pod Pod Pod Pod PodSaturday, December 1, 2012 9
  47. 47. Availability Zone Architecture End users Admin/User API CloudStack DC Edge MySQL L3 core Access Sw Hypervisor (Xen) Primary Storage ZFS Pod Pod Pod Pod PodSaturday, December 1, 2012 9
  48. 48. Availability Zone Architecture End users Admin/User API CloudStack DC Edge MySQL ECMP/BGP L3 core Access Sw Hypervisor (Xen) Primary Storage ZFS Pod Pod Pod Pod PodSaturday, December 1, 2012 9
  49. 49. Multiple zones CloudStack MySQL Zone1Saturday, December 1, 2012 10
  50. 50. Multiple zones CloudStack MySQL Internet Zone1 Zone2 Zone3Saturday, December 1, 2012 10
  51. 51. Network VirtualizationSaturday, December 1, 2012 11
  52. 52. Network Virtualization Tenant    1  Virtual  Network  10.1.1.0/24 Tenant  1    10.1.1.2 Gateway  address   VM  1 10.1.1.1 Tenant  1    10.1.1.3 VM  2 Tenant  1    10.1.1.4 VM  3 Tenant  1    10.1.1.5 VM  4Saturday, December 1, 2012 11
  53. 53. Network Virtualization Tenant    1  Virtual  Network  10.1.1.0/24 Tenant  1    10.1.1.2 Gateway  address   VM  1 10.1.1.1 Tenant  1   Tenant  1    10.1.1.3 Virtual  Router VM  2 NAT Internet DHCP Tenant  1    10.1.1.4 FW VM  3 Load  Balancing Tenant  1    10.1.1.5 VM  4Saturday, December 1, 2012 11
  54. 54. Network Virtualization Tenant    1  Virtual  Network  10.1.1.0/24 Public  IP  address   Tenant  1    10.1.1.2 65.37.141.11 Gateway  address   VM  1 65.37.141.36 10.1.1.1 Public  Network Tenant  1   Tenant  1    10.1.1.3 Virtual  Router VM  2 NAT Internet DHCP Tenant  1    10.1.1.4 FW VM  3 Load  Balancing Tenant  1    10.1.1.5 VM  4Saturday, December 1, 2012 11
  55. 55. Network Virtualization Tenant    1  Virtual  Network  10.1.1.0/24 Public  IP  address   Tenant  1    10.1.1.2 65.37.141.11 Gateway  address   VM  1 65.37.141.36 10.1.1.1 Public  Network Tenant  1   Tenant  1    10.1.1.3 Virtual  Router VM  2 NAT Internet DHCP Tenant  1    10.1.1.4 FW VM  3 Load  Balancing Tenant  1    10.1.1.5 VM  4 Tenant  2  Virtual  Network  10.1.1.0/24 Public  IP  address   65.37.141.24 Gateway  address   Tenant  2    10.1.1.2 10.1.1.1 VM  1 65.37.141.80 Tenant  2   Tenant  2    10.1.1.3 Virtual  Router   VM  2 Appliance FW NAT Tenant  2    10.1.1.4 DHCP VM  3Saturday, December 1, 2012 11
  56. 56. Virtual Machine Placement End users CloudStack DC Edge MySQL L3/L2 core Hypervisor (Xen) Primary Storage ZFS Pod Pod Pod Pod PodSaturday, December 1, 2012 12
  57. 57. Network Virtualization - why? AWS-style security groups “unfamiliar” Adopt traditional L2 model for end-users VLANs do not scale use network virtualization to realize thisSaturday, December 1, 2012 13
  58. 58. Network Virtualization UDP encapsulation of Ethernet Frames similar to VXLAN (128-bit address vs 24-bit for VXLAN) Driver in dom0 Virtual Router to provide edge services including Load Balancing Hand-rolled Fedora 8 para-virtualizedSaturday, December 1, 2012 14
  59. 59. ZFS (Volume Service) ZFS for highly scalable, reliable storage RAID-Z SSD cache NFS for shared storage Hand-rolled version of OpenSolaris 2008.11Saturday, December 1, 2012 15
  60. 60. OrchestrationSaturday, December 1, 2012 16
  61. 61. Orchestration JVMSaturday, December 1, 2012 16
  62. 62. Orchestration JVM MySQLSaturday, December 1, 2012 16
  63. 63. Orchestration JVM Message Bus MySQLSaturday, December 1, 2012 16
  64. 64. Orchestration JVM Front- end Message Bus MySQLSaturday, December 1, 2012 16
  65. 65. Orchestration JVM VM Front- Orchest end rator Message Bus MySQLSaturday, December 1, 2012 16
  66. 66. Orchestration JVM VM Network Front- Orchest Orchestr end rator ator Message Bus MySQLSaturday, December 1, 2012 16
  67. 67. Orchestration JVM VM Network Storage Front- Orchest Orchestr Orchest end rator ator rator Message Bus MySQLSaturday, December 1, 2012 16
  68. 68. Orchestration JVM VM Network Storage Host/ Front- Orchest Orchestr Orchest Resource end rator ator rator Manager Message Bus MySQLSaturday, December 1, 2012 16
  69. 69. Orchestration JVM VM Network Storage Host/ Front- Orchest Orchestr Orchest Resource end rator ator rator Manager Message Bus Hypervi Hypervi Hypervisor sor Agent sor (JVM) MySQLSaturday, December 1, 2012 16
  70. 70. Orchestration JVM VM Network Storage Host/ Front- Orchest Orchestr Orchest Resource end rator ator rator Manager Message Bus Hypervi Hypervi Hypervisor Storage Storage Storage sor Agent sor Agent Agent Agent (JVM) (JVM) MySQLSaturday, December 1, 2012 16
  71. 71. Orchestration JVM VM Network Storage Host/ Front- Orchest Orchestr Orchest Resource end rator ator rator Manager Message Bus Hypervi Hypervi Hypervisor Storage Storage Storage Console Console sor Agent sor Agent Agent Agent Agent Agent (JVM) (JVM) MySQLSaturday, December 1, 2012 16
  72. 72. Orchestration JVM VM Network Storage Host/ GWT-RPC Front- Browser Orchest Orchestr Orchest Resource end rator ator rator Manager Message Bus Hypervi Hypervi Hypervisor Storage Storage Storage Console Console sor Agent sor Agent Agent Agent Agent Agent (JVM) (JVM) MySQLSaturday, December 1, 2012 16
  73. 73. Reject Pile (2009) Local disks of hypervisors presented as network block storage with network RAID Too complex (sheepdog-like) Split-brain issues DRBD Too complex to automate recovery Commercial Arrays Limited volumes, limited snapshotsSaturday, December 1, 2012 17
  74. 74. Reject Pile (2009) Security Groups Not well understood VLANs 4K limit Physical Network Appliances No APIs (Cisco at least) Can’t decapsulate UDP tunnelSaturday, December 1, 2012 18
  75. 75. Reject Pile (early 2009) KVM Unstable Windows support No CPU allocationSaturday, December 1, 2012 19
  76. 76. April 2009Saturday, December 1, 2012 20
  77. 77. April 2009 “5 guys in a garage”Saturday, December 1, 2012 20
  78. 78. Ship It! ( Beta 6/2009) ISO/CD with Xen 3.3 / CentOs customized ISO/CD with OSOL 2008.11 + patches + drivers Java binaries System VM: Fedora Core 8 VM bits including dnsmasq, iptables, HAProxy Installers, documentationSaturday, December 1, 2012 21
  79. 79. Road to 1.0 ISCSI - ZFS zvols -> ∞ snapshots Performance tuning (UDP tunnels, zvol) Usage, metering, Formal user / admin Query-based API JQuery-based GUI - skinnable Windows PV driversSaturday, December 1, 2012 22
  80. 80. More rejects (2009) Network RAID of zvols over ISCSI/md Performance Re-mirror time / penalty Consistent snapshots Split-brainSaturday, December 1, 2012 23
  81. 81. 1.0 - Ship It (Nov 2009) 6 engineers + 3 QA + 1 sales 3 beta sites Traction with SPs interested in competing with AWSSaturday, December 1, 2012 24
  82. 82. Orchestration + Usage server JVM Front- endSaturday, December 1, 2012 25
  83. 83. Orchestration + Usage server JVM MySQL Front- endSaturday, December 1, 2012 25
  84. 84. Orchestration + Usage server Browser LoadBalan JVM MySQL ced HTTP Query Front- API endSaturday, December 1, 2012 25
  85. 85. Orchestration + Usage server Browser LoadBalan JVM MySQL ced HTTP VM Network Storage Host/ Query Front- API Orchest Orchestr Orchest Resource end rator ator rator ManagerSaturday, December 1, 2012 25
  86. 86. Orchestration + Usage server Browser LoadBalan JVM MySQL ced HTTP VM Network Storage Host/ Query Front- API Orchest Orchestr Orchest Resource end rator ator rator Manager Message Bus Hypervi Hypervi Hypervisor Storage Storage Storage Console Console sor Agent sor Agent Agent Agent Agent Agent (JVM) (JVM)Saturday, December 1, 2012 25
  87. 87. Orchestration + Usage server Browser LoadBalan JVM MySQL ced Storage VM VM Network Network Storage HTTP Front- Front- Orchest Network Storage Host/ VM Orchestr Orchest Host/ Query Front- Orchest Orchestr Orchest Resource end API end Orchest Orchestr rator rator ator Orchest Resource end rator ator rator rator ator rator Manager Manager Clustered Message Bus Orchestration Message Bus Message Bus Server Hypervi Hypervi Hypervisor Storage Storage Storage Console Console sor Agent sor Agent Agent Agent Agent Agent (JVM) (JVM)Saturday, December 1, 2012 25
  88. 88. Orchestration + Usage server Browser LoadBalan JVM MySQL ced Storage VM VM Network Network Storage HTTP Front- Front- Orchest Network Storage Host/ VM Orchestr Orchest Host/ Query Front- Orchest Orchestr Orchest Resource end API end Orchest Orchestr rator rator ator Orchest Resource end rator ator rator rator ator rator Manager Manager Usage Clustered Message Bus Orchestration Message Bus Message Bus Server Hypervi Hypervi Hypervisor Storage Storage Storage Console Console sor Agent sor Agent Agent Agent Agent Agent (JVM) (JVM)Saturday, December 1, 2012 25
  89. 89. 1.0 Issues Multicast scaling Oracle acquisition of OpenSolaris Hiring “full-stack” developers 5 startups - Hypervisor + Storage + Network Virtualization + Network appliance + OrchestrationSaturday, December 1, 2012 26
  90. 90. Multicast Scaling Broadcast (ARP) / Multicast from VM translates to UDP multicast Switches snoop on IGMP to prevent flooding all ports Most switches fall-over after 64 Multicast groups each tenant -> 1 multicast groupSaturday, December 1, 2012 27
  91. 91. Road Ahead Multi-hypervisor, SAN, and VLAN support Use commercial XenServer Commercial filers / ISCSI arrays Support KVM + NFS Product first, architecture second From web hosting to enterprise workloadSaturday, December 1, 2012 28
  92. 92. Road Ahead Throw out custom hypervisor and OpenSolaris VLANs Secondary Storage to store snapshots and imagesSaturday, December 1, 2012 29
  93. 93. Circa 3/2010 GUI Proprietary Orchestration Services Proprietary Control Plane Open Source Xen Storage Agent Virtual Networks Agent (custom) Arrays !"#$%&( !"#$%&( !"#$%&(Saturday, December 1, 2012 30
  94. 94. Circa 3/2010 JQuery GUI Proprietary Orchestration Services Proprietary Control Plane Open Source Xen Storage Agent Virtual Networks Agent (custom) Arrays !"#$%&( !"#$%&( !"#$%&(Saturday, December 1, 2012 30
  95. 95. Circa 3/2010 JQuery GUI Proprietary Orchestration Services Proprietary Control Plane VLANs Open Source Xen Storage Agent Virtual Networks Agent (custom) Arrays !"#$%&( !"#$%&( !"#$%&(Saturday, December 1, 2012 30
  96. 96. Circa 3/2010 JQuery GUI Proprietary Orchestration Services Proprietary Control Plane VLANs Open Source Xen Storage Virtual Networks Agent (custom) Arrays !"#$%&( !"#$%&( !"#$%&(Saturday, December 1, 2012 30
  97. 97. Circa 3/2010 JQuery GUI Proprietary Orchestration Services Proprietary Control Plane VLANs Open Source Xen Storage Virtual Networks (custom) Arrays !"#$%&( !"#$%&( !"#$%&(Saturday, December 1, 2012 30
  98. 98. Circa 3/2010 JQuery GUI Proprietary Orchestration Services Proprietary Control Plane VLANs Storage Virtual Networks Arrays !"#$%&( !"#$%&( !"#$%&(Saturday, December 1, 2012 30
  99. 99. Circa 3/2010 JQuery GUI Proprietary Orchestration Services Proprietary Control Plane VLANs Commercial Storage Virtual Networks XenServer Arrays !"#$%&( !"#$%&( !"#$%&(Saturday, December 1, 2012 30
  100. 100. Circa 3/2010 JQuery GUI Proprietary Orchestration Services Proprietary Control Plane ISCSI/ VLANs NFS Commercial Storage Virtual Networks XenServer Arrays !"#$%&( !"#$%&( !"#$%&(Saturday, December 1, 2012 30
  101. 101. Orchestration + Usage server Browser LoadBalan JVM MySQL ced Front- HTTP end VM Network Storage Host/ Query API Orchest Orchestr Orchest Resource rator ator rator Manager Job Queue Usage Clustered Message Bus Orchestration Server Hypervi XS sor Agent XAPI KVM Console Console Agent Secondary Agent Agent Storage Agent XenServers (JVM) (JVM) (JVM)Saturday, December 1, 2012 31
  102. 102. Availability Zone Architecture End users Admin/User API CloudStack DC Edge MySQL Access Sw Hypervisor (Xen /VMWare/KVM) Pod Pod Pod Pod PodSaturday, December 1, 2012 32
  103. 103. Availability Zone Architecture End users Admin/User API CloudStack DC Edge MySQL Access Sw Hypervisor (Xen /VMWare/KVM) Secondary Storage Pod Pod Pod Pod PodSaturday, December 1, 2012 32
  104. 104. Availability Zone Architecture End users Admin/User API CloudStack DC Edge MySQL L2/L3 core Access Sw Hypervisor (Xen /VMWare/KVM) Secondary Storage Pod Pod Pod Pod PodSaturday, December 1, 2012 32
  105. 105. Availability Zone Architecture End users Admin/User API CloudStack DC Edge MySQL L2/L3 core Access Sw Hypervisor (Xen /VMWare/KVM) Secondary Storage Primary Storage NFS/ISCSI/FC Pod Pod Pod Pod PodSaturday, December 1, 2012 32
  106. 106. System VM Data Path problem Need to move bits between storage types Present consoles to end users Scale up / down these data path servicesSaturday, December 1, 2012 33
  107. 107. System VMs Virtual Routers, Console Proxy VMs, Secondary Storage VMs based on the same vm image Orchestration and scaling baked inSaturday, December 1, 2012 34
  108. 108. 2.0 6/2010 Shipping just java binaries + system vm image Open sourced (GPL) Company name changed to Cloud.com New product - Cloud PortalSaturday, December 1, 2012 35
  109. 109. Citrix Cloud Portal Business and operations support platform for Service Providers Account management Self-service portal Billing and metering CRM and support ticketing Not OSSSaturday, December 1, 2012 36
  110. 110. Reject Pile (2010) “Zone” and “Pod” VLANs to go beyond 4k limit Pod VLANs bridged using zone VLANs Another SDN attempt using GRE on Open vSwitchSaturday, December 1, 2012 37
  111. 111. Flexibility and Scale (2010-2011) Network, hypervisor and vm placement flexibility Tens of thousands of hypervisors AWS APISaturday, December 1, 2012 38
  112. 112. Network Flexibility (2.2 refactor) Hardware appliances instead of virtual router Upgrade / downgrade to/from hardware to virtual Pick and choose services (L2-L7) Security GroupsSaturday, December 1, 2012 39
  113. 113. Network Flexibility Scale  out  edge  services  using  virtual  appliances 10.1.1.0/24 VLAN  100 VM  1 10.1.1.2 65.37.141.111 10.1.1.1 65.37.141.112 CS Virtual   10.1.1.3 VM  2 Router DHCP,  DNS NAT 10.1.1.4 VM  3 Load  Balancing VM  4 10.1.1.5Saturday, December 1, 2012 40
  114. 114. Network Flexibility Scale  out  edge  services  using  virtual  appliances Scale  up  using  hardware  devices 10.1.1.0/24 10.1.1.0/24 VLAN  100 VLAN  100 65.37.141.111 10.1.1.1 10.1.1.2 VM  1 VM  1 10.1.1.2 Juniper  SRX 65.37.141.111 10.1.1.1 Firewall NAT,   65.37.141.112 VPN CS VM  2 Virtual   10.1.1.3 VM  2 10.1.1.3 Router 65.37.141.112 10.1.1.112 DHCP,  DNS Netscaler NAT Load   10.1.1.4 VM  3 VM  3 Load  Balancing Balancer 10.1.1.4 VM  4 VM  4 10.1.1.5 10.1.1.5 CS DHCP,   Virtual   DNS RouterSaturday, December 1, 2012 40
  115. 115. VM Placement Customize vm placement Pick hosts and storage according to rules Affinity / anti-affinitySaturday, December 1, 2012 41
  116. 116. Multi-hypervisor XenServer, KVM and VMWare in the same zone Differences in network and storage behavior Different image formats Same system vm imageSaturday, December 1, 2012 42
  117. 117. Plugin Architecture (2.2) Orchestration EngineSaturday, December 1, 2012 43
  118. 118. Plugin Architecture (2.2) Plugin Framework Orchestration EngineSaturday, December 1, 2012 43
  119. 119. Plugin Architecture (2.2) Hypervisor Hypervisor Plugins Plugins Plugin Framework Orchestration EngineSaturday, December 1, 2012 43
  120. 120. Plugin Architecture (2.2) Hypervisor Hypervisor Plugins Plugins Plugin Framework Orchestration Network Network Plugins Plugins EngineSaturday, December 1, 2012 43
  121. 121. Plugin Architecture (2.2) Hypervisor Hypervisor Plugins Plugins Plugin Framework Orchestration Network Network Plugins Plugins Engine Allocator Allocator Plugins PluginsSaturday, December 1, 2012 43
  122. 122. Plugin Architecture (2.2) Hypervisor Hypervisor Plugins Plugins Plugin Framework Orchestration Network Network Plugins Plugins Engine Allocator Allocator Plugins Plugins Storage PluginsSaturday, December 1, 2012 43
  123. 123. Plugin Architecture Hypervisor Hypervisor Plugins Plugins Plugin Framework Orchestration Network Network Plugins Plugins Engine Allocator Allocator Plugins PluginsSaturday, December 1, 2012 44
  124. 124. Plugin Architecture •XenServer •VMWare •KVM •OracleVM Hypervisor Hypervisor Plugins Plugins Plugin Framework Orchestration Network Network Plugins Plugins Engine Allocator Allocator Plugins PluginsSaturday, December 1, 2012 44
  125. 125. Plugin Architecture •XenServer •VMWare •KVM •OracleVM Hypervisor Hypervisor Plugins Plugins Plugin •Nicira Framework •Netscaler Orchestration Network Network Plugins •Brocade •MidoNet Plugins Engine Allocator Allocator Plugins PluginsSaturday, December 1, 2012 44
  126. 126. Plugin Architecture •XenServer •VMWare •KVM •OracleVM Hypervisor Hypervisor Plugins Plugins Plugin •Nicira Framework •Netscaler Orchestration Network Network Plugins •Brocade •MidoNet Plugins Engine Random • Allocator Allocator •User- Plugins Plugins concentrated •Intel TXT •AffinitySaturday, December 1, 2012 44
  127. 127. Security Groups Web   DB   Web   VM VM VM Web   Web   DB   VM VM VM … … … Web   Web   VM VM Ingress  Rule:  Allow  VMs  in  Web  Security  Group  access  to  VMs  in  DB  Security  Group  on  Port  3306Saturday, December 1, 2012 45
  128. 128. Security Groups Web   DB   Web   VM VM VM Web Security   Group Web   Web   DB   VM VM VM … … … Web   Web   VM VM Ingress  Rule:  Allow  VMs  in  Web  Security  Group  access  to  VMs  in  DB  Security  Group  on  Port  3306Saturday, December 1, 2012 45
  129. 129. Security Groups Web   DB   Web   VM VM VM Web DB   Security   Security   Group Group Web   Web   DB   VM VM VM … … … Web   Web   VM VM Ingress  Rule:  Allow  VMs  in  Web  Security  Group  access  to  VMs  in  DB  Security  Group  on  Port  3306Saturday, December 1, 2012 45

×