SlideShare a Scribd company logo
The Evolution of
                               CloudStack
                             The roads taken and not taken
                                      @chiradeep
                                      Collab 2012




Saturday, December 1, 2012                                   1
We remember the most painful and pleasurable
                      parts of an experience, not the duration




Saturday, December 1, 2012                                        2
In the Beginning




Saturday, December 1, 2012                      3
Circa
                             2008




Saturday, December 1, 2012           4
Circa
                                        2008




                             !"#$%&'(


Saturday, December 1, 2012                      4
Circa
                                        2008




                             !"#$%&'(       !"#$%&'(


Saturday, December 1, 2012                             4
Circa
                                        2008




                             !"#$%&'(       !"#$%&'(   !"#$%&'(


Saturday, December 1, 2012                                        4
Circa
                                               2008




                             Open Source Xen
                                 (custom)



                                !"#$%&'(           !"#$%&'(   !"#$%&'(


Saturday, December 1, 2012                                               4
Circa
                                               2008




                             Open Source Xen
                                                 Security Groups
                                 (custom)



                                !"#$%&'(           !"#$%&'(        !"#$%&'(


Saturday, December 1, 2012                                                    4
Circa
                                               2008




                             Open Source Xen
                                 (custom)
                                                 Security Groups   EBS


                                !"#$%&'(           !"#$%&'(              !"#$%&'(


Saturday, December 1, 2012                                                          4
Circa
                                               2008




                             Open Source Xen
                                 (custom)
                                                 Security Groups   EBS          S3


                                !"#$%&'(           !"#$%&'(              !"#$%&'(


Saturday, December 1, 2012                                                           4
Circa
                                               2008




                                    Proprietary Orchestration Services




                             Open Source Xen
                                 (custom)
                                                 Security Groups   EBS          S3


                                !"#$%&'(           !"#$%&'(              !"#$%&'(


Saturday, December 1, 2012                                                           4
Circa
                                               2008



                             EC2
                             API




                                    Proprietary Orchestration Services




                             Open Source Xen
                                 (custom)
                                                 Security Groups   EBS          S3


                                !"#$%&'(           !"#$%&'(              !"#$%&'(


Saturday, December 1, 2012                                                           4
Circa
                                                 2008



                             EC2           EBS
                             API           API




                                    Proprietary Orchestration Services




                             Open Source Xen
                                 (custom)
                                                   Security Groups   EBS          S3


                                !"#$%&'(             !"#$%&'(              !"#$%&'(


Saturday, December 1, 2012                                                             4
Circa
                                                 2008



                             EC2           EBS
                                                   S3 API
                             API           API




                                    Proprietary Orchestration Services




                             Open Source Xen
                                 (custom)
                                                   Security Groups   EBS          S3


                                !"#$%&'(             !"#$%&'(              !"#$%&'(


Saturday, December 1, 2012                                                             4
Circa
                                                 2008



                             EC2           EBS
                                                   S3 API
                             API           API




                                    Proprietary Orchestration Services




  Secret
  Sauce                      Open Source Xen
                                 (custom)
                                                   Security Groups   EBS          S3


                                !"#$%&'(             !"#$%&'(              !"#$%&'(


Saturday, December 1, 2012                                                             4
The	
  Virtual	
  Datacenter	
  OS	
  allows	
  businesses	
  to	
  
                efficiently	
  pool	
  all	
  types	
  of	
  hardware	
  resources	
  -­‐	
  
                servers,	
  storage	
  and	
  network	
  –	
  into	
  an	
  aggregated	
  
                on-­‐premise	
  cloud
                  -­‐	
  VMWare	
  Press	
  Release	
  9/2008




Saturday, December 1, 2012                                                                    5
The	
  Virtual	
  Datacenter	
  OS	
  allows	
  businesses	
  to	
  
                efficiently	
  pool	
  all	
  types	
  of	
  hardware	
  resources	
  -­‐	
  
                servers,	
  storage	
  and	
  network	
  –	
  into	
  an	
  aggregated	
  
                on-­‐premise	
  cloud
                  -­‐	
  VMWare	
  Press	
  Release	
  9/2008

              Eucalyptus	
  is	
  the	
  only	
  cloud	
  architecture	
  to	
  support	
  
              the	
  same	
  applicaKon	
  programming	
  interfaces	
  (APIs)	
  
              as	
  public	
  clouds,	
  and	
  today	
  Eucalyptus	
  is	
  fully	
  
              compaKble	
  with	
  the	
  Amazon	
  AWS	
  public	
  cloud	
  
              infrastructure.	
  

              Eucalyptus	
  Systems	
  Press	
  Release	
  4/2009

Saturday, December 1, 2012                                                                    5
2.0          AWS             2.2       3.0 Quality     4.X
                                 Prototype   1.0 GA
                                                      Refactor     Compatibility   Refactor   Improvements   Refactor

                        2008                 2009                 2010                   2011                2012




                             Sept 2008:         Nov 2009:        May 2010:           July 2011:       April 2012:
                             VMOps              CloudStack       Cloud.com           Citrix           Apache
                             Founded            1.0 GA           Launch &            Acquires         CloudStack
                                                                 CloudStack          Cloud.com
                                                                 2.0 GA




Saturday, December 1, 2012                                                                                              6
Circa
                             3/2009




Saturday, December 1, 2012            7
Circa
                                        3/2009




                             !"#$%&'(


Saturday, December 1, 2012                       7
Circa
                                                   3/2009




                             !"#$%&'(   !"#$%&'(


Saturday, December 1, 2012                                  7
Circa
                                                   3/2009




                             !"#$%&'(   !"#$%&'(     !"#$%&'(


Saturday, December 1, 2012                                      7
Circa
                                                          3/2009




                             Open Source Xen
                                 (custom)



                                !"#$%&'(       !"#$%&'(     !"#$%&'(


Saturday, December 1, 2012                                             7
Circa
                                                                  3/2009




                             Open Source Xen
                                               Virtual Networks
                                 (custom)



                                !"#$%&'(          !"#$%&'(          !"#$%&'(


Saturday, December 1, 2012                                                     7
Circa
                                                                  3/2009




                             Open Source Xen
                                 (custom)
                                               Virtual Networks   NFS


                                !"#$%&'(          !"#$%&'(              !"#$%&'(


Saturday, December 1, 2012                                                         7
Circa
                                                                     3/2009




                                      Proprietary Control Plane


                             Open Source Xen
                                 (custom)
                                                  Virtual Networks   NFS


                                !"#$%&'(             !"#$%&'(              !"#$%&'(


Saturday, December 1, 2012                                                            7
Circa
                                                                      3/2009




                             Proprietary Orchestration Services



                                       Proprietary Control Plane


                              Open Source Xen
                                  (custom)
                                                   Virtual Networks   NFS


                                 !"#$%&'(             !"#$%&'(              !"#$%&'(


Saturday, December 1, 2012                                                             7
Circa
                                                                      3/2009




                                                 GUI




                             Proprietary Orchestration Services



                                       Proprietary Control Plane


                              Open Source Xen
                                  (custom)
                                                   Virtual Networks   NFS


                                 !"#$%&'(              !"#$%&'(             !"#$%&'(


Saturday, December 1, 2012                                                             7
Circa
                                                                      3/2009




                                                 GUI




                             Proprietary Orchestration Services



                                       Proprietary Control Plane


                              Open Source Xen
                                  (custom)
                                                   Virtual Networks   NFS


                                 !"#$%&'(              !"#$%&'(         !"#$%&'(


Saturday, December 1, 2012                                                         8
Circa
                                                                      3/2009




                                                              GWT
                                                 GUI




                             Proprietary Orchestration Services



                                       Proprietary Control Plane


                              Open Source Xen
                                  (custom)
                                                   Virtual Networks   NFS


                                 !"#$%&'(              !"#$%&'(         !"#$%&'(


Saturday, December 1, 2012                                                         8
Circa
                                                                      3/2009




                                                              GWT
                                                 GUI




                                                                            Java App
                                                                             Server
                             Proprietary Orchestration Services



                                       Proprietary Control Plane


                              Open Source Xen
                                  (custom)
                                                   Virtual Networks   NFS


                                 !"#$%&'(              !"#$%&'(         !"#$%&'(


Saturday, December 1, 2012                                                             8
Circa
                                                                      3/2009




                                                              GWT
                                                 GUI




                                                                            Java App
                                                                             Server
                             Proprietary Orchestration Services

                                                                                 JSON
                                                                               over TCP
                                       Proprietary Control Plane


                              Open Source Xen
                                  (custom)
                                                   Virtual Networks   NFS


                                 !"#$%&'(              !"#$%&'(         !"#$%&'(


Saturday, December 1, 2012                                                                8
Circa
                                                                      3/2009




                                                              GWT
                                                 GUI




                                                                            Java App
                                                                             Server
                             Proprietary Orchestration Services

                                                                                 JSON
                                                                               over TCP
                                       Proprietary Control Plane

                                                                                       NFS (ZFS on
                              Open Source Xen                                          OpenSolaris)
                                  (custom)
                                                   Virtual Networks   NFS


                                 !"#$%&'(              !"#$%&'(         !"#$%&'(


Saturday, December 1, 2012                                                                            8
Circa
                                                                            3/2009




                                                              GWT
                                                 GUI




                                                                                  Java App
                                                                                   Server
                             Proprietary Orchestration Services

                                                                                       JSON
                                                                                     over TCP
                                       Proprietary Control Plane
                                                                    UDP
                                                                  Tunnels                    NFS (ZFS on
                              Open Source Xen                                                OpenSolaris)
                                  (custom)
                                                   Virtual Networks         NFS


                                 !"#$%&'(              !"#$%&'(               !"#$%&'(


Saturday, December 1, 2012                                                                                  8
Circa
                                                                            3/2009




                                                              GWT
                                                 GUI




                                                                                  Java App
                                                                                   Server
                             Proprietary Orchestration Services

                                                                                       JSON
                                                                                     over TCP
                                       Proprietary Control Plane
                                                                    UDP
                                                                  Tunnels                    NFS (ZFS on
                               Open Source Xen                                               OpenSolaris)
                             Agent (custom)
                                                   Virtual Networks         NFS


                                 !"#$%&'(              !"#$%&'(               !"#$%&'(


Saturday, December 1, 2012                                                                                  8
Circa
                                                                            3/2009




                                                              GWT
                                                 GUI




                                                                                  Java App
                                                                                   Server
                             Proprietary Orchestration Services

                                                                                        JSON
                                                                                      over TCP
                                       Proprietary Control Plane
                                                                    UDP
                                                                  Tunnels                    NFS (ZFS on
                               Open Source Xen                                Agent          OpenSolaris)
                             Agent (custom)
                                                   Virtual Networks         NFS


                                 !"#$%&'(              !"#$%&'(               !"#$%&'(


Saturday, December 1, 2012                                                                                  8
Availability Zone Architecture




Saturday, December 1, 2012                           9
Availability Zone Architecture




                                              L3 core




Saturday, December 1, 2012                              9
Availability Zone Architecture
                                                End users




                                              DC Edge


                                              L3 core




Saturday, December 1, 2012                                  9
Availability Zone Architecture
                                                End users




                                              DC Edge


                                              L3 core

             Access Sw

       Hypervisor (Xen)




   Primary Storage
         ZFS

                             Pod

Saturday, December 1, 2012                                  9
Availability Zone Architecture
                                                End users




                                              DC Edge


                                              L3 core

             Access Sw

       Hypervisor (Xen)




   Primary Storage
         ZFS

                             Pod   Pod

Saturday, December 1, 2012                                  9
Availability Zone Architecture
                                                 End users




                                               DC Edge


                                               L3 core

             Access Sw

       Hypervisor (Xen)




   Primary Storage
         ZFS

                             Pod   Pod   Pod

Saturday, December 1, 2012                                   9
Availability Zone Architecture
                                                       End users




                                                     DC Edge


                                                     L3 core

             Access Sw

       Hypervisor (Xen)




   Primary Storage
         ZFS

                             Pod   Pod   Pod   Pod

Saturday, December 1, 2012                                         9
Availability Zone Architecture
                                                       End users




                                                     DC Edge


                                                     L3 core

             Access Sw

       Hypervisor (Xen)




   Primary Storage
         ZFS

                             Pod   Pod   Pod   Pod

Saturday, December 1, 2012                                         9
Availability Zone Architecture
                                                       End users




                                                     DC Edge


                                                     L3 core

             Access Sw

       Hypervisor (Xen)




   Primary Storage
         ZFS

                             Pod   Pod   Pod   Pod                 Pod

Saturday, December 1, 2012                                               9
Availability Zone Architecture
                                                          End users
        Admin/User API

                             CloudStack
                                                        DC Edge
                                 MySQL

                                                        L3 core

             Access Sw

       Hypervisor (Xen)




   Primary Storage
         ZFS

                              Pod     Pod   Pod   Pod                 Pod

Saturday, December 1, 2012                                                  9
Availability Zone Architecture
                                                          End users
        Admin/User API

                             CloudStack
                                                        DC Edge
                                 MySQL
                                                         ECMP/BGP
                                                        L3 core

             Access Sw

       Hypervisor (Xen)




   Primary Storage
         ZFS

                              Pod     Pod   Pod   Pod                 Pod

Saturday, December 1, 2012                                                  9
Multiple zones
                             CloudStack

                                 MySQL




                                 Zone1




Saturday, December 1, 2012                            10
Multiple zones
                             CloudStack

                                 MySQL




                                          Internet




                                 Zone1      Zone2    Zone3




Saturday, December 1, 2012                                   10
Network Virtualization




Saturday, December 1, 2012                            11
Network Virtualization
                                                   Tenant	
  	
  1	
  Virtual	
  Network	
  10.1.1.0/24


                                                                                       Tenant	
  1	
      	
  10.1.1.2
                                      Gateway	
  address	
                              VM	
  1
                                      10.1.1.1

                                                                                       Tenant	
  1	
      	
  10.1.1.3
                                                                                        VM	
  2


                                                                                       Tenant	
  1	
   	
  10.1.1.4
                                                                                        VM	
  3


                                                                                       Tenant	
  1	
      	
  10.1.1.5
                                                                                        VM	
  4




Saturday, December 1, 2012                                                                                               11
Network Virtualization
                                                                 Tenant	
  	
  1	
  Virtual	
  Network	
  10.1.1.0/24


                                                                                                     Tenant	
  1	
      	
  10.1.1.2
                                                    Gateway	
  address	
                              VM	
  1
                                                    10.1.1.1

                                         Tenant	
  1	
                                               Tenant	
  1	
      	
  10.1.1.3
                                      Virtual	
  Router                                               VM	
  2


                                   NAT
            Internet               DHCP                                                              Tenant	
  1	
   	
  10.1.1.4
                                   FW                                                                 VM	
  3
                                   Load	
  Balancing

                                                                                                     Tenant	
  1	
      	
  10.1.1.5
                                                                                                      VM	
  4




Saturday, December 1, 2012                                                                                                             11
Network Virtualization
                                                                                                       Tenant	
  	
  1	
  Virtual	
  Network	
  10.1.1.0/24


                                              Public	
  IP	
  address	
                                                                    Tenant	
  1	
      	
  10.1.1.2
                                              65.37.141.11                                Gateway	
  address	
                              VM	
  1
                                              65.37.141.36                                10.1.1.1
                             Public	
  Network
                                                                               Tenant	
  1	
                                               Tenant	
  1	
      	
  10.1.1.3
                                                                            Virtual	
  Router                                               VM	
  2


                                                                      NAT
            Internet                                                  DHCP                                                                 Tenant	
  1	
   	
  10.1.1.4
                                                                      FW                                                                    VM	
  3
                                                                      Load	
  Balancing

                                                                                                                                           Tenant	
  1	
      	
  10.1.1.5
                                                                                                                                            VM	
  4




Saturday, December 1, 2012                                                                                                                                                   11
Network Virtualization
                                                                                                          Tenant	
  	
  1	
  Virtual	
  Network	
  10.1.1.0/24


                                              Public	
  IP	
  address	
                                                                       Tenant	
  1	
      	
  10.1.1.2
                                              65.37.141.11                                   Gateway	
  address	
                              VM	
  1
                                              65.37.141.36                                   10.1.1.1
                             Public	
  Network
                                                                                  Tenant	
  1	
                                               Tenant	
  1	
      	
  10.1.1.3
                                                                               Virtual	
  Router                                               VM	
  2


                                                                        NAT
            Internet                                                    DHCP                                                                  Tenant	
  1	
   	
  10.1.1.4
                                                                        FW                                                                     VM	
  3
                                                                        Load	
  Balancing

                                                                                                                                              Tenant	
  1	
      	
  10.1.1.5
                                                                                                                                               VM	
  4


                                                                                                          Tenant	
  2	
  Virtual	
  Network	
  10.1.1.0/24
                                                 Public	
  IP	
  address	
  
                                                 65.37.141.24                                Gateway	
  address	
                             Tenant	
  2	
  
                                                                                                                                                                 	
  10.1.1.2
                                                                                             10.1.1.1                                          VM	
  1
                                                 65.37.141.80

                                                                              Tenant	
  2	
                                                   Tenant	
  2	
      	
  10.1.1.3
                                                                           Virtual	
  Router	
                                                 VM	
  2
                                                                             Appliance

                                                                        FW
                                                                        NAT                                                                   Tenant	
  2	
      	
  10.1.1.4
                                                                        DHCP                                                                   VM	
  3




Saturday, December 1, 2012                                                                                                                                                      11
Virtual Machine Placement
                                                           End users


                             CloudStack
                                                         DC Edge
                                 MySQL

                                                        L3/L2 core




       Hypervisor (Xen)




   Primary Storage
         ZFS

                              Pod     Pod   Pod   Pod                  Pod

Saturday, December 1, 2012                                                   12
Network Virtualization - why?


                    AWS-style security groups “unfamiliar”

                    Adopt traditional L2 model for end-users

                             VLANs do not scale

                             use network virtualization to realize this




Saturday, December 1, 2012                                                13
Network Virtualization
                    UDP encapsulation of Ethernet Frames

                             similar to VXLAN (128-bit address vs 24-bit
                             for VXLAN)

                             Driver in dom0

                    Virtual Router to provide edge services

                             including Load Balancing

                             Hand-rolled Fedora 8 para-virtualized




Saturday, December 1, 2012                                                 14
ZFS (Volume Service)


                             ZFS for highly scalable, reliable storage

                               RAID-Z

                               SSD cache

                             NFS for shared storage

                             Hand-rolled version of OpenSolaris 2008.11




Saturday, December 1, 2012                                                15
Orchestration




Saturday, December 1, 2012                   16
Orchestration
                                             JVM




Saturday, December 1, 2012                         16
Orchestration
                                             JVM




                                              MySQL



Saturday, December 1, 2012                            16
Orchestration
                                                   JVM




                                     Message Bus




                                                    MySQL



Saturday, December 1, 2012                                  16
Orchestration
                                                            JVM
                             Front-
                              end




                                              Message Bus




                                                             MySQL



Saturday, December 1, 2012                                           16
Orchestration
                                                              JVM
                                        VM
                             Front-
                                      Orchest
                              end
                                       rator




                                                Message Bus




                                                               MySQL



Saturday, December 1, 2012                                             16
Orchestration
                                                                   JVM
                                        VM      Network
                             Front-
                                      Orchest   Orchestr
                              end
                                       rator      ator




                                                     Message Bus




                                                                    MySQL



Saturday, December 1, 2012                                                  16
Orchestration
                                                                     JVM
                                        VM      Network    Storage
                             Front-
                                      Orchest   Orchestr   Orchest
                              end
                                       rator      ator      rator




                                                     Message Bus




                                                                      MySQL



Saturday, December 1, 2012                                                    16
Orchestration
                                                                                JVM
                                        VM      Network    Storage    Host/
                             Front-
                                      Orchest   Orchestr   Orchest   Resource
                              end
                                       rator      ator      rator    Manager




                                                     Message Bus




                                                                                 MySQL



Saturday, December 1, 2012                                                               16
Orchestration
                                                                                    JVM
                                            VM      Network    Storage    Host/
                             Front-
                                          Orchest   Orchestr   Orchest   Resource
                              end
                                           rator      ator      rator    Manager




                                                         Message Bus




                               Hypervi
                                Hypervi
                                 Hypervisor
                                  sor
                                   Agent
                                   sor
                                      (JVM)


                                                                                     MySQL



Saturday, December 1, 2012                                                                   16
Orchestration
                                                                                       JVM
                                            VM      Network       Storage    Host/
                             Front-
                                          Orchest   Orchestr      Orchest   Resource
                              end
                                           rator      ator         rator    Manager




                                                             Message Bus




                               Hypervi
                                Hypervi
                                 Hypervisor     Storage
                                                 Storage
                                                   Storage
                                  sor
                                   Agent
                                   sor           Agent
                                                    Agent
                                                  Agent
                                      (JVM)         (JVM)


                                                                                        MySQL



Saturday, December 1, 2012                                                                      16
Orchestration
                                                                                       JVM
                                            VM      Network       Storage    Host/
                             Front-
                                          Orchest   Orchestr      Orchest   Resource
                              end
                                           rator      ator         rator    Manager




                                                             Message Bus




                               Hypervi
                                Hypervi
                                 Hypervisor     Storage
                                                 Storage
                                                   Storage       Console
                                                                  Console
                                  sor
                                   Agent
                                   sor           Agent
                                                    Agent
                                                  Agent           Agent
                                                                   Agent
                                      (JVM)         (JVM)


                                                                                        MySQL



Saturday, December 1, 2012                                                                      16
Orchestration
                                                                                           JVM
                                                VM      Network       Storage    Host/
                       GWT-RPC   Front-
     Browser                                  Orchest   Orchestr      Orchest   Resource
                                  end
                                               rator      ator         rator    Manager




                                                                 Message Bus




                                   Hypervi
                                    Hypervi
                                     Hypervisor     Storage
                                                     Storage
                                                       Storage       Console
                                                                      Console
                                      sor
                                       Agent
                                       sor           Agent
                                                        Agent
                                                      Agent           Agent
                                                                       Agent
                                          (JVM)         (JVM)


                                                                                            MySQL



Saturday, December 1, 2012                                                                          16
Reject Pile (2009)
                             Local disks of hypervisors presented as
                             network block storage with network RAID

                               Too complex (sheepdog-like)

                               Split-brain issues

                             DRBD

                               Too complex to automate recovery

                             Commercial Arrays

                               Limited volumes, limited snapshots
Saturday, December 1, 2012                                             17
Reject Pile (2009)
                             Security Groups

                               Not well understood

                             VLANs

                               4K limit

                             Physical Network Appliances

                               No APIs (Cisco at least)

                               Can’t decapsulate UDP tunnel

Saturday, December 1, 2012                                    18
Reject Pile (early 2009)

                             KVM

                              Unstable

                              Windows support

                              No CPU allocation




Saturday, December 1, 2012                        19
April 2009




Saturday, December 1, 2012                20
April 2009
                                          “5 guys in
                                          a garage”




Saturday, December 1, 2012                             20
Ship It! ( Beta 6/2009)

                             ISO/CD with Xen 3.3 / CentOs customized

                             ISO/CD with OSOL 2008.11 + patches +
                             drivers

                             Java binaries

                             System VM: Fedora Core 8 VM bits including
                               dnsmasq, iptables, HAProxy


                             Installers, documentation



Saturday, December 1, 2012                                                21
Road to 1.0
                             ISCSI - ZFS zvols -> ∞ snapshots

                             Performance tuning (UDP tunnels, zvol)

                             Usage, metering,

                             Formal user / admin Query-based API

                             JQuery-based GUI - skinnable

                             Windows PV drivers




Saturday, December 1, 2012                                            22
More rejects (2009)

                             Network RAID of zvols over ISCSI/md

                               Performance

                               Re-mirror time / penalty

                               Consistent snapshots

                               Split-brain



Saturday, December 1, 2012                                         23
1.0 - Ship It (Nov 2009)


                             6 engineers + 3 QA + 1 sales

                             3 beta sites

                             Traction with SPs interested in competing
                             with AWS




Saturday, December 1, 2012                                               24
Orchestration + Usage server

                                                      JVM

                              Front-
                               end




Saturday, December 1, 2012                                  25
Orchestration + Usage server

                                                      JVM   MySQL



                              Front-
                               end




Saturday, December 1, 2012                                          25
Orchestration + Usage server
     Browser



           LoadBalan
                                                      JVM   MySQL

              ced
             HTTP
             Query            Front-
              API              end




Saturday, December 1, 2012                                          25
Orchestration + Usage server
     Browser



           LoadBalan
                                                                                 JVM   MySQL

              ced
             HTTP                        VM      Network    Storage    Host/
             Query            Front-
              API
                                       Orchest   Orchestr   Orchest   Resource
                               end
                                        rator      ator      rator    Manager




Saturday, December 1, 2012                                                                     25
Orchestration + Usage server
     Browser



           LoadBalan
                                                                                       JVM   MySQL

              ced
             HTTP                            VM        Network    Storage    Host/
             Query            Front-
              API
                                           Orchest     Orchestr   Orchest   Resource
                               end
                                            rator        ator      rator    Manager




                                                             Message Bus




                               Hypervi
                                Hypervi
                                 Hypervisor    Storage
                                                Storage
                                                  Storage       Console
                                                               Console
                                  sor
                                   Agent
                                   sor          Agent
                                                   Agent
                                                 Agent           Agent
                                                                Agent
                                   (JVM)             (JVM)




Saturday, December 1, 2012                                                                           25
Orchestration + Usage server
     Browser



           LoadBalan
                                                                            JVM   MySQL

              ced                                      Storage
                                       VM
                                        VM    Network
                                               Network Storage
             HTTP            Front-
                              Front- Orchest    Network Storage Host/
                                         VM Orchestr Orchest      Host/
             Query             Front- Orchest Orchestr Orchest Resource
                              end
              API              end     Orchest Orchestr rator
                                      rator     ator     Orchest Resource
                                end    rator     ator    rator
                                        rator     ator    rator Manager
                                                                 Manager


    Clustered
                                                           Message Bus
  Orchestration                                             Message Bus
                                                             Message Bus
     Server


                               Hypervi
                                Hypervi
                                 Hypervisor   Storage
                                               Storage
                                                 Storage        Console
                                                               Console
                                  sor
                                   Agent
                                   sor         Agent
                                                  Agent
                                                Agent            Agent
                                                                Agent
                                   (JVM)          (JVM)




Saturday, December 1, 2012                                                                25
Orchestration + Usage server
     Browser



           LoadBalan
                                                                            JVM   MySQL

              ced                                      Storage
                                       VM
                                        VM    Network
                                               Network Storage
             HTTP            Front-
                              Front- Orchest    Network Storage Host/
                                         VM Orchestr Orchest      Host/
             Query             Front- Orchest Orchestr Orchest Resource
                              end
              API              end     Orchest Orchestr rator
                                      rator     ator     Orchest Resource
                                end    rator     ator    rator
                                        rator     ator    rator Manager
                                                                 Manager

                                                                                  Usage
    Clustered
                                                           Message Bus
  Orchestration                                             Message Bus
                                                             Message Bus
     Server


                               Hypervi
                                Hypervi
                                 Hypervisor   Storage
                                               Storage
                                                 Storage        Console
                                                               Console
                                  sor
                                   Agent
                                   sor         Agent
                                                  Agent
                                                Agent            Agent
                                                                Agent
                                   (JVM)          (JVM)




Saturday, December 1, 2012                                                                25
1.0 Issues
                             Multicast scaling

                             Oracle acquisition of OpenSolaris

                             Hiring “full-stack” developers

                             5 startups - Hypervisor + Storage + Network
                             Virtualization + Network appliance +
                             Orchestration




Saturday, December 1, 2012                                                 26
Multicast Scaling
                             Broadcast (ARP) / Multicast from VM
                             translates to UDP multicast

                             Switches snoop on IGMP to prevent flooding
                             all ports

                             Most switches fall-over after 64 Multicast
                             groups

                               each tenant -> 1 multicast group




Saturday, December 1, 2012                                                27
Road Ahead
                             Multi-hypervisor, SAN, and VLAN support

                               Use commercial XenServer

                               Commercial filers / ISCSI arrays

                               Support KVM + NFS

                             Product first, architecture second

                             From web hosting to enterprise workload




Saturday, December 1, 2012                                             28
Road Ahead

                             Throw out custom hypervisor and
                             OpenSolaris

                             VLANs

                             Secondary Storage to store snapshots and
                             images




Saturday, December 1, 2012                                              29
Circa
                                                                      3/2010




                                                 GUI




                             Proprietary Orchestration Services



                                       Proprietary Control Plane


                               Open Source Xen                          Storage
                                                                      Agent
                                                   Virtual Networks
                             Agent (custom)                             Arrays


                                 !"#$%&'(              !"#$%&'(               !"#$%&'(


Saturday, December 1, 2012                                                               30
Circa
                                                                       3/2010




                                                              JQuery
                                                 GUI




                             Proprietary Orchestration Services



                                       Proprietary Control Plane


                               Open Source Xen                           Storage
                                                                       Agent
                                                   Virtual Networks
                             Agent (custom)                              Arrays


                                 !"#$%&'(              !"#$%&'(                !"#$%&'(


Saturday, December 1, 2012                                                                30
Circa
                                                                           3/2010




                                                              JQuery
                                                 GUI




                             Proprietary Orchestration Services



                                       Proprietary Control Plane

                                                                   VLANs
                               Open Source Xen                               Storage
                                                                           Agent
                                                   Virtual Networks
                             Agent (custom)                                  Arrays


                                 !"#$%&'(              !"#$%&'(                    !"#$%&'(


Saturday, December 1, 2012                                                                    30
Circa
                                                                           3/2010




                                                              JQuery
                                                 GUI




                             Proprietary Orchestration Services



                                       Proprietary Control Plane

                                                                   VLANs
                               Open Source Xen                              Storage
                                                   Virtual Networks
                             Agent (custom)                                 Arrays


                                 !"#$%&'(              !"#$%&'(              !"#$%&'(


Saturday, December 1, 2012                                                              30
Circa
                                                                           3/2010




                                                              JQuery
                                                 GUI




                             Proprietary Orchestration Services



                                       Proprietary Control Plane

                                                                   VLANs
                              Open Source Xen                               Storage
                                                   Virtual Networks
                                  (custom)                                  Arrays


                                 !"#$%&'(              !"#$%&'(              !"#$%&'(


Saturday, December 1, 2012                                                              30
Circa
                                                                         3/2010




                                                            JQuery
                                               GUI




                             Proprietary Orchestration Services



                                     Proprietary Control Plane

                                                                 VLANs
                                                                          Storage
                                                 Virtual Networks
                                                                          Arrays


                                !"#$%&'(             !"#$%&'(              !"#$%&'(


Saturday, December 1, 2012                                                            30
Circa
                                                                          3/2010




                                                             JQuery
                                                GUI




                             Proprietary Orchestration Services



                                      Proprietary Control Plane

                                                                  VLANs
                                Commercial                                 Storage
                                                  Virtual Networks
                                XenServer                                  Arrays


                                !"#$%&'(              !"#$%&'(              !"#$%&'(


Saturday, December 1, 2012                                                             30
Circa
                                                                          3/2010




                                                             JQuery
                                                GUI




                             Proprietary Orchestration Services



                                      Proprietary Control Plane
                                                                                       ISCSI/
                                                                  VLANs                  NFS
                                Commercial                                 Storage
                                                  Virtual Networks
                                XenServer                                  Arrays


                                !"#$%&'(              !"#$%&'(              !"#$%&'(


Saturday, December 1, 2012                                                                      30
Orchestration + Usage server
     Browser



           LoadBalan
                                                                                     JVM             MySQL

              ced              Front-
             HTTP               end         VM      Network    Storage     Host/
             Query
              API
                                          Orchest   Orchestr   Orchest    Resource
                                           rator      ator      rator     Manager
                              Job Queue
                                                                                                     Usage
     Clustered
                                                       Message Bus
   Orchestration
      Server


                                Hypervi
                                   XS
                                  sor
                                 Agent


                             XAPI
                                           KVM                       Console
                                                                 Console Agent
                                                                                       Secondary
                                          Agent                      Agent           Storage Agent
    XenServers                            (JVM)
                                                                    (JVM)
                                                                                         (JVM)

Saturday, December 1, 2012                                                                                   31
Availability Zone Architecture
                                                          End users
        Admin/User API

                             CloudStack
                                                        DC Edge
                                 MySQL




             Access Sw

    Hypervisor (Xen
    /VMWare/KVM)




                              Pod     Pod   Pod   Pod                 Pod

Saturday, December 1, 2012                                                  32
Availability Zone Architecture
                                                          End users
        Admin/User API

                             CloudStack
                                                        DC Edge
                                 MySQL




             Access Sw

    Hypervisor (Xen
    /VMWare/KVM)
                                                                            Secondary Storage




                              Pod     Pod   Pod   Pod                 Pod

Saturday, December 1, 2012                                                                 32
Availability Zone Architecture
                                                          End users
        Admin/User API

                             CloudStack
                                                        DC Edge
                                 MySQL
                                                        L2/L3 core


             Access Sw

    Hypervisor (Xen
    /VMWare/KVM)
                                                                            Secondary Storage




                              Pod     Pod   Pod   Pod                 Pod

Saturday, December 1, 2012                                                                 32
Availability Zone Architecture
                                                          End users
        Admin/User API

                             CloudStack
                                                        DC Edge
                                 MySQL
                                                        L2/L3 core


             Access Sw

    Hypervisor (Xen
    /VMWare/KVM)
                                                                            Secondary Storage




    Primary Storage
     NFS/ISCSI/FC

                              Pod     Pod   Pod   Pod                 Pod

Saturday, December 1, 2012                                                                 32
System VM

                             Data Path problem

                               Need to move bits between storage types

                               Present consoles to end users

                               Scale up / down these data path services




Saturday, December 1, 2012                                                33
System VMs


                             Virtual Routers, Console Proxy VMs,
                             Secondary Storage VMs

                               based on the same vm image

                             Orchestration and scaling baked in




Saturday, December 1, 2012                                         34
2.0 6/2010

                             Shipping just java binaries + system vm
                             image

                             Open sourced (GPL)

                             Company name changed to Cloud.com

                             New product - Cloud Portal



Saturday, December 1, 2012                                             35
Citrix Cloud Portal
                             Business and operations support platform for
                             Service Providers

                             Account management

                             Self-service portal

                             Billing and metering

                             CRM and support ticketing

                             Not OSS

Saturday, December 1, 2012                                                  36
Reject Pile (2010)
                             “Zone” and “Pod” VLANs to go beyond 4k
                             limit

                               Pod VLANs bridged using zone VLANs

                             Another SDN attempt using GRE on Open
                             vSwitch




Saturday, December 1, 2012                                            37
Flexibility and Scale
                                  (2010-2011)

                             Network, hypervisor and vm placement
                             flexibility

                             Tens of thousands of hypervisors

                             AWS API




Saturday, December 1, 2012                                          38
Network Flexibility (2.2
                          refactor)

                             Hardware appliances instead of virtual router

                             Upgrade / downgrade to/from hardware to
                             virtual

                             Pick and choose services (L2-L7)

                             Security Groups



Saturday, December 1, 2012                                                   39
Network Flexibility
          Scale	
  out	
  edge	
  services	
  using	
  virtual	
  appliances

                                                              10.1.1.0/24
                                                              VLAN	
  100




                                                                                         VM	
  1
                                                                            10.1.1.2

               65.37.141.111                       10.1.1.1
               65.37.141.112
                                       CS
                                  Virtual	
                                 10.1.1.3
                                                                                         VM	
  2
                                  Router
                               DHCP,	
  DNS
                               NAT
                                                                            10.1.1.4     VM	
  3
                               Load	
  Balancing



                                                                                         VM	
  4
                                                                              10.1.1.5




Saturday, December 1, 2012                                                                         40
Network Flexibility
          Scale	
  out	
  edge	
  services	
  using	
  virtual	
  appliances                           Scale	
  up	
  using	
  hardware	
  devices
                                                                                                                                                 10.1.1.0/24
                                                              10.1.1.0/24                                                                        VLAN	
  100
                                                              VLAN	
  100




                                                                                                   65.37.141.111                      10.1.1.1                 10.1.1.2   VM	
  1
                                                                                         VM	
  1
                                                                            10.1.1.2
                                                                                                                    Juniper	
  SRX
               65.37.141.111                       10.1.1.1                                                           Firewall       NAT,	
  
               65.37.141.112                                                                                                         VPN
                                       CS
                                                                                                                                                                          VM	
  2
                                  Virtual	
                                 10.1.1.3
                                                                                         VM	
  2                                                               10.1.1.3
                                  Router                                                            65.37.141.112                    10.1.1.112
                               DHCP,	
  DNS                                                                          Netscaler
                               NAT                                                                                     Load	
  
                                                                            10.1.1.4     VM	
  3                                                                          VM	
  3
                               Load	
  Balancing                                                                     Balancer                                  10.1.1.4




                                                                                         VM	
  4                                                                          VM	
  4
                                                                              10.1.1.5                                                                         10.1.1.5



                                                                                                                                                            CS
                                                                                                                                          DHCP,	
  
                                                                                                                                                          Virtual	
  
                                                                                                                                          DNS
                                                                                                                                                          Router




Saturday, December 1, 2012                                                                                                                                                          40
VM Placement


                             Customize vm placement

                               Pick hosts and storage according to rules

                               Affinity / anti-affinity




Saturday, December 1, 2012                                                 41
Multi-hypervisor

                             XenServer, KVM and VMWare in the same
                             zone

                             Differences in network and storage behavior

                             Different image formats

                             Same system vm image



Saturday, December 1, 2012                                                 42
Plugin Architecture (2.2)


                             Orchestration
                                Engine




Saturday, December 1, 2012                   43
Plugin Architecture (2.2)

                                               Plugin
                                             Framework

                             Orchestration
                                Engine




Saturday, December 1, 2012                               43
Plugin Architecture (2.2)
                                                         Hypervisor
                                                          Hypervisor
                                                          Plugins
                                                           Plugins
                                               Plugin
                                             Framework

                             Orchestration
                                Engine




Saturday, December 1, 2012                                             43
Plugin Architecture (2.2)
                                                         Hypervisor
                                                          Hypervisor
                                                          Plugins
                                                           Plugins
                                               Plugin
                                             Framework

                             Orchestration                Network
                                                           Network
                                                           Plugins
                                                            Plugins
                                Engine




Saturday, December 1, 2012                                             43
Plugin Architecture (2.2)
                                                         Hypervisor
                                                          Hypervisor
                                                          Plugins
                                                           Plugins
                                               Plugin
                                             Framework

                             Orchestration                Network
                                                           Network
                                                           Plugins
                                                            Plugins
                                Engine
                                                         Allocator
                                                          Allocator
                                                          Plugins
                                                           Plugins




Saturday, December 1, 2012                                             43
Plugin Architecture (2.2)
                                                         Hypervisor
                                                          Hypervisor
                                                          Plugins
                                                           Plugins
                                               Plugin
                                             Framework

                             Orchestration                Network
                                                           Network
                                                           Plugins
                                                            Plugins
                                Engine
                                                         Allocator
                                                          Allocator
                                                          Plugins
                                                           Plugins


                                                          Storage
                                                          Plugins




Saturday, December 1, 2012                                             43
Plugin Architecture
                                                          Hypervisor
                                                           Hypervisor
                                                           Plugins
                                                            Plugins
                                                Plugin
                                              Framework

                              Orchestration                Network
                                                            Network
                                                            Plugins
                                                             Plugins
                                 Engine
                                                          Allocator
                                                           Allocator
                                                           Plugins
                                                            Plugins




Saturday, December 1, 2012                                              44
Plugin Architecture                       •XenServer
                                                                       •VMWare

                                                                       •KVM

                                                                       •OracleVM


                                                          Hypervisor
                                                           Hypervisor
                                                           Plugins
                                                            Plugins
                                                Plugin
                                              Framework

                              Orchestration                Network
                                                            Network
                                                            Plugins
                                                             Plugins
                                 Engine
                                                          Allocator
                                                           Allocator
                                                           Plugins
                                                            Plugins




Saturday, December 1, 2012                                                          44
Plugin Architecture                       •XenServer
                                                                       •VMWare

                                                                       •KVM

                                                                       •OracleVM


                                                          Hypervisor
                                                           Hypervisor
                                                           Plugins
                                                            Plugins
                                                Plugin                     •Nicira
                                              Framework                    •Netscaler


                              Orchestration                Network
                                                            Network
                                                            Plugins
                                                                           •Brocade

                                                                           •MidoNet
                                                             Plugins
                                 Engine
                                                          Allocator
                                                           Allocator
                                                           Plugins
                                                            Plugins




Saturday, December 1, 2012                                                              44
Plugin Architecture                       •XenServer
                                                                       •VMWare

                                                                       •KVM

                                                                       •OracleVM


                                                          Hypervisor
                                                           Hypervisor
                                                           Plugins
                                                            Plugins
                                                Plugin                     •Nicira
                                              Framework                    •Netscaler


                              Orchestration                Network
                                                            Network
                                                            Plugins
                                                                           •Brocade

                                                                           •MidoNet
                                                             Plugins
                                 Engine
                                                                      Random
                                                                       •
                                                          Allocator
                                                           Allocator •User-
                                                           Plugins
                                                            Plugins
                                                                     concentrated
                                                                     •Intel TXT

                                                                     •Affinity




Saturday, December 1, 2012                                                              44
Security Groups


                                          Web	
                                      DB	
                                      Web	
  
                                          VM                                         VM                                        VM




                                         Web	
                                       Web	
                                     DB	
  
                                         VM                                          VM                                        VM

                                    …                                           …                                         …

                                         Web	
                                       Web	
  
                                         VM                                          VM



                             Ingress	
  Rule:	
  Allow	
  VMs	
  in	
  Web	
  Security	
  Group	
  access	
  to	
  VMs	
  in	
  DB	
  Security	
  Group	
  on	
  Port	
  3306




Saturday, December 1, 2012                                                                                                                                                      45
Security Groups


                                          Web	
                                      DB	
                                      Web	
  
                                          VM                                         VM                                        VM
                                                       Web
                                                     Security	
  
                                                      Group
                                         Web	
                                       Web	
                                     DB	
  
                                         VM                                          VM                                        VM

                                    …                                           …                                         …

                                         Web	
                                       Web	
  
                                         VM                                          VM



                             Ingress	
  Rule:	
  Allow	
  VMs	
  in	
  Web	
  Security	
  Group	
  access	
  to	
  VMs	
  in	
  DB	
  Security	
  Group	
  on	
  Port	
  3306




Saturday, December 1, 2012                                                                                                                                                      45
Security Groups


                                          Web	
                                      DB	
                                      Web	
  
                                          VM                                         VM                                        VM
                                                       Web                                        DB	
  
                                                     Security	
                                 Security	
  
                                                                                                 Group
                                                      Group
                                         Web	
                                       Web	
                                     DB	
  
                                         VM                                          VM                                        VM

                                    …                                           …                                         …

                                         Web	
                                       Web	
  
                                         VM                                          VM



                             Ingress	
  Rule:	
  Allow	
  VMs	
  in	
  Web	
  Security	
  Group	
  access	
  to	
  VMs	
  in	
  DB	
  Security	
  Group	
  on	
  Port	
  3306




Saturday, December 1, 2012                                                                                                                                                      45
Security Groups


                                          Web	
                                      DB	
                                      Web	
  
                                          VM                                         VM                                        VM
                                                       Web                                        DB	
  
                                                     Security	
                                 Security	
  
                                                                                                 Group
                                                      Group
                                         Web	
                                       Web	
                                     DB	
  
                                         VM                                          VM                                        VM

                                    …                                           …                                         …

                                         Web	
                                       Web	
  
                                         VM                                          VM



                             Ingress	
  Rule:	
  Allow	
  VMs	
  in	
  Web	
  Security	
  Group	
  access	
  to	
  VMs	
  in	
  DB	
  Security	
  Group	
  on	
  Port	
  3306




Saturday, December 1, 2012                                                                                                                                                      45
A million firewalls?

                             Manage the config state of 10^6 firewalls?

                               “Eventual consistency”

                             Iptable Rule explosion on hypervisor

                               use ‘ipset’ to optimize lookup




Saturday, December 1, 2012                                              46
2.1 -> 3.02

                             Features, features features

                               VMWare, Oracle VM, Baremetal

                               Multi-tier networking, VPC, VPN, more
                               vendors

                               EC2 API, S3 API

                               SDN

                               Focus on quality
Saturday, December 1, 2012                                             47
Evolution of CloudStack Architecture (Collab 2012)
Evolution of CloudStack Architecture (Collab 2012)
Evolution of CloudStack Architecture (Collab 2012)
Evolution of CloudStack Architecture (Collab 2012)
Evolution of CloudStack Architecture (Collab 2012)
Evolution of CloudStack Architecture (Collab 2012)
Evolution of CloudStack Architecture (Collab 2012)
Evolution of CloudStack Architecture (Collab 2012)
Evolution of CloudStack Architecture (Collab 2012)
Evolution of CloudStack Architecture (Collab 2012)
Evolution of CloudStack Architecture (Collab 2012)
Evolution of CloudStack Architecture (Collab 2012)
Evolution of CloudStack Architecture (Collab 2012)
Evolution of CloudStack Architecture (Collab 2012)
Evolution of CloudStack Architecture (Collab 2012)
Evolution of CloudStack Architecture (Collab 2012)
Evolution of CloudStack Architecture (Collab 2012)

More Related Content

Viewers also liked

Bare Metal to OpenStack with Razor and Chef
Bare Metal to OpenStack with Razor and ChefBare Metal to OpenStack with Razor and Chef
Bare Metal to OpenStack with Razor and Chef
Matt Ray
 
Architecting data center networks in the era of big data and cloud
Architecting data center networks in the era of big data and cloudArchitecting data center networks in the era of big data and cloud
Architecting data center networks in the era of big data and cloud
bradhedlund
 
CouchDB Vs MongoDB
CouchDB Vs MongoDBCouchDB Vs MongoDB
CouchDB Vs MongoDB
Gabriele Lana
 
AWS Re:Invent - High Availability Architecture at Netflix
AWS Re:Invent - High Availability Architecture at NetflixAWS Re:Invent - High Availability Architecture at Netflix
AWS Re:Invent - High Availability Architecture at Netflix
Adrian Cockcroft
 
La gestión de los recursos humanos
La gestión de los recursos humanosLa gestión de los recursos humanos
La gestión de los recursos humanos
Toni Cañabate
 
【初心者向け】Go言語勉強会資料
 【初心者向け】Go言語勉強会資料 【初心者向け】Go言語勉強会資料
【初心者向け】Go言語勉強会資料
Yuji Otani
 
WordPress Security from WordCamp NYC 2012
WordPress Security from WordCamp NYC 2012WordPress Security from WordCamp NYC 2012
WordPress Security from WordCamp NYC 2012
Brad Williams
 
Gensim
GensimGensim
Gensim
saireya _
 
Apache Arrow (Strata-Hadoop World San Jose 2016)
Apache Arrow (Strata-Hadoop World San Jose 2016)Apache Arrow (Strata-Hadoop World San Jose 2016)
Apache Arrow (Strata-Hadoop World San Jose 2016)
Wes McKinney
 
50 communicationactivitiesicebreakersandexercises
50 communicationactivitiesicebreakersandexercises50 communicationactivitiesicebreakersandexercises
50 communicationactivitiesicebreakersandexercises
Jerry Saribun
 
MongoDB, E-commerce and Transactions
MongoDB, E-commerce and TransactionsMongoDB, E-commerce and Transactions
MongoDB, E-commerce and Transactions
Steven Francia
 
[DSC 2016] 系列活動:李宏毅 / 一天搞懂深度學習
[DSC 2016] 系列活動:李宏毅 / 一天搞懂深度學習[DSC 2016] 系列活動:李宏毅 / 一天搞懂深度學習
[DSC 2016] 系列活動:李宏毅 / 一天搞懂深度學習
台灣資料科學年會
 

Viewers also liked (12)

Bare Metal to OpenStack with Razor and Chef
Bare Metal to OpenStack with Razor and ChefBare Metal to OpenStack with Razor and Chef
Bare Metal to OpenStack with Razor and Chef
 
Architecting data center networks in the era of big data and cloud
Architecting data center networks in the era of big data and cloudArchitecting data center networks in the era of big data and cloud
Architecting data center networks in the era of big data and cloud
 
CouchDB Vs MongoDB
CouchDB Vs MongoDBCouchDB Vs MongoDB
CouchDB Vs MongoDB
 
AWS Re:Invent - High Availability Architecture at Netflix
AWS Re:Invent - High Availability Architecture at NetflixAWS Re:Invent - High Availability Architecture at Netflix
AWS Re:Invent - High Availability Architecture at Netflix
 
La gestión de los recursos humanos
La gestión de los recursos humanosLa gestión de los recursos humanos
La gestión de los recursos humanos
 
【初心者向け】Go言語勉強会資料
 【初心者向け】Go言語勉強会資料 【初心者向け】Go言語勉強会資料
【初心者向け】Go言語勉強会資料
 
WordPress Security from WordCamp NYC 2012
WordPress Security from WordCamp NYC 2012WordPress Security from WordCamp NYC 2012
WordPress Security from WordCamp NYC 2012
 
Gensim
GensimGensim
Gensim
 
Apache Arrow (Strata-Hadoop World San Jose 2016)
Apache Arrow (Strata-Hadoop World San Jose 2016)Apache Arrow (Strata-Hadoop World San Jose 2016)
Apache Arrow (Strata-Hadoop World San Jose 2016)
 
50 communicationactivitiesicebreakersandexercises
50 communicationactivitiesicebreakersandexercises50 communicationactivitiesicebreakersandexercises
50 communicationactivitiesicebreakersandexercises
 
MongoDB, E-commerce and Transactions
MongoDB, E-commerce and TransactionsMongoDB, E-commerce and Transactions
MongoDB, E-commerce and Transactions
 
[DSC 2016] 系列活動:李宏毅 / 一天搞懂深度學習
[DSC 2016] 系列活動:李宏毅 / 一天搞懂深度學習[DSC 2016] 系列活動:李宏毅 / 一天搞懂深度學習
[DSC 2016] 系列活動:李宏毅 / 一天搞懂深度學習
 

More from Chiradeep Vittal

Loadbalancers: The fabric for your micro services
Loadbalancers: The fabric for your micro servicesLoadbalancers: The fabric for your micro services
Loadbalancers: The fabric for your micro services
Chiradeep Vittal
 
Load Balancing for Containers and Cloud Native Architecture
Load Balancing for Containers and Cloud Native ArchitectureLoad Balancing for Containers and Cloud Native Architecture
Load Balancing for Containers and Cloud Native Architecture
Chiradeep Vittal
 
Load Balancing for Containers and Cloud Native Architecture
Load Balancing for Containers and Cloud Native ArchitectureLoad Balancing for Containers and Cloud Native Architecture
Load Balancing for Containers and Cloud Native Architecture
Chiradeep Vittal
 
Directions for CloudStack Networking
Directions for CloudStack  NetworkingDirections for CloudStack  Networking
Directions for CloudStack Networking
Chiradeep Vittal
 
Private cloud networking_cloudstack_days_austin
Private cloud networking_cloudstack_days_austinPrivate cloud networking_cloudstack_days_austin
Private cloud networking_cloudstack_days_austin
Chiradeep Vittal
 
StackWatch: A prototype CloudWatch service for CloudStack
StackWatch: A prototype CloudWatch service for CloudStackStackWatch: A prototype CloudWatch service for CloudStack
StackWatch: A prototype CloudWatch service for CloudStack
Chiradeep Vittal
 
Network Functions Virtualization and CloudStack
Network Functions Virtualization and CloudStackNetwork Functions Virtualization and CloudStack
Network Functions Virtualization and CloudStack
Chiradeep Vittal
 
CloudStack Networking Deepdive CCCEU13
CloudStack Networking Deepdive CCCEU13CloudStack Networking Deepdive CCCEU13
CloudStack Networking Deepdive CCCEU13
Chiradeep Vittal
 
StackMate - CloudFormation for CloudStack
StackMate - CloudFormation for CloudStackStackMate - CloudFormation for CloudStack
StackMate - CloudFormation for CloudStack
Chiradeep Vittal
 
SDN in Apache CloudStack (ApacheCon NA 2013)
SDN in Apache CloudStack (ApacheCon NA 2013)SDN in Apache CloudStack (ApacheCon NA 2013)
SDN in Apache CloudStack (ApacheCon NA 2013)
Chiradeep Vittal
 
Scalable Object Storage with Apache CloudStack and Apache Hadoop
Scalable Object Storage with Apache CloudStack and Apache HadoopScalable Object Storage with Apache CloudStack and Apache Hadoop
Scalable Object Storage with Apache CloudStack and Apache Hadoop
Chiradeep Vittal
 

More from Chiradeep Vittal (11)

Loadbalancers: The fabric for your micro services
Loadbalancers: The fabric for your micro servicesLoadbalancers: The fabric for your micro services
Loadbalancers: The fabric for your micro services
 
Load Balancing for Containers and Cloud Native Architecture
Load Balancing for Containers and Cloud Native ArchitectureLoad Balancing for Containers and Cloud Native Architecture
Load Balancing for Containers and Cloud Native Architecture
 
Load Balancing for Containers and Cloud Native Architecture
Load Balancing for Containers and Cloud Native ArchitectureLoad Balancing for Containers and Cloud Native Architecture
Load Balancing for Containers and Cloud Native Architecture
 
Directions for CloudStack Networking
Directions for CloudStack  NetworkingDirections for CloudStack  Networking
Directions for CloudStack Networking
 
Private cloud networking_cloudstack_days_austin
Private cloud networking_cloudstack_days_austinPrivate cloud networking_cloudstack_days_austin
Private cloud networking_cloudstack_days_austin
 
StackWatch: A prototype CloudWatch service for CloudStack
StackWatch: A prototype CloudWatch service for CloudStackStackWatch: A prototype CloudWatch service for CloudStack
StackWatch: A prototype CloudWatch service for CloudStack
 
Network Functions Virtualization and CloudStack
Network Functions Virtualization and CloudStackNetwork Functions Virtualization and CloudStack
Network Functions Virtualization and CloudStack
 
CloudStack Networking Deepdive CCCEU13
CloudStack Networking Deepdive CCCEU13CloudStack Networking Deepdive CCCEU13
CloudStack Networking Deepdive CCCEU13
 
StackMate - CloudFormation for CloudStack
StackMate - CloudFormation for CloudStackStackMate - CloudFormation for CloudStack
StackMate - CloudFormation for CloudStack
 
SDN in Apache CloudStack (ApacheCon NA 2013)
SDN in Apache CloudStack (ApacheCon NA 2013)SDN in Apache CloudStack (ApacheCon NA 2013)
SDN in Apache CloudStack (ApacheCon NA 2013)
 
Scalable Object Storage with Apache CloudStack and Apache Hadoop
Scalable Object Storage with Apache CloudStack and Apache HadoopScalable Object Storage with Apache CloudStack and Apache Hadoop
Scalable Object Storage with Apache CloudStack and Apache Hadoop
 

Recently uploaded

Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
maigasapphire
 
find out more about the role of autonomous vehicles in facing global challenges
find out more about the role of autonomous vehicles in facing global challengesfind out more about the role of autonomous vehicles in facing global challenges
find out more about the role of autonomous vehicles in facing global challenges
huseindihon
 
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyyActive Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
RaminGhanbari2
 
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python CodebaseEuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
Jimmy Lai
 
Best Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdfBest Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdf
Tatiana Al-Chueyr
 
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and OllamaTirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
Zilliz
 
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite SolutionIPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Networks
 
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptxIntroduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
313mohammedarshad
 
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
Kief Morris
 
CiscoIconsLibrary cours de réseau VLAN.ppt
CiscoIconsLibrary cours de réseau VLAN.pptCiscoIconsLibrary cours de réseau VLAN.ppt
CiscoIconsLibrary cours de réseau VLAN.ppt
moinahousna
 
Figma AI Design Generator_ In-Depth Review.pdf
Figma AI Design Generator_ In-Depth Review.pdfFigma AI Design Generator_ In-Depth Review.pdf
Figma AI Design Generator_ In-Depth Review.pdf
Management Institute of Skills Development
 
Pigging Unit Lubricant Oil Blending Plant
Pigging Unit Lubricant Oil Blending PlantPigging Unit Lubricant Oil Blending Plant
Pigging Unit Lubricant Oil Blending Plant
LINUS PROJECTS (INDIA)
 
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
digitalxplive
 
How to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptxHow to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptx
Adam Dunkels
 
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
Priyanka Aash
 
Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024
Nicolás Lopéz
 
Data Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining DataData Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining Data
Safe Software
 
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdfAcumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
BrainSell Technologies
 
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptxUse Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
SynapseIndia
 
Recent Advancements in the NIST-JARVIS Infrastructure
Recent Advancements in the NIST-JARVIS InfrastructureRecent Advancements in the NIST-JARVIS Infrastructure
Recent Advancements in the NIST-JARVIS Infrastructure
KAMAL CHOUDHARY
 

Recently uploaded (20)

Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
 
find out more about the role of autonomous vehicles in facing global challenges
find out more about the role of autonomous vehicles in facing global challengesfind out more about the role of autonomous vehicles in facing global challenges
find out more about the role of autonomous vehicles in facing global challenges
 
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyyActive Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
 
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python CodebaseEuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
 
Best Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdfBest Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdf
 
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and OllamaTirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
 
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite SolutionIPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite Solution
 
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptxIntroduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
 
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
 
CiscoIconsLibrary cours de réseau VLAN.ppt
CiscoIconsLibrary cours de réseau VLAN.pptCiscoIconsLibrary cours de réseau VLAN.ppt
CiscoIconsLibrary cours de réseau VLAN.ppt
 
Figma AI Design Generator_ In-Depth Review.pdf
Figma AI Design Generator_ In-Depth Review.pdfFigma AI Design Generator_ In-Depth Review.pdf
Figma AI Design Generator_ In-Depth Review.pdf
 
Pigging Unit Lubricant Oil Blending Plant
Pigging Unit Lubricant Oil Blending PlantPigging Unit Lubricant Oil Blending Plant
Pigging Unit Lubricant Oil Blending Plant
 
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
 
How to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptxHow to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptx
 
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
 
Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024
 
Data Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining DataData Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining Data
 
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdfAcumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
 
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptxUse Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
 
Recent Advancements in the NIST-JARVIS Infrastructure
Recent Advancements in the NIST-JARVIS InfrastructureRecent Advancements in the NIST-JARVIS Infrastructure
Recent Advancements in the NIST-JARVIS Infrastructure
 

Evolution of CloudStack Architecture (Collab 2012)

  • 1. The Evolution of CloudStack The roads taken and not taken @chiradeep Collab 2012 Saturday, December 1, 2012 1
  • 2. We remember the most painful and pleasurable parts of an experience, not the duration Saturday, December 1, 2012 2
  • 3. In the Beginning Saturday, December 1, 2012 3
  • 4. Circa 2008 Saturday, December 1, 2012 4
  • 5. Circa 2008 !"#$%&'( Saturday, December 1, 2012 4
  • 6. Circa 2008 !"#$%&'( !"#$%&'( Saturday, December 1, 2012 4
  • 7. Circa 2008 !"#$%&'( !"#$%&'( !"#$%&'( Saturday, December 1, 2012 4
  • 8. Circa 2008 Open Source Xen (custom) !"#$%&'( !"#$%&'( !"#$%&'( Saturday, December 1, 2012 4
  • 9. Circa 2008 Open Source Xen Security Groups (custom) !"#$%&'( !"#$%&'( !"#$%&'( Saturday, December 1, 2012 4
  • 10. Circa 2008 Open Source Xen (custom) Security Groups EBS !"#$%&'( !"#$%&'( !"#$%&'( Saturday, December 1, 2012 4
  • 11. Circa 2008 Open Source Xen (custom) Security Groups EBS S3 !"#$%&'( !"#$%&'( !"#$%&'( Saturday, December 1, 2012 4
  • 12. Circa 2008 Proprietary Orchestration Services Open Source Xen (custom) Security Groups EBS S3 !"#$%&'( !"#$%&'( !"#$%&'( Saturday, December 1, 2012 4
  • 13. Circa 2008 EC2 API Proprietary Orchestration Services Open Source Xen (custom) Security Groups EBS S3 !"#$%&'( !"#$%&'( !"#$%&'( Saturday, December 1, 2012 4
  • 14. Circa 2008 EC2 EBS API API Proprietary Orchestration Services Open Source Xen (custom) Security Groups EBS S3 !"#$%&'( !"#$%&'( !"#$%&'( Saturday, December 1, 2012 4
  • 15. Circa 2008 EC2 EBS S3 API API API Proprietary Orchestration Services Open Source Xen (custom) Security Groups EBS S3 !"#$%&'( !"#$%&'( !"#$%&'( Saturday, December 1, 2012 4
  • 16. Circa 2008 EC2 EBS S3 API API API Proprietary Orchestration Services Secret Sauce Open Source Xen (custom) Security Groups EBS S3 !"#$%&'( !"#$%&'( !"#$%&'( Saturday, December 1, 2012 4
  • 17. The  Virtual  Datacenter  OS  allows  businesses  to   efficiently  pool  all  types  of  hardware  resources  -­‐   servers,  storage  and  network  –  into  an  aggregated   on-­‐premise  cloud -­‐  VMWare  Press  Release  9/2008 Saturday, December 1, 2012 5
  • 18. The  Virtual  Datacenter  OS  allows  businesses  to   efficiently  pool  all  types  of  hardware  resources  -­‐   servers,  storage  and  network  –  into  an  aggregated   on-­‐premise  cloud -­‐  VMWare  Press  Release  9/2008 Eucalyptus  is  the  only  cloud  architecture  to  support   the  same  applicaKon  programming  interfaces  (APIs)   as  public  clouds,  and  today  Eucalyptus  is  fully   compaKble  with  the  Amazon  AWS  public  cloud   infrastructure.   Eucalyptus  Systems  Press  Release  4/2009 Saturday, December 1, 2012 5
  • 19. 2.0 AWS 2.2 3.0 Quality 4.X Prototype 1.0 GA Refactor Compatibility Refactor Improvements Refactor 2008 2009 2010 2011 2012 Sept 2008: Nov 2009: May 2010: July 2011: April 2012: VMOps CloudStack Cloud.com Citrix Apache Founded 1.0 GA Launch & Acquires CloudStack CloudStack Cloud.com 2.0 GA Saturday, December 1, 2012 6
  • 20. Circa 3/2009 Saturday, December 1, 2012 7
  • 21. Circa 3/2009 !"#$%&'( Saturday, December 1, 2012 7
  • 22. Circa 3/2009 !"#$%&'( !"#$%&'( Saturday, December 1, 2012 7
  • 23. Circa 3/2009 !"#$%&'( !"#$%&'( !"#$%&'( Saturday, December 1, 2012 7
  • 24. Circa 3/2009 Open Source Xen (custom) !"#$%&'( !"#$%&'( !"#$%&'( Saturday, December 1, 2012 7
  • 25. Circa 3/2009 Open Source Xen Virtual Networks (custom) !"#$%&'( !"#$%&'( !"#$%&'( Saturday, December 1, 2012 7
  • 26. Circa 3/2009 Open Source Xen (custom) Virtual Networks NFS !"#$%&'( !"#$%&'( !"#$%&'( Saturday, December 1, 2012 7
  • 27. Circa 3/2009 Proprietary Control Plane Open Source Xen (custom) Virtual Networks NFS !"#$%&'( !"#$%&'( !"#$%&'( Saturday, December 1, 2012 7
  • 28. Circa 3/2009 Proprietary Orchestration Services Proprietary Control Plane Open Source Xen (custom) Virtual Networks NFS !"#$%&'( !"#$%&'( !"#$%&'( Saturday, December 1, 2012 7
  • 29. Circa 3/2009 GUI Proprietary Orchestration Services Proprietary Control Plane Open Source Xen (custom) Virtual Networks NFS !"#$%&'( !"#$%&'( !"#$%&'( Saturday, December 1, 2012 7
  • 30. Circa 3/2009 GUI Proprietary Orchestration Services Proprietary Control Plane Open Source Xen (custom) Virtual Networks NFS !"#$%&'( !"#$%&'( !"#$%&'( Saturday, December 1, 2012 8
  • 31. Circa 3/2009 GWT GUI Proprietary Orchestration Services Proprietary Control Plane Open Source Xen (custom) Virtual Networks NFS !"#$%&'( !"#$%&'( !"#$%&'( Saturday, December 1, 2012 8
  • 32. Circa 3/2009 GWT GUI Java App Server Proprietary Orchestration Services Proprietary Control Plane Open Source Xen (custom) Virtual Networks NFS !"#$%&'( !"#$%&'( !"#$%&'( Saturday, December 1, 2012 8
  • 33. Circa 3/2009 GWT GUI Java App Server Proprietary Orchestration Services JSON over TCP Proprietary Control Plane Open Source Xen (custom) Virtual Networks NFS !"#$%&'( !"#$%&'( !"#$%&'( Saturday, December 1, 2012 8
  • 34. Circa 3/2009 GWT GUI Java App Server Proprietary Orchestration Services JSON over TCP Proprietary Control Plane NFS (ZFS on Open Source Xen OpenSolaris) (custom) Virtual Networks NFS !"#$%&'( !"#$%&'( !"#$%&'( Saturday, December 1, 2012 8
  • 35. Circa 3/2009 GWT GUI Java App Server Proprietary Orchestration Services JSON over TCP Proprietary Control Plane UDP Tunnels NFS (ZFS on Open Source Xen OpenSolaris) (custom) Virtual Networks NFS !"#$%&'( !"#$%&'( !"#$%&'( Saturday, December 1, 2012 8
  • 36. Circa 3/2009 GWT GUI Java App Server Proprietary Orchestration Services JSON over TCP Proprietary Control Plane UDP Tunnels NFS (ZFS on Open Source Xen OpenSolaris) Agent (custom) Virtual Networks NFS !"#$%&'( !"#$%&'( !"#$%&'( Saturday, December 1, 2012 8
  • 37. Circa 3/2009 GWT GUI Java App Server Proprietary Orchestration Services JSON over TCP Proprietary Control Plane UDP Tunnels NFS (ZFS on Open Source Xen Agent OpenSolaris) Agent (custom) Virtual Networks NFS !"#$%&'( !"#$%&'( !"#$%&'( Saturday, December 1, 2012 8
  • 39. Availability Zone Architecture L3 core Saturday, December 1, 2012 9
  • 40. Availability Zone Architecture End users DC Edge L3 core Saturday, December 1, 2012 9
  • 41. Availability Zone Architecture End users DC Edge L3 core Access Sw Hypervisor (Xen) Primary Storage ZFS Pod Saturday, December 1, 2012 9
  • 42. Availability Zone Architecture End users DC Edge L3 core Access Sw Hypervisor (Xen) Primary Storage ZFS Pod Pod Saturday, December 1, 2012 9
  • 43. Availability Zone Architecture End users DC Edge L3 core Access Sw Hypervisor (Xen) Primary Storage ZFS Pod Pod Pod Saturday, December 1, 2012 9
  • 44. Availability Zone Architecture End users DC Edge L3 core Access Sw Hypervisor (Xen) Primary Storage ZFS Pod Pod Pod Pod Saturday, December 1, 2012 9
  • 45. Availability Zone Architecture End users DC Edge L3 core Access Sw Hypervisor (Xen) Primary Storage ZFS Pod Pod Pod Pod Saturday, December 1, 2012 9
  • 46. Availability Zone Architecture End users DC Edge L3 core Access Sw Hypervisor (Xen) Primary Storage ZFS Pod Pod Pod Pod Pod Saturday, December 1, 2012 9
  • 47. Availability Zone Architecture End users Admin/User API CloudStack DC Edge MySQL L3 core Access Sw Hypervisor (Xen) Primary Storage ZFS Pod Pod Pod Pod Pod Saturday, December 1, 2012 9
  • 48. Availability Zone Architecture End users Admin/User API CloudStack DC Edge MySQL ECMP/BGP L3 core Access Sw Hypervisor (Xen) Primary Storage ZFS Pod Pod Pod Pod Pod Saturday, December 1, 2012 9
  • 49. Multiple zones CloudStack MySQL Zone1 Saturday, December 1, 2012 10
  • 50. Multiple zones CloudStack MySQL Internet Zone1 Zone2 Zone3 Saturday, December 1, 2012 10
  • 52. Network Virtualization Tenant    1  Virtual  Network  10.1.1.0/24 Tenant  1    10.1.1.2 Gateway  address   VM  1 10.1.1.1 Tenant  1    10.1.1.3 VM  2 Tenant  1    10.1.1.4 VM  3 Tenant  1    10.1.1.5 VM  4 Saturday, December 1, 2012 11
  • 53. Network Virtualization Tenant    1  Virtual  Network  10.1.1.0/24 Tenant  1    10.1.1.2 Gateway  address   VM  1 10.1.1.1 Tenant  1   Tenant  1    10.1.1.3 Virtual  Router VM  2 NAT Internet DHCP Tenant  1    10.1.1.4 FW VM  3 Load  Balancing Tenant  1    10.1.1.5 VM  4 Saturday, December 1, 2012 11
  • 54. Network Virtualization Tenant    1  Virtual  Network  10.1.1.0/24 Public  IP  address   Tenant  1    10.1.1.2 65.37.141.11 Gateway  address   VM  1 65.37.141.36 10.1.1.1 Public  Network Tenant  1   Tenant  1    10.1.1.3 Virtual  Router VM  2 NAT Internet DHCP Tenant  1    10.1.1.4 FW VM  3 Load  Balancing Tenant  1    10.1.1.5 VM  4 Saturday, December 1, 2012 11
  • 55. Network Virtualization Tenant    1  Virtual  Network  10.1.1.0/24 Public  IP  address   Tenant  1    10.1.1.2 65.37.141.11 Gateway  address   VM  1 65.37.141.36 10.1.1.1 Public  Network Tenant  1   Tenant  1    10.1.1.3 Virtual  Router VM  2 NAT Internet DHCP Tenant  1    10.1.1.4 FW VM  3 Load  Balancing Tenant  1    10.1.1.5 VM  4 Tenant  2  Virtual  Network  10.1.1.0/24 Public  IP  address   65.37.141.24 Gateway  address   Tenant  2    10.1.1.2 10.1.1.1 VM  1 65.37.141.80 Tenant  2   Tenant  2    10.1.1.3 Virtual  Router   VM  2 Appliance FW NAT Tenant  2    10.1.1.4 DHCP VM  3 Saturday, December 1, 2012 11
  • 56. Virtual Machine Placement End users CloudStack DC Edge MySQL L3/L2 core Hypervisor (Xen) Primary Storage ZFS Pod Pod Pod Pod Pod Saturday, December 1, 2012 12
  • 57. Network Virtualization - why? AWS-style security groups “unfamiliar” Adopt traditional L2 model for end-users VLANs do not scale use network virtualization to realize this Saturday, December 1, 2012 13
  • 58. Network Virtualization UDP encapsulation of Ethernet Frames similar to VXLAN (128-bit address vs 24-bit for VXLAN) Driver in dom0 Virtual Router to provide edge services including Load Balancing Hand-rolled Fedora 8 para-virtualized Saturday, December 1, 2012 14
  • 59. ZFS (Volume Service) ZFS for highly scalable, reliable storage RAID-Z SSD cache NFS for shared storage Hand-rolled version of OpenSolaris 2008.11 Saturday, December 1, 2012 15
  • 61. Orchestration JVM Saturday, December 1, 2012 16
  • 62. Orchestration JVM MySQL Saturday, December 1, 2012 16
  • 63. Orchestration JVM Message Bus MySQL Saturday, December 1, 2012 16
  • 64. Orchestration JVM Front- end Message Bus MySQL Saturday, December 1, 2012 16
  • 65. Orchestration JVM VM Front- Orchest end rator Message Bus MySQL Saturday, December 1, 2012 16
  • 66. Orchestration JVM VM Network Front- Orchest Orchestr end rator ator Message Bus MySQL Saturday, December 1, 2012 16
  • 67. Orchestration JVM VM Network Storage Front- Orchest Orchestr Orchest end rator ator rator Message Bus MySQL Saturday, December 1, 2012 16
  • 68. Orchestration JVM VM Network Storage Host/ Front- Orchest Orchestr Orchest Resource end rator ator rator Manager Message Bus MySQL Saturday, December 1, 2012 16
  • 69. Orchestration JVM VM Network Storage Host/ Front- Orchest Orchestr Orchest Resource end rator ator rator Manager Message Bus Hypervi Hypervi Hypervisor sor Agent sor (JVM) MySQL Saturday, December 1, 2012 16
  • 70. Orchestration JVM VM Network Storage Host/ Front- Orchest Orchestr Orchest Resource end rator ator rator Manager Message Bus Hypervi Hypervi Hypervisor Storage Storage Storage sor Agent sor Agent Agent Agent (JVM) (JVM) MySQL Saturday, December 1, 2012 16
  • 71. Orchestration JVM VM Network Storage Host/ Front- Orchest Orchestr Orchest Resource end rator ator rator Manager Message Bus Hypervi Hypervi Hypervisor Storage Storage Storage Console Console sor Agent sor Agent Agent Agent Agent Agent (JVM) (JVM) MySQL Saturday, December 1, 2012 16
  • 72. Orchestration JVM VM Network Storage Host/ GWT-RPC Front- Browser Orchest Orchestr Orchest Resource end rator ator rator Manager Message Bus Hypervi Hypervi Hypervisor Storage Storage Storage Console Console sor Agent sor Agent Agent Agent Agent Agent (JVM) (JVM) MySQL Saturday, December 1, 2012 16
  • 73. Reject Pile (2009) Local disks of hypervisors presented as network block storage with network RAID Too complex (sheepdog-like) Split-brain issues DRBD Too complex to automate recovery Commercial Arrays Limited volumes, limited snapshots Saturday, December 1, 2012 17
  • 74. Reject Pile (2009) Security Groups Not well understood VLANs 4K limit Physical Network Appliances No APIs (Cisco at least) Can’t decapsulate UDP tunnel Saturday, December 1, 2012 18
  • 75. Reject Pile (early 2009) KVM Unstable Windows support No CPU allocation Saturday, December 1, 2012 19
  • 77. April 2009 “5 guys in a garage” Saturday, December 1, 2012 20
  • 78. Ship It! ( Beta 6/2009) ISO/CD with Xen 3.3 / CentOs customized ISO/CD with OSOL 2008.11 + patches + drivers Java binaries System VM: Fedora Core 8 VM bits including dnsmasq, iptables, HAProxy Installers, documentation Saturday, December 1, 2012 21
  • 79. Road to 1.0 ISCSI - ZFS zvols -> ∞ snapshots Performance tuning (UDP tunnels, zvol) Usage, metering, Formal user / admin Query-based API JQuery-based GUI - skinnable Windows PV drivers Saturday, December 1, 2012 22
  • 80. More rejects (2009) Network RAID of zvols over ISCSI/md Performance Re-mirror time / penalty Consistent snapshots Split-brain Saturday, December 1, 2012 23
  • 81. 1.0 - Ship It (Nov 2009) 6 engineers + 3 QA + 1 sales 3 beta sites Traction with SPs interested in competing with AWS Saturday, December 1, 2012 24
  • 82. Orchestration + Usage server JVM Front- end Saturday, December 1, 2012 25
  • 83. Orchestration + Usage server JVM MySQL Front- end Saturday, December 1, 2012 25
  • 84. Orchestration + Usage server Browser LoadBalan JVM MySQL ced HTTP Query Front- API end Saturday, December 1, 2012 25
  • 85. Orchestration + Usage server Browser LoadBalan JVM MySQL ced HTTP VM Network Storage Host/ Query Front- API Orchest Orchestr Orchest Resource end rator ator rator Manager Saturday, December 1, 2012 25
  • 86. Orchestration + Usage server Browser LoadBalan JVM MySQL ced HTTP VM Network Storage Host/ Query Front- API Orchest Orchestr Orchest Resource end rator ator rator Manager Message Bus Hypervi Hypervi Hypervisor Storage Storage Storage Console Console sor Agent sor Agent Agent Agent Agent Agent (JVM) (JVM) Saturday, December 1, 2012 25
  • 87. Orchestration + Usage server Browser LoadBalan JVM MySQL ced Storage VM VM Network Network Storage HTTP Front- Front- Orchest Network Storage Host/ VM Orchestr Orchest Host/ Query Front- Orchest Orchestr Orchest Resource end API end Orchest Orchestr rator rator ator Orchest Resource end rator ator rator rator ator rator Manager Manager Clustered Message Bus Orchestration Message Bus Message Bus Server Hypervi Hypervi Hypervisor Storage Storage Storage Console Console sor Agent sor Agent Agent Agent Agent Agent (JVM) (JVM) Saturday, December 1, 2012 25
  • 88. Orchestration + Usage server Browser LoadBalan JVM MySQL ced Storage VM VM Network Network Storage HTTP Front- Front- Orchest Network Storage Host/ VM Orchestr Orchest Host/ Query Front- Orchest Orchestr Orchest Resource end API end Orchest Orchestr rator rator ator Orchest Resource end rator ator rator rator ator rator Manager Manager Usage Clustered Message Bus Orchestration Message Bus Message Bus Server Hypervi Hypervi Hypervisor Storage Storage Storage Console Console sor Agent sor Agent Agent Agent Agent Agent (JVM) (JVM) Saturday, December 1, 2012 25
  • 89. 1.0 Issues Multicast scaling Oracle acquisition of OpenSolaris Hiring “full-stack” developers 5 startups - Hypervisor + Storage + Network Virtualization + Network appliance + Orchestration Saturday, December 1, 2012 26
  • 90. Multicast Scaling Broadcast (ARP) / Multicast from VM translates to UDP multicast Switches snoop on IGMP to prevent flooding all ports Most switches fall-over after 64 Multicast groups each tenant -> 1 multicast group Saturday, December 1, 2012 27
  • 91. Road Ahead Multi-hypervisor, SAN, and VLAN support Use commercial XenServer Commercial filers / ISCSI arrays Support KVM + NFS Product first, architecture second From web hosting to enterprise workload Saturday, December 1, 2012 28
  • 92. Road Ahead Throw out custom hypervisor and OpenSolaris VLANs Secondary Storage to store snapshots and images Saturday, December 1, 2012 29
  • 93. Circa 3/2010 GUI Proprietary Orchestration Services Proprietary Control Plane Open Source Xen Storage Agent Virtual Networks Agent (custom) Arrays !"#$%&'( !"#$%&'( !"#$%&'( Saturday, December 1, 2012 30
  • 94. Circa 3/2010 JQuery GUI Proprietary Orchestration Services Proprietary Control Plane Open Source Xen Storage Agent Virtual Networks Agent (custom) Arrays !"#$%&'( !"#$%&'( !"#$%&'( Saturday, December 1, 2012 30
  • 95. Circa 3/2010 JQuery GUI Proprietary Orchestration Services Proprietary Control Plane VLANs Open Source Xen Storage Agent Virtual Networks Agent (custom) Arrays !"#$%&'( !"#$%&'( !"#$%&'( Saturday, December 1, 2012 30
  • 96. Circa 3/2010 JQuery GUI Proprietary Orchestration Services Proprietary Control Plane VLANs Open Source Xen Storage Virtual Networks Agent (custom) Arrays !"#$%&'( !"#$%&'( !"#$%&'( Saturday, December 1, 2012 30
  • 97. Circa 3/2010 JQuery GUI Proprietary Orchestration Services Proprietary Control Plane VLANs Open Source Xen Storage Virtual Networks (custom) Arrays !"#$%&'( !"#$%&'( !"#$%&'( Saturday, December 1, 2012 30
  • 98. Circa 3/2010 JQuery GUI Proprietary Orchestration Services Proprietary Control Plane VLANs Storage Virtual Networks Arrays !"#$%&'( !"#$%&'( !"#$%&'( Saturday, December 1, 2012 30
  • 99. Circa 3/2010 JQuery GUI Proprietary Orchestration Services Proprietary Control Plane VLANs Commercial Storage Virtual Networks XenServer Arrays !"#$%&'( !"#$%&'( !"#$%&'( Saturday, December 1, 2012 30
  • 100. Circa 3/2010 JQuery GUI Proprietary Orchestration Services Proprietary Control Plane ISCSI/ VLANs NFS Commercial Storage Virtual Networks XenServer Arrays !"#$%&'( !"#$%&'( !"#$%&'( Saturday, December 1, 2012 30
  • 101. Orchestration + Usage server Browser LoadBalan JVM MySQL ced Front- HTTP end VM Network Storage Host/ Query API Orchest Orchestr Orchest Resource rator ator rator Manager Job Queue Usage Clustered Message Bus Orchestration Server Hypervi XS sor Agent XAPI KVM Console Console Agent Secondary Agent Agent Storage Agent XenServers (JVM) (JVM) (JVM) Saturday, December 1, 2012 31
  • 102. Availability Zone Architecture End users Admin/User API CloudStack DC Edge MySQL Access Sw Hypervisor (Xen /VMWare/KVM) Pod Pod Pod Pod Pod Saturday, December 1, 2012 32
  • 103. Availability Zone Architecture End users Admin/User API CloudStack DC Edge MySQL Access Sw Hypervisor (Xen /VMWare/KVM) Secondary Storage Pod Pod Pod Pod Pod Saturday, December 1, 2012 32
  • 104. Availability Zone Architecture End users Admin/User API CloudStack DC Edge MySQL L2/L3 core Access Sw Hypervisor (Xen /VMWare/KVM) Secondary Storage Pod Pod Pod Pod Pod Saturday, December 1, 2012 32
  • 105. Availability Zone Architecture End users Admin/User API CloudStack DC Edge MySQL L2/L3 core Access Sw Hypervisor (Xen /VMWare/KVM) Secondary Storage Primary Storage NFS/ISCSI/FC Pod Pod Pod Pod Pod Saturday, December 1, 2012 32
  • 106. System VM Data Path problem Need to move bits between storage types Present consoles to end users Scale up / down these data path services Saturday, December 1, 2012 33
  • 107. System VMs Virtual Routers, Console Proxy VMs, Secondary Storage VMs based on the same vm image Orchestration and scaling baked in Saturday, December 1, 2012 34
  • 108. 2.0 6/2010 Shipping just java binaries + system vm image Open sourced (GPL) Company name changed to Cloud.com New product - Cloud Portal Saturday, December 1, 2012 35
  • 109. Citrix Cloud Portal Business and operations support platform for Service Providers Account management Self-service portal Billing and metering CRM and support ticketing Not OSS Saturday, December 1, 2012 36
  • 110. Reject Pile (2010) “Zone” and “Pod” VLANs to go beyond 4k limit Pod VLANs bridged using zone VLANs Another SDN attempt using GRE on Open vSwitch Saturday, December 1, 2012 37
  • 111. Flexibility and Scale (2010-2011) Network, hypervisor and vm placement flexibility Tens of thousands of hypervisors AWS API Saturday, December 1, 2012 38
  • 112. Network Flexibility (2.2 refactor) Hardware appliances instead of virtual router Upgrade / downgrade to/from hardware to virtual Pick and choose services (L2-L7) Security Groups Saturday, December 1, 2012 39
  • 113. Network Flexibility Scale  out  edge  services  using  virtual  appliances 10.1.1.0/24 VLAN  100 VM  1 10.1.1.2 65.37.141.111 10.1.1.1 65.37.141.112 CS Virtual   10.1.1.3 VM  2 Router DHCP,  DNS NAT 10.1.1.4 VM  3 Load  Balancing VM  4 10.1.1.5 Saturday, December 1, 2012 40
  • 114. Network Flexibility Scale  out  edge  services  using  virtual  appliances Scale  up  using  hardware  devices 10.1.1.0/24 10.1.1.0/24 VLAN  100 VLAN  100 65.37.141.111 10.1.1.1 10.1.1.2 VM  1 VM  1 10.1.1.2 Juniper  SRX 65.37.141.111 10.1.1.1 Firewall NAT,   65.37.141.112 VPN CS VM  2 Virtual   10.1.1.3 VM  2 10.1.1.3 Router 65.37.141.112 10.1.1.112 DHCP,  DNS Netscaler NAT Load   10.1.1.4 VM  3 VM  3 Load  Balancing Balancer 10.1.1.4 VM  4 VM  4 10.1.1.5 10.1.1.5 CS DHCP,   Virtual   DNS Router Saturday, December 1, 2012 40
  • 115. VM Placement Customize vm placement Pick hosts and storage according to rules Affinity / anti-affinity Saturday, December 1, 2012 41
  • 116. Multi-hypervisor XenServer, KVM and VMWare in the same zone Differences in network and storage behavior Different image formats Same system vm image Saturday, December 1, 2012 42
  • 117. Plugin Architecture (2.2) Orchestration Engine Saturday, December 1, 2012 43
  • 118. Plugin Architecture (2.2) Plugin Framework Orchestration Engine Saturday, December 1, 2012 43
  • 119. Plugin Architecture (2.2) Hypervisor Hypervisor Plugins Plugins Plugin Framework Orchestration Engine Saturday, December 1, 2012 43
  • 120. Plugin Architecture (2.2) Hypervisor Hypervisor Plugins Plugins Plugin Framework Orchestration Network Network Plugins Plugins Engine Saturday, December 1, 2012 43
  • 121. Plugin Architecture (2.2) Hypervisor Hypervisor Plugins Plugins Plugin Framework Orchestration Network Network Plugins Plugins Engine Allocator Allocator Plugins Plugins Saturday, December 1, 2012 43
  • 122. Plugin Architecture (2.2) Hypervisor Hypervisor Plugins Plugins Plugin Framework Orchestration Network Network Plugins Plugins Engine Allocator Allocator Plugins Plugins Storage Plugins Saturday, December 1, 2012 43
  • 123. Plugin Architecture Hypervisor Hypervisor Plugins Plugins Plugin Framework Orchestration Network Network Plugins Plugins Engine Allocator Allocator Plugins Plugins Saturday, December 1, 2012 44
  • 124. Plugin Architecture •XenServer •VMWare •KVM •OracleVM Hypervisor Hypervisor Plugins Plugins Plugin Framework Orchestration Network Network Plugins Plugins Engine Allocator Allocator Plugins Plugins Saturday, December 1, 2012 44
  • 125. Plugin Architecture •XenServer •VMWare •KVM •OracleVM Hypervisor Hypervisor Plugins Plugins Plugin •Nicira Framework •Netscaler Orchestration Network Network Plugins •Brocade •MidoNet Plugins Engine Allocator Allocator Plugins Plugins Saturday, December 1, 2012 44
  • 126. Plugin Architecture •XenServer •VMWare •KVM •OracleVM Hypervisor Hypervisor Plugins Plugins Plugin •Nicira Framework •Netscaler Orchestration Network Network Plugins •Brocade •MidoNet Plugins Engine Random • Allocator Allocator •User- Plugins Plugins concentrated •Intel TXT •Affinity Saturday, December 1, 2012 44
  • 127. Security Groups Web   DB   Web   VM VM VM Web   Web   DB   VM VM VM … … … Web   Web   VM VM Ingress  Rule:  Allow  VMs  in  Web  Security  Group  access  to  VMs  in  DB  Security  Group  on  Port  3306 Saturday, December 1, 2012 45
  • 128. Security Groups Web   DB   Web   VM VM VM Web Security   Group Web   Web   DB   VM VM VM … … … Web   Web   VM VM Ingress  Rule:  Allow  VMs  in  Web  Security  Group  access  to  VMs  in  DB  Security  Group  on  Port  3306 Saturday, December 1, 2012 45
  • 129. Security Groups Web   DB   Web   VM VM VM Web DB   Security   Security   Group Group Web   Web   DB   VM VM VM … … … Web   Web   VM VM Ingress  Rule:  Allow  VMs  in  Web  Security  Group  access  to  VMs  in  DB  Security  Group  on  Port  3306 Saturday, December 1, 2012 45
  • 130. Security Groups Web   DB   Web   VM VM VM Web DB   Security   Security   Group Group Web   Web   DB   VM VM VM … … … Web   Web   VM VM Ingress  Rule:  Allow  VMs  in  Web  Security  Group  access  to  VMs  in  DB  Security  Group  on  Port  3306 Saturday, December 1, 2012 45
  • 131. A million firewalls? Manage the config state of 10^6 firewalls? “Eventual consistency” Iptable Rule explosion on hypervisor use ‘ipset’ to optimize lookup Saturday, December 1, 2012 46
  • 132. 2.1 -> 3.02 Features, features features VMWare, Oracle VM, Baremetal Multi-tier networking, VPC, VPN, more vendors EC2 API, S3 API SDN Focus on quality Saturday, December 1, 2012 47