Community Connect:
Apache CloudStack
Ram Chinta
Director – CloudStack Engineering
Cloud Platforms Group, Citrix

Hyderabad Hadoop User Group Meetup
September 22, 2012
Agenda

• Introduction to CloudStack
• Essentials of a Cloud
• CloudStack Architecture
• Designing for diverse Workloads
• Hadoop and CloudStack
• Apache CloudStack Community
Introducing
Apache CloudStack
CloudStack Background

• A proven IaaS Cloud Platform
   – Developed since 2008 by Cloud.com
   – Cloud.com development center in Hyderabad since 2009
   – Cloud.com acquired by Citrix in 2011
   – Donated to the Apache Software Foundation (ASF) in April
     2012, where it is now an Incubator project
   – First ASF community release in October 2012
   – Powering some of the world’s largest clouds today
Enterprises are embracing Cloud

Service Providers     Web 2.0         Enterprise
I’ve already virtualized my data center,
why do I need a cloud?
How is cloud really different from
server virtualization?
Virtualization does not make a cloud

   Server Virtualization++                                Cloud




Built for traditional enterprise          Designed around big data,
apps & client-server compute              massive scale & next-gen apps
 •Enterprise arch for 100s of hosts       •   Cloud arch for 1000s of hosts
 •Scale-up (pool-based resourcing)        •   Scale-out (horizontal resourcing)
 •IT management-centric                   •   Autonomic management
 •1 administrator for Dozens of servers   •   1 administrator for 1,000’s of servers
 •Apps assume reliability                 •   Apps assume failure
 •Proprietary vendor stack                •   Open, value-added stack
Essentials of Cloud

        • On-Demand and Self Serve
        • Resource pooling, Multi-tenancy
        • Massive Scale
        • Rapid Elasticity
        • APIs and Extreme Automation
        • Total Homogeneous
CloudStack Architecture
How did Amazon build its Cloud?

                    Amazon eCommerce Platform
                    Amazon eCommerce Platform


                          AWS API (EC2, S3, …)
                          AWS API (EC2, S3, …)


              Amazon Proprietary Orchestration Software
              Amazon Proprietary Orchestration Software


                     Open Source Xen Hypervisor
                     Open Source Xen Hypervisor

           Networking
           Networking         Commodity
                              Commodity          Commodity
                                                 Commodity
           (NetScaler)
            (NetScaler)         Servers
                                Servers            Storage
                                                   Storage
Apache CloudStack

      Portal and End User Experience
       Amazon eCommerce Platform
      Portal and End User Experience
       Amazon eCommerce Platform

            AWS API (EC2, S3, …)
            AWS Cloud APIS3, …)
                API (EC2,
                Cloud API

Amazon Proprietary CloudStack Software
Amazon Proprietary Orchestration Software
          Apache CloudStack
           Apache Orchestration

  vSphere XenServerXen Hypervisor
       Open Source KVM OVM Hyper-V
  vSphere XenServerXen Hypervisor
       Open Source KVM OVM Hyper-V

Any Networking
Any Networking   Any Servers
                 Any Servers   Any Storage
                               Any Storage
Key Highlights
• Broad Hypervisor support
   —XenServer, KVM, VMware, OracleVM. Baremetal support.
• Scalable architecture
   —Support thousands of hosts and VMs
• Flexible and extensible architecture
   —Plugins and Adapters
• Rich networking capability
   • Firewall, LoadB, VPN, NAT, DNS, DHCP, VLAN/L3 isolation, physical devices
• Storage options
  • Local, Shared, Snapshots and backup
ᵒ Fine-Grained Service Offerings (compute, network, storage, HA, affinity)
• Choice of interfaces
   — RESTful API, Web UI, command line, AWS Compatibility
Looking Closer at an IaaS Cloud built on CloudStack
Multiple levels of Scaling
                           Availability Zone




                         VMOps                    VMOps Pod
                    VMOps                         CloudStack
                         Pod
            CloudStack Pod       CloudStack Pod   Pod CloudStack Pod
                    Pod
Cloud Scalability Must Span Datacenters
                                 CloudStack Management Cluster




      San Jose                         Barcelona
                 Miami                                Private Dehli   Tokyo




                         Private Rio
CloudStack Management
                         Server Cluster

                             MySQL                          Internet


                                                                 Router


                                        Availability Zone        Load Balancer

                                                                  L3 Core Switch

Top of Rack Switch



           Servers
                        …               …           …           …                     …

                     Pod 1           Pod 2       Pod 3                             Pod N
Object store is critical for Amazon-style cloud
                                 Availability Zone 1




                                                           Storage Cloud

          ELB/GSLB                                     ?
 Users
                     Availability Zone 2
Designing
       for
Diverse Workloads
Traditional Server-Virtualization workload

 Traditional-Style Deployment                Hypervisor


               vCenter/XenCenter
                vCenter/XenCenter
                                             Storage

     Enterprise Networking (e.g., VLAN)
      Enterprise Networking (e.g., VLAN)

                                             Networking
  Hypervisor
  Hypervisor      Hypervisor
                  Hypervisor    Hypervisor
                                Hypervisor
   Cluster
    Cluster        Cluster
                    Cluster      Cluster
                                  Cluster

                                             Network Services
       Enterprise Storage (e.g., SAN)
        Enterprise Storage (e.g., SAN)

                                             Multi-tier Apps
Amazon-style workload
        Amazon-Style Availability Zone
           Software Defined Networks
            Software Defined Networks                 Hypervisor
        (e.g., Security Groups, EIP, ELB,...)
         (e.g., Security Groups, EIP, ELB,...)

  Server
   Server      Server
                Server        Server
                               Server       Server
                                             Server
  Racks        Racks          Racks         Racks     Storage
   Racks        Racks          Racks         Racks

  Server
   Server      Server
                Server        Server
                               Server       Server
                                             Server
  Racks
   Racks       Racks
                Racks         Racks
                               Racks        Racks
                                             Racks     Networking



  Server
   Server      Server
                Server        Server
                               Server       Server
                                             Server
  Racks
   Racks       Racks
                Racks         Racks
                               Racks        Racks
                                             Racks    Network Services



               Elastic Block Storage
               Elastic Block Storage                  Multi-tier Apps
Amazon-Style Cloud

                                                       Amazon-Style Availability Zone
                CloudStack Mgmt
                CloudStack Mgmt                                           vCloud Director
                                                            Software Defined Networks
                                                                           vCloud Director
                                                             Software Defined Networks
                     Server
                      Server                               (e.g., Security Groups, VPC,...)
                                                            (e.g., Security Groups, VPC,...)

                                                 Server
                                                  Server        Server
                                                                 Server      vSphere
                                                                               Server
                                                                              vSphere
                                                                                Server         Server
                                                                                                Server
                                                 Racks
                                                  Racks         Racks
                                                                 Racks         Racks
                                                                                Racks          Racks
                                                                                                Racks
                                                              Enterprise Networking (e.g., VLAN)
                                                              Enterprise Networking (e.g., VLAN)
                                                 Server
                                                  Server       Server
                                                                Server      Server
                                                                             Server       Server
                                                                                           Server
Availability
Availability   Availability
               Availability       Availability
                                  Availability   Racks          Racks        Racks         Racks
                 Zone                             Racks         Racks        Racks         Racks
  Zone
   Zone           Zone              Zone
                                     Zone
                                                          ESXi
                                                          ESXi              ESXi
                                                                             ESXi                   ESXi
                                                                                                    ESXi
                                                 Server Cluster
                                                             Server
                                                  Server Cluster
                                                               Server         Server
                                                                           Cluster
                                                                               Server
                                                                           Cluster             Server
                                                                                                  Cluster
                                                                                                Server
                                                                                                   Cluster
                                                 Racks
                                                  Racks       Racks
                                                               Racks           Racks
                                                                               Racks           Racks
                                                                                                Racks

                 Object Storage
                 Object Storage                                 Elastic BlockStorage (e.g., SAN)
                                                                 Enterprise Storage
                                                                Elastic BlockStorage (e.g., SAN)
                                                                  Enterprise Storage
Architecture to embrace Server Virtualization Workloads

               Amazon-Style Cloud                    Server Virtualization++

                  CloudStack Mgmt
                  CloudStack Mgmt
                       Server
                        Server
                                                   Server Virtualization Availability Zone

                                                                  vSphere
                                                                   vSphere


                                                     Enterprise Networking (e.g., VLAN)
                                                     Enterprise Networking (e.g., VLAN)
Availability
Availability     Availability
                 Availability       Availability
                                    Availability
  Zone
   Zone            Zone
                    Zone              Zone
                                       Zone
                                                    ESXi
                                                     ESXi         ESXi
                                                                   ESXi              ESXi
                                                                                      ESXi
                                                   Cluster
                                                   Cluster       Cluster
                                                                 Cluster            Cluster
                                                                                    Cluster


                   Object Storage
                   Object Storage                       Enterprise Storage (e.g., SAN)
                                                        Enterprise Storage (e.g., SAN)
Hadoop and CloudStack
Common interest
•Apache Foundation project
•Written in Java
•Runs on Linux, Mac OS/X, Windows, and Solaris
•Cloud-scale workloads
•Optimized for commodity hardware
•Hadoop Core includes:
 ᵒ HDFS - Distributed file system derived from Google File System
 ᵒ Map/Reduce – Distributed compute engine
Opportunities
•HDFS as Object Store solution for CloudStack cloud
•Hadoop on CloudStack
    • Implement services similar to Amazon’s Elastic MapReduce
      (EMR)
    • Big Data workload readily moved to IaaS clouds
•CloudStack’s baremetal provisioning capabilities to build high
performance Hadoop clusters
•Feedback?
CloudStack Community

          Nitin Mehta
 Committer, CS Meetup Coordinator


         Rohit Yadav
 Developer, CS Meetup Coordinator
CloudStack Joins the Apache
             Foundation

Joins Apache Webserver, Tomcat, Hadoop, Cassendra
       10+years, 100+ projects, 800 developers

   cwiki.apache.org/confluence/display/CLOUDSTACK/Index
                     www.cloudstack.org
Developers Love CloudStack
• Open source (ASL 2.0)
• Written in Java
• Multiple APIs
   ᵒ Native CloudStack API
   ᵒ Amazon EC2 compatible API
• Customizable Web UI
   ᵒ Rich AJAX client interface
   ᵒ Built on HTML/JSP, CSS, Javascript, jQuery
• Active development community
   ᵒ http://www.cloudstack.org
   ᵒ Download binaries, source code.
ASF Community Status

•   In Incubation since April 2012
•   PPMC and Mentors setup in April
•   Code migrated to Apache git
•   Migration underway from cloudstack.org:
    – Jira (Bugs), wiki, forums
• New committers, partners, developers every week
• Apache CloudStack 4.0: First ASF community release coming
  in October 2012
Join the community!
• Why not?
     ᵒ Interested in learning the coolest technologies shaping the next decade.
     ᵒ Being part of a young, passionate community working at ground
         breaking speed.
     ᵒ Interested in solving cool real-world problems.
     ᵒ For the love of open source.
     ᵒ If you want to rise and shine.
• Still don’t believe us – Let the numbers speak
     ᵒ 30k active community members in 8 months, ~100 IRC users
     ᵒ Exponentially growing dev/user community discussions
    ᵒ Apache is the industry standard for 10+ years in OSS foundations, 100+ projects,
      800 developers
    ᵒ Well known projects: Apache webserver, Tomcat, Hadoop, Cassandra
So lets get started ….
•   Check us out @
      ᵒ cwiki.apache.org/confluence/display/CLOUDSTACK/Index
      ᵒ http://cloudstack.org
•   IRC: #CloudStack on irc.freenode.net
•   Forums: http://cloudstack.org/forum/
•   Subscribe to Mailing Lists @
    cloudstack-dev@incubator.apache.org
    cloudstack-users@incubator.apache.org
•   Join CloudStack Hyderabad Meetup Group @
    http://www.meetup.com/CloudStack-Hyderabad-Group/
Thank You

ram.chinta@citrix.com
nitin.mehta@citrix.com
rohit.yadav@citrix.com

Ram chinta hug-20120922-v1

  • 1.
    Community Connect: Apache CloudStack RamChinta Director – CloudStack Engineering Cloud Platforms Group, Citrix Hyderabad Hadoop User Group Meetup September 22, 2012
  • 2.
    Agenda • Introduction toCloudStack • Essentials of a Cloud • CloudStack Architecture • Designing for diverse Workloads • Hadoop and CloudStack • Apache CloudStack Community
  • 3.
  • 4.
    CloudStack Background • Aproven IaaS Cloud Platform – Developed since 2008 by Cloud.com – Cloud.com development center in Hyderabad since 2009 – Cloud.com acquired by Citrix in 2011 – Donated to the Apache Software Foundation (ASF) in April 2012, where it is now an Incubator project – First ASF community release in October 2012 – Powering some of the world’s largest clouds today
  • 5.
    Enterprises are embracingCloud Service Providers Web 2.0 Enterprise
  • 6.
    I’ve already virtualizedmy data center, why do I need a cloud?
  • 7.
    How is cloudreally different from server virtualization?
  • 8.
    Virtualization does notmake a cloud Server Virtualization++ Cloud Built for traditional enterprise Designed around big data, apps & client-server compute massive scale & next-gen apps •Enterprise arch for 100s of hosts • Cloud arch for 1000s of hosts •Scale-up (pool-based resourcing) • Scale-out (horizontal resourcing) •IT management-centric • Autonomic management •1 administrator for Dozens of servers • 1 administrator for 1,000’s of servers •Apps assume reliability • Apps assume failure •Proprietary vendor stack • Open, value-added stack
  • 9.
    Essentials of Cloud • On-Demand and Self Serve • Resource pooling, Multi-tenancy • Massive Scale • Rapid Elasticity • APIs and Extreme Automation • Total Homogeneous
  • 10.
  • 11.
    How did Amazonbuild its Cloud? Amazon eCommerce Platform Amazon eCommerce Platform AWS API (EC2, S3, …) AWS API (EC2, S3, …) Amazon Proprietary Orchestration Software Amazon Proprietary Orchestration Software Open Source Xen Hypervisor Open Source Xen Hypervisor Networking Networking Commodity Commodity Commodity Commodity (NetScaler) (NetScaler) Servers Servers Storage Storage
  • 12.
    Apache CloudStack Portal and End User Experience Amazon eCommerce Platform Portal and End User Experience Amazon eCommerce Platform AWS API (EC2, S3, …) AWS Cloud APIS3, …) API (EC2, Cloud API Amazon Proprietary CloudStack Software Amazon Proprietary Orchestration Software Apache CloudStack Apache Orchestration vSphere XenServerXen Hypervisor Open Source KVM OVM Hyper-V vSphere XenServerXen Hypervisor Open Source KVM OVM Hyper-V Any Networking Any Networking Any Servers Any Servers Any Storage Any Storage
  • 13.
    Key Highlights • BroadHypervisor support —XenServer, KVM, VMware, OracleVM. Baremetal support. • Scalable architecture —Support thousands of hosts and VMs • Flexible and extensible architecture —Plugins and Adapters • Rich networking capability • Firewall, LoadB, VPN, NAT, DNS, DHCP, VLAN/L3 isolation, physical devices • Storage options • Local, Shared, Snapshots and backup ᵒ Fine-Grained Service Offerings (compute, network, storage, HA, affinity) • Choice of interfaces — RESTful API, Web UI, command line, AWS Compatibility
  • 14.
    Looking Closer atan IaaS Cloud built on CloudStack
  • 15.
    Multiple levels ofScaling Availability Zone VMOps VMOps Pod VMOps CloudStack Pod CloudStack Pod CloudStack Pod Pod CloudStack Pod Pod
  • 16.
    Cloud Scalability MustSpan Datacenters CloudStack Management Cluster San Jose Barcelona Miami Private Dehli Tokyo Private Rio
  • 17.
    CloudStack Management Server Cluster MySQL Internet Router Availability Zone Load Balancer L3 Core Switch Top of Rack Switch Servers … … … … … Pod 1 Pod 2 Pod 3 Pod N
  • 18.
    Object store iscritical for Amazon-style cloud Availability Zone 1 Storage Cloud ELB/GSLB ? Users Availability Zone 2
  • 19.
    Designing for Diverse Workloads
  • 20.
    Traditional Server-Virtualization workload Traditional-Style Deployment Hypervisor vCenter/XenCenter vCenter/XenCenter Storage Enterprise Networking (e.g., VLAN) Enterprise Networking (e.g., VLAN) Networking Hypervisor Hypervisor Hypervisor Hypervisor Hypervisor Hypervisor Cluster Cluster Cluster Cluster Cluster Cluster Network Services Enterprise Storage (e.g., SAN) Enterprise Storage (e.g., SAN) Multi-tier Apps
  • 21.
    Amazon-style workload Amazon-Style Availability Zone Software Defined Networks Software Defined Networks Hypervisor (e.g., Security Groups, EIP, ELB,...) (e.g., Security Groups, EIP, ELB,...) Server Server Server Server Server Server Server Server Racks Racks Racks Racks Storage Racks Racks Racks Racks Server Server Server Server Server Server Server Server Racks Racks Racks Racks Racks Racks Racks Racks Networking Server Server Server Server Server Server Server Server Racks Racks Racks Racks Racks Racks Racks Racks Network Services Elastic Block Storage Elastic Block Storage Multi-tier Apps
  • 22.
    Amazon-Style Cloud Amazon-Style Availability Zone CloudStack Mgmt CloudStack Mgmt vCloud Director Software Defined Networks vCloud Director Software Defined Networks Server Server (e.g., Security Groups, VPC,...) (e.g., Security Groups, VPC,...) Server Server Server Server vSphere Server vSphere Server Server Server Racks Racks Racks Racks Racks Racks Racks Racks Enterprise Networking (e.g., VLAN) Enterprise Networking (e.g., VLAN) Server Server Server Server Server Server Server Server Availability Availability Availability Availability Availability Availability Racks Racks Racks Racks Zone Racks Racks Racks Racks Zone Zone Zone Zone Zone ESXi ESXi ESXi ESXi ESXi ESXi Server Cluster Server Server Cluster Server Server Cluster Server Cluster Server Cluster Server Cluster Racks Racks Racks Racks Racks Racks Racks Racks Object Storage Object Storage Elastic BlockStorage (e.g., SAN) Enterprise Storage Elastic BlockStorage (e.g., SAN) Enterprise Storage
  • 23.
    Architecture to embraceServer Virtualization Workloads Amazon-Style Cloud Server Virtualization++ CloudStack Mgmt CloudStack Mgmt Server Server Server Virtualization Availability Zone vSphere vSphere Enterprise Networking (e.g., VLAN) Enterprise Networking (e.g., VLAN) Availability Availability Availability Availability Availability Availability Zone Zone Zone Zone Zone Zone ESXi ESXi ESXi ESXi ESXi ESXi Cluster Cluster Cluster Cluster Cluster Cluster Object Storage Object Storage Enterprise Storage (e.g., SAN) Enterprise Storage (e.g., SAN)
  • 24.
  • 25.
    Common interest •Apache Foundationproject •Written in Java •Runs on Linux, Mac OS/X, Windows, and Solaris •Cloud-scale workloads •Optimized for commodity hardware •Hadoop Core includes: ᵒ HDFS - Distributed file system derived from Google File System ᵒ Map/Reduce – Distributed compute engine
  • 26.
    Opportunities •HDFS as ObjectStore solution for CloudStack cloud •Hadoop on CloudStack • Implement services similar to Amazon’s Elastic MapReduce (EMR) • Big Data workload readily moved to IaaS clouds •CloudStack’s baremetal provisioning capabilities to build high performance Hadoop clusters •Feedback?
  • 27.
    CloudStack Community Nitin Mehta Committer, CS Meetup Coordinator Rohit Yadav Developer, CS Meetup Coordinator
  • 28.
    CloudStack Joins theApache Foundation Joins Apache Webserver, Tomcat, Hadoop, Cassendra 10+years, 100+ projects, 800 developers cwiki.apache.org/confluence/display/CLOUDSTACK/Index www.cloudstack.org
  • 29.
    Developers Love CloudStack •Open source (ASL 2.0) • Written in Java • Multiple APIs ᵒ Native CloudStack API ᵒ Amazon EC2 compatible API • Customizable Web UI ᵒ Rich AJAX client interface ᵒ Built on HTML/JSP, CSS, Javascript, jQuery • Active development community ᵒ http://www.cloudstack.org ᵒ Download binaries, source code.
  • 30.
    ASF Community Status • In Incubation since April 2012 • PPMC and Mentors setup in April • Code migrated to Apache git • Migration underway from cloudstack.org: – Jira (Bugs), wiki, forums • New committers, partners, developers every week • Apache CloudStack 4.0: First ASF community release coming in October 2012
  • 31.
    Join the community! •Why not? ᵒ Interested in learning the coolest technologies shaping the next decade. ᵒ Being part of a young, passionate community working at ground breaking speed. ᵒ Interested in solving cool real-world problems. ᵒ For the love of open source. ᵒ If you want to rise and shine. • Still don’t believe us – Let the numbers speak ᵒ 30k active community members in 8 months, ~100 IRC users ᵒ Exponentially growing dev/user community discussions ᵒ Apache is the industry standard for 10+ years in OSS foundations, 100+ projects, 800 developers ᵒ Well known projects: Apache webserver, Tomcat, Hadoop, Cassandra
  • 32.
    So lets getstarted …. • Check us out @ ᵒ cwiki.apache.org/confluence/display/CLOUDSTACK/Index ᵒ http://cloudstack.org • IRC: #CloudStack on irc.freenode.net • Forums: http://cloudstack.org/forum/ • Subscribe to Mailing Lists @ cloudstack-dev@incubator.apache.org cloudstack-users@incubator.apache.org • Join CloudStack Hyderabad Meetup Group @ http://www.meetup.com/CloudStack-Hyderabad-Group/
  • 33.

Editor's Notes

  • #6 Intro: Cloud computing is transforming IT and business for some of the largest companies… Key Points: Major service providers and Telcos around the globe are powered by Citrix and are enabling cloud services for Enterprise and SMB customers. Web 2.0 and Internet centric companies running core business services on the web are benefiting the efficiency, scale and cost benefits that come from right-sizing, cloud bursting, and provisioning on-demand. Enterprise customers are building private clouds for workloads such as Dev/test environments, product application, training and demos, among others while also connecting to public cloud services for access to computing resources. Transition: Let’s take a look at the solution a bit closer…  
  • #9 Intro: First and foremost, the solution you choose must have the right architecture… Key Points: Cloud is vastly different from server virtualization, the IT drivers alone are unique. Server virtualization is about scaling-up your server environment whereas Cloud is designed to scale-out. Server virtualization requires more IT management with 1 administrator for 100’s of servers vs Cloud which is more autonomic only requiring 1 administrator for 1000’s of servers. Finally cloud architecture requires an open stack, where various solutions can easily plug in to bring added value. Transition: Server virtualization is a key enabling component but a true cloud requires a platform that is purpose built…
  • #10 Intro: Cloud elements Key Points: Key elements of cloud Transition: Let’s look at customers utilizing cloud