Open Source Toolchains to Manage Cloud Infrastructure


Published on

Open Source Toolchains to Manage Cloud Infrastructure presentation for Cloud Computing Expo East - June 6, 2011.

Added APIs (jclouds, fog, libcloud, deltacloud)

Published in: Technology, Business
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • How many users are using
  • IN the late 1800s English economist William Stanley Jevons noticed that despite huge increases in the efficiency of coal burning, demand actually increased rather than decreased. In cloud computing despite huge cost savings the consumption of cloud computing is and will continue to grow as demand increases. The downside is that this increased consumption has the ability to increase management burden if users don’t adopt leveraged methods for managing cloud computing.
  • With a credit card and a single administrator can spin up 100s of servers in minutes.
  • How does the management personnel for the cloud scale? …Automation.
  • DevOps is a popular movement among cloud operations professionals who are advocating many of the same methodologes used in Agile software development.
  • Ideally for the cloud you create management toolchains that automate the management of your cloud.
  • Cloud computing promises highly available systems, but if you have a reactive approach you won’t achieve that goal. Build redundant, highly automated systems that can self-heal.
  • Other disciplines like back-up, log management, performance and security (virus,intrusion detection) are important but not core to the delivery of cloud computing systems
  • Tools that can automatically build your systems/install systems
  • Configuration management tools abstract infrastructure into variables that are implemented via “recipes”Interesting new development is the PXE boot Crowbar extension for Chef Server used to provision virtual machines in OpenStack.
  • API lets you develop to one API and move between cloudseasilyTypesof Tasks Accomplished by an APIProvisioning (creating, re-creating, moving, or deleting components e.g. virtual machines, vlans)Configuration (assigning or changing attributes of the architecture such as security and network settings)Cloud ProvidersJclouds – java API Abstraction Libcloud – started by CloudKick (now Rackspace) to abstract clouds, Apache incubator projectDeltacloud – started by Red Hat to abstract clouds, Apache incubator projectFog – started by EngineYard is a ruby gem that allows you to interact with number of cloud API.
  • Java applications are developed and checked into Source Control Manager (SCM)Applications built and dependencies resolved via MavenBuilt and unit-tested by HudsonChanges recorded in Change Management Database (CMDB)Deployed by ControlTier (RunDeck)
  • Open Source Toolchains to Manage Cloud Infrastructure

    1. 1. Open Source Toolchainsto Manage CloudInfrastructureMark R. HinkleVP of
    2. 2. 2Introduction•  Responsible for Driving Adoption of CloudStack Open Source Cloud Computing Software•  Former manager of Zenoss Open Source project 100,000 users, 1.5 million downloads•  Former Linux Desktop Advocate (Zealot?)•  Former LinuxWorld Magazine Editor-in-Chief•  Open Management Consortium Conspirator•  Open Desktop Consortium Instigator•  Author - “Windows to Linux Business Desktop Migration” - Thomson•  NetDirector Project - Open Source Con guration Management Project•  Sometimes Author and Blogger at•  Start-up junkie, Glutton for punishment
    3. 3. 3Agenda•  Introductions•  Cloud & Systems Management Overview•  Open Source Tools for Cloud Management•  Creating Open Source Toolchains•  Questions (and Hopefully Answers)
    4. 4. Cloud Computing &Systems Management Overview
    5. 5. 5Jevon’s Paradox and the Cloud…the efficiency with which a resource is used tendsto increase (rather than decrease) the rate ofconsumption of that resource.…in cloud computing increasedconsumption can introduces signi cantmanagement overhead despite higherhardware utilization and other bene ts 1865, the English economist William Stanley Jevons
    6. 6. 6Balancing the Equation:Creating the Cloud + =
    7. 7. 7Balancing the Equation:Managing the Cloud + ? =
    8. 8. Automation Balances the 8Equation•  MeatCloud, Can’t Keep up with Cloud Computing•  Devops & Agile IT Philosophy•  Script Repetitive Tasks•  Integrate Tools•  Automate, Automate, Automate
    9. 9. DevOps – Agile Management for 9the Cloud•  Agile, fast moving•  Smaller incremental change•  Increased coordination across groups (development/operations)•  Eye towards Automation
    10. 10. 10Management ToolchainsToolchain (n): MonitoringA set of tools where Patching and Provisioningthe output of onetool becomes the Con gurationinput of another tool
    11. 11. 11The Myth of the NinesAvailability % Downtime per Downtime per Downtime per Year Month Week99.9% (three nines) 8.76 hours 43.2 minutes 10.1 minutes99.95% 4.38 hours 21.56 minutes 5.04 minutes99.99% (four nines) 52.6 minutes 4.32 minutes 1.01 minutes99.999% ( ve nines) 5.26 minutes 25.9 seconds 6.05 seconds99.9999% (six nines) 31.5 seconds 2.59 seconds .0605 seconds Average polling interval for monitoring? 5 minutes? Even superhuman operations people can’t be alerted and take action in under 5 minutes. One outage per year could drop service level to three nines or worse.
    12. 12. Open Source Tools toManage the Cloud
    14. 14. 144 Types of Management Tools Provisioning Installation of operating systems and other softwareConfiguration ManagementSets the parameters for servers, can specifyinstallation parameters Orchestration/Automation Automate tasks across systemsMonitoringRecords errors and health of IT infrastructure
    15. 15. 15Cloudy Tools•  Network Capable•  Cloud “Aware”•  Easy-to-Integrate•  Adhere to Open Standards•  Lend Themselves to Automation
    16. 16. Open Source 16Provisioning Tools Year Started Language License Installation TargetsCobbler 2007 Python GPL Red Hat, OpenSUSE Fedora, Debian, UbuntuFully Automatic 2000 Perl GPL DebianInstallation (FAI)Kickstart ? Python GPL Most .dep and RPM based Linux distrosSpacewalk 2008 Perl, Python, Java GPL Fedora, CentosCrowbar/Chef 2011 Ruby Apache ?
    17. 17. Open Source 17Configuration Management Tools Year Started Language License Client/ServerBcfg2 2003 Python BSD YesCfengine 1993 C Apache YesChef 2009 Ruby Apache Chef Solo – No Chef Server - YesPuppet 2004 Ruby GPL yes
    18. 18. Open Source 18Monitoring Tools Year Started License Language Type of Collection Monitoring MethodsCacti (RRDTool) 2001 GPL PHP Performance SNMP, syslogNagios 1999 GPL C/PHP Availability SNMP,TCP, ICMP, IPMI, syslogOpenNMS 2000 GPL Java Availability/ SNMP, PerformanceZabbix 2001 GPL C/PHP Availability/ SNMP, TCP/ICMP, Performance and IPMI, Synthetic more TransactionsZenoss 2005 GPL Python Availability, SNMP, ICMP, SSH, Performance, Event syslog, WMI Management
    19. 19. Open Source 19Automation/Orchestration Tools Year Language License Client/ Support Started Server OrganizationCapistrano 2006 Ruby MIT Yes NoneRunDeck 2010 Java Apache Yes DTO SolutionsFunc 2007 Python GPL Yes Fedora ProjectMCollective 2009 Ruby Apache Yes PuppetLabs
    20. 20. 20Cloud APIs Aren’t Created EqualOpen Source AbstractionsAPI Languagejclouds javadeltacloud pythonLib cloud pythonFog Ruby
    21. 21. Open Source ManagementToolchains
    22. 22. 22Toolchain Concept Source:
    23. 23. 23Automated Toolchain Capistrano Command and Application Service RunDeck Fabric Control Orchestration Func Provisioning Activity BCFG2 Con guration Cfengine System Con guration Chef Puppet Cloud: OS Install: Eucalyptus Kickstart Cloud Image OS OpenStack Cobbler Bootstrapping Launch Install CloudStack Spacewalk Abiquo
    24. 24. 24Example: GeekNet•  Servers are automatically built using con guration management software (Puppet)•  Discovery tool (Zenoss) nds infrastructure and populates a CMDB then spits out Hundreds of servers, information to scripts that translate serving web, databases, information to BIND con gurations for DNS and other infrastructure•  Monitoring tool adds hosts to polling tool for some of the world’s (Nagios) to check servers for availability most highly trafficked•  As infrastructure changes systems are updated websites – over 40 million automatically visitors per month.•  Servers can be spun up and managed in minutes, not hours automatically with little or no human interaction
    25. 25. Example Toolchain: 25Online Gaming Studio
    26. 26. Questions?
    27. 27. 27Contact Me Professional: Personal: Professional: Personal: Professional: Personal: Mark R. Hinkle VP of Community Twitter: Open Source Enthusiast @clouddotcom @mrhinkle
    28. 28. Open Source Toolchains to Manage Cloud Computing Mark R. Hinkle is licensed under a Creative Commons Attribution-ShareAlike 3.0 United States License.