OSCON 2010: Open Source Tool Chains for Cloud Computing

What is Devops?

Devops •Culture People •Automation Process •Measurement •Sharing Tools

Devops Sharing Automation

Devops Sharing Automation •Tool Chain Project •Monitoring and control •Provisioning, Conﬁg and Systems Integrations

DevOps Toolchain Project Alex Honor Project Leader, ControlTier Open Source Project Co-Founder, DTO Solutions

DevOps Toolchain Project Goal Share and discuss... DevOps problems Toolchains Best practices and lessons learned

DevOps Toolchain Project Goal Share and discuss... DevOps problems Toolchains Best practices and lessons learned needs a name!

History of DevOps Toolchain Project Noticed clients building ad-hoc toolchains: 2008 - 2009 Version 1.1 August 15, 2009 Fully automated provisioning paper: Oct 2009 Web Ops 2.0: Achieving Fully Automated Provisioning Contributors: Damon Edwards, DTO Solutions Andrew Schafer, Reductive Labs Anthony Shortland, DTO Solutions Alex Honor, ControlTier Project Lee Thompson, Former VP & Chief Technologist of E*TRADE Financial Creative Commons Licensed (Attribution - Share Alike) OpsCamp Austin: Jan 2010 O’Reilly Velocity Online: Mar 2010 Google group “devops-toolchain”: Mar 2010 OpsCamp SF: May 2010 O’Reilly Velocity Conference: June 2010

Who are the ‘devops-toolchain’ group members? http://groups.google.com/group/devops-toolchain

Who are the ‘devops-toolchain’ group members? People: System administrators Application developers Open source software tool developers Software product managers Generalists and process methodologists http://groups.google.com/group/devops-toolchain

Who are the ‘devops-toolchain’ group members? People: Organizations: System administrators E-Commerce Application developers Search Open source software tool Social media developers Gaming Software product managers Industrial process Generalists and process methodologists Financial Commercial / OSS ISVs http://groups.google.com/group/devops-toolchain

Lots of interesting contributions and discussions Ernest Mueller’s Scott Mcarty’s Vlad’s case study http://groups.google.com/group/devops-toolchain

Sample of Discussion Topics • Unix Like Tool Chains • Open questions on unified pipe architecture • Taxonomy (still TBD!) • Distribution methods: package vs file, rscyn/murder vs yum/rpm vs DFS • Configuration management: RPMs vs puppet/cfengine/chef tool? • Rollback methodologies for package and config mgt tools • Sizing a Devops team (what is a devops team?) • Controlling and timing package release and config mgt tools • Log management (aggregating, crunching, charting) • Change detection • Scripting language choices http://groups.google.com/group/devops-toolchain

We’re inspired by other conceptual models

quality of the web operations that support these businesses has lagged behind. Outages are all too common. High variability and defect rates are bemoaned but have We’re inspired by other conceptual models become an accepted reality. Key engineers spend all day (and sometimes all night) mired in deployment issues and bottlenecks. And topping it all off, what tooling that does exist are usually a custom one-offs that are brittle and expensive to maintain. Today’s business of operating software over the Web as a revenue producing service is a ” Today’s business of dramatic departure from the days when software was primarily produced for delivery on operating software over the physical mediums and IT Operations was considered a back-of-the-house support Web as a revenue producing function. Shouldn't we be completely rethinking our tooling and operational capabilities service is a dramatic to match these new innovations? departure from the days when software was primarily In short, we need to get out of Web Operations 1.0 -- mired in legacy tools, outdated produced for delivery on approaches, and low expectations -- and into Web Operations 2.0 where tools and procedures are built from the ground up for highly efﬁcient, reliable, and agile Industrial Control Automation physical mediums...” operations. Runbook Automation Control Eventing, Alarm Mgmt Charting, History, SPC Measurement Instrumentation System There are multiple factors that go into achieving excellence in Web Operations, but the linchpin that holds it all together is a fully automated provisioning system. In this paper we will be: 1. Deﬁning what we mean by "fully automated provisioning" 2. Explaining why virtualization and cloud computing efforts fail without fully automated provisioning capabilities 3. Proposing a reference open source tool chain for fully automated provisioning 4. Describing a live implementation where a leading online retailer is actively rolling out a fully automated provisioning system using all open source tools 2 Web Ops 2.0: Achieving Fully Automated Provisioning

quality of the web operations that support these businesses has lagged behind. Outages are all too common. High variability and defect rates are bemoaned but have We’re inspired by other conceptual models become an accepted reality. Key engineers spend all day (and sometimes all night) mired in deployment issues and bottlenecks. And topping it all off, what tooling that does exist are usually a custom one-offs that are brittle and expensive to maintain. Today’s business of operating software over the Web as a revenue producing service is a ” Today’s business of dramatic departure from the days when software was primarily produced for delivery on operating software over the physical mediums and IT Operations was considered a back-of-the-house support Web as a revenue producing function. Shouldn't we be completely rethinking our tooling and operational capabilities service is a dramatic to match these new innovations? departure from the days when software was primarily In short, we need to get out of Web Operations 1.0 -- mired in legacy tools, outdated produced for delivery on approaches, and low expectations -- and into Web Operations 2.0 where tools and procedures are built from the ground up for highly efﬁcient, reliable, and agile Industrial Control Automation physical mediums...” operations. Runbook Automation Control Eventing, Alarm Mgmt Charting, History, SPC Measurement Instrumentation System There are multiple factors that go into achieving excellence in Web Operations, but the linchpin that holds it all together is a fully automated provisioning system. In this paper we will be: Unix Tool Pipeline 1. Deﬁning what we mean by "fully automated provisioning" 2. Explaining why virtualization and cloud computing efforts fail without fully automated provisioning capabilities 3. Proposing a reference open source tool chain for fully automated provisioning 4. Describing a live implementation where a leading online retailer is actively rolling out a fully automated provisioning system using all open source tools 2 Web Ops 2.0: Achieving Fully Automated Provisioning

quality of the web operations that support these businesses has lagged behind. Outages are all too common. High variability and defect rates are bemoaned but have We’re inspired by other conceptual models become an accepted reality. Key engineers spend all day (and sometimes all night) mired in deployment issues and bottlenecks. And topping it all off, what tooling that does exist are usually a custom one-offs that are brittle and expensive to maintain. Today’s business of operating software over the Web as a revenue producing service is a ” Today’s business of dramatic departure from the days when software was primarily produced for delivery on operating software over the physical mediums and IT Operations was considered a back-of-the-house support Web as a revenue producing function. Shouldn't we be completely rethinking our tooling and operational capabilities service is a dramatic to match these new innovations? departure from the days when software was primarily In short, we need to get out of Web Operations 1.0 -- mired in legacy tools, outdated produced for delivery on approaches, and low expectations -- and into Web Operations 2.0 where tools and procedures are built from the ground up for highly efﬁcient, reliable, and agile Industrial Control Automation physical mediums...” operations. Runbook Automation Control Eventing, Alarm Mgmt Charting, History, SPC Measurement Instrumentation System There are multiple factors that go into achieving excellence in Web Operations, but the linchpin that holds it all together is a fully automated provisioning system. In this paper we will be: Unix Tool Pipeline 1. Deﬁning what we mean by "fully automated provisioning" 2. Explaining why virtualization and cloud computing efforts fail without fully automated provisioning capabilities 3. Proposing a reference open source tool chain for fully automated provisioning 4. Describing a live implementation where a leading online retailer is actively rolling out a fully automated provisioning system using all open source tools 2 Web Ops 2.0: Achieving Fully Automated Provisioning Brent Chapman’s Incident Command System

Integrated vs “integrate-able” tools

Integrated vs “integrate-able” tools Integrated Turn key solution that provides end to end functionality for the problem domain

Integrated vs “integrate-able” tools vs. Integrated Turn key solution that provides end to end functionality for the problem domain

Integrated vs “integrate-able” tools vs. Integrated Integrate-able Turn key solution that Chosen set of complementary provides end to end independent parts than can functionality for the problem be joined to solve a problem domain domain

Commercial vs. Open Source Tools

Commercial vs. Open Source Tools Commercial (Integrated)

Commercial vs. Open Source Tools vs. Commercial (Integrated)

Commercial vs. Open Source Tools vs. Commercial Open Source (Integrated) (Integrate-able)

Example: “Programmable Infrastructure” http://www.webadminblog.com/

Example: Game operator’s framework Monitoring: nagios, jcollectd, rrd, jmx Deployment: ControlTier, Liquibase Build: Hudson, SVN, Maven

Example: KaChing’s continuous deployment Monitoring: nagios, jcollectd, rrd, jmx Deployment: Custom app, rpm/yum Build: Hudson, SVN, ant http://eng.kaching.com/

Generalized architecture Control Provisioning Release Orchestration Deploy CI Server Issue tracker Artifact Config repository Dispatcher management SCM Build Repository OS boot/ Scheduler install Code Workflows Model Resources Configuration Events Trending Asset Identity inventory Topology Trending Reporting Host naming CMDB Sources Monitoring

Release management toolchain yum/yast artifactory Repository archiva DFS/openEFS HTTTP eggs tgz/zip Artifact gems rpm/deb/pkg/msi perlmod jar/war/ear make sbt Build maven rake ant/ivy phing svn cvs SCM git hg bugzilla Tracker mantis trac

Monitoring and control toolchain ControlTier Runbook Automation Jobscheduler OpenScheduler nagios Op Console, Control zenoss opennms Alarm Mgmt rrdtool Charting, History, SPC cacti Measurement Instrumentation

Provisioning toolchain Capistrano ControlTier Command and Application Service Fabric Control Orchestration Func mCollective Provisioning Activity Bcfg2 cfengine System Conﬁguration Chef Conﬁguration Puppet Smart Frog Cloud: OS Install: Cloud or VM Xen Kickstart Bootstrapping OS lxc Jumpstart Image Install openVZ Cobbler Launch Eucalyptus OpenQRM KVM xCAT

Join us! • Deﬁne missing tool chains • Fix tool lists • Keep working on taxonomy • Document and share experience about your tool chain http://groups.google.com/group/devops-toolchain

Monitoring and Control Mark Hinkle VP of Community, Zenoss Inc.

Legacy IT Different perspective, lack of coordination Cartoon originally copyrighted by the authors; G. Renee Guzlas, artist

Legacy Monitoring Perspective Types of Monitoring Data Collection • Availability Monitoring – Binary, Moment in Time • Performance Monitoring – Two Dimensions, Time • SNMP and State • Change Management – Comparisons of states in • SSH Time • Event Management – Normalizing Randomness • WMI • • Synthetic Transactions – Simulated Experiences Business Service Management (BSM) – • Syslog $$$ Consequences of IT Performance • Proprietary Agents

The Myth of the Nines Availability % Downtime per Year Downtime per Month Downtime per Week 99.9% (three nines) 8.76 hours 43.2 minutes 10.1 minutes 99.95% 4.38 hours 21.56 minutes 5.04 minutes 99.99% (four nines) 52.6 minutes 4.32 minutes 1.01 minutes 99.999% (five 5.26 minutes 25.9 seconds 6.05 minutes nines) 99.9999% (six 31.5 seconds 2.59 seconds .0605 seconds nines) •Average polling interval for monitoring? 5 minutes? •Even super human operations people can’t be alerted and take action in under 5 minutes. •One outage per year could drop service level to three nines.

Legacy Systems Management: Fragmented Awareness Global dashboard is a difficult mash-up of disparate systems or doesn’t exist. No communication, No automation Database Provisioning Configuration Management Performance & Availability Management Analytics server Analytics server Process server Process server Configuration Process server Database database Database Multiple data models across disciplines with no Each management discipline common object model managed has its own separate Agent Agent Agent product (UI, process, database, and domain specific Multiple agents required for each discipline language) and platform

Systems Management DevOps Style: Integrated Model, Interactive, Automated Application Application Op. System Op. System Virtual Machine Virtual Machine Physical/Virtual/Cloud Infrastructure

Example – Broadcast Company Large premium television content provider serves national cable network with content served from Linux servers. • Servers are automatically built using configuration management software • As servers are brought into service configuration management inserts hosts into CMDB used by monitoring database • One way interaction between configuration management and monitoring system • Reports are generated to determine which systems are compliant

Example - Gameday

Example - Geeknet Hundreds of servers, serving web, databases, and other infrastructure for some of the world’s most highly trafficked websites – over 40 million visitors per month. • Servers are automatically built using configuration management software • Discovery tool finds infrastructure and populates a CMDB then spits out information to scripts that translate information to BIND configurations for DNS • Monitoring tool adds hosts to polling tool to start monitoring servers for availability • As infrastructure changes systems are updated automatically • Servers can be spun up and managed in minutes, not hours automatically with little or no human interaction

Unlegacy Future: Devops Development Operations

Provisioning, Configuration Management and Systems Integration John Willis VP of Services - Opscode, Inc.

What Does Configuration Management In The Cloud Mean?

Caveat Emptor • Provisioning • Configuration Management •Systems Integration

For Developers...

For Developers... • Self Service Operations

For Developers... • Self Service Operations • The infrastructure is the application (and vice versa)

For Developers... • Self Service Operations • The infrastructure is the application (and vice versa) • Minimize Bottlenecks

For Developers... • Self Service Operations • The infrastructure is the application (and vice versa) • Minimize Bottlenecks • The “Right” Tools

Operations http://covers.oreilly.com/images/9780596007836/lrg.jpg Lean into it appears courtesy of Cliff Moon, of Dynomite fame: http://twitter.com/moonpolysoft

Operations • Say “Yes”. http://covers.oreilly.com/images/9780596007836/lrg.jpg Lean into it appears courtesy of Cliff Moon, of Dynomite fame: http://twitter.com/moonpolysoft

Operations • Say “Yes”. • You never liked rack and stack that much anyway. http://covers.oreilly.com/images/9780596007836/lrg.jpg Lean into it appears courtesy of Cliff Moon, of Dynomite fame: http://twitter.com/moonpolysoft

Operations • Say “Yes”. • You never liked rack and stack that much anyway. • You have never been more critical. http://covers.oreilly.com/images/9780596007836/lrg.jpg Lean into it appears courtesy of Cliff Moon, of Dynomite fame: http://twitter.com/moonpolysoft

Operations • Say “Yes”. • You never liked rack and stack that much anyway. • You have never been more critical. • Just get out of the way. http://covers.oreilly.com/images/9780596007836/lrg.jpg Lean into it appears courtesy of Cliff Moon, of Dynomite fame: http://twitter.com/moonpolysoft

Role Based Services name "webserver" description "Systems that serve HTTP traffic" run_list( "role[base]", "recipe[apache2]", "recipe[apache2::mod_ssl]" ) default_attributes( "apache" => { "listen_ports" => [ "80", "443" ] } ) override_attributes( "apache" => { "max_children" => "50" } ) Copyright © 2010 Opscode, Inc - All Rights Reserved 49

Resources Is a declarative description of the state you desire a part of your node to be in. http://www.ﬂickr.com/photos/xiaming/382205902/sizes/l/

Resources • Is of a given type. Is a declarative description of the state you desire a part of your node to be in. http://www.ﬂickr.com/photos/xiaming/382205902/sizes/l/

Resources • Is of a given type. • Has a name. Is a declarative description of the state you desire a part of your node to be in. http://www.ﬂickr.com/photos/xiaming/382205902/sizes/l/

Resources • Is of a given type. • Has a name. • Has attributes. Is a declarative description of the state you desire a part of your node to be in. http://www.ﬂickr.com/photos/xiaming/382205902/sizes/l/

Resources • Is of a given type. • Has a name. • Has attributes. • Takes actions to bring the resource to a declared state. Is a declarative description of the state you desire a part of your node to be in. http://www.ﬂickr.com/photos/xiaming/382205902/sizes/l/

Recipies Applies resources in the order they are speciﬁed http://www.ﬂickr.com/photos/roadsidepictures/2478953342/sizes/o/

Recipies Applies resources in the order they are speciﬁed • Can include other recipes. http://www.ﬂickr.com/photos/roadsidepictures/2478953342/sizes/o/

Recipies Applies resources in the order they are speciﬁed • Can include other recipes. • Is just Ruby. http://www.ﬂickr.com/photos/roadsidepictures/2478953342/sizes/o/

Tale of Two Startups “Traditional” Operations Operations - The “Secret Sauce” 50 50 40 40 # of Hours 30 30 20 20 Hardware OS Install 10 10 Conﬁg Upkeep 20 20 15 15 Servers 10 10 New 5 5 Existing 0 0 1 2 3 4 5 6 7 9 10 11 12 1 2 3 4 5 6 7 9 10 11 12 Week # Week # (http://radar.oreilly.com/archives/2007/10/operations-advantage.html) Copyright © 2010 Opscode, Inc - All Rights Reserved 54

Tale of Two Startups “Traditional” Operations Operations - The “Secret Sauce” 50 50 This is the secret of 40 40 Cloud Computing. Every other virtue stems from # of Hours 30 30 here. 20 20 Hardware OS Install 10 10 Conﬁg Upkeep 20 20 15 15 Servers 10 10 New 5 5 Existing 0 0 1 2 3 4 5 6 7 9 10 11 12 1 2 3 4 5 6 7 9 10 11 12 Week # Week # (http://radar.oreilly.com/archives/2007/10/operations-advantage.html) Copyright © 2010 Opscode, Inc - All Rights Reserved 54

http://www.oscon.com	73
http://socializedsoftware.com	42
http://feeds.feedburner.com	24
http://cloudcomputing.sys-con.com	21
http://www.cloud24by7.com	11
http://opensourceclouds.ulitzer.com	2
http://markrhinkle.sys-con.com	2
http://markrhinkle.ulitzer.com	1
http://e-collaboration.ulitzer.com	1
http://www.netvibes.com	1
http://paper.li	1
http://www.slideshare.net	1
http://feeds2.feedburner.com	1
http://127.0.0.1:8795	1
http://www.linkedin.com	1

OSCON 2010: Open Source Tool Chains for Cloud Computing

by Mark Hinkle

Accessibility

Categories

Upload Details

Usage Rights

15 Embeds 183

Statistics

OSCON 2010: Open Source Tool Chains for Cloud Computing Presentation Transcript