OSCON 2010: Open Source Tool Chains for Cloud Computing

Open Source Tool
Chain for Cloud
Computing

Copyright © 2010 Opscode, Inc - All Rights Reserved 1

John Willis
Opscode VP of Services
@botchagalupe

Mark Hinkle
Zenoss VP of Community
@mrhinkle

Alex Honor
DTO Founder
@alexhonor

Devops

•Culture People

•Automation
Process
•Measurement

•Sharing Tools

Devops Sharing Automation
•Tool Chain
Project

•Monitoring and
control

•Provisioning,
Conﬁg and
Systems
Integrations

DevOps Toolchain Project

Alex Honor
Project Leader, ControlTier Open Source Project
Co-Founder, DTO Solutions

DevOps Toolchain Project Goal

Share and discuss...

DevOps problems

Toolchains

Best practices and lessons learned

DevOps Toolchain Project Goal

Share and discuss...

DevOps problems

Toolchains

Best practices and lessons learned

needs a name!

History of DevOps Toolchain Project
Noticed clients building ad-hoc toolchains: 2008 - 2009
Version 1.1 August 15, 2009

Fully automated provisioning paper: Oct 2009
Web Ops 2.0:
Achieving Fully Automated Provisioning

Contributors:

Damon Edwards, DTO Solutions

Andrew Schafer, Reductive Labs

Anthony Shortland, DTO Solutions

Alex Honor, ControlTier Project

Lee Thompson, Former VP & Chief Technologist of E*TRADE Financial

Creative Commons Licensed (Attribution - Share Alike)

OpsCamp Austin: Jan 2010

O’Reilly Velocity Online: Mar 2010

Google group “devops-toolchain”: Mar 2010

OpsCamp SF: May 2010

O’Reilly Velocity Conference: June 2010

Who are the ‘devops-toolchain’ group members?

http://groups.google.com/group/devops-toolchain


People:
System administrators

Application developers

Open source software tool
developers

Software product managers

Generalists and process
methodologists



People: Organizations:
System administrators E-Commerce

Application developers Search

Open source software tool Social media
developers
Gaming
Software product managers
Industrial process
Generalists and process
methodologists Financial

Commercial / OSS ISVs


Lots of interesting contributions and discussions
Ernest Mueller’s Scott Mcarty’s

Vlad’s case study


Sample of Discussion Topics
• Unix Like Tool Chains
• Open questions on unified pipe architecture
• Taxonomy (still TBD!)
• Distribution methods: package vs file, rscyn/murder vs yum/rpm vs DFS
• Configuration management: RPMs vs puppet/cfengine/chef tool?
• Rollback methodologies for package and config mgt tools
• Sizing a Devops team (what is a devops team?)
• Controlling and timing package release and config mgt tools
• Log management (aggregating, crunching, charting)
• Change detection
• Scripting language choices


We’re inspired by other conceptual models

quality of the web operations that support these businesses has lagged behind.
Outages are all too common. High variability and defect rates are bemoaned but have

become an accepted reality. Key engineers spend all day (and sometimes all night)
mired in deployment issues and bottlenecks. And topping it all off, what tooling that
does exist are usually a custom one-offs that are brittle and expensive to maintain.
Today’s business of operating software over the Web as a revenue producing service is a
” Today’s business of dramatic departure from the days when software was primarily produced for delivery on
operating software over the physical mediums and IT Operations was considered a back-of-the-house support
Web as a revenue producing function. Shouldn't we be completely rethinking our tooling and operational capabilities
service is a dramatic to match these new innovations?
departure from the days
when software was primarily In short, we need to get out of Web Operations 1.0 -- mired in legacy tools, outdated
produced for delivery on approaches, and low expectations -- and into Web Operations 2.0 where tools and
procedures are built from the ground up for highly efﬁcient, reliable, and agile

Industrial Control Automation
physical mediums...”
operations.

Runbook Automation

Control

Eventing, Alarm Mgmt

Charting, History, SPC

Measurement Instrumentation

System

There are multiple factors that go into achieving excellence in Web Operations, but the
linchpin that holds it all together is a fully automated provisioning system.
In this paper we will be:
1. Deﬁning what we mean by "fully automated provisioning"
2. Explaining why virtualization and cloud computing efforts fail without fully
automated provisioning capabilities
3. Proposing a reference open source tool chain for fully automated provisioning
4. Describing a live implementation where a leading online retailer is actively rolling
out a fully automated provisioning system using all open source tools

2 Web Ops 2.0: Achieving Fully Automated Provisioning



operations.

Runbook Automation

Control




System


Unix Tool Pipeline




operations.

Runbook Automation

Control




System


Unix Tool Pipeline


Brent Chapman’s Incident Command System

Integrated vs “integrate-able” tools


Integrated

Turn key solution that
provides end to end
functionality for the problem
domain


vs.
Integrated

Turn key solution that
provides end to end
functionality for the problem
domain


vs.
Integrated Integrate-able

Turn key solution that Chosen set of complementary
provides end to end independent parts than can
functionality for the problem be joined to solve a problem
domain domain

Commercial vs. Open Source Tools


Commercial
(Integrated)


vs.

Commercial
(Integrated)


vs.

Commercial Open Source
(Integrated) (Integrate-able)

Example: “Programmable Infrastructure”

http://www.webadminblog.com/

Example: Game operator’s framework

Monitoring: nagios, jcollectd, rrd, jmx

Deployment: ControlTier, Liquibase

Build: Hudson, SVN, Maven

Example: KaChing’s continuous deployment

Monitoring: nagios, jcollectd, rrd, jmx

Deployment: Custom app, rpm/yum

Build: Hudson, SVN, ant http://eng.kaching.com/

Generalized architecture

Control Provisioning Release
Orchestration Deploy
CI Server Issue tracker

Artifact
Config repository
Dispatcher management SCM
Build
Repository
OS boot/
Scheduler install

Code Workflows

Model Resources Configuration
Events Trending
Asset
Identity
inventory Topology
Trending Reporting

Host naming CMDB Sources
Monitoring

Release management toolchain

yum/yast
artifactory
Repository archiva
DFS/openEFS
HTTTP

eggs tgz/zip
Artifact gems rpm/deb/pkg/msi
perlmod jar/war/ear

make sbt
Build maven rake
ant/ivy phing

svn
cvs
SCM git
hg

bugzilla
Tracker mantis
trac

Monitoring and control toolchain

ControlTier
Runbook Automation Jobscheduler
OpenScheduler

nagios
Op Console, Control zenoss
opennms

Alarm Mgmt

rrdtool
Charting, History, SPC cacti

Measurement
Instrumentation

Provisioning toolchain

Capistrano
ControlTier
Command and
Application Service
Fabric
Control Orchestration Func
mCollective
Provisioning Activity

Bcfg2
cfengine
System Conﬁguration Chef
Conﬁguration Puppet
Smart Frog

Cloud: OS Install:
Cloud or VM Xen Kickstart
Bootstrapping OS lxc Jumpstart
Image
Install openVZ Cobbler
Launch Eucalyptus OpenQRM
KVM xCAT

Join us!

• Deﬁne missing tool chains

• Fix tool lists

• Keep working on taxonomy

• Document and share experience about your tool chain


Monitoring and Control

Mark Hinkle
VP of Community, Zenoss Inc.

Legacy IT

Different perspective, lack of coordination

Cartoon originally copyrighted by the authors; G. Renee Guzlas, artist

Legacy Monitoring Perspective

Types of Monitoring Data Collection
• Availability Monitoring – Binary, Moment in Time
• Performance Monitoring – Two Dimensions, Time
• SNMP
and State
• Change Management – Comparisons of states in
• SSH
Time
• Event Management – Normalizing Randomness
• WMI
•
•
Synthetic Transactions – Simulated Experiences
Business Service Management (BSM) –
• Syslog
$$$ Consequences of IT Performance
• Proprietary Agents

The Myth of the Nines

Availability % Downtime per Year Downtime per Month Downtime per Week

99.9% (three nines) 8.76 hours 43.2 minutes 10.1 minutes

99.95% 4.38 hours 21.56 minutes 5.04 minutes
99.99% (four nines) 52.6 minutes 4.32 minutes 1.01 minutes

99.999% (five 5.26 minutes 25.9 seconds 6.05 minutes
nines)
99.9999% (six 31.5 seconds 2.59 seconds .0605 seconds
nines)

•Average polling interval for monitoring? 5 minutes?
•Even super human operations people can’t be alerted and take action in under 5 minutes.
•One outage per year could drop service level to three nines.

Legacy Systems Management:
Fragmented Awareness
Global dashboard is a difficult mash-up of
disparate systems or doesn’t exist. No
communication, No automation
Database

Provisioning Configuration Management Performance & Availability Management

Analytics server Analytics server

Process server Process server Configuration Process server
Database database
Database

Multiple data models across disciplines with no Each management discipline
common object model managed has its own separate
Agent Agent Agent
product (UI, process,
database, and domain specific
Multiple agents required for each discipline language)
and platform

Systems Management DevOps Style:
Integrated Model, Interactive, Automated

Application Application

Op. System Op. System

Virtual Machine Virtual Machine

Physical/Virtual/Cloud Infrastructure

Example – Broadcast Company

Large premium television content provider serves national cable network with content
served from Linux servers.

• Servers are automatically built using configuration
management software
• As servers are brought into service configuration
management inserts hosts into CMDB used by
monitoring database
• One way interaction between configuration
management and monitoring system
• Reports are generated to determine which systems
are compliant

Example - Geeknet

Hundreds of servers, serving web, databases, and other infrastructure for some of the
world’s most highly trafficked websites – over 40 million visitors per month.
• Servers are automatically built using configuration
management software
• Discovery tool finds infrastructure and populates a
CMDB then spits out information to scripts that
translate information to BIND configurations for DNS
• Monitoring tool adds hosts to polling tool to start
monitoring servers for availability
• As infrastructure changes systems are updated
automatically
• Servers can be spun up and managed in minutes,
not hours automatically with little or no human
interaction

Unlegacy Future: Devops

Development Operations

Provisioning, Configuration
Management and Systems
Integration

John Willis
VP of Services - Opscode, Inc.

What Does
Configuration
Management
In The Cloud
Mean?

Did They Lie?


Did They Lie?

I did’ not have “operational”
relations with that provider

Caveat Emptor

• Provisioning
• Configuration Management
•Systems Integration

Provisioning
Nodes

opslb01

opsws01
opsws02

opsdm01
opsds01
opsds02 Copyright © 2010 Opscode, Inc - All Rights Reserved 37

Conﬁguration Management

Roles

loadbalancer
webserver
dbmaster
dbslave


Systems Integration

Load Balancer
Recipes

haproxy
Web Server Web Server

apache2
myssql DB Master

DB Slave DB Slave

Disk Disk Disk


What Do Developers
Want?


For Developers...

• Self Service Operations

For Developers...

• The infrastructure is the application
(and vice versa)

For Developers...

(and vice versa)

• Minimize Bottlenecks

For Developers...

(and vice versa)

• Minimize Bottlenecks
• The “Right” Tools

What Does
Operations Want?


Operations

http://covers.oreilly.com/images/9780596007836/lrg.jpg
Lean into it appears courtesy of Cliff Moon, of Dynomite fame: http://twitter.com/moonpolysoft

Operations
• Say “Yes”.


Operations
• Say “Yes”.
• You never liked rack
and stack that much
anyway.


Operations
• Say “Yes”.
and stack that much
anyway.

• You have never
been more critical.


Operations
• Say “Yes”.
and stack that much
anyway.

• You have never
been more critical.

• Just get out of the
way.

Industry Shifts

Be bold-and mighty forces
will come to your aid
Basil King


Infrastructure as Code
Development

Team focus
IDE/Workbench
Agile methodology
Source Control


Devops and Automation

Agile Operations
Operations as Code
Conﬁguration Management
Infrastructure as Code
Agile Infrastructure

Operations as Code


Role Based Services
name "webserver"
description "Systems that serve HTTP traffic"

run_list(
"role[base]",
"recipe[apache2]",
"recipe[apache2::mod_ssl]"
)

default_attributes(
"apache" => {
"listen_ports" => [ "80", "443" ]
}
)

override_attributes(
"apache" => {
"max_children" => "50"
}
)


Resources

Is a declarative description of the state you desire a part
of your node to be in.
http://www.ﬂickr.com/photos/xiaming/382205902/sizes/l/

Resources

• Is of a given type.


Resources


• Has a name.


Resources


• Has a name.

• Has attributes.


Resources


• Has a name.

• Has attributes.

• Takes actions to
bring the resource to
a declared state.


Recipies
Applies resources in the order they are speciﬁed

http://www.ﬂickr.com/photos/roadsidepictures/2478953342/sizes/o/

Recipies

• Can include other
recipes.


Recipies

• Can include other
recipes.

• Is just Ruby.


Load Balancer Example


Systems Integration

Load Balancer

Web Server Web Server

DB Master

DB Slave DB Slave

Disk Disk Disk


Tale of Two Startups
“Traditional” Operations Operations - The “Secret Sauce”
50 50

40 40
# of Hours

30 30

20 20
Hardware
OS Install
10 10 Conﬁg
Upkeep

20 20
15 15
Servers

10 10 New
5 5 Existing
0 0
1 2 3 4 5 6 7 9 10 11 12 1 2 3 4 5 6 7 9 10 11 12
Week # Week #
(http://radar.oreilly.com/archives/2007/10/operations-advantage.html)


Tale of Two Startups
“Traditional” Operations Operations - The “Secret Sauce”
50 50

This is the secret of
40 40
Cloud Computing.
Every other virtue stems from
# of Hours

30 30
here.
20 20
Hardware
OS Install
10 10 Conﬁg
Upkeep

20 20
15 15
Servers

10 10 New
5 5 Existing
0 0
1 2 3 4 5 6 7 9 10 11 12 1 2 3 4 5 6 7 9 10 11 12
Week # Week #
(http://radar.oreilly.com/archives/2007/10/operations-advantage.html)


OSCON 2010: Open Source Tool Chains for Cloud Computing

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (19)

More from Mark Hinkle

More from Mark Hinkle (20)

Recently uploaded

Recently uploaded (20)

OSCON 2010: Open Source Tool Chains for Cloud Computing