Austin agile conf 2012 infrastructure automation-gmiranda

Agility Through
Infrastructure Automation
George Miranda
gmiranda@opscode.com
Austin Agile Conference 2012
November 16, 2012
Friday, November 16, 12

Introductions
# finger $(whoami)
Login: gmiranda! ! ! ! ! ! ! Name: George Miranda
Directory: /home/gmiranda! ! ! Shell: /bin/bash
On since Mon 14 Apr 1997 18:01 (GMT) on tty1 from :0
No mail on gmiranda@opscode.com
Plan:
twitter:! gmiranda23
github:!! gmiranda23
irc:!! ! gmiranda23! (irc.freenode.net - #chef)
community:!gmiranda23! (community.opscode.com)
role:! ! consultant, evangelist, trainer, *:*

Scope
Automation + Culture = Agility

Scope
• Infrastructure Automation Approaches

Scope
• Infrastructure & Automation Best
Practices

Scope
Practices
• Cultural Pitfalls

Scope
Practices
• Making more awesome

Scope
Practices
• Making more awesome
What this talk is not
• Chef vs. Puppet
• Cloud All The Things!!!
• How to structure your
Organization
• Which Development Model to
adopt

System Build Approaches
http://www.ﬂickr.com/photos/dancedaoc/3083836988/sizes/z/in/photostream/

Complications
• “That one host” you know you can’t rebuild
• Untracked conﬁguration change
• Collections of Bash, PERL, Python, ???
• Rebuild from: wiki, cheatsheets, folklore
http://www.ﬂickr.com/photos/humblog/4996661110/sizes/l/in/photostream/

Unprecedented Growth
1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 201980 1981 1982 1983 1984 1985 1986 1987 1988 1989
Virtual Nodes
Physical Hardware
1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015

1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 201980 1981 1982 1983 1984 1985 1986 1987 1988 1989
Virtual Nodes
Physical Hardware
1980
Mainframe
1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015

1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 201980 1981 1982 1983 1984 1985 1986 1987 1988 1989
Virtual Nodes
Physical Hardware
1980
Mainframe
1990
Client/Server
1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015

1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 201980 1981 1982 1983 1984 1985 1986 1987 1988 1989
Virtual Nodes
Physical Hardware
1980
Mainframe
1990
Client/Server
2000
Datacenter
1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015

1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 201980 1981 1982 1983 1984 1985 1986 1987 1988 1989
Virtual Nodes
Physical Hardware
1980
Mainframe
1990
Client/Server
2000
Datacenter
2010+
Cloud
1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015

1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 201980 1981 1982 1983 1984 1985 1986 1987 1988 1989
Virtual Nodes
Physical Hardware
1980
Mainframe
1990
Client/Server
2000
Datacenter
2010+
Cloud
The things that got us here…
…must change to get us here!
1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015

The Rise of Conﬁguration Management
http://www.ﬂickr.com/photos/24375810@N06/6611017007

We have a problem at scale

Here’s a hint

Cabling?

Close...

http://www.ﬂickr.com/photos/michaelheiss/3090102907/
Complexity

Infrastructure

Items we manipulate
• Routes
• Users
• Groups
• Tasks
• Packages
• Software
• Services
• Nodes
• Networking
• Files
• Directories
• Symlinks
• Mounts
• Ruby Gems
• Python Modules
• Java Artifacts
• Disks
• Volumes
• Filesystems
• Firewall Rules

See Node
Application

See Nodes
Application
Application Database

See Nodes Grow
Application
Application Databases

Grow Nodes
App Servers

...Grow
Load Balancer
App Servers

Grow Nodes, Grow
Load Balancers
App Servers

Grow Nodes, Grow
Load Balancers
App DB Cache
App Servers

Infrastructure has a Topology
Load Balancers
App DB Cache
App Servers

Infrastructure IS a Snowﬂake
Load Balancers
App DB Cache
App Servers
Floating IP?

Complexity Increases Quickly
App LBs
App Servers
NoSQL
DB slaves
Cache
DB Cache
DBs

... Increases Very Quickly
DC1 DC3
DC2

Conﬁguration Management
http://www.ﬂickr.com/photos/philliecasablanca/3354734116/

Sysadmins

The Past

• Labor intensive
• Error prone
• Hard to reproduce
• Unsustainable
http://www.ﬂickr.com/photos/pureimaginations/4805330106/
Manual Conﬁguration

• Typically very brittle
• Throw away, one off scripts
• grep sed awk perl
• curl | bash
http://www.ﬂickr.com/photos/40389360@N00/2428706650/
Scripting

• NFS mounts
• rdist
• scp-on-a-for-loop
• rsync on cron
http://www.ﬂickr.com/photos/walkadog/4317655660
File Distribution

for i in `cat servers.txt` ;
do scp ntp.conf root@
$i:/etc/ntpd.conf ; done
do ssh root@$i /etc/
init.d/ntpd restart ; done
do ssh root@$i
chkconfig ntpd on ; done
• ^ does not scale
http://www.ﬂickr.com/photos/alexerde/3479006495
This used to be awesome.

• Cluster SSH
• ISConf
• Golden Images
Execution Management

Typical Boring Infrastructure
Jboss App
Memcache
Postgres Slaves
Postgres Master
NagiosGraphite

Jboss App
Memcache
Postgres Slaves
Postgres Master
NagiosGraphite
• Move SSH off port 22
• Lets put it on 2022

Jboss App
Memcache
Postgres Slaves
Postgres Master
NagiosGraphite
• edit /etc/ssh/sshd_conﬁg

Jboss App
Memcache
Postgres Slaves
Postgres Master
NagiosGraphite
• edit /etc/ssh/sshd_conﬁg
1 2
3
4
5
6

Maintenance Window
Jboss App
Memcache
Postgres Slaves
Postgres Master
NagiosGraphite

Maintenance Window
Jboss App
Memcache
Postgres Slaves
Postgres Master
NagiosGraphite 1 2
3
8
5 64 7
9
10 11
12
• Launch, Delete
• Repeat
• Typically manually

Maintenance Window
Jboss App
Memcache
Postgres Slaves
Postgres Master
NagiosGraphite 1 2
3
8
5 64 7
9
10 11
12
• Launch, Delete
• Repeat
• Typically manually
• Don’t break anything!
• Bob just got ﬁred =(

Different IP Addresses?
Jboss App
Memcache
Postgres Slaves
Postgres Master
NagiosGraphite
• Invalid conﬁgs!

Systems Integration
• Keep a list of current
resources
• Collect vast amounts of data
on those resources
• Quickly search through stacks
of current resource data
• Generate your Infrastructure
Topology from a current
source of truth
http://www.flickr.com/photos/fotos_medem/3399096196/

So when this...
Jboss App
Memcache
Postgres Slaves
Postgres Master
NagiosGraphite

... becomes this...
Jboss App
Memcache
Postgres Slaves
Postgres Master
NagiosGraphite

That can happen automatically
Jboss App
Memcache
Postgres Slaves
Postgres Master
NagiosGraphite

Managing Complexity Today

Managing Complexity Today
How Do we Manage This
at Cloud Scale?
• Thousands of infrastructure
dependencies and
conﬁgurations needed for
each change.
• Huge Amounts of Time
• Increased Cost of
Correction of Manual Errors
• Huge Need for Talent
• Risk of Critical Skills
Shortage

Google,Amazon, Microsoft,Yahoo
built their own tools

but it was “secret sauce”

everyone else was here
... inexperienced & poorly equipped
for the world they must now
operate in.

Infrastructure
"It is common to think in terms of individual
machines rather than view an entire
infrastructure as a combined whole"
“A good infrastructure, whether departmental,
divisional, or enterprise-wide, is a single loosely-
coupled virtual machine, with hundreds or
thousands of hard drives and CPU's.”
-- Bootstrapping an Infrastructure USENIX LISA ’98

Infrastructure as Code
• Programmatically provision and
conﬁgure
• Treat like any other code base
• Gives you tools to manage
complexity while being ﬂexible
enough to evolve with your
Infrastructure
• Reconstruct the business from
code repository, data backup, and
baremetal resources

Declarative Syntax
• Deﬁne policy
• Say what, not how
• Abstraction between platforms
• Many positive side effects

Idempotence
• You’ll hear this a lot
• Property of declarative interface
• Eliminates brittleness of
scripting
• Identity function: f(x)=x
• Safe to repeat

Chef is a Tool
http://www.ﬂickr.com/photos/wessexandy/7690486884/sizes/c/in/pool-96164123@N00/

Wax Philosophical
• We are artists & masters of our
craft
• Everyone needs great tools
• Nobody remember’s Picasso’s
paintbrush
http://www.ﬂickr.com/photos/vgm8383/2686128924/sizes/l/

The core ideas in this talk:
Automation + Culture =
AGILITY!

Pitfalls
http://www.ﬂickr.com/photos/nesposit/2787559303/sizes/o/in/photostream/

This should sound familiar...

Traditional thinking
Dev’s job is to add new features
Ops’ job is to keep the site stable and fast
http://www.ﬂickr.com/photos/stewart/461099066/
Slide Courtesy of John Allspaw - http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-ﬂickr

Ops’ job is NOT to keep the site
stable and fast
Dev’s job is NOT to add new
features

OUR job
is to ENABLE our business

Our business
REQUIRES change

BUT CHANGE IS THE CAUSE
OF MOST OUTAGES!

Choose:
Discourage change in the interests of
stability
OR
Allow change to happen as often as it
needs to

http://www.ﬂickr.com/photos/gsfc/6795048198/sizes/o/in/photostream/
The Great Abyss

The right culture is a requirement
for survival & success at web
scale.

Lessons Learned:
Every Post-mortem

Lessons Learned:
Every Post-mortem
Ever...

Root Cause:
“Bad Luck... it was a perfect
storm of impossible events”

Lesson #1
“We have a bunch of manual
processes which we need to
automate”

Lesson #2
“We introduced too many changes
at once”

Slide Courtesy of John Allspaw - http://www.slideshare.net/jallspaw/ops-metametrics-the-currency-you-pay-for-change

Image Courtesy of John Allspaw - http://www.slideshare.net/jallspaw/ops-metametrics-the-currency-you-pay-for-change
RAAAWR!!!
I’m SCARY!

Images Courtesy of John Allspaw - http://www.slideshare.net/jallspaw/ops-metametrics-the-currency-you-pay-for-change
I can haz
cuddle?

MAKE
MORE
AWESOME!!!

Continuous Delivery
Faster Time toValue
Higher Availability
Happier Teams
More Cool Stuff

Stuff Suits Care About
• Visibility & Accountability
• Reduce Risk
• Business Agility

Stuff Engineers care about
• Change when we need it
• Innovate Faster
• Constant Improvements
• Application & Site Resiliency

Recap
•Step 1) Automate your Infrastructure

Recap
•Step 2) Bridge the Cultural Divide

Recap
•Step 3) Proﬁt!

Recap
•Step 3) Proﬁt!
•Automation + Culture = Agility

Try it out!
• Hosted Chef is a SaaS product hosted by Opscode
• http://manage.opscode.com
• Our wiki: http://wiki.opscode.com
• Fast start guide:
• http://wiki.opscode.com/display/chef/Fast+Start+Guide
• Our Community site: http://community.opscode.com
• Cookbooks in our Github account: http://github.com/opscode/cookbooks
• The materials for our 3-day Chef Fundamentals class are online:
• https://github.com/opscode/chef-fundamentals

Supported Platforms
• Ubuntu (10.04, 10.10, 11.04, 11.10, 12.04)
• Debian (5.x, 6.x)
• RHEL & CentOS (5.x, 6.x)
• Fedora 10+
• SUSE Enterprise (11.2)
• openSUSE (12.1)
• Solaris (5.9, 5.10, 5.11 -- x86 and SPARC)
• Mac OS X (10.4, 10.5, 10.6, 10.7)
• Windows 7
• Windows Server 2003 R2, 2008, 2008 R2

Additional Resources
• OpscodeYoutube Channel:
• http://www.youtube.com/opscode
• Jesse Robbins, Changing Culture & Being a force for Awesome
• http://www.youtube.com/watch?v=OU8ihx3nT6I
• Matt Ray on Automating Continuous Deployment
• http://www.opscode.com/blog/2012/11/13/automating-continuous-
deployment-wchef/
• Continuous Delivery by Jez Humble & David Farley
• http://continuousdelivery.com/

Thanks!
• George Miranda
• gmiranda@opscode.com
• @gmiranda23

Questions?
• On freenode: #chef and #chef-hacking
• http://lists.opscode.com
• http://tickets.opscode.com
• http://help.opscode.com
• @opscode and @opscode_status on Twitter

Austin agile conf 2012 infrastructure automation-gmiranda

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (13)

Similar to Austin agile conf 2012 infrastructure automation-gmiranda

Similar to Austin agile conf 2012 infrastructure automation-gmiranda (20)

Recently uploaded

Recently uploaded (20)

Austin agile conf 2012 infrastructure automation-gmiranda