This document discusses how to tame infrastructure that has grown wild over time. It describes two types of tangled infrastructure - visible and obfuscated "rat's nests". It then outlines several common "infrastructure smells" like configuration drift, systems entropy, staffing atrophy, reliance on gold images, and unstable software stacks. The document proposes tools and strategies to address these issues, including system configuration tools, OS installation tools, virtualization tools, separating concerns, continuous training, avoiding gold images, and container partitioning through automation. The overall message is to bring order to infrastructure chaos through automation, separation of duties, and continuous improvement practices.

1. TAMING THE INFRASTRUCTURE
GONE WILD
Isaac Christoffersen
Contributing Partner - Open Source Solutions,
Colloquium
This work is licensed under the Creative Commons Attribution 3.0 United States License. To view a
copy of this license, visit http://creativecommons.org/licenses/by/3.0/us/

2. OVERVIEW
– THE VISIBLE RAT’S NEST
– THE OBFUSCATED RAT’S NEST
INFRASTRUCTURE SMELLS
–
–
–
–
–
CONFIGURATION DRIFT
SYSTEMS ENTROPY
STAFFING ATROPHY
GOLD IMAGE
SOFTWARE STACKS
KEY TOOLS
– SYSTEM CONFIGURATION TOOLS
– OS INSTALLATION TOOLS
– VIRTUALIZATION TOOLS
SUMMARY

3. A NEW SERVER ROOM – COMPLETE WITH THAT “NEW CAR” SMELL
Room for
expansion
Cables all neatly
tied and out of
the way.

4. THE VISIBLE RAT’S NEST

5. THE OBFUSCATED RAT’S NEST

6. PROVISIONING INFRASTRUCTURE IS HARD.
I need 3 servers to use
for the development of a
proof of concept. It's
going to be the next big
thing!
Process
Procurement Process
Order Fulfillment Process
Staff Recruitment & Hiring
Policy
Service Level Agreements
Budgets
Staffing Guidance
Infrastructure
Tools
Linux
Xen
Hyperic & Drools
Cobbler & Koan
Puppet
Commodity Servers
SANs
Networking Equipment
Power
HVAC Systems
People
Linux Administrators
SAN Engineers
Network Engineers

7. OVERVIEW
– THE VISIBLE RAT’S NEST
– THE OBFUSCATED RAT’S NEST
INFRASTRUCTURE SMELLS
–
–
–
–
–
CONFIGURATION DRIFT
SYSTEMS ENTROPY
STAFFING ATROPHY
GOLD IMAGE
TEETERING STACKS
KEY TOOLS
– SYSTEM CONFIGURATION TOOLS
– OS INSTALLATION TOOLS
– VIRTUALIZATION TOOLS
SUMMARY

8. SMELL :: CONFIGURATION DRIFT
• Any change outside a baseline
configuration is considered a
configuration drift.
• Could be an accidental or incidental or
intentional.
• Early identification will minimize
impact and reduce the risks around
business continuity.

9. PRESCRIPTION :: AUTOMATED CONFIGURATION

10. SMELL :: SYSTEMS ENTROPY
As a system is modified, its disorder, or entropy,
always increases.
• Similar to theories in software development, a
system that is used will be modified.
• When a system is modified, its complexity will
increase, provided that one is not actively working
to reduce complexity.

11. PRESCRIPTION :: SEPARATION OF CONCERNS
How can I build my web hosting business?
How can I manage my customer relationships?
How can I run my company's IT department?
How can I collaborate across the enterprise?
How can I view my corporate sales data?
How do I store & retrieve my inventory data?
How do I deploy my web services?
How do I get the most of my IT and
personnel resources?
Data
Center
Software
Platform
Infrastructure

12. SMELL :: STAFFING ATROPHY
“Atrophy is the partial or complete
wasting away of a part of the body.
Causes include lack of exercise and/or
use”

13. PRESCRIPTION :: DRILL, BABY. DRILL.

14. SMELL :: GOLD IMAGE

15. PRESCRIPTION :: USE CONTINUOUS PROVISIONING
Consistent, repeatable builds encourage fast
deployment and consistently identical
systems without the burden of managing
images
A repeatable build improves product
quality, reduces the mundane tasks
and minimizes dependencies on key
people.
build
cycle
Software
packages

16. SMELL :: TEETERING STACK

17. PRESCRIPTION :: CONTAINER PARTITIONING
Virtualization can reduce cost while also improving flexibility…
Cluster multiple physical servers to gain failover capabilities
Share resources across physical servers
source: “Automated failover and recovery of virtualized guests
in Advanced Platform”, Rob Kenna, Redhat Magazine

18. OVERVIEW
– THE VISIBLE RAT’S NEST
– THE OBFUSCATED RAT’S NEST
INFRASTRUCTURE SMELLS
–
–
–
–
–
CONFIGURATION DRIFT
SYSTEMS ENTROPY
STAFFING ATROPHY
GOLD IMAGE
SOFTWARE STACKS
KEY TOOLS
– SYSTEM CONFIGURATION TOOLS
– OS INSTALLATION TOOLS
– VIRTUALIZATION TOOLS
SUMMARY

19. SYSTEM CONFIGURATION TOOLS
Puppet
http://www.puppetlabs.com
Cfengine
http://www.cfengine.org/
Chef
http://www.opscode.com/
And many more…

20. OS INSTALLATION TOOLS
Cobbler
https://fedorahosted.org/cobbler/
Spacewalk
http://spacewalk.redhat.com/
And many more…

21. VIRTUALIZATION TOOLS
And many more…

22. OVERVIEW
– THE VISIBLE RAT’S NEST
– THE OBFUSCATED RAT’S NEST
INFRASTRUCTURE SMELLS
–
–
–
–
–
CONFIGURATION DRIFT
SYSTEMS ENTROPY
STAFFING ATROPHY
GOLD IMAGE
SOFTWARE STACKS
KEY TOOLS
– SYSTEM CONFIGURATION TOOLS
– OS INSTALLATION TOOLS
– VIRTUALIZATION TOOLS
SUMMARY

23. •
•
•
•
•
SEPARATE CONCERNS.
DRILL, BABY. DRILL.
ALL THAT GLITTERS IS NOT GOLD.
PARTITION CONTAINERS.
AUTOMATE. AUTOMATE. AUTOMATE.

25. Isaac Christoffersen
Colloquium
Contributing Partner, Open Source Solutions
email: isaac.christoffersen@colloquium.co
twitter: @colloquiumco
web: http://www.colloquium.co
twitter: @colloquiumco
identica: @colloquiumco