AGILE INFRASTRUCTURE
ANDREW SHAFER - PAUL NASRAT

ANDREW CLAY
SHAFER

ANDREW CLAY
SHAFER
DEVELOPER, ONCE UPON A TIME

ANDREW CLAY
SHAFER
DEVELOPER, ONCE UPON A TIME
CO-FOUNDER
REDUCTIVE LABS INC.

ANDREW CLAY
SHAFER
DEVELOPER, ONCE UPON A TIME
CO-FOUNDER
REDUCTIVE LABS INC.
ALL AROUND TROUBLE MAKER

ANDREW CLAY
SHAFER
DEVELOPER, ONCE UPON A TIME
CO-FOUNDER
REDUCTIVE LABS INC.
ALL AROUND TROUBLE MAKER
THE REST IS COMPLICATED...

AGILE
INFLUENCES
Alistair Cockburn
Brian Marick
Israel Gat
Zhon Johansen
Kay Johansen
Salt Lake Agile Roundtable

INFRASTRUCTURE
INFLUENCES
Luke Kanies
Teyo Tyree
James Turnbull
Paul Lathrop
John Alspaw
Adam Jacob
Puppet Community

PAUL
NASRAT

PAUL
NASRAT
GUARDIAN OPERATIONS LEAD

PAUL
NASRAT
GUARDIAN OPERATIONS LEAD
THOUGHTWORKER

PAUL
NASRAT
GUARDIAN OPERATIONS LEAD
THOUGHTWORKER
XTREME TUESDAY CLUB

PAUL
NASRAT
GUARDIAN OPERATIONS LEAD
THOUGHTWORKER
XTREME TUESDAY CLUB
INFRASTRUCTURE DEVELOPER

WHY DOES IT MATTER?

WHY DOES IT MATTER?
WEB APPLICATIONS ARE
THE INFRASTRUCTURE

WHY DOES IT MATTER?
WEB APPLICATIONS ARE
THE INFRASTRUCTURE
OPERATIONS IS THE
SPECIAL SAUCE

DISCLAIMERS

DISCLAIMERS
WEB OPERATIONS BIAS

DISCLAIMERS
WEB OPERATIONS BIAS
*NIX BIAS

DISCLAIMERS
WEB OPERATIONS BIAS
*NIX BIAS
OPEN SOURCE BIAS

DAY BREAK

DAY BREAK
APPLIED PRINCIPLES

DAY BREAK
APPLIED PRINCIPLES
ENABLE CHANGE

DAY BREAK
APPLIED PRINCIPLES
ENABLE CHANGE
ENABLE COMMUNICATION

DAY BREAK
APPLIED PRINCIPLES
ENABLE CHANGE
ENABLE COMMUNICATION
ENABLE FEEDBACK

DAY BREAK
APPLIED PRINCIPLES
ENABLE CHANGE
ENABLE COMMUNICATION
ENABLE FEEDBACK
ENABLE CHOICES

YOU ARE A
SPECIAL
SNOWFLAKE

JUST LIKE EVERYBODY
EVERYTHING THE SAME,
EVERYTHING DISTINCT

PROBLEMS

CONFIGURATION
DRIFT

CHANGE CAUSES OUTAGES
INTERNAL AND EXTERNAL

DEVELOPERS OPERATIONS

DEVELOPERS OPERATIONS

DEVELOPERS OPERATIONS

DEVELOPERS OPERATIONS

DEVELOPERS OPERATIONS

HERO CULTURE
RUNNING ON ADRENALINE
REACTIVE FIREFIGHTING
PURELY TACTICAL
CONTRIBUTES TO CONFIG DRIFT

DON’T NEED ANOTHER HERO...

NON-FUNCTIONAL
REQUIREMENTS

DIFFERENT ENVIRONMENTS

THE MYSTERY MACHINE
THE ONE IN THE CORNER THAN EVERYONE IS AFRAID TO
TURN OFF, BUT NO ONE KNOWS WHY IT IS ON.

DONE, DONE, DONE
DONE IS DEPLOYED

INFRASTRUCTURE IS CODE

INFRASTRUCTURE IS CODE
API DRIVEN
ABSTRACTIONS

INFRASTRUCTURE IS CODE
API DRIVEN
ABSTRACTIONS
THE INFRASTRUCTURE
IS AN APPLICATION

WHAT IS
AGILE?

WHAT IS
AGILE?
PLANNING

WHAT IS
AGILE?
PLANNING ENGINEERING

WHAT IS
AGILE?
PLANNING ENGINEERING
DEVELOPERS

WHAT IS
AGILE?
PLANNING ENGINEERING
DEVELOPERS
PRODUCT
OWNERS

WHAT IS
AGILE?
PLANNING ENGINEERING
DEVELOPERS
PRODUCT
OWNERS
TESTERS

WHAT IS
AGILE?
PLANNING ENGINEERING
CIRCLE OF HAPPINESS
DEVELOPERS
PRODUCT
OWNERS
TESTERS

WHAT IS
AGILE?
PLANNING ENGINEERING
CIRCLE OF HAPPINESS
DEVELOPERS EXECUTIVES
PRODUCT
OWNERS
TESTERS

WHAT IS
AGILE?
PLANNING ENGINEERING
CIRCLE OF HAPPINESS
DEVELOPERS EXECUTIVES
SYSTEM ADMINISTRATORS
PRODUCT
OWNERS
TESTERS

WHAT IS
AGILE?
PLANNING ENGINEERING
CIRCLE OF HAPPINESS
DEVELOPERS EXECUTIVES
SYSTEM ADMINISTRATORS
PRODUCT
OWNERS DATABASE ADMINISTRATORS
TESTERS

WHAT IS
AGILE?
PLANNING ENGINEERING
CIRCLE OF HAPPINESS
DEVELOPERS EXECUTIVES
SYSTEM ADMINISTRATORS
PRODUCT
OWNERS DATABASE ADMINISTRATORS
NETWORK ENGINEERS
TESTERS

WHAT IS
AGILE?
PLANNING ENGINEERING
CIRCLE OF HAPPINESS
DEVELOPERS EXECUTIVES
SYSTEM ADMINISTRATORS
PRODUCT
OWNERS DATABASE ADMINISTRATORS
NETWORK ENGINEERS
TESTERS
DESIGNERS

WHAT IS
AGILE?
PLANNING ENGINEERING
CIRCLE OF HAPPINESS
DEVELOPERS EXECUTIVES
SYSTEM ADMINISTRATORS
PRODUCT
OWNERS DATABASE ADMINISTRATORS
NETWORK ENGINEERS
TESTERS
DESIGNERS
USABILITY EXPERTS

WHAT IS
AGILE?
PLANNING ENGINEERING
CIRCLE OF HAPPINESS
DEVELOPERS EXECUTIVES
SYSTEM ADMINISTRATORS
PRODUCT
OWNERS DATABASE ADMINISTRATORS
NETWORK ENGINEERS
TESTERS
DESIGNERS
USABILITY EXPERTS

AGILE INFRASTRUCTURE

AGILE INFRASTRUCTURE
INFRASTRUCTURE IS CODE: APPLY
THE LESSONS LEARNED FROM
SOFTWARE DEVELOPMENT

AGILE INFRASTRUCTURE
INFRASTRUCTURE IS CODE: APPLY
THE LESSONS LEARNED FROM
SOFTWARE DEVELOPMENT
LEVERAGE THE INFRASTRUCTURE
RENAISSANCE TO BUILD MORE
FLEXIBLE SYSTEMS AND FASTER
FEEDBACK

AGILE INFRASTRUCTURE
INFRASTRUCTURE IS CODE: APPLY
THE LESSONS LEARNED FROM
SOFTWARE DEVELOPMENT
LEVERAGE THE INFRASTRUCTURE
RENAISSANCE TO BUILD MORE
FLEXIBLE SYSTEMS AND FASTER
FEEDBACK
TAKE ADVANTAGE OF MORE
PERSONAS AND IDEAS TO DELIVER
VALUE STREAM

TECHNIQUES

VERSION CONTROL

VERSION

VERSION
NETWORK CONFIGURATIONS

VERSION
NETWORK CONFIGURATIONS
SYSTEM CONFIGURATIONS

VERSION
NETWORK CONFIGURATIONS
SYSTEM CONFIGURATIONS
APPLICATIONS CONFIGURATIONS

VERSION
NETWORK CONFIGURATIONS
SYSTEM CONFIGURATIONS
APPLICATIONS CONFIGURATIONS
APPLICATION CODE

VERSION
NETWORK CONFIGURATIONS
SYSTEM CONFIGURATIONS
APPLICATIONS CONFIGURATIONS
APPLICATION CODE
DATABASE SCHEMA

VERSION
NETWORK CONFIGURATIONS
SYSTEM CONFIGURATIONS
APPLICATIONS CONFIGURATIONS
APPLICATION CODE
DATABASE SCHEMA
DOCUMENTATION

VERSION
NETWORK CONFIGURATIONS
SYSTEM CONFIGURATIONS
APPLICATIONS CONFIGURATIONS
APPLICATION CODE
DATABASE SCHEMA
DOCUMENTATION
ANYTHING THAT MATTERS

VERSION EVERYTHING

VERSION EVERYTHING
SERIOUSLY...

VERSION EVERYTHING
SERIOUSLY...
EVERYTHING

VERSION EVERYTHING
SERIOUSLY...
EVERYTHING
THE TOOL DOESN’T MATTER
(BUT YOU SHOULD USE GIT)

CONFIGURATION MANAGEMENT

CONFIGURATION MANAGEMENT

CONFIGURATION MANAGEMENT
PUT SYSTEMS INTO A KNOWN STATE

CONFIGURATION MANAGEMENT
PUT SYSTEMS INTO A KNOWN STATE
AUDIT AND ENFORCE CONSISTENCY

CONFIGURATION MANAGEMENT
PUT SYSTEMS INTO A KNOWN STATE
AUDIT AND ENFORCE CONSISTENCY
MANAGE SERVER LIFECYCLE

CONFIGURATION MANAGEMENT
PUT SYSTEMS INTO A KNOWN STATE
AUDIT AND ENFORCE CONSISTENCY
MANAGE SERVER LIFECYCLE
REASON ABOUT SERVICES, INSTEAD OF SYSTEMS

CONFIGURATION MANAGEMENT
PUT SYSTEMS INTO A KNOWN STATE
AUDIT AND ENFORCE CONSISTENCY
MANAGE SERVER LIFECYCLE
REASON ABOUT SERVICES, INSTEAD OF SYSTEMS
APPLY DEV-TEST-PROD CYCLE TO INFRASTRUCTURE

CONFIGURATION MANAGEMENT
ISCONF CHEF
RADMIND

BUILD FROM SOURCE
BARE METAL TO RUNNING SERVICES

BUILD FROM SOURCE

BUILD FROM SOURCE
AUTOMATED PROVISIONING AND DEPLOYMENT OF
SERVICES

BUILD FROM SOURCE
AUTOMATED PROVISIONING AND DEPLOYMENT OF
SERVICES
ROLL CONFIG CHANGES FORWARD (DEV-TEST-PROD)

BUILD FROM SOURCE
AUTOMATED PROVISIONING AND DEPLOYMENT OF
SERVICES
ROLL CONFIG CHANGES FORWARD (DEV-TEST-PROD)
DEV, TEST AND PROD NOT OUT OF SYNC

BUILD FROM SOURCE
AUTOMATED PROVISIONING AND DEPLOYMENT OF
SERVICES
ROLL CONFIG CHANGES FORWARD (DEV-TEST-PROD)
DEV, TEST AND PROD NOT OUT OF SYNC
TEST FROM A KNOWN STATE

BUILD FROM SOURCE
AUTOMATED PROVISIONING AND DEPLOYMENT OF
SERVICES
ROLL CONFIG CHANGES FORWARD (DEV-TEST-PROD)
DEV, TEST AND PROD NOT OUT OF SYNC
TEST FROM A KNOWN STATE
SCALING

BUILD FROM SOURCE
AUTOMATED PROVISIONING AND DEPLOYMENT OF
SERVICES
ROLL CONFIG CHANGES FORWARD (DEV-TEST-PROD)
DEV, TEST AND PROD NOT OUT OF SYNC
TEST FROM A KNOWN STATE
SCALING
DISASTER RECOVERY

ONE STEP DEPLOY

ONE STEP DEPLOY
ONE AUTOMATED PROCESS
FROM VERSION CONTROL TO
LIVE SERVICES.

ONE STEP DEPLOY
ONE AUTOMATED PROCESS
FROM VERSION CONTROL TO
LIVE SERVICES.
COMPUTERS ARE REALLY
GOOD AT RUNNING THE SAME
COMMANDS OVER AND OVER.

ONE STEP DEPLOY
ONE AUTOMATED PROCESS
FROM VERSION CONTROL TO
LIVE SERVICES.
COMPUTERS ARE REALLY
GOOD AT RUNNING THE SAME
COMMANDS OVER AND OVER.
LOWER THE FIXED COST OF
DEPLOY.

MONITORING

MONITORING
WHAT DOES 'NORMAL'
LOOKS LIKE?

MONITORING
WHAT DOES 'NORMAL'
LOOKS LIKE?
DON’T JUST LOOK AT
THE DATA WHEN
THINGS ARE BAD

MONITORING
WHAT DOES 'NORMAL'
LOOKS LIKE?
DON’T JUST LOOK AT
THE DATA WHEN
THINGS ARE BAD
NEED BASELINE,
CHART, TRENDS

CONTINUOUS INTEGRATION

CONTINUOUS INTEGRATION
TEST NEW BUILDS

CONTINUOUS INTEGRATION
TEST NEW BUILDS
ASSERT SERVICES
ARE RUNNING

CONTINUOUS INTEGRATION
TEST NEW BUILDS
ASSERT SERVICES
ARE RUNNING
RUN FUNCTIONAL
TESTS

DEPLOY EARLY AND OFTEN

TAG EVERYTHING - WHO? WHAT? WHEN?

SYNCHRONIZATION
GET ALL MACHINES SYNC’D

CORRELATE
VISUALIZE ON THE SAME TIMELINE
TAGGED WITH WHO? WHAT? WHEN?

INFORMATION RADIATORS
SHARE METRICS
DEV AND OPS SEE THE SAME THING, IN THE SAME PLACE

SHARE THE REPOSITORY

SHARE THE REPOSITORY
KEEP CONFIGS IN SYNC WITH
APPLICATION CODE

SHARE THE REPOSITORY
KEEP CONFIGS IN SYNC WITH
APPLICATION CODE
EVERYONE KNOWS WHERE
TO LOOK

SHARE THE REPOSITORY
KEEP CONFIGS IN SYNC WITH
APPLICATION CODE
EVERYONE KNOWS WHERE
TO LOOK
EVERYONE SEES EVERYONE
ELSE WORKING

SHARE THE REPOSITORY
KEEP CONFIGS IN SYNC WITH
APPLICATION CODE
EVERYONE KNOWS WHERE
TO LOOK
EVERYONE SEES EVERYONE
ELSE WORKING
MINIMIZE SURPRISE

SHARE THE REPOSITORY
KEEP CONFIGS IN SYNC WITH
APPLICATION CODE
EVERYONE KNOWS WHERE
TO LOOK
EVERYONE SEES EVERYONE
ELSE WORKING
MINIMIZE SURPRISE
BOUNDARY OBJECT

ALWAYS SHIP TRUNK
EVERYONE ALWAYS KNOWS WHICH BRANCH IS LIVE

BRANCH FEATURES IN APPLICATION CODE

DARK LAUNCH

DECOUPLE SERVICES
ISOLATE THINGS THAT CHANGE FREQUENTLY
FROM THINGS THAT DON'T

CACHING
DON'T GENERATE THE SAME CONTENT TWICE
(DRY COMPUTATION)

FAIL HAPPENS

FAIL HAPPENS

FAIL HAPPENS
CAN YOU AFFORD TO BE
DOWN?

FAIL HAPPENS
CAN YOU AFFORD TO BE
DOWN?
HOW LONG?

FAIL HAPPENS
CAN YOU AFFORD TO BE
DOWN?
HOW LONG?
HOW FAST CAN YOU BE
BACK UP?

FAIL HAPPENS
CAN YOU AFFORD TO BE
DOWN?
HOW LONG?
HOW FAST CAN YOU BE
BACK UP?
TRY NOT TO CAUSE IT

FAIL SAFE
PRACTICE MAKES PERFECT

OUT THE
WINDOW

FIRE DRILLS

WORKING EFFECTIVELY
WITH LEGACY SYSTEMS

CULTURE
THERE IS ONLY US

LEARNING AND RESPECT
CULTIVATION CULTURE

WORK TOGETHER

WORK TOGETHER
DEVS NEED TO
UNDERSTAND THE
INFRASTRUCTURE

WORK TOGETHER
DEVS NEED TO
UNDERSTAND THE
INFRASTRUCTURE
OPS NEED TO
UNDERSTAND THE
APPLICATION

WORK TOGETHER
DEVS NEED TO
UNDERSTAND THE
INFRASTRUCTURE
OPS NEED TO
UNDERSTAND THE
APPLICATION
EVERYONE HAS
SOMETHING TO
CONTRIBUTE

PLANNING FOR FIRES IS HARD

PLANNING FOR FIRES IS HARD
DEAR DIARY,

PLANNING FOR FIRES IS HARD
DEAR DIARY,
TODAY I WAS ON FIRE FOR 12
HOURS...

PLANNING FOR FIRES IS HARD
DEAR DIARY,
TODAY I WAS ON FIRE FOR 12
HOURS...
IT WASN’T AS PLEASANT AS IT
SOUNDS.

PLANNING FOR FIRES IS HARD
DEAR DIARY,
TODAY I WAS ON FIRE FOR 12
HOURS...
IT WASN’T AS PLEASANT AS IT
SOUNDS.
--EBEN BRINSON SMITH III

MANAGE FLOW
THE BEST WAY TO FIGHT FIRES IS
NEVER LET THEM GET STARTED

SHARED METAPHORS

CONCLUSIONS

CONCLUSIONS
LEARN FROM EVERYONE YOU CAN

CONCLUSIONS
LEARN FROM EVERYONE YOU CAN
NEW TOOLS AND IDEAS WON'T SOLVE ALL THE
PROBLEMS... BUT THEY CAN SOLVE A LOT.

CONCLUSIONS
LEARN FROM EVERYONE YOU CAN
NEW TOOLS AND IDEAS WON'T SOLVE ALL THE
PROBLEMS... BUT THEY CAN SOLVE A LOT.
THE SOLUTIONS TO SOME PROBLEMS REQUIRES SOCIAL
ENGINEERING AS MUCH AS TECHNICAL

CONCLUSIONS
LEARN FROM EVERYONE YOU CAN
NEW TOOLS AND IDEAS WON'T SOLVE ALL THE
PROBLEMS... BUT THEY CAN SOLVE A LOT.
THE SOLUTIONS TO SOME PROBLEMS REQUIRES SOCIAL
ENGINEERING AS MUCH AS TECHNICAL
EVERYONE’S JOB IS TO ENABLE BUSINESS VALUE

CONCLUSIONS
LEARN FROM EVERYONE YOU CAN
NEW TOOLS AND IDEAS WON'T SOLVE ALL THE
PROBLEMS... BUT THEY CAN SOLVE A LOT.
THE SOLUTIONS TO SOME PROBLEMS REQUIRES SOCIAL
ENGINEERING AS MUCH AS TECHNICAL
EVERYONE’S JOB IS TO ENABLE BUSINESS VALUE
THERE IS ONLY US

WE ARE UNCOVERING BETTER WAYS OF DEVELOPING
SOFTWARE BY DOING IT AND HELPING OTHERS DO IT.

WE ARE UNCOVERING BETTER WAYS OF DEVELOPING
SOFTWARE BY DOING IT AND HELPING OTHERS DO IT.

DELIVERING
WE ARE UNCOVERING BETTER WAYS OF DEVELOPING
SOFTWARE BY DOING IT AND HELPING OTHERS DO IT.

KEEP UNCOVERING
KEEP HELPING PEOPLE

RESOURCES
AGILE SYSADMIN GROUP
HTTP://GROUPS.GOOGLE.COM/GROUP/AGILE-SYSTEM-ADMINISTRATION/
PUPPET USERS
HTTP://GROUPS.GOOGLE.COM/GROUP/PUPPET-USERS/
VELOCITY CONF
HTTP://EN.OREILLY.COM/VELOCITY2009
PATRICK DEBOIS
HTTP://WWW.DEVOPSDAYS.ORG/
JOHN ALLSPAW AND PAUL HAMMOND
HTTP://BLIP.TV/FILE/2284377

QUESTIONS?
ANDREW@REDUCTIVELABS.COM
PAUL.NASRAT@GUARDIAN.CO.UK