Kim IT Pro Forum Eugene: IT at Ludicrous Speeds - rugged dev ops

IT At Ludicrous Speeds:
Rugged DevOps and More……

Gene Kim
Author, Visible Ops Handbook
IT Pro Forum Eugene
July 17, 2012

Session ID:

@RealGeneKim, genek@realgenekim.me

Where Did The High Performers Come From?


Visible Ops: Playbook of High Performers

 The IT Process Institute has
been studying high-performing
organizations since 1999
 What is common to all the high
performers?
 What is different between them
and average and low
performers?
 How did they become great?
 Answers have been codified in
the Visible Ops Methodology

www.ITPI.org

DevOps:
Engage Ludicrous Speed!


Source: John Allspaw

Source: Theo Schlossnagle

Source: John Jenkins, Amazon.com

Ludicrous Speed!

16

Ludicrous Fail?!

17

Source: James Wickett


Why DevOps Is So Important To Me


Since 1999, We’ve Benchmarked 1500+
IT Organizations

Source: EMA (2009)
Source: IT Process Institute (2008)

High Performing IT Organizations
 High performers maintain a posture of compliance
 Fewest number of repeat audit findings
 One-third amount of audit preparation effort
 High performers find and fix security breaches faster
 5 times more likely to detect breaches by automated control
 5 times less likely to have breaches result in a loss event
 When high performers implement changes…
 14 times more changes
 One-half the change failure rate
 One-quarter the first fix failure rate
 10x faster MTTR for Sev 1 outages
 When high performers manage IT resources…
 One-third the amount of unplanned work
 8 times more projects and IT services
 6 times more applications

Source: IT Process Institute, 2008


2007: Three Controls Predict 60% Of
Performance

 To what extent does an organization
define, monitor and enforce the following?
 Standardized configuration strategy
 Process discipline
 Controlled access to production systems

Source: IT Process Institute, 2008

Tough Love From Ari Balogh


The Downward Spiral
Operations Sees… Dev Sees…
 Too many fragile and insecure  More urgent, date-driven projects
applications in production put into the queue
 Too much time required to restore  Even more fragile code (less
service secure) put into production
 Too much firefighting and unplanned  More releases have increasingly
work “turbulent installs”
 Planned project work cannot complete  Release cycles lengthen to
amortize “cost of deployments”
 Frustrated customers leave
 Bigger deployment failures
 Market share goes down
 More time spent on firefighting
 Business misses Wall Street
commitments  Ever increasing backlog of work
that cold help the business win
 Business makes even larger promises
to Wall Street  Ever increasing amount of
tension between IT
Ops, Development, Design…

These aren’t ITSM or IT Operations problems…
These are business problems!

My Mission: Figure Out How Break The IT Core
Chronic Conflict

 Every IT organization is pressured to
simultaneously:
 Respond more quickly to urgent business needs
 Provide stable, secure and predictable IT service

Words often used to describe process improvement:
“hysterical, irrelevant, bureaucratic, bottleneck, difficult to understand, not
aligned with the business, immature, shrill, perpetually focused on irrelevant
technical minutiae…”

Source: The authors acknowledge Dr. Eliyahu Goldratt, creator of the Theory of Constraints and
author of The Goal, has written extensively on the theory and practice of identifying and resolving
core, chronic conflicts.
26

Good News: It Can Be Done

Bad News: You Can’t Do It Alone


Ops


QA And Test

Source: Flickr: vandyll

Development


Process And Controls


Product Management And Design

Source: Flickr: birdsandanchors

DevOps: It’s A Real Movement
 I would never do another startup that didn’t
employ DevOps like principles
 It’s not just startups – it’s happening in the
enterprise and in public sector, too
 I believe working in DevOps environments will
be a necessary skillset 5 years from now
 Agile helped Dev regain trust with the business;
DevOps will help all of IT
 IT becoming more automated relies on DevOps
practices (especially PaaS)


If I Could Wave A Magic Wand, Everyone Will…

 Become conversant with DevOps and recognize
the practices when you see them
 Be energized about how ITSM practitioners can
contribute in this organizational journey
 Leave with some concrete steps to get some
great outcomes
 Become a part of a team that starts putting
DevOps practices into place

34

How Do You Do
DevOps?

35

The Prescriptive DevOps Cookbook

 “DevOps Cookbook” Authors
 Patrick DeBois, Mike Orzen,
John Willis

 Goals
 Codify how to start and finish
DevOps transformations
 How does Development, IT
Operations and Infosec
become dependable partners
 Describe in detail how to
replicate the transformations
describe in “When IT Fails: The
Novel”


“The Goal” by Dr. Eliyahu Goldratt


38

39

The First Way:
Systems Thinking


The First Way:
Systems Thinking

(Business) (Customer)


The First Way:
Systems Thinking (Left To Right)

 Don’t pass defects downstream
 Don’t optimize locally
 Always increase flow: elevate
bottlenecks, reduce WIP, throttle release of
work, reduce batch sizes
 Understanding where reliance is placed


Phase 1: Extend the Agile CI/CR Processes
 Create one-step Dev, Test and Production
environment creation procedure in Sprint 0
 Create the one-step automated code
deployment procedure
 Properly integrate release, configuration and
change into the value stream (as well as QA and
infosec)
 Ensure developers don’t leave until production
change is successful
 Assign Ops person into Dev team


Definition: Kanban Board
 Signaling tool to reduce WIP and increase flow

44

The First Way:
Systems Thinking: ITSM Insurgency
 Have someone attend the daily Agile standups
 Gain awareness of what the team is working on
 Find the automated infrastructure project team (e.g.,
puppet, chef)
 Release managers can provide hardening guidance
 Integrate and extend their production configuration monitoring
 Find where code packaging is performed
 Integrate security testing pre- and post-deployment
 Integrate testing into continuous integration and release
process
 Add security test scripts to automated test library
 Define what changes/deploys cannot be made without
triggering full retest


The First Way:
Outcomes
 Determinism in the release process
 Creating single repository for code and environments
 Consistent Dev, QA, Int, and Staging environments, all
properly built before deployment begins
 Decreased cycle time
 Reduce deployment times from 6 hours to 45 minutes
 Refactor deployment process that had 1300+ steps
spanning 4 weeks
 Faster release cadence


The Second Way:
Amplify Feedback Loops


The Second Way:
Amplify Feedback Loops (Right to Left)

 Expose visual data so everyone can see how
their decisions affect the entire system
 Get Development closer to Operations and
customers
 Create a reliable system system of work that
improves itself


Phase 2: Extend Release Process And Create
Right -> Left Feedback Loops

 Embed Dev into Ops escalation process
 Invite Dev to post-mortems/root cause analysis
meeting
 Have Dev cross-train IT Operations
 Ensure application monitoring/metrics to aid in
Ops and Infosec work (e.g., incident/problem
management


The Second Way:
Amplify Feedback Loops: ITSM Insurgency

 Find areas in the incident and problem
management processes where Development
knowledge could help
 Ensure that countermeasures are captured in
the Agile backlog
 Find that developer who really cares about the
production environment


The Second Way:
Outcomes

 Defects and security issues getting fixed faster
than ever
 Reusable Ops and Infosec user stories now part
of the Agile process
 All groups communicating and coordinating
better
 Everybody is getting more work done


The Third Way:
Culture Of Continual Experimentation And
Learning


The Third Way:
Learning

 Foster a culture that rewards:
 Experimentation (taking risks) and learning from
failure
 Repetition is the prerequisite to mastery
 Why?
 You need a culture that keeps pushing into the danger
zone
 And have the habits that enable you to survive in the
danger zone


Help IT Operations…

 “The best way to avoid failure is
to fail constantly”
 Harden the production
environment
 Have scheduled drills to “crash
the data center”
 Create your “chaos monkeys” to
introduce faults into the system
(e.g., randomly kill processes,
take out servers, etc.)
 Rehearse and improve
responding to unplanned work
 NetFlix: Hardened AWS service

 StackOverflow
 Amazon firedrills (Jesse Allspaw)
 The Monkey (Mac)


You Don’t Choose Chaos Monkey…
Chaos Monkey Chooses You


Help Product Management…

Lesson: Allocate 20% of Dev cycles to paying down
technical debt

Phase 3: Organize Dev and Ops To Achieve
Organizational Goals

 Allocate 20% of Dev cycles to non-functional
requirements
 Integrate fault injection and resilience into
design, development and production (e.g.,
Chaos Monkey)


The Third Way:
Learning: ITSM

 Ensure that process improvement projects are in
the Agile backlog
 Make technical debt visible
 Help prioritize work against features and other non-functional
requirements
 Release your Chaos Monkey
 Rehearse cleaning up after the Chaos Monkey
 Find processes that waste everyone’s time


The Third Way:
Outcomes
 Technical debt is being paid off
 Exploitable attack surface area decreases
 Continual reduction of unplanned work
 More cycles for planned work
 More resilient code and environments
 Balancing nimbleness and practiced repetition
 Enabling wider range of risk/reward balance


What Does Transformation Feel Like?

63

Find What’s Most Important First


Quickly Find What Is Different…


Before Something Bad Happens…


Find Risk Early…


Communicate It Effectively To Peers…


Hold People Accountable…


Based On Objective Evidence…


Answer Important Questions…


Recognize Compounding Technical Debt…


That Gets Worse…


And Fixing It…

Source: Pingdom

Have What We Need, When When We Need
It…


Big Things Get Done Quickly…


Ever Increasing Situational Mastery…


Help The Business Win…


With Support From Your Peers…


And Do More With Less Effort…


This Is An Important Problem
Operations Sees… Dev Sees…
 Fragile applications are prone to  More urgent, date-driven projects
failure put into the queue
 Long time required to figure out “which  Even more fragile code (less
bit got flipped” secure) put into production
 Detective control is a salesperson  More releases have increasingly
“turbulent installs”
 Too much time required to restore
service  Release cycles lengthen to
amortize “cost of deployments”
 Too much firefighting and unplanned
work  Failing bigger deployments more
difficult to diagnose
 Urgent security rework and
remediation  Most senior and constrained IT
ops resources have less time to
 Planned project work cannot complete fix underlying process problems
 Frustrated customers leave  Ever increasing backlog of work
 Market share goes down that cold help the business win
 Business misses Wall Street  Ever increasing amount of
commitments tension between IT Ops,
Development, Design…
 Business makes even larger promises
to Wall Street


82

If I Could Wave A Magic Wand, Everyone Will…

 Become conversant with DevOps and recognize
the practices when you see them
 Be energized about how ITSM practitioners can
contribute in this organizational journey
 Leave with some concrete steps to get some
great outcomes
 Become a part of a team that starts putting
DevOps practices into place

…And fill out the survey
forms! 84

When IT Fails: The Novel and The DevOps
Cookbook

 Coming in Fall 2012

 “In the tradition of the best MBA case studies, this
book should be mandatory reading for business
and IT graduates alike.”
Paul Muller, VP Software Marketing, Hewlett-
Packard

Gene Kim, Tripwire founder,
 “The greatest IT management book of our
Visible Ops co-author generation.”
Branden Williams, CTO Marketing, RSA


When IT Fails: The Novel and The DevOps
Cookbook

 Our mission is to positively affect the
lives of 1 million IT workers by 2017

 If you would like the “Top 10 Things
Infosec Needs To Know About DevOps,”
sample chapters and updates on the
book:

Gene Kim, Tripwire founder,
Visible Ops co-author
 Sign up at http://itrevolution.com
 Email genek@realgenekim.me
 Hand me a business card


If you’d like the slides from today’s
presentation…

 Text your first name, email
address and “68383” to:
+1 (858) 598-3980

 Or visit:
http://www.instantcustomer.c
om/go/68383

 Or scan this QR Code:

87

Kim IT Pro Forum Eugene: IT at Ludicrous Speeds - rugged dev ops

More Related Content

What's hot

Viewers also liked

Similar to Kim IT Pro Forum Eugene: IT at Ludicrous Speeds - rugged dev ops

Recently uploaded

Kim IT Pro Forum Eugene: IT at Ludicrous Speeds - rugged dev ops

Editor's Notes