Limited WIP Meeting presentation - The Phoenix Project book review
The Phoenix Project: A Novel About
IT, DevOps, and Helping Your
How many have read “The Phoenix Project”?
What is the cycle time in your org from code to
How many of you have Ops represented in Dev
How long has Ops people been embedded in Dev
Who is here today?
Why are you here?
Interest in book / event / Limited WIP
What do you want to get out of tonight?
Get more people to read “The Phoenix Project”
Provide some thinking tools and references
Make a difference to more people
Speed up your learning and adoption
Learn from you and your experiences
Why is this important?
High Performing DevOps Teams
They’re more agile
30x more frequent deployments
8,000x faster cycle time than their peers
They’re more reliable
2x the change success rate
12x faster MTTR
MTTR = Mean time to Restore/Repair/recovery
Source: Puppet Labs 2012 State Of DevOps: http://puppetlabs.com/2013-state-of-devops-infographic
Top 100 Books
What does Gene Kim Say about the
http://www.youtube.com/watch?v=FpEd-QKVIrw 5 min
The Phoenix Project - The Authors
Gene Kim is a multiple award-winning entrepreneur, the founder
and former CTO of Tripwire and a researcher. He is passionate
about IT operations, security and compliance, and how IT
organizations successfully transform from “good to great”.
George is an author and speaker, consulting and conducting
training on strategy, IT management, information security and
overall service improvement in the U.S., Canada, Australia,
New Zealand and China. Co-author of “The Visible Ops
Handbook” and “Visible Ops Security,” George is a certified
ITIL Expert, TOCICO Jonah and a Certified Information
Systems Auditor (CISA).
Kevin Behr is the founder of the Information Technology
Process Institute (ITPI) and the CTO of Assemblage Pointe.
Kevin has twenty years of IT management experience and is a
mentor and advisor to Chief Executive Officers and Chief
Bill is an IT manager at Parts Unlimited. It’s Tuesday morning and on his drive into
the office, Bill gets a call from the CEO.
The company’s new IT initiative, code named Phoenix Project, is critical to the
future of Parts Unlimited, but the project is massively over budget and very late.
The CEO wants Bill to report directly to him and fix the mess in ninety days or else
Bill’s entire department will be outsourced.
With the help of a prospective board member and his mysterious philosophy of The
Three Ways, Bill starts to see that IT work has more in common with manufacturing
plant work than he ever imagined. With the clock ticking, Bill must organize work
flow streamline interdepartmental communications, and effectively serve the other
business functions at Parts Unlimited.
In a fast-paced and entertaining style, three luminaries of the DevOps movement
deliver a story that anyone who works in IT will recognize. Readers will not only
learn how to improve their own IT organizations, they’ll never view IT the same way
Cast at Parts Unlimited
Parts Unlimited: Business Executives
Dick Landry, CFO
Sarah Moulton, SVP of Retail Operations
Maggie Lee, Senior Director of Retail Program Management
Bill Palmer, VP of IT Operations , former Director of Midrange Technology Operations
Wes Davis, Director of Distributed Technology Operations
Brent Geller, Lead Engineer
Patty McKee, Director of IT Service Support
John Pesche, Chief Information Security Officer (CISO)
Steve Masters, CEO, acting CIO
Chris Allers, VP of Application Development
Parts Unlimited: Board
Bob Strauss, Lead Director, former Chairman, former CEO
Erik Reid, Board Candidate
Nancy Mailer, Chief Audit Executive
Why a novel?
Who has read “The Goal”?
Is The Most Effective Means Of
Creating A Shared Understanding
The pattern is everywhere:
The core of the solution-selling,
good copy-writing and the best TED talks: Describe the problem, show
its significance, describe how the problem is solved and the value it
Storytelling becomes very important when the problem we’re trying to
point out is not fully recognized, or when the solution being proposed
flies in the face of common wisdom.
How to create a “Problem”?
The ERM framework by the Commission of Sponsoring Organizations of the Treadway
Commission (COSO) provides a more disciplined and consistent standard against which to
implement and assess a company’s ERM program.
Strategy: as embodied by the Phoenix Project,
which the entire future of the company depends
upon. It requires that IT be a core competency.
Accurate financial reporting: as embodied by
the third year repeat audit findings around the
IT general controls, but also the loss of
accounts payable and inventory management
systems which prevents the closing the
financial books at the end of quarter. Even the
payroll failure resulted in inaccurate payroll
numbers, which led to financial reporting errors.
Compliance with laws and regulations (e.g.,
SOX-404, PCI DSS): failing the SOX-404 audit
results in an adverse footnotes by the external
auditor in the SEC 10-K statement, and there
are grave implications for failing the PCI DSS
Operations (e.g., IT application and project
delivery, IT operations, information security)
The Project Phoenix was $20 million overbudget and three years late, and when it finally
was deployed into production, everyone would
have been better off it hadn’t. Furthermore,
virtually all of the critical business systems that
run daily operations require that IT services be
running correctly, which they often weren’t.
Generic problems – Downward Spirals
Fragile artefacts become more fragile
Technical debt grows
Date-driven application projects focus only on features, sacrificing
non-functional requirements, which results in more fragile artifacts in
Application deployment take longer, more difficult, getting gets worse
IT Ops is stuck fire fighting and therefore cannot do preventive work
or new projects
Long feature delivery cycle times result in more political decision
making, meaning more focus on features (vs. non-functional
Breakthrough 1 - 3
Breakthrough 1: discovering too much WIP in IT
Operations and gob-smacking amounts of reliance
on Brent for both project and recovery work
Breakthrough 2: elevating preventive work to
prevent unplanned work (especially for Brent) and
making work visible
Breakthrough 3: throttling the flow of work into IT
Operations by the project freeze (subordinate)
Breakthrough 4 - 6
Breakthrough 4: identifying and limiting the flows of work to
Brent (it was genuinely surprising to find out after the fact at
how much this resembles the green/red tag pattern used in
“The Goal”) Kanban Board
Breakthrough 5: documenting and defining standardized work,
and managing handoffs to reduce time spent in queue
(through use of kanbans), and further reducing reliance on
Breakthrough 6: correctly identifying reliance on IT systems to
hit Dick’s corporate objectives (via Gartner RVM model) and
correctly scoping audit and infosec (via GAIT and GAIT-R)
Breakthrough 7 & 8
Breakthrough 7: building DevOps flow of work to better
design in non-functional requirements in Dev, reduce batch
sizes and enable single-piece flow through Development and
IT Operations and better enable operational resilience
(replicating the work described by Jez Humble, David Farley,
Paul Hammond and John Allspaw)
Breakthrough 8: when they discover the constraint moves
outside the organization, they bring back those resources inhouse (replicating one of my favourite Theory of Constraints
plays: when there’s idle plant capacity, it’s often cheaper to
fabricate parts in-house than to pay a supplier. Lovely.)
4 types of work
Business project work,
IT project work,
Thee Three Ways
Way No. 1: The flow
Way No. 2: Consistent feedback
Way No. 3: Continual learning
Page 99 – Erik Introduces the concept.
Page 297 – Bill learns how important The Three Ways are.
“But I learned that the practices that Allspaw and Hammond
espoused are the inevitable outcome of applying the Three
Ways to the IT value stream. It totally changed how we managed
IT and it saved our company. “How did they do it?” I ask,
Way No. 1: The flow
1. Understand the flow of work.
Without understanding, any changes will have random
2. Always increase the flow
3. Never pass on defects downstream
4. Never allow local optimization to cause global
5. Achieve profound understanding of the entire system
Way No 2: Consistent Feedback
Understand and respond to the needs of all customers,
both internal and external.
Shorten and amplify all feedback loops.
Create quality at the source.
Kim said this rule is modelled after the Toyota Lean
manufacturing line, where any employee can stop the line
immediately if they see any defect.
This means pro-actively building quality into everything. There's
no need for developers to rely on QA to find their errors.
Create and embed knowledge where we need it.
Way No. 3: Continual learning
Create a culture that encourages
Learns from failure
Recognizes repetition as the prerequisite to
Thanks Barclays Bank for providing the venue
Thanks to the authors of “The Phoenix Project”
Thanks for your participation
Free 170 page excerpt:http://itrevolution.com/the-phoenix-project-excerpt/
Dr Holt's TOC courses at Washington State University