Wanted to talk to you guys about this book I am reading…
I am a big fan of one of the authors… there is a good looking bloke… <click> I saw him speak at ChefConf last year… He is mimicking pulling an Andon Cord there… That was when he worked at ThoughtWorks with this dude named <click> Martin Fowler Fowler is likely more recognizable with this hat <click>
Jez has since become VP of Development Velocity at Chef.
So here he is giving a keynote this year at ChefConf.
But I also want to talk to you about this other book that I am reading… also by Jez Humble
At our company there is a collection of applications that are in the Exploit phase – Namely Millennium There are also those in the Explore phase – Namely Population Health and Millennium+ Both are important – I have found myself for the last several years in the Explore projects – which is what I am talking about today
The culture is different in each. For Explore – we have READ and exploit we have READ
Lean Enterprise is of course a follow on to Lean Startup, where we learn about the Runway. (or at least I did).
Lean Startup, in turn has roots from “The Four Steps to the Epiphany” by Steve Blank
So when you hear your startup buddies talking about “The Pivot” that term likely has it’s roots in popularity from this book by Steve Blank
Enterprises will not “Pivot” as often as Startups, but there is a lot to learn about being Lean from Startups.
The big point that Lean Enterprise brings to the table, the “sound bite” that we should be discussing in our Enterprise is” Explore vs. Exploit”. To reiterate – I am talking about the Explore portion of the company.
I lifted this slide from a Jez Humble presentation, this is actually a screenshot of a pdf so I apologize for the readability. We need to build a high trust culture to be lean. Here is where we need to be.
It is your responsibility to aid in building the culture around you. Blameless Post Mortems should be part of every team for every incident, aimed at identifying causes, learning
I want to talk to you about the Fremont Assembly Plant. A 411-acre manufacturing plant in California.
At the time of its closure, the Fremont employees were "considered the worst workforce in the automobile industry in the United States", according to the United Auto Workers. Employees drank alcohol on the job, were frequently absent (enough so that the production line couldn't be started), and even committed petty acts of sabotage such as putting "Coke bottles inside the door panels, so they'd rattle and annoy the customer."
In spite of the history and reputation, when NUMMI reopened the factory for production in 1984, most of the troublesome GM workforce was rehired, with some sent to Japan to learn the Toyota Production System. Workers who made the transition identified the emphasis on quality and teamwork by Toyota management as what motivated a change in work ethic. almost right away, the NUMMI factory was producing cars with as few defects per 100 vehicles as those produced in Japan.
Coincidentally, this historic NUMMI plant is now the home of Tesla. The electric car manufacturer that open sourced to a degree their designs and patents recently that may change the world of car manufacturing. Tesla as you may know discarded the idea of model long ago and moves faster than Toyota using a continuous model where consumers can download updates to get new features.
From Lean Enterprise READ
Friction will always be part of our situation.
Friction is ultimately a consequence of the human condition—the fact that organizations are composed of people with independent wills and limited information. Thus friction cannot be overcome.
Many believe that the way through the friction we face is best done with brute force. Let’s define exactly what everyone should do so we can eliminate friction thus creating a command and control world.
Many believe that army’s use command and control.
Army’s… at least those that have won have not used Command and Control for a couple centuries since the Prussian’s lost to Napoleon.
In fact Prussia’s loss to Napoleon launched a lot of research into the subject of Mission Control.
It is crucial to understand that when we work in a complex adaptive system where friction dominates, the scientific management remedies cannot work. In fact, they make things worse. Creating ever more detailed plans delays the feedback that would tells us which of our assumptions are invalid. Complex sets of rules and controls punish the innocent but can be evaded by the guilty, all the while destroying morale, innovation, and entrepreneurialism.
It means Think Purpose!
In 2013, PuppetLabs, IT Revolution Press, and ThoughtWorks surveyed 9,200 technologists worldwide to find out what made high-performing organizations successful.
The resulting 2014 State of DevOps Report is based on analysis of answers from people working in a variety of industries including finance, tele-coms, retail, government, technology, education, and healthcare
The most important of these turned out to be whether people were satisfied with their jobs, based on the extent to which they agreed with the following statement.
It went a step farther used science to test Westrum’s model by asking respondents to rate the extent to which they agreed with statements such as “On my team, failure causes enquiry.”
Statistical analysis of the results showed that team culture was not only strongly correlated with organizational performance, it was also a strong predictor of job satisfaction.
We are not ready for Continous Deployment at Cerner. We need to have the manual intervention, however we also need to minimize it.
This is directly from the Continuous Delivery Book.
It took them 18 months to build this pipeline. Some reports state that they did not ship any new features during this time, only critical fixes.
On the left - The place where the instructions are determined is not the place where they must be carried out, so the instructor does not have the information about conditions where the local work needs to take place. The instructors might not even be aware of each other to resolve the conflict.
In a promise-based design, each part behaves only according to the promises it makes to others. Instead of instructions from without, we have behavior promised from within.
Jenkins Workflow Plugin is open source
* Wait time cut * Double speed
With Enterprise Jenkins support from CloudBee’s we would get this awesome view.
This is cutting edge. I had to upgrade the version of Jenkins, and many plugins to get my pipeline to work. I even had to upgrade git past the level that is packaged with Red Hat 6 (they ship versions 1.7.1, and I needed credentials in verision 1.7.9, so I needed to compile version 1.9.5 from source, using the community git cookbook of course. That and many other Yak’s were shaved in the making of this pipeline.
Authorization – only I can release code, and we will likely tear this down.
One thing that I learned along the way working with the Workflow Plugin is that Jenkins can get mad… Very mad…
This slide comes from a deck of Matthew Swindells VP of Population Health.
Whether a new heart procedure, a new physical instrument, or a new software innovation, it takes 17 years to hit the tipping point of adoption for physicians. We need to shrink this down to 17 months.
I often here pushback that Etsy, Netflix, and some of these other companies are smaller than Cerner. While this is true, Google is not smaller than Cerner in regard to the number of developers.
Let’s go back to our example from HP Laser Jet Printers.
Likely 2/3’s of work we do today provides 0 or negative value to the client.
What if we were able to replace this manual effort that is being done now, with doing actual live A/B testing in production. Comparing the current implementation with a change that we plan to work on. We can know some actual scientific data about whether that new feature is needed, wanted or will work for the client faster by many orders of magnitude.
This past weekend I updated the status of the document I created last fall stating the short list of things we need to make Continuous Delivery a reality at Cerner.
Quite frankly for our Explore areas of the company we cannot get to a cloud like environment with programmatic provisioning soon enough. We need OpenStack or something similar to be a real solution that we can use.
Nearly all of the other concepts have been proven and there is code in place, we likely will spend the next year refining and moving this into Chef Delivery when we can get our hands on it.
We have been focusing on the tools that will sit on top of the Continuous Delivery tool.
So in summary…
Whether it is Military Science, Modern Organizational Leadership Science, or Configuration Management for Systems… Command and Control is not the optimal approach… it may work but it is cumbersome and usually cracks under the weight of complexity it produces.
We need to continue to work collaboratively to get the workflow right.
DevOps does not mean that the developer would have any access to the production systems Quite the contrary, what we are working toward is automating to the point where no one needs direct access to a production machine. As we learned years ago with software packaging, manually doing this type of work is error prone, inconsistent and not a decent use of time.
It is simple to be stuck in the day to day of the current system and loop this for years. Investment needs to occur now to get the benefit years from now. Imagine freeing up 8X the capacity for innovation. This is what we need to push the meter on
Continuous Delivery: Delivering Client Value at Light Speed - DevCon 2015
This dude promises to
Stand on his head.
“The long-term value of an enterprise
is not captured by the value of its
products and intellectual property but
rather by its ability to continuously
increase the value it provides to
customers-and to create new
• So when can you say you’re doing continuous
delivery? I’d say it’s when you could flip a
switch to go to continuous deployment if you
decided that was the best way to deliver value
to your customers.
232 Highlights – Kindle Version
• It should always be cheaper to create a new
environment than to repair an old one.
Humble, Jez; Farley, David (2010-07-27).
Continuous Delivery: Reliable Software Releases
through Build, Test, and Deployment
Automation (Addison-Wesley Signature Series
(Fowler)) (Kindle Location 1633). Pearson
Education. Kindle Edition.
Let’s Take a Test!!!!11!!
If your configuration management process is sound, you
should be able to answer “yes” to the following
• Could you completely re-create your production
system, excluding production data, from scratch from
the version -controlled assets that you store?
• Could you regress to an earlier, known good state of
• Can you be sure that each deployed environment in
production, in staging, and in test is set up in precisely
the same way?
If not, then your organization is at risk.
Address slow innovation adoption
From time new knowledge discovered until ½ of physicians act on that
knowledge = 15 - 17 years
Everett Rogers, Diffusion of Innovations, 1995
Balas, Boren. Managing Clinical Knowledge for Health Care
Improvement. Yearbook of Medical Informatics 2000
Adoption Half-life = 17y
Knowledge Half-life = 10y
“Finish medical school and residency knowing
everything…read and retain 2 articles every single
night…at the end of 1 year you’re only 1,225 years
W Stead. JAMIA 2005;12:113-20
Alper BS, Hand JA, ElliottSG, et al.
J Med Lib Assoc 2004;92:429-37
How far along are we?
• From October (7 months ago)
– Continuous Delivery: What Do We Need to Get
There - October 2014 Meetup
Continuous Delivery Tool
Roll Out test-kitchen
rake deploy kitchen test knife push
Command and Control…
… does not lead to Success
Culture can …
… radically change in positive ways
• DevOps != Devs In Production
• DevOps == Few, if anyone In Production
• DevOps != Manual steps
• DevOps == Automate to achieve quality
Patient but Persistent
It’s worth it
• HP 18 months to build continuous
• Microsoft 10 years to build continuous