Successfully reported this slideshow.
Your SlideShare is downloading. ×

Continuous Delivery: Delivering Client Value at Light Speed - DevCon 2015


More Related Content


Continuous Delivery: Delivering Client Value at Light Speed - DevCon 2015

  1. 1. This year… This dude promises to NOT Stand on his head.
  2. 2. • m8n4
  3. 3. Continuous Delivery: Delivering Client Value at Light Speed Aaron Blythe
  4. 4. Aaron Blythe - Cerner • Writing Code • Answering questions • Sharing Thoughts • Running Meetups @ablythe
  5. 5. Aaron Blythe – Outside Cerner • Writing Code • Answering questions • Running Meetups @ablythe
  6. 6. Three Things to Get Right •Culture •Workflow •Tooling
  7. 7. •
  8. 8. Part 1 Culture 2. Workflow 3. Tooling
  9. 9. Fremont Assembly Plant
  10. 10. NUMMI plant
  11. 11. Tesla Factory
  12. 12. “The long-term value of an enterprise is not captured by the value of its products and intellectual property but rather by its ability to continuously increase the value it provides to customers-and to create new customers-through innovation.”
  13. 13. Friction
  14. 14. Mission Control vs. Command and Control The Prussion Army lost to Napoleon in 1809 Picture courtesy:
  15. 15. Auftragstaktik
  16. 16. 2014 State of Devops Report 9,200 technologists
  17. 17. • I would recommend this organization as a good place to work. • I have the tools and resources to do my job well. • I am satisfied with my job. • My job makes good use of my skills and abilities.
  18. 18. Part 2 Workflow 3. Tooling 1. Culture
  19. 19. deployment
  20. 20. • So when can you say you’re doing continuous delivery? I’d say it’s when you could flip a switch to go to continuous deployment if you decided that was the best way to deliver value to your customers. • nuous-delivery-vs-continuous-deployment/
  21. 21. deployment
  22. 22. 232 Highlights – Kindle Version • It should always be cheaper to create a new environment than to repair an old one. Humble, Jez; Farley, David (2010-07-27). Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation (Addison-Wesley Signature Series (Fowler)) (Kindle Location 1633). Pearson Education. Kindle Edition.
  23. 23. Let’s Take a Test!!!!11!! If your configuration management process is sound, you should be able to answer “yes” to the following questions: • Could you completely re-create your production system, excluding production data, from scratch from the version -controlled assets that you store? • Could you regress to an earlier, known good state of your application? • Can you be sure that each deployed environment in production, in staging, and in test is set up in precisely the same way? If not, then your organization is at risk.
  24. 24. Command and Control vs. Promise Theory
  25. 25. mvn deploy rake deploy
  26. 26. Continuous Delivery Tool Roll Out test-kitchen Chef Push Jobs rake deploy kitchen test knife push Project Project.yml Kitchen.yml Chef roles
  27. 27. Part 3 Tooling 1. Culture 2. Workflow
  28. 28. Continuous Delivery Tools Workflow PluginDelivery Many more over the next couple years…
  29. 29. Chef Delivery
  30. 30. Go (from ThoughtWorks)
  31. 31. Jenkins
  32. 32. Demo
  33. 33. • Jenkins – Open Source “Workflow Plugin” • Enterprise Support needed for “Workflow Stage View” – Working POC – Open Source • Job – t_ops_clientlog_server/configure • Source Code –
  34. 34. • operations-and-continuous.html
  35. 35. Workflow Plugin TODO’s • Permissions for input approval – • Option for stage step to cancel older executions – • Visual Distinction of Steps –
  36. 36. Chef Delivery • • ChefConf 2015 Keynote: – 4&index=7&list=PL11cZfNdwNyO9CpTWH2qjYfzys EtpfOCd – @ about 23:30
  37. 37. Verify Stage
  38. 38. Code Review
  39. 39. Chef Delivery TODO’s • Everything • We do have a demo version for the next 2 weeks.
  40. 40. Why is this important??
  41. 41. Address slow innovation adoption From time new knowledge discovered until ½ of physicians act on that knowledge = 15 - 17 years Everett Rogers, Diffusion of Innovations, 1995 Balas, Boren. Managing Clinical Knowledge for Health Care Improvement. Yearbook of Medical Informatics 2000 %ofpopulation Time Adoption Half-life = 17y Knowledge Half-life = 10y “Finish medical school and residency knowing everything…read and retain 2 articles every single night…at the end of 1 year you’re only 1,225 years behind.” W Stead. JAMIA 2005;12:113-20 Alper BS, Hand JA, ElliottSG, et al. J Med Lib Assoc 2004;92:429-37
  42. 42. Can this actually be done?
  43. 43. If we delivery faster so what?
  44. 44. How far along are we? • From October (7 months ago) – Continuous Delivery: What Do We Need to Get There - October 2014 Meetup – • Status –
  45. 45. Continuous Delivery Tool Roll Out test-kitchen Chef Push Jobs rake deploy kitchen test knife push Project Project.yml Kitchen.yml Chef roles
  46. 46. Science Says: Command and Control… … does not lead to Success Culture can … … radically change in positive ways 1. Culture
  47. 47. DevOps means? • DevOps != Devs In Production • DevOps == Few, if anyone In Production • DevOps != Manual steps • DevOps == Automate to achieve quality 2. Workflow
  48. 48. Patient but Persistent It’s worth it • HP  18 months to build continuous delivery • Microsoft  10 years to build continuous delivery 3. Tooling
  49. 49.

Editor's Notes

  • Wanted to talk to you guys about this book I am reading…
  • I am a big fan of one of the authors… there is a good looking bloke… <click>
    I saw him speak at ChefConf last year… He is mimicking pulling an Andon Cord there… That was when he worked at ThoughtWorks with this dude named <click> Martin Fowler
    Fowler is likely more recognizable with this hat <click>
  • Jez has since become VP of Development Velocity at Chef.

    So here he is giving a keynote this year at ChefConf.
  • But I also want to talk to you about this other book that I am reading… also by Jez Humble

    At our company there is a collection of applications that are in the Exploit phase – Namely Millennium
    There are also those in the Explore phase – Namely Population Health and Millennium+
    Both are important – I have found myself for the last several years in the Explore projects – which is what I am talking about today

    The culture is different in each. For Explore – we have READ and exploit we have READ
  • Lean Enterprise is of course a follow on to Lean Startup, where we learn about the Runway. (or at least I did).

    Lean Startup, in turn has roots from “The Four Steps to the Epiphany” by Steve Blank
  • So when you hear your startup buddies talking about “The Pivot” that term likely has it’s roots in popularity from this book by Steve Blank

    Enterprises will not “Pivot” as often as Startups, but there is a lot to learn about being Lean from Startups.
  • The big point that Lean Enterprise brings to the table, the “sound bite” that we should be discussing in our Enterprise is” Explore vs. Exploit”.
    To reiterate – I am talking about the Explore portion of the company.
  • I lifted this slide from a Jez Humble presentation, this is actually a screenshot of a pdf so I apologize for the readability.
    We need to build a high trust culture to be lean. Here is where we need to be.


    It is your responsibility to aid in building the culture around you. Blameless Post Mortems should be part of every team for every incident, aimed at identifying causes, learning
  • I want to talk to you about the Fremont Assembly Plant.
    A 411-acre manufacturing plant in California.

    At the time of its closure, the Fremont employees were "considered the worst workforce in the automobile industry in the United States", according to the United Auto Workers.[6][7] Employees drank alcohol on the job, were frequently absent (enough so that the production line couldn't be started), and even committed petty acts of sabotage such as putting "Coke bottles inside the door panels, so they'd rattle and annoy the customer."
  • In spite of the history and reputation, when NUMMI reopened the factory for production in 1984, most of the troublesome GM workforce was rehired, with some sent to Japan to learn the Toyota Production System.[6][7] Workers who made the transition identified the emphasis on quality and teamwork by Toyota management as what motivated a change in work ethic. almost right away, the NUMMI factory was producing cars with as few defects per 100 vehicles as those produced in Japan.
  • Coincidentally, this historic NUMMI plant is now the home of Tesla. The electric car manufacturer that open sourced to a degree their designs and patents recently that may change the world of car manufacturing. Tesla as you may know discarded the idea of model long ago and moves faster than Toyota using a continuous model where consumers can download updates to get new features.
  • From Lean Enterprise
  • Friction will always be part of our situation.

    Friction is ultimately a consequence of the human condition—the fact that organizations are composed of people with independent wills and limited information. Thus friction cannot be overcome.

    Many believe that the way through the friction we face is best done with brute force. Let’s define exactly what everyone should do so we can eliminate friction thus creating a command and control world.
  • Many believe that army’s use command and control.

    Army’s… at least those that have won have not used Command and Control for a couple centuries since the Prussian’s lost to Napoleon.
  • In fact Prussia’s loss to Napoleon launched a lot of research into the subject of Mission Control.


    It is crucial to understand that when we work in a complex adaptive system where friction dominates, the scientific management remedies cannot work. In fact, they make things worse.
    Creating ever more detailed plans delays the feedback that would tells us which of our assumptions are invalid.
    Complex sets of rules and controls punish the innocent but can be evaded by the guilty, all the while destroying morale, innovation, and entrepreneurialism.

    It means Think Purpose!
  • In 2013, PuppetLabs, IT Revolution Press, and ThoughtWorks surveyed 9,200 technologists worldwide to find out what made high-performing organizations successful.

    The resulting 2014 State of DevOps Report is based on analysis of answers from people working in a variety of industries including finance, tele-coms, retail, government, technology, education, and healthcare
  • The most important of these turned out to be whether people were satisfied with their jobs, based on the extent to which they agreed with the following statement.
  • It went a step farther used science to test Westrum’s model by asking respondents to rate the extent to which they agreed with statements such as “On my team, failure causes enquiry.”

    Statistical analysis of the results showed that team culture was not only strongly correlated with organizational performance, it was also a strong predictor of job satisfaction.
  • Combined culture is super important to continuous delivery considering how early and often we integrate.

    So we now move into workflow
  • Grow engineering.

    Purple Diamond

  • Put in more change Management.
  • Here we have the numbers from Amazon in May…

    These numbers are impressive… However they are from 2011, 4 years ago, the year before we started using Chef at Cerner.
  • From a Blog by Jez Humble
  • We are not ready for Continous Deployment at Cerner.
    We need to have the manual intervention, however we also need to minimize it.
  • This is directly from the Continuous Delivery Book.
  • It took them 18 months to build this pipeline. Some reports state that they did not ship any new features during this time, only critical fixes.
  • On the left - The place where the instructions are determined is not the place where they must be carried out, so the instructor does not have the information about conditions where the local work needs to take place. The instructors might not even be aware of each other to resolve the conflict.

    In a promise-based design, each part behaves only according to the promises it makes to others. Instead of instructions from without, we have behavior promised from within.
  • Jenkins Workflow Plugin is open source
  • * Wait time cut
    * Double speed
  • With Enterprise Jenkins support from CloudBee’s we would get this awesome view.

    This is cutting edge. I had to upgrade the version of Jenkins, and many plugins to get my pipeline to work. I even had to upgrade git past the level that is packaged with Red Hat 6 (they ship versions 1.7.1, and I needed credentials in verision 1.7.9, so I needed to compile version 1.9.5 from source, using the community git cookbook of course. That and many other Yak’s were shaved in the making of this pipeline.
  • Authorization – only I can release code, and we will likely tear this down.

  • One thing that I learned along the way working with the Workflow Plugin is that Jenkins can get mad…
    Very mad…
  • This slide comes from a deck of Matthew Swindells VP of Population Health.

    Whether a new heart procedure, a new physical instrument, or a new software innovation, it takes 17 years to hit the tipping point of adoption for physicians. We need to shrink this down to 17 months.
  • I often here pushback that Etsy, Netflix, and some of these other companies are smaller than Cerner. While this is true, Google is not smaller than Cerner in regard to the number of developers.
  • Let’s go back to our example from HP Laser Jet Printers.
  • Likely 2/3’s of work we do today provides 0 or negative value to the client.

    What if we were able to replace this manual effort that is being done now, with doing actual live A/B testing in production. Comparing the current implementation with a change that we plan to work on. We can know some actual scientific data about whether that new feature is needed, wanted or will work for the client faster by many orders of magnitude.
  • This past weekend I updated the status of the document I created last fall stating the short list of things we need to make Continuous Delivery a reality at Cerner.

    Quite frankly for our Explore areas of the company we cannot get to a cloud like environment with programmatic provisioning soon enough. We need OpenStack or something similar to be a real solution that we can use.

    Nearly all of the other concepts have been proven and there is code in place, we likely will spend the next year refining and moving this into Chef Delivery when we can get our hands on it.
  • We have been focusing on the tools that will sit on top of the Continuous Delivery tool.
  • So in summary…

    Whether it is Military Science, Modern Organizational Leadership Science, or Configuration Management for Systems…
    Command and Control is not the optimal approach… it may work but it is cumbersome and usually cracks under the weight of complexity it produces.
  • We need to continue to work collaboratively to get the workflow right.

    DevOps does not mean that the developer would have any access to the production systems
    Quite the contrary, what we are working toward is automating to the point where no one needs direct access to a production machine.
    As we learned years ago with software packaging, manually doing this type of work is error prone, inconsistent and not a decent use of time.
  • It is simple to be stuck in the day to day of the current system and loop this for years. Investment needs to occur now to get the benefit years from now.
    Imagine freeing up 8X the capacity for innovation.
    This is what we need to push the meter on