Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
What to Upload to SlideShare
What to Upload to SlideShare
Loading in …3
×
1 of 32

Getting out of quicksand with devops - DevOps unicorns Riga 2019

Dec. 30, 2019
• 0 likes • 209 views

0

Share

Download to read offline

Software

Join me in this talk about why high workload leads to increasing waiting times and is detrimental to your project’s success.

Teams often find themselves in high workload situations, showing symptoms of overload and being too busy to implement the necessary changes or adopt the state-of-the-art tools to get the upper hand.
This is a story of how we applied the Three Ways of DevOps to get out of a state we referred to as quicksand (The more you fight it, the more it pulls you in).

I will not only talk about queuing theory and capacity management, but also about DevOps strategies to cope with high utilization and how to start a virtuous circle.

Key Takeaways are:

* Crisis situations are opportunities for change
* Often it takes fresh eyes or at least courage to see the problems
* High workload leads to increasing waiting times
* Get out of the quicksand with devops:
* Quantify the work and set goals
* Measure
* Less is more
* Automate the right things
* Establish a continuous improvement process
* Start a virtuous circle

Recommended

Related Books

Free with a 30 day trial from Scribd

See all
Hamlet's BlackBerry: A Practical Philosophy for Building a Good Life in the Digital Age William Powers
(4/5)
Free
In the Plex: How Google Thinks, Works, and Shapes Our Lives Steven Levy
(4.5/5)
Free
An Army of Davids: How Markets and Technology Empower Ordinary People to Beat Big Media, Big Government, and Other Goliaths Glenn Reynolds
(4/5)
Free
World Wide Mind: The Coming Integration of Humanity, Machines, and the Internet Michael Chorost
(4/5)
Free
Emergence: The Connected Lives of Ants, Brains, Cities, and Software Steven Johnson
(4.5/5)
Free
Tubes: A Journey to the Center of the Internet Andrew Blum
(4/5)
Free
The Impulse Economy: Understanding Mobile Shoppers and What Makes Them Buy Gary Schwartz
(4.5/5)
Free
Blog Schmog: The Truth About What Blogs Can (and Can't) Do for Your Business Robert W. Bly
(4/5)
Free
The End of Business As Usual: Rewire the Way You Work to Succeed in the Consumer Revolution Brian Solis
(5/5)
Free
Socialnomics: How Social Media Transforms the Way We Live and Do Business Erik Qualman
(3/5)
Free
The Nature of the Future: Dispatches from the Socialstructed World Marina Gorbis
(4/5)
Free
Public Parts: How Sharing in the Digital Age Improves the Way We Work and Live Jeff Jarvis
(3.5/5)
Free
Talking Back to Facebook: The Common Sense Guide to Raising Kids in the Digital Age James P. Steyer
(4.5/5)
Free
The Thank You Economy Gary Vaynerchuk
(4/5)
Free
Blog, Inc.: Blogging for Passion, Profit, and to Create Community Joy Deangdeelert Cho
(3.5/5)
Free
What Video Games Have to Teach Us About Learning and Literacy. Second Edition James Paul Gee
(4/5)
Free

Related Audiobooks

Free with a 30 day trial from Scribd

See all
New Dark Age: Technology and the End of the Future James Bridle
(4.5/5)
Free
Blockchain Revolution: How the Technology Behind Bitcoin Is Changing Money, Business, and the World Don Tapscott
(4/5)
Free
The Emperor's New Mind: Concerning Computers, Minds, and the Laws of Physics Roger Penrose
(3.5/5)
Free
The Death of Expertise: The Campaign Against Established Knowledge and Why it Matters Tom Nichols
(4.5/5)
Free
Alone Together: Why We Expect More from Technology and Less from Each Other Sherry Turkle
(4/5)
Free
Algorithms to Live By: The Computer Science of Human Decisions Tom Griffiths
(4.5/5)
Free
Kill All Normies: Online Culture Wars From 4Chan And Tumblr To Trump And The Alt-Right Angela Nagle
(4/5)
Free
The Art of Social Media: Power Tips for Power Users Guy Kawasaki
(4/5)
Free
An Introduction to Information Theory: Symbols, Signals and Noise John R. Pierce
(4.5/5)
Free
Who Owns the Future? Jaron Lanier
(4/5)
Free
The Dark Net: Inside the Digital Underworld Jamie Bartlett
(3.5/5)
Free
So You Want to Start a Podcast: Finding Your Voice, Telling Your Story, and Building a Community that Will Listen Kristen Meinzer
(4.5/5)
Free
Cognitive Surplus: Creativity and Generosity in a Connected Age Clay Shirky
(3.5/5)
Free
Everybody Lies: Big Data, New Data, and What the Internet Can Tell Us About Who We Really Are Seth Stephens-Davidowitz
(4.5/5)
Free
The New New Thing: A Silicon Valley Story Michael Lewis
(4.5/5)
Free
Once Upon an Algorithm: How Stories Explain Computing Martin Erwig
(4/5)
Free

Getting out of quicksand with devops - DevOps unicorns Riga 2019

  1. 1. 03.10.2019 Roman Pickl Getting Out of Quicksand, With DevOps!
  2. 2. 2© Elektrobit (EB) 2018 • Technical Project Manager @ Elektrobit • Former CTO @ Fluidtime (now a Kapsch company) • Loves CI/CD/DevOps • Here to learn Roman Pickl (@rompic) How did I end up here? Getting Out of Quicksand, With DevOps!
  3. 3. 3© Elektrobit (EB) 2018 https://youtu.be/BeNdum6u6eg?t=2081 Andrew Shafer, DevOpsDays Cairo 2018 Getting Out of Quicksand, With DevOps!
  4. 4. 4© Elektrobit (EB) 2018 Tom DeMarco, 2001, Slack: Getting Past Burnout, Busywork, and the Myth of Total Efficiency Efficiency vs. Effectiveness Getting Out of Quicksand, With DevOps! Efficiency vs. Effectiveness By Xwizard35 - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=58355021
  5. 5. 5© Elektrobit (EB) 2018 Symptoms of overload • Decreased team morale • Working long hours / when sick • More frequent illness • Unhealthy tasks queue • Imbalanced metrics • … How the team currently looks like Getting Out of Quicksand, With DevOps! 100 90130
  6. 6. 6© Elektrobit (EB) 2018 It hurts … Getting Out of Quicksand, With DevOps! … again and again
  7. 7. 7© Elektrobit (EB) 2018 Source unknown No thanks, too busy Getting Out of Quicksand, With DevOps!
  8. 8. 8© Elektrobit (EB) 2018 What your manager thinks should be done Getting Out of Quicksand, With DevOps! 100 100 100 20 20 From another team with a project on hold
  9. 9. 9© Elektrobit (EB) 2018 https://en.wikipedia.org/wiki/Kingman%27s_formula https://less.works/less/principles/queueing_theory.html https://observablehq.com/@troymagennis/how-does-utilization-impact-lead-time-of-work?collection=@troymagennis/agile-software-development - Troy Magennis • Different types of queues (e.g. G/G/1) • Waiting Time: – Service time – Utilization – Variation in process and arrival • Kingman Equation / approximation • Little’s Law: – Avg. Lead Time = Avg. WiP / Avg. Throughput  Focus on throughput rather than utilization Queuing theory Wait a moment? How high workload kills efficiency! 0 2 4 6 8 10 12 14 16 18 20 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% Waittime Wait Time = (%Busy)/(% Idle)
  10. 10. 11© Elektrobit (EB) 2018 Source: https://www.eaglecreek.com/blog/how-escape-quicksand Docker: Ops unleashed by Mario Kleinsasser referring to Quicksand: https://www.youtube.com/watch?v=Ocmx7nifh2I&feature=youtu.be&t=400 Feels like quicksand Wait a moment? How high workload kills efficiency! The more you fight it, the more it pulls you in.
  11. 11. 12© Elektrobit (EB) 2018 “slack is the missing ingredient required for all change” Invest roughly a day per week to - Spur innovation - Rethink - Practice new ways - Master new skills - Improve efficiency. Scrum: sustainable pace 3Ms 15% time Google’s 20% time Hard limit on toil at Google SRE: 50% #2 SRE principle: SREs must have time to make tomorrow better than today What you think should be done: hire more people Wait a moment? How high workload kills efficiency! 80 80 80 80
  12. 12. 13© Elektrobit (EB) 2018 What actually happens Getting Out of Quicksand, With DevOps! YOU „People under time pressure don‘t think faster“ - Tim Lister
  13. 13. 14© Elektrobit (EB) 2018 KC Green It sometimes takes a crisis Getting Out of Quicksand, With DevOps!
  14. 14. 15© Elektrobit (EB) 2018 2018 Accelerate State of DevOps Report: https://cloudplatformonline.com/2018-state-of-devops.html Quantify the work: Activity Accounting Getting Out of Quicksand, With DevOps!
  15. 15. 16© Elektrobit (EB) 2018 Hydrargyrum Flow time – Lead time – Cycle time Frequency of deployment Mean time to restore services WiP is the biggest deterrent to flow What is the biggest bottleneck? Can we fix the bottleneck? Ask why (5 times), Measure Getting Out of Quicksand, With DevOps!
  16. 16. 17© Elektrobit (EB) 2018 https://xkcd.com/303/ XKCD - Compiling Getting Out of Quicksand, With DevOps!
  17. 17. 18© Elektrobit (EB) 2018 https://www.youtube.com/watch?v=5S90x6CCGJ0&feature=youtu.be&t=2696 - Carmen DeArdo Getting Out of Quicksand, With DevOps!
  18. 18. 19© Elektrobit (EB) 2018 A Practical Approach to Large-Scale Agile Development – Gruver, Young, Fulghum Principles of Lean Product Managment by Jez Humble - https://youtu.be/cH6bnQzJojo?t=1083 "A Practical Approach to Large Scale Agile Development" - Gary Gruver at Spark 2013 https://www.youtube.com/watch?v=2QGYEwghRSM • Tendency to do more planning / estimation and more control • Accept uncertainty and appreciate flexibility • Decreasing marginal utility of each hour spend planning • Every hour spent planning is an hour not delivering (as resources often overlap) • Lightweight capacity planning • Just-in-time approach • Prioritize e.g. by cost of delay Lightweight Capacity Management Getting Out of Quicksand, With DevOps!
  19. 19. 20© Elektrobit (EB) 2018 • Reduce WiP • Reduce batch size • Reduce variability / increase predictability • Do less • Self service platform • Eliminate before you try to automate Getting Out of Quicksand, With DevOps! Do less (at the same time) - = + Less is More
  20. 20. 21© Elektrobit (EB) 2018 Picture: https://moduscreate.com/blog/automation-what-why-when/ • Make time to create automation • Documenting the process and making it repeatable is the first step • Automate, eliminate or engineer out the drivers that are not key to the value proposition – New work – hard to automate – Triage work – automate for consistency – Repetitive work – automate • Fix the biggest bottleneck first Automate Getting Out of Quicksand, With DevOps!
  21. 21. 22© Elektrobit (EB) 2018 The Practice of Cloud System Administration Getting Out of Quicksand, With DevOps! Left-Over Principle DifficultEasy Frequent Rare Acquire Document Manual Automate Hire Create Tools Eliminate
  22. 22. 23© Elektrobit (EB) 2018 Demarco T, Slack: Getting Past Burnout, Busywork and the Myth of Total Efficiency, page 108ff Forsgren N., Humble J., Gene K. Accelerate p. 12 ff Accelerate State of DevOps Report 2018 p. 28ff A word of caution Getting Out of Quicksand, With DevOps! The paradox of automation
  23. 23. 24© Elektrobit (EB) 2018 “Improving your software delivery effectiveness will improve your ability to work in small batches and incorporate customer feedback along the way” “Lean product management practices positively impact software delivery performance, stimulate a generative culture, and decrease burnout” “[…] software delivery performance drives Lean product management practices” “[…] it becomes a reciprocal model or, colloquially, a virtuous cycle” – Accelerate Virtuous circle Getting Out of Quicksand, With DevOps!
  24. 24. 25© Elektrobit (EB) 2018 http://www-personal.umich.edu/~mrother/Homepage.html Improvement Kata Getting Out of Quicksand, With DevOps!
  25. 25. 26© Elektrobit (EB) 2018 The Practice of Cloud System Administration Getting Out of Quicksand, With DevOps! Prioritization of Improvement High ImpactLow Impact Easy Hard Priority 1 Priority 2 Priority 4 Priority 3
  26. 26. 27© Elektrobit (EB) 2018 27 Based on Virtuous Circle of CI/CD The Site Reliability Workbook Virtuous Circle Getting Out of Quicksand, With DevOps! More release / test automation / improvements Faster releases / more time for new work More frequent releases / smaller batches / less WiP Fewer changes per release / lower lead time Fewer bugs per release / faster feedback/ higher confidence / lower variability and rework More time to automate / test / lower utilization / decreased burnout / higher job satisfaction
  27. 27. 28© Elektrobit (EB) 2018 Spent 1 day a week to improve “something” Created self service pipelines Used Information radiators Clear priorities and weekly goals How it ended up working: Getting Out of Quicksand, With DevOps! 80 80 80
  28. 28. 29© Elektrobit (EB) 2018 Crisis situations are opportunities for change Often it takes fresh eyes or at least courage to see the problems Get out of the quicksand, the sky is the limit Quantify the work and set goals Measure Less is more Automate the right things Establish a continuous improvement process Start a virtuous circle Conclusion Getting Out of Quicksand, With DevOps!
  29. 29. 30© Elektrobit (EB) 2018 Read these if you want to know more Getting Out of Quicksand, With DevOps!
  30. 30. 34© Elektrobit (EB) 2018 We don‘t let our servers get to 100% capacity utilization, so let‘s not do that to ourselves Dominica DeGrandis – Making work visible Getting Out of Quicksand, With DevOps!
  31. 31. 35© Elektrobit (EB) 2018 • Daniel Vacanti: Little‘s (F)law: https://vimeo.com/52683659 • Håkan Forss, Queueing theory in software development - ALEBathtub 2011-06-30, https://de.slideshare.net/HkanForss/queueing-theory-in- software-development-ale-bathtub-2011-0630 • GOTO 2015 • Why Scaling Agile Doesn't Work • Jez Humble - https://www.youtube.com/watch?v=2zYxWEZ0gYg&feature=youtu.be&t=3003 (it often takes a disaster for lasting change) • John Cutler – Too Busy to Improve https://anchor.fm/john-cutler/episodes/Too-Busy-To-Improve-for-Roman-e2ap3i • 2018 Accelerate State of DevOps Report: https://cloudplatformonline.com/2018-state-of-devops.html • Dominica DeGrandis - Making Work Visible: How to Unmask Capacity Killing WIP - https://www.youtube.com/watch?v=KR7Y8IUgyyA • Keynote: Continuous Delivery Sounds Great By Jez Humble @ Agile India 2017 https://www.youtube.com/watch?v=SjVV3xuYKJs • A Practical Approach to Large Scale Agile Development"- Gary Gruver at Spark 2013 https://www.youtube.com/watch?v=2QGYEwghRSM • High-Performance Team, Management, and Leadership Behaviors and practices https://devops- research.com/assets/transformation_practices.pdf • John Cutler - This feels like going faster vs. this actually makes us faster https://twitter.com/johncutlefish/status/1029757026895720449 • John Cutler – Slack vs. Blocked https://medium.com/@johnpcutler/slack-vs-blocked-16c294938b1 • Mike Rother - http://www-personal.umich.edu/~mrother/Homepage.html • Troy Magennis - https://observablehq.com/@troymagennis/how-does-utilization-impact-lead-time-of-work?collection=@troymagennis/agile- software-development Further references & information Getting Out of Quicksand, With DevOps!
  32. 32. Roman Pickl (@rompic) roman.pickl@elektrobit.com www.elektrobit.com Paldies! Don’t wait and get in touch

×