Upcoming SlideShare
×

# Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing

1,443 views

Published on

Finding bottlenecks in our software delivery processes is often pretty easy. But once we squash one bottleneck, another team becomes the limiting factor. This presentation looks how bottlenecks work, and how to predict the next bottleneck you'll need to work on.

3 Likes
Statistics
Notes
• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

Views
Total views
1,443
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
27
0
Likes
3
Embeds 0
No embeds

No notes for slide
• http://commons.wikimedia.org/wiki/File:Empty_Wine_bottle.jpg
• http://commons.wikimedia.org/wiki/File:Empty_Wine_bottle.jpg
• ### Pushing the Bottleneck: Predicting and Addressing the Next, Next Thing

1. 1. Pushing the bottleneck Predicting what will break next in your SDLC
2. 2. Lead Consultant & Tech Evangelist Eric is Lead Consultant at IBM UrbanCode Products where I help customers get the most out of their build, deploy and release processes. Today he works with customers and industry leaders to figure out this DevOps thing. Eric Minick eminick@us.ibm.com @EricMinick
3. 3. The plan  Theory of constraints in a nutshell  Finding bottlenecks  Predicting the next bottleneck  Common bottleneck pushing patterns  Q&A
4. 4. Theory of constraints in a nutshell
5. 5. A bottleneck is a constraint Maximum pouring speed
6. 6. Only by increasing flow through the constraint can overall throughput be increased* Making the bottle wide down here does not help * The Goal: a process of ongoing improvement. Goldratt eta al
7. 7. Optimize at your constraint Can pour faster now
8. 8. Your system must have a measureable goal My goal is to empty a wine bottle quickly
9. 9. Simplified five step plan 1. Identify the system’s most severe constraint 2. Decide how to get the most out of the constraint 3. Subordinate everything else to the above constraint 4. Make changes to expand constraint’s capacity 5. Once constraint is relieved, return to step 1
10. 10. Example: Slow QA cycles 1. Identify the system’s most severe constraint 2. Decide how to get the most out of the constraint 3. Subordinate everything else to the above constraint 4. Make changes to expand constraint’s capacity 5. Once constraint is relieved, return to step 1 1. It takes too long to test our changes. Everyone else waits 2. Testers should focus on exploratory testing 3. Devs help with regression tests. Ops prioritizes QA. 4. Dev & QA work together to automate regression tests 5. Find the next bottleneck
11. 11. Wait, what’s our goal in software?  Generally: turn ideas into business value  Measuring “business value” hard Emptying wine bottles only relevant if dev speed is the constraint and you believe in the Ballmer Peak (http://xkcd.com/323/) ?
12. 12. Wait, what’s our goal in software?  Generally: turn ideas into business value  Measuring “business value” hard  Features delivered minus bugs is a decent approximation – …but rewards building useless features. Emptying wine bottles only relevant if dev speed is the constraint and you believe in the Ballmer Peak (http://xkcd.com/323/) ?
13. 13. Three key measures  Lag time: how long from idea to value?  Throughput: how much delivered value per unit time?  Cost: what does it cost to deliver value? ?
14. 14. Finding the bottlenecks
15. 15. Most teams can feel the constraint  What are you waiting on?  Where’s the pain? Constraints before you in a process feel like not enough work. Constraints after you in a process are annoying or painful
16. 16. If it hurts, do it more often  Painful processes often grow exponentially worse with large batch sizes.  Examples – Integration work – Releases – Bug Triage – Updating databases – Visiting the dentist http://martinfowler.com/bliki/FrequencyReducesDifficulty.html Time between doing it Pain
17. 17. Use Lean techniques to measure  What does it take to get a change from idea to production? –At each phase measure wait time and work time  Long wait times indicate large batch sizes or backlogs image credits: http://commons.wikimedia.org/wiki/File:Diagram_spaghetti_kilka_produktow.PNG http://www.michaelnygard.com/blog/2008/02/outrunning_your_headlights.html
18. 18. Manual fix & verify spaghetti
19. 19. Bug fix & verify value stream 720 3600 240 2880 720 3600 240 15 15 120 60 15 15 60 Waiting Working 12000 300 1. Feature build 2. Build deployed 3. Bug reported 4. Dev fixes 5. Fix built 6. Fix deployed 7. Tester verifies
20. 20. Seeing the “next” bottleneck
21. 21. QA Scenario Dev Produces a nightly build Twice weekly, 2 hour deploy to Test Lab 3 Days to test each drop
22. 22. Dev produces a nightly build Twice weekly, 2 hour deploy to Test Lab 1.5 Days to test each drop If we improve test speed, our constraint moves.
23. 23. Examining a constraint • Manual process • Limited staff • Production releases have priority Why can we only deploy twice a week to QA?
24. 24. Tackling the constraint • Manual process • Limited staff • Production releases have priority Why can we only deploy twice a week to QA? • Automate processes • Hire more staff • Prioritize QA Releases Options
25. 25. Imagine a 1 day test cycle Dev produces a nightly build 2 hour deploy to Test Lab 1.0 Days to test each drop ¼ day deploy downtime becomes turns 1 day test cycle into two days.
26. 26. Tackling the constraint • Manual process • Limited staff • Production releases have priority Why can we only deploy twice a week to QA? • Automate processes • Hire more staff • Prioritize QA deploys Options Short term approach Long term approach
27. 27. Measuring utilization helps with this prediction  “Feeling” the pain isn’t enough to predict the next constraint  There may be no pain at the next constraint today
28. 28. When something is free, it is used more  Example: Amazon Prime. For \$79/yr, customers get free 2 day shipping on everything. “…Customers spent as much as 150% more at Amazon after they became Prime members. Subscribers not only ordered more often … they started buying things at Amazon that they probably wouldn’t have in the past” * * http://business.time.com/2013/03/18/amazon-prime-bigger-more-powerful-more-profitable-than- anyone-imagined/
29. 29. Consider implications of making something free If builds, deploy, regression tests are free… We’re going to be testing lots more. Better buy some hardware.
30. 30. Common “next bottleneck” patterns
31. 31. After build, deploy is next  Build guy & deploy guy used to be in sync  A CI server can do hundreds of builds per day  Agile tends to make Operations a constraint Knowing my stuff compiles at all times is great. I want to know if it passes functional tests too.
32. 32. As tests shift left, expensive tests constrain  Developers run more integration tests  Some tests use expensive resources –pay per use web services, mainframes, production systems…  Stubbing those resources becomes important – known as “service virtualization”
33. 33. Concurrent agile dev requires more test labs  More code is compiling, and set to be released “soon”  Need more environments to test changes in – Pressure for Platform as a Service
34. 34. Frequent releases demand fast feedback  Frequent releases enable experimentation  Monitoring of business outcomes required  Architectural pressures to support A/B testing ♫ Stop ignoring difference between features delivered and value delivered
35. 35. If we keep chasing constraints, where does it end?
36. 36. You may end up with Continuous Deployment Build Run thousands of tests Deploy to some servers Monitor Deploy to remaining servers
37. 37. Summary  Optimizations other than at the constraint don’t help  “Breaking” one constraint will expose the next  Patterns or analysis can be used to predict next constraint  Continual Improvement, Continual Improvement, Continual Improvement, Continual Improvement…
38. 38. IBM will collaborate with you to understand your current situation, goals and constraints. The Assessment and Planning Workshop will aim to capture sufficient information to make specific recommendations for improvement and implementation.  Intended Audience: Key leadership from practice areas and stakeholder organizations  Value Proposition Clear recommendations for capability improvements aligned to your business goals Initial Architecture Adoption roadmap based on proven best practices  Activities Workshop planning Assessment and Planning Workshop Collaborative discussion on current status, future goals and adoption requirements Produce Deliverable  Deliverables Current Status and Improvement Recommendations Architecture Adoption Roadmap Assessment and Planning Workshop
39. 39. More resources Urbancode.com/resources Continuous Delivery Maturity Model Deployment Automation Basics Applying Lean Principals to Software Delivery Blogs.urbancode.com Twitter.com/UrbanCode Facebook.com/IBMUrbanCodeProducts SlideShare.net/Urbancode
40. 40. 40 SlideShare.net/Urbancode @EricMinick