Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Queues: An Invisible Money Drain

1,433 views

Published on

In The Principles of Product Development Flow, Don Reinertsen writes, "Queues lie at the root of a large number of product development problems. They increase variability, risk, and cycle time. They decrease efficiency, quality, and motivation.”

Queues hide in plain sight, and most companies ignore them. Queues are hard to see initially but, once you're aware of them and their costs, they're hard to ignore.

In this talk, I'll review some theory on what queues are and why they matter. I'll then discuss some examples that we've seen at GameChanger. We've found that there's no silver bullet to crushing queues, and we've had to be creative when devising strategies to eliminate them.

Published in: Technology

Queues: An Invisible Money Drain

  1. 1. Queues: An Invisible Money Drain Phil Sarin VP Engineering, GameChanger Media @philsarin http://216ways.net
  2. 2. “It ain’t what you don’t know that gets you into trouble. It’s what you know for sure that just ain’t so. — Mark Twain
  3. 3. Things I once believed You shouldn’t release often because releases have high overhead. You need to track all of your bugs. Slack is bad. Your team should be busy. Specialists will always be more productive than generalists.
  4. 4. The Biggest Misunderstanding: We think queues are free.
  5. 5. What we’ll cover tonight What a queue is Identifying the queues in your organization Tactics for handling queues Some war stories
  6. 6. Part 1: Queues and why they matter Flickr: Jeff Turner
  7. 7. “Queues are the root cause of the majority of economic waste in product development.” — Donald G. Reinertsen
  8. 8. Examples of queues Feature Queues Waiting to get designed Waiting to get developed Waiting for testing Waiting to get released Waiting to get validated Bug Queues Waiting for triage Waiting to get fixed Waiting to get verified Waiting for release
  9. 9. What’s wrong with big queues? Big queues increase cycle time and delay cost Big queues increase variability Big queues require management Big queues delay feedback Big queues hurt morale
  10. 10. The conservative team vs The aggressive team
  11. 11. Variability Flickr: State Farm
  12. 12. Flickr: State Farm
  13. 13. Flickr: State Farm
  14. 14. Flickr: State Farm
  15. 15. Batch Sizes Flickr: kbeil
  16. 16. 1 year 3 months vs 3 months 3 months 3 months vs
  17. 17. Why small batches are (usually) better Realize benefits sooner Faster feedback Higher motivation Less schedule slippage
  18. 18. Old School: Releases are expensive so let’s batch up work for release. New School: Let’s drive down the cost of releases so that we can release more often.
  19. 19. Queue takeaways Queues introduce delay. Queues have economic and psychological costs. High capacity utilization drives up queue size. Small batches can reduce the cost of queues.
  20. 20. Part 2: How we apply this theory at GameChanger Flickr: Jeff Turner
  21. 21. How we’ve managed queues 1. Swarm 2. Generalize queue handling 3. Obliterate queues 4. Reduce testing batch sizes
  22. 22. 1. Swarm
  23. 23. Flickr: State Farm
  24. 24. The stages of swarming 1. Skepticism/suspension of disbelief 2. Guilt and busywork 3. Scope blows up 4. The “surge capacity effect”
  25. 25. Announcements Invite Tools Alerts Announcements Invite Tools Alerts vs
  26. 26. No silver bullet
  27. 27. 2. Generalize Flickr: Kaleb Fulgham
  28. 28. Multiple queues, 1 server each Single queue, Single server Single queue, pool of servers
  29. 29. Multiple queues, 1 server each Single queue, Single server Single queue, pool of servers Sustained huge queues are least likely
  30. 30. Pressures to specialize A specialty differentiates your business “I don’t want to touch that code!” Initial builder owns it forever. Early adopters become de-facto specialists. Some people want specialist careers.
  31. 31. Resisting specialist queues Outsource non-core specialties Despecialize Challenge someone Staff up
  32. 32. Generalist teams and morale
  33. 33. 3. Obliterate queues http://provocateurs.ca/2014/11/16/delmar-dont-be-afraid-to-hit-delete/
  34. 34. 4. Reduce testing batch sizes
  35. 35. That time we missed our deadline by four months
  36. 36. 2 months development 2 weeks testing 2 months development 2 weeks testing 4 months bug-driven development
  37. 37. 2 weeks dev test 2 weeks dev test 2 weeks dev test 2 weeks dev test 2 weeks dev test Can we reduce this batch size further? Our first batch size reduction
  38. 38. 2 weeks dev test 2 weeks dev test 2 weeks dev test 2 weeks dev test 2 weeks dev test Can we reduce this batch size further?
  39. 39. 2 weeks dev+test T 2 weeks dev+test T 2 weeks dev+test T 2 weeks dev+test T 2 weeks dev+test T 2 weeks dev+test T Continuous pre-release delivery?
  40. 40. It didn’t work
  41. 41. Nobody used these
  42. 42. How we fixed it More automated test coverage Instituted small batch manual testing process Hired more testers
  43. 43. Release Days 5.15 21 5.14 14 5.13 26 5.12 21 5.11 23 5.10 4 5.9 66 5.8 7 5.6 21 It worked, mostly, eventually
  44. 44. Parting Thoughts Flickr: Jeff Turner
  45. 45. Things I once believed You shouldn’t release often because releases have high overhead. You need to track all of your bugs. Slack is bad. Your team should be busy. Specialists will always be more productive than generalists.
  46. 46. Mindset shifts Being busy vs being productive Individual vs team Rituals vs principles
  47. 47. Takeaways Queues are real and expensive You should know where your queues are There are a bunch of ways to reduce the cost of queues
  48. 48. Further Reading The Principles of Product Development Flow, Donald G. Reinertsen “Software Inventory” by Joel Spolsky: http://www.joelonsoftware.com/ items/2012/07/09.html “How we fixed more bugs by deleting our bug DB” and “Building around generalists” on my blog (216ways.net)

×