Recommended
More Related Content
What's hot
What's hot (20)
Similar to BA World Boston: Evening the Odds with Monte Carlo Project Forecasting
Similar to BA World Boston: Evening the Odds with Monte Carlo Project Forecasting (20)
Recently uploaded
Recently uploaded (20)
BA World Boston: Evening the Odds with Monte Carlo Project Forecasting
- 1. Evening the Odds: Monte Carlo Forecasting Hunter Tammaro / @htammaro excella.com | @excellaco
- 2. excella.com | @excellaco Learning Objectives • What a Monte Carlo forecast is • How the technique illustrates Agile concepts • How Monte Carlo is compatible with Agile
- 3. excella.com | @excellaco How do you forecast today?
- 4. excella.com | @excellaco Whydo you make forecasts?
- 5. excella.com | @excellaco Setting the Stage
- 6. excella.com | @excellaco GettingStarted • Team funded for one year • Only high-level, long-term goals • Built a“releasable” product over a few months • Started to improve ad-hoc,based on feedback
- 7. excella.com | @excellaco GettingFinished • Question of when to move on • What does it mean to be “complete?” • When willit happen? • How sure can we be?
- 8. excella.com | @excellaco Forecasting with (or without) MonteCarlo
- 9. excella.com | @excellaco • Estimate allbacklog items • Add up to get the total backlog size • Divide by average velocity • That’syour release date! • Thatisyour release date, right? Forecasting with Averages
- 10. excella.com | @excellaco • How long did forecasting take? • How accurate is the forecast? • How sure are you? • What happens if you miss the forecast? BUT…
- 11. excella.com | @excellaco Determine a range of future outcomes and their probabilities based onrepeated, random sampling of existingdata Enter the Monte Carlo method
- 13. excella.com | @excellaco Acknowledgeand estimateuncertaintyin theforecast Weunderstandthisimplicitlyforother types of predictions! Embracing Variability
- 17. excella.com | @excellaco Creating a Monte Carlo forecast
- 18. excella.com | @excellaco Gather Data • Measure the variability in what you’re trying topredict • Takttimes –the time between completed PBIs • Takt accounts foractual output of the whole team
- 19. excella.com | @excellaco Gather Data For software teams, measure takt in weekdays between completed items M T W T F M T WS S 3 days 1 day 0 days 1 day 2 days
- 20. excella.com | @excellaco Run one simulation Randomly choose takttimes for each item in your backlog Add up the total time PBI Takt 1 2 1 3 3 4 0 5 1 6 2 PBI Takt 7 8 9 10 11 12 Total Measured Simulated 3 2 1 0 1 3 10
- 21. excella.com | @excellaco Run a lot more simulations Repeat a few dozen times… PBI Takt 7 1 8 0 9 0 10 3 11 0 12 2 Total 6 PBI Takt 7 3 8 2 9 1 10 3 11 3 12 3 Total 15 PBI Takt 7 1 8 3 9 3 10 1 11 3 12 0 Total 11 PBI Takt 7 3 8 3 9 3 10 3 11 1 12 0 Total 13 PBI Takt 7 3 PBI Takt 7 1 PBI Takt 7 2 PBI Takt 7 2
- 22. excella.com | @excellaco Results Count the number of simulations of a given duration Determine cumulative percentages Days N Runs Running % 0-4 4 4% 5-6 24 28% 7-8 27 55% 9-10 24 79% 11-12 15 94% 13-14 5 99% 15+ 1 100%
- 23. excella.com | @excellaco And graph it 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 0 5 10 15 20 25 30 0-4 5-6 7-8 9-10 11-12 13-14 15+ Days N Runs Cumulative %
- 24. excella.com | @excellaco Let a computer do it! Activity
- 25. excella.com | @excellaco A simple forecasting tool Input tab: takt times, start date, size of forecasted backlog
- 26. excella.com | @excellaco A simple forecasting tool Simulation tab: simulated PBIs and their takt times, totals for each simulated backlog
- 27. excella.com | @excellaco A simple forecasting tool Results tab: Distribution of simulations by date range
- 28. excella.com | @excellaco You will need… • A phone with the Google Sheets or MS Excel app • Or a laptop • Or a neighbor with one of the above Conference Wifi: SSID: MCTGuest Password: MCT2019Guest
- 29. excella.com | @excellaco Monte Carlo activity • Sheets: duplicate the spreadsheet Web: File > Make a copy Mobile: menu > Share & export > Make a copy • Excel: download a copy • Play with the parameters and see how the predictions change https://cutt.ly/ montecarlo
- 30. excella.com | @excellaco Things to try • Add more input takt times in the same range of values • Replace some of the input takt times to use a broader range of values • Replace just one input takt time with an outlier
- 31. excella.com | @excellaco • What do you notice about your forecasts? • When could you have used this on a project? • How could you use it on your current project? • What would you need to start using this technique? Discuss with a partner
- 32. excella.com | @excellaco Number of takt inputs Simulations, 10 takts Simulations, 30 takts
- 33. excella.com | @excellaco Range of takt inputs Simulations, narrow range Simulations, broad range
- 34. excella.com | @excellaco Adding an outlier Simulations, normal range Simulations, with outlier
- 35. excella.com | @excellaco How we used the forecast
- 36. excella.com | @excellaco • Understand how sure we are that we’ll finish on time • Explicitly acknowledges risk • Tailor the forecast you communicate to your audience Uncertainty
- 37. excella.com | @excellaco Release planning Gave us the information we needed to plan a release Made the significance of good prioritization more obvious
- 38. excella.com | @excellaco Pace setting • Understand impact of scope changes throughout the project • Gave us a better feel for our status than “red/yellow/green” • Easy to update once created
- 39. excella.com | @excellaco Tracking over time 10/27/18 12/16/18 2/4/19 3/26/19 5/15/19 7/4/19 8/23/19 8/18/18 9/7/18 9/27/18 10/17/18 11/6/18 11/26/18 12/16/18 1/5/19 forecastcompletedate date of simulation forecast evolution 50% confidence 80% confidence 95% confidence
- 40. excella.com | @excellaco Tracking over time
- 41. excella.com | @excellaco Limitations and how to work with them
- 42. excella.com | @excellaco The usual suspects Planning is guessing • Don’t mistake math for certainty • Even if you’re 80% sure, you’re wrong 1 in 5 times You need historical data • Take a few weeks to track if you need to • A “gut” estimate may get you over this hump
- 43. excella.com | @excellaco The past dictates the future Will work be completed at the same rate? • Keep the team clear of impediments • Keep team membership consistent Will work items be the same size? • Keep an eye on story splitting and estimation • Discuss when work items are unusually large
- 44. excella.com | @excellaco What are we building? Does the backlog we’re simulating adequately represent the project? • Build in a buffer of nice-to-haves so you have features to trade out if you need to • Adjust your projected backlog using historical data
- 46. excella.com | @excellaco Use the Monte Carlo technique to predict the size of your backlog: • Create epics for the entire product • Break a handful into stories • Use Monte Carlo to model story counts for the others Use Monte Carlo again to simulate the simulated backlog Double Monte Carlo
- 47. excella.com | @excellaco • Estimate the PBIs you can complete by a date, not the date to complete the PBIs • Choose simulated times until you hit your forecast date • Useful for fixed timelines Fixed-Date Monte Carlo
- 48. excella.com | @excellaco Other sources of variability
- 49. excella.com | @excellaco Takeaways • Understand all possibilities • Validates Agile concepts • Replaces a data point with a conversation
- 50. How to connect hunter.tamaro@excella.com @htammaro https://linkedin.com/in/wmhunter Excella Consulting @excellaconsulting Connect With Us! @excellaco
Editor's Notes
- Ref Adam Yuret’s talk
- Average velocity per sprint? Average throughput per sprint? Average takt time? Others…?
- Plan releases and related activities Plan when to take on future projects Project budgeting and staffing decisions Others…?
- Wanted to know when to go live to help roll out public engagement We needed a forecast!
- Probably not! Our estimates that far out are really bad, and only gives you one day. What are the odds you deliver on that day?
- Centuries old, but took off in 1940s as computers came into use One of the first Monte Carlo algorithms ran on the ENIAC! Especially useful for modeling random processes or those with high uncertainty
- Examples on individual slides: weather forecast, retirement “It’s hard to make predictions, especially about the future” – Yogi Berra (?)
- Weather forecasting uses Monte Carlo
- Financial planning uses Monte Carlo
- Here is a Monte Carlo forecast being used to track Godzilla.
- Let’s step through the process!
- We’re predicting a date, so we’ll use takt time, but we’ll look at other possibilities later Cycle time looks at time to complete an individual item, not the rate at which the team works through the backlog Takt times roll up a lot of other factors in variability – impediments, vacations, etc
- For software teams this is probably in weekdays
- Emphasize: we are using takt times, not point estimates or other – that’s rolled up in time
- How many times? Hard to say, because it depends on your inputs and the size of the backlog you’re trying to simulate, but in general, more runs will get you more precision in your results. There are a few approaches and even academic papers that attempt to find a formula for the ideal number of runs. But nowadays, computers are fast enough that you can just add more and more runs until you get a reasonable-looking output.
- Percentages are cumulative
- The graph is a good way to tell if you have enough simulations – this one has a pretty smooth graph with a nice recognizable bell curve. If it’s jumping all over the place, that’s a tell that you need more simulations. Grouping the outputs like we have here – rather than single days – is an easy way to smooth out a curve without having to add simulations.
- Point of this activity: make it very simple, to demystify it; make it clear how it works, so you know how to adapt it yourself; make it useable on a phone
- Tripling the number of inputs doesn’t really affect the outcome! From Adam Yuret’s talk on Monday: you have a decent sense of the range of inputs after just seven samples.
- Broader range of inputs leads to broader range of outputs – obvious, but stresses the value of consistent sizing for enhancing predictability
- Adding just one outlier is enough to push out your median timeline, but also adds a “long tail,” making it take longer and longer to add certainty
- Need to start this section by about 3:38 pm (10-12 mins left)
- Turned out the devs’ gut estimate lined up with the “most likely” date, but with just a 50/50 chance of making it Accepted that internally, but communicated the 80% date externally
- Not just when we were releasing, but what to include – and what other factors to consider. If this is the date we’re feature complete, what else do we need to do? How much overlap? Validates the “gut check” from the devs
- Watch the forecast change over time – going up or down as the team gets faster or slower, or as the backlog evolves; cone of possibility narrowing as the team gets more reliable, or as real time closes in on the prediction
- Another view
- Will the team keep working at roughly the same rate? Will PBIs be broken down to roughly the same size? No PO so longer stories; shifts after she starts and we start doing feedback New Agile teams not as good at breaking things down, but get better In both cases, use recent data, not necessarily more, to make sure it’s accurate
- Mixture of refining stories and scope expansion
- If you don’t have a well-developed backlog, or any backlog, use Double Monte Carlo to forecast what the ultimate size will be
- Useful for fixed-date events, like trade shows or prescheduled releases