Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

LKNA 2014 Risk and Impediment Analysis and Analytics - Troy Magennis

992 views

Published on

Software risk impact is more predictable than you might think. This session discusses similarities of uncertainty in various industries and relates this back to how we can measure and analyze impediments and risk for agile software teams.

Published in: Business, Technology
  • Be the first to comment

LKNA 2014 Risk and Impediment Analysis and Analytics - Troy Magennis

  1. 1. Risk, Options and Cost of Delay Troy Magennis LKNA 2014 San Francisco. May 2014
  2. 2. risk events 1 2 3 Performance AND Vendor Delay Performance OR Vendor Delay Nothing Goes Wrong Time Probability
  3. 3. Definition: Risk The impact of uncertainty on an outcome
  4. 4. Technical Risk Financial Risk Market Risk • Real Options • Right Staff / liquidity • Dev Practices • Dependencies • Constraints • Lean Startup • Agile Processes • Competitive Awareness • Having funding/cash • Having a strategy • Economic prioritization • Real Options “Aleatory Risk” Cannot be reduce by more info
  5. 5. Delay (Technical Risk) Low Adoption (Market Risk) Low Cashflow (Financial Risk) Less Resources (Financial Risk) Risk Positive Feedback Loop
  6. 6. Key Point Occurrence of a risk Increases exposure to other risks Break the chain early AKA: Early and meaningful contact with enemy – RISK (source: quote from Reinertsen, but sources from US marines?)
  7. 7. Correlation != Causation We can see average flight delay matches the shape of “Late Aircraft,” but don’t yet know why…
  8. 8. Key Point Serialized dependencies cascade delays, but are not the root cause – Why was the aircraft late? The later you are, the later you get.
  9. 9. Four people arrange a restaurant booking after work Q. What is the chance they arrive on-time to be seated?
  10. 10. Commercial in confidence Person 1 Person 2 Person 3 Person 4 1in16EVERYONEisON-TIME 15TIMESmorelikelyatleastonpersonislate
  11. 11. 1 2 3 4 5 6 7 Team Dependency Diagram
  12. 12. 1 in 2n or 1 in 27 or 1 in 128
  13. 13. 7 dependencies 1 chance in 128
  14. 14. 6 dependencies 1 chance in 64
  15. 15. 5 dependencies 1 chance in 32
  16. 16. Key Point Risk of being impacted decreases by half for every risk vector/factor removed But, not all risks have the same likelihood (or impact)…
  17. 17. Frequency Recency Impact
  18. 18. If you haven’t seen an event after testing for it n times, you can be 95% sure that its probability of happening is less than 3/n References: Wikipedia: Statistical Rule of Three and Thanks to John Cook: Estimating the chances of something that hasn’t happened yet, http://www.johndcook.com/blog/2010/03/30/statistical-rule-of-three/ The Math: (1-p)n = 0.05 for p. Taking logs of both sides, n ln (1-p) = ln(0.05) ≈ -3. Since log(1-p) is approximately -p for small values of p, we have p ≈ 3/n.
  19. 19. Statistical Rule of Three • Example: Proofreading a book, you find no grammatical errors in n pages • Error decreases as a proportion to the number of independent test cases examined • It hard to be independent! n percentage 20 15% (3/20) 100 3% (3/100) 200 1.5% (3/200) 500 0.6% (3/500) 1000 0.3% (3/1000) 0.00000 0.10000 0.20000 0.30000 0.40000 0.50000 0.60000 0.70000 0.80000 1 21 41 61 81 101 121 141 161 181 201 221 241 261 281 301 321 341 361 381 401 421 441 461 481 p
  20. 20. ‘s Absence of Evidence isn’t Evidence of Absence But, it does demonstrate the occurrence is rare with growing certainty Depends on consequence…. Ps. The most common Black Swan is project on-time delivery!
  21. 21. CONSEQUENCE MATTERS
  22. 22. Capture Actual Impacts Calculate “Impact” Order from highest to lowest Discuss, Root cause Top 10 Prioritize Sum of Days impacted for 3 last months Sum of Days impacted for 3 last months Category Start End
  23. 23. “Value” Cost of Delay Product 1 Product 2 Product 3 Complete Order? 3 2 1 “Time” Remaining Time/Effort to solve Economic Prioritization – same time, different value
  24. 24. Product 1 Product 2 Product 3 1 2 3 Economic Prioritization – same value different time “Value” Cost of Delay Complete Order? “Time” Remaining Time/Effort to solve
  25. 25. W.S.R.F. = Prioritization Heuristic to optimize reward “Do Highest First” Impact of risk Time to resolve/mitigate Weighted Shortest Risk First Sum of delay time of same risk causes over the last 3 (?) months Effort estimate of the resolution time of risk root cause
  26. 26. All Sheep in Scotland Are Black • A psychologist, a biologist, a mathematician, and a physicist were riding a train through the Scottish countryside. Looking out the window, they all noticed a lone black sheep on a hill. • The psychologist intoned, “Well, what do you know. I didn’t realize the sheep in Scotland were black.” • The biologist corrected him, saying, “You don’t know that all the sheep in Scotland are black – just some of them.” • Piping in, the mathematician retorted, “Tut, tut, tut, to be correct you must say, ‘At least one’ sheep in Scotland is black.” • The physicist had the last word, though, stating, “Gentlemen, all we know with certainty based on our observations is that at least one sheep in Scotland is black on at least one side, at least part of the time.” • Moral: There are hard and soft sciences, and extrapolation is not always justified. http://creationsafaris.com/humor.htm
  27. 27. Total Story Lead Time 30 days Story / Feature Inception 5 Days Waiting in Backlog 25 days System Regression Testing & Staging 5 Days Waiting for Release Window 5 Days “Active Development” 30 days Pre Work 30 days Post Work 10 days 9 days (70 total) approx 13%
  28. 28. THE SHAPE OF CYCLE TIME What distribution fits cycle time data and why…
  29. 29. If we understand how cycle time is statistically distributed, then an initial guess of maximum allows an inference to be made Alternatives - • Borrow a similar project’s data • Borrow industry data • Fake it until you make it… (AKA guess range)
  30. 30. Why Weibull • Now for some Math – I know, I’m excited too! • Simple Model • All units of work between 1 and 3 days • A unit of work can be a task, story, feature, project • Base Scope of 50 units of work – Always Normal • 5 Delays / Risks, each with – 25% Likelihood of occurring – 10 units of work (same as 20% scope increase each)
  31. 31. Normal, or it will be after a few thousand more simulations
  32. 32. Base + 1 Delay
  33. 33. Base + 2 Delays
  34. 34. Base + 3 Delays
  35. 35. Base + 4 Delays
  36. 36. Base + 5 Delays
  37. 37. Exponential Distribution (Weibull shape = 1) The person who gets the work can complete the work Teams with no external dependencies Teams doing repetitive work E.g. DevOps, Database teams,
  38. 38. Weibull Distribution (shape = 1.5) Typical dev team ranges between 1.2 and 1.8
  39. 39. Rayleigh Distribution (Weibull shape = 2) Teams with MANY external dependencies Teams that have many delays and re-work. E.g. Test teams
  40. 40. What Distribution To Use... • No Data at All, or Less than < 11 Samples (why 11?) – Uniform Range with Boundaries Guessed (safest) – Weibull Range with Boundaries Guessed (likely) • 11 to 30 Samples – Uniform Range with Boundaries at 5th and 95th CI – Weibull Range with Boundaries at 5th and 95th CI • More than 30 Samples – Use historical data as bootstrap reference – Curve Fitting software
  41. 41. Probability Density Function Histogram Weibull x 1201101009080706050403020100 f(x) 0.28 0.24 0.2 0.16 0.12 0.08 0.04 0 Scale – How Wide in Range. Related to the Upper Bound. *Rough* Guess: (High – Low) / 4 Shape – How Fat the distribution. 1.5 is a good starting point. Location – The Lower Bound

×