Modeling, Simulation & DataMining: Answering Tough Cost, Date    & Staff Forecasts Questions   Troy Magennis (Focused Obje...
Pain PointMy Boss“Needs”A Date…
GettingQuantitative Evidence
Assessing & CommunicatingRisk / Uncertainty
My Mission Arm my teams (and yours)with the tools and techniques   to solve these problems
2 Minutes About Larry• Larry is a Pisces who enjoys skiing, reading  and wine (red, or white in outdoor setting)• We have ...
Metrics &Measurement
Why measure?    Feedback   Diagnostics   Forecasting     Lever
When to NOT take a shotGood players?• Monta Ellis  – 9th highest scorer (8th    last season)• Carmelo Anthony  (Melo)  – 8...
The ODIM Frameworkbetter Measurement  better Insight    better Decisions       better Outcomes
What is normal?Cumulative -> 0.1%   2.3%   15.9%   50.0%   84.1%   97.7%   99.9%Roughly ->                                ...
Are you normal?
You will be wrong by…• 3x-10x when assuming Normal distribution• 2.5x-5x when assuming Poisson distribution• 7x-20x if you...
Bad application of control chart    Control is an illusion, you infantileegomaniac. Nobody knows whats gonna happen next: ...
Time in Process (TIP) ChartA good alternative to control chart
Collection• Perceived cost  is high• Little need for  explicit collection  activities• Use a 1-question NPS survey for cus...
Summary of how to make good           metric choices• Start with outcomes and  use ODIM to make metrics         Data visua...
Flaw of Averages, Risk  & Monte Carlo Sim
A model is a tool used to mimic areal world process  A tool for low-cost   experimentation
Monte Carlo Simulation?Performing a simulation of a model multiple times usingrandom input conditions and recording the fr...
Scrum                                        Run   Sim TotalBacklog     This Iteration   Deployed         Iterations      ...
Kanban                                                            Run Time TotalBacklog   Design       Develop        Test...
Result versus Frequency (50 runs)                                                                     More Often          ...
Result versus Frequency (250 runs)                                                                     More Often         ...
Result versus Frequency (1000+ runs)                                                                     More Often       ...
Key PointThere is NO single forecast result   There will always be manypossible results, some more likely
50%                50%             Possible           PossibleLikelihood             Outcomes           Outcomes          ...
95% Outcomes               5% Likelihood              Time to Complete BacklogMonte Carlo Simulation Yields More  Informat...
Key Point  “Average” isNEVER an optionWARNING: Regression lines are most often “average”
But, I.T. gets worse
Time to DeliveryPromised           New                  Average                                         50% Possible      ...
Key PointRisks play a BIGrole in forecasts  Yes, more than backlog.
Velocity is NOT Linear nor is defect rate, scope-creep, storyexpertise requirements, team skill, etc.
Likelihood (0-100%)        Date for likelihood
Key PointForecasts should bepresented with the  right amount of     uncertainty
Demo: Forecasting… PAIN POINTMy Boss “Needs” a Date…
In this demo• Basic Scrum and Kanban Modeling• How to build a simple model  – SimML Modeling Language  – Visual checking o...
Demo: Finding WhatMatters MostCost of Defects & Staff Analysis
Sensitivity ReportActively            Ignore for theManage                   moment
Staff Skill Impact Report              Explore what staff              changes have the              greatest impact
Key Point  Modeling helpsfind what matters  Fewer estimates required
In this demo• Finding what matters most  – Manual experiments  – Sensitivity Testing• Finding the next best 3 staff skill ...
Demo: Finding the Cost /Benefit of Outsourcing
Outsourcing Cost & Benefits• Outsourcing often controversial  – Often fails when pursued for cost savings alone  – Doesn’t...
Build Date & Cost Matrix               1x        1.5 x     2x               Estimates Estimates Estimates1 x Staff       B...
NOT LINEAR & NOT YOUR PROJECT$150,000$100,000  $50,000                              1x Multiplier      $-                 ...
In this demo• Model the impact of various outsourcing  models
New Project Rules of Thumb…• Cost of Delay plays a significant role  – High cost of delay project poor candidates  – Incre...
Assessing andCommunicating Risk
Speaking Risk To Executives• Buy them a copy of “Flaw of Averages”• Show them you are tracking & managing risk• Do  – “We ...
**Major risk events have the predominate role       in deciding where deliver actually occurs **  We spend all ourtime est...
Risk likelihood changes constantly                              95th                          Confidence                  ...
Risk likelihood changes constantly                          95th                      Confidence                        In...
Risk likelihood changes constantly                        95th                    Confidence                      Interval...
Risk likelihood changes constantly                    95th                Confidence                  Interval      1     ...
Key Points• There is no single release date forecast• Never use Average as a quoted forecast• Risk factors play a major ro...
Call to action• Read these books• Download the software FocusedObjective.com• Follow @AgileSimulation• Follow @LMaccherone
Please Submit an   Eval Form! We want to learn too!
BEST PRACTICES
Sensitivity            Model   Test               (a little)          The Model           Creation            Cycle Monte-...
Make Informed           BaselineDecision(s)               The           Experiment              Cycle                     ...
Best Practice 1 Start simple and add ONE input condition at a time. Visually / Monte-carlo testeach input to verify it works
Best Practice 2 Find the likelihood of major  events and estimate delay  E.g. vendor dependencies,performance/memory issue...
Best Practice 3Only obtain and add detailed estimates and opinion to amodel if Sensitivity Analysis says that input is mat...
Best Practice 4Use a uniform random inputdistribution UNTIL sensitivity  analysis says that input is    influencing the ou...
Best Practice 5   Educate your managers’about risk. They will still want a “single” date for planning,  but let them decid...
SIMULATION EXAMPLESReturn to main presentation…
unlikely   Forecasts   Return to main presentation…certain
unlikely     Forecasts    Return to main presentation…           50%             50%           Possible    Possible       ...
Return to main presentation…    Sensitivity ReportActively             Ignore for theManage                    moment
Return to main presentation…Staff Skill Impact Report              Explore what staff              changes have the       ...
Return to mainpresentation…
Focused Objective•   Risk Tools for Software Dev•   Scrum/Agile Simulation•   Kanban/Lean Simulation•   Forecasting Staff,...
We Use & Recommend: EasyFit• MathWave.com• Invaluable for  – Analyzing data  – Fitting Distributions  – Generating Random ...
Free Images: MorgueFile.com• http://www.morguefile.com/corporate/about  – Calendar: http://www.opticgroove.com.au  – Calcu...
Agile Metrics - Modeling, Simulation, and Data Mining
Agile Metrics - Modeling, Simulation, and Data Mining
Agile Metrics - Modeling, Simulation, and Data Mining
Agile Metrics - Modeling, Simulation, and Data Mining
Upcoming SlideShare
Loading in...5
×

Agile Metrics - Modeling, Simulation, and Data Mining

2,540

Published on

Modeling, Simulation & Data Mining: Answering Tough Cost, Date & Staff Forecasts Questions

Published in: Technology
1 Comment
8 Likes
Statistics
Notes
No Downloads
Views
Total Views
2,540
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
57
Comments
1
Likes
8
Embeds 0
No embeds

No notes for slide

Agile Metrics - Modeling, Simulation, and Data Mining

  1. 1. Modeling, Simulation & DataMining: Answering Tough Cost, Date & Staff Forecasts Questions Troy Magennis (Focused Objective) Larry Maccherone (Rally)
  2. 2. Pain PointMy Boss“Needs”A Date…
  3. 3. GettingQuantitative Evidence
  4. 4. Assessing & CommunicatingRisk / Uncertainty
  5. 5. My Mission Arm my teams (and yours)with the tools and techniques to solve these problems
  6. 6. 2 Minutes About Larry• Larry is a Pisces who enjoys skiing, reading and wine (red, or white in outdoor setting)• We have a lot in common… over to Larry!
  7. 7. Metrics &Measurement
  8. 8. Why measure? Feedback Diagnostics Forecasting Lever
  9. 9. When to NOT take a shotGood players?• Monta Ellis – 9th highest scorer (8th last season)• Carmelo Anthony (Melo) – 8th highest scorer (3rd last season)
  10. 10. The ODIM Frameworkbetter Measurement better Insight better Decisions better Outcomes
  11. 11. What is normal?Cumulative -> 0.1% 2.3% 15.9% 50.0% 84.1% 97.7% 99.9%Roughly -> 85% 98%
  12. 12. Are you normal?
  13. 13. You will be wrong by…• 3x-10x when assuming Normal distribution• 2.5x-5x when assuming Poisson distribution• 7x-20x if you use Shewhart’s method Heavy tail phenomena are not incomprehensible… but they cannot be understood with traditional statistical tools. Using the wrong tools is incomprehensible. ~ Roger Cooke and Daan Nieboer
  14. 14. Bad application of control chart Control is an illusion, you infantileegomaniac. Nobody knows whats gonna happen next: not on a freeway, not in an airplane, not inside our own bodies andcertainly not on a racetrack with 40 other infantile egomaniacs. ~Days of Thunder
  15. 15. Time in Process (TIP) ChartA good alternative to control chart
  16. 16. Collection• Perceived cost is high• Little need for explicit collection activities• Use a 1-question NPS survey for customer and employee satisfaction• Plenty to learn in passive data from ALM and other tools• How you use the tools will drive your use of metrics from them
  17. 17. Summary of how to make good metric choices• Start with outcomes and use ODIM to make metrics Data visualization is like choices. photography. Impact is a• Make sure your metrics are function of perspective, balanced so you don’t illumination, and focus. over-emphasize one at the ~Larry Maccherone cost of others.• Be careful in your analysis. The TIP chart is a good alternative to control chart. Troy’s approach is excellent for forecasting. We’ve shown that there are many out there that are not so good.• Consider collection costs. Get maximal value out of passively gathered data.
  18. 18. Flaw of Averages, Risk & Monte Carlo Sim
  19. 19. A model is a tool used to mimic areal world process A tool for low-cost experimentation
  20. 20. Monte Carlo Simulation?Performing a simulation of a model multiple times usingrandom input conditions and recording the frequency of each result occurrence
  21. 21. Scrum Run Sim TotalBacklog This Iteration Deployed Iterations 1 3 2 2 3 55 2 4 3 5 4 6 2 8 … …
  22. 22. Kanban Run Time TotalBacklog Design Develop Test 1 – 2 days 1 – 5 days 1 – 2 days Deployed 1 5 2 4 3 3 4 9 2 5 5 6 6 … …
  23. 23. Result versus Frequency (50 runs) More Often 25Frequency of Result 20 15 10 5 1 10 15 20 Less Often Result Values – For example, Days
  24. 24. Result versus Frequency (250 runs) More Often 25Frequency of Result 20 15 10 5 1 10 15 20 Less Often Result Values – For example, Days
  25. 25. Result versus Frequency (1000+ runs) More Often 25Frequency of Result 20 15 10 5 1 10 15 20 Less Often Result Values – For example, Days
  26. 26. Key PointThere is NO single forecast result There will always be manypossible results, some more likely
  27. 27. 50% 50% Possible PossibleLikelihood Outcomes Outcomes Time to Complete BacklogWhen pressed for a single number, we often give the average.
  28. 28. 95% Outcomes 5% Likelihood Time to Complete BacklogMonte Carlo Simulation Yields More Information – 95% Common.
  29. 29. Key Point “Average” isNEVER an optionWARNING: Regression lines are most often “average”
  30. 30. But, I.T. gets worse
  31. 31. Time to DeliveryPromised New Average 50% Possible Outcomes Likelihood Planned Perf. Vendor Backlog Issues Delay 1 2 3
  32. 32. Key PointRisks play a BIGrole in forecasts Yes, more than backlog.
  33. 33. Velocity is NOT Linear nor is defect rate, scope-creep, storyexpertise requirements, team skill, etc.
  34. 34. Likelihood (0-100%) Date for likelihood
  35. 35. Key PointForecasts should bepresented with the right amount of uncertainty
  36. 36. Demo: Forecasting… PAIN POINTMy Boss “Needs” a Date…
  37. 37. In this demo• Basic Scrum and Kanban Modeling• How to build a simple model – SimML Modeling Language – Visual checking of models – Forecasting Date and Cost – The “Law of Large Numbers”
  38. 38. Demo: Finding WhatMatters MostCost of Defects & Staff Analysis
  39. 39. Sensitivity ReportActively Ignore for theManage moment
  40. 40. Staff Skill Impact Report Explore what staff changes have the greatest impact
  41. 41. Key Point Modeling helpsfind what matters Fewer estimates required
  42. 42. In this demo• Finding what matters most – Manual experiments – Sensitivity Testing• Finding the next best 3 staff skill hires• Minimizing and simplifying estimation – Grouping backlog – Range Estimates – Deleting un-important model elements
  43. 43. Demo: Finding the Cost /Benefit of Outsourcing
  44. 44. Outsourcing Cost & Benefits• Outsourcing often controversial – Often fails when pursued for cost savings alone – Doesn’t always reduce local employment – An important tool to remain competitive – I.Q. has no geographic boundaries• Many models – Entire project – Augmentation of local team
  45. 45. Build Date & Cost Matrix 1x 1.5 x 2x Estimates Estimates Estimates1 x Staff Best Case1.5 x Staff Midpoint2 x Staff Worst CaseBenefit = (Baseline Dev Cost – New Dev Cost) - Cost of Delay + Local Staff Cost Savings
  46. 46. NOT LINEAR & NOT YOUR PROJECT$150,000$100,000 $50,000 1x Multiplier $- 1.5x Multiplier 1 1.5 2 2x Multiplier $(50,000)$(100,000)$(150,000)
  47. 47. In this demo• Model the impact of various outsourcing models
  48. 48. New Project Rules of Thumb…• Cost of Delay plays a significant role – High cost of delay project poor candidates – Increase staffing some compensation• Knowledge transfer and ramp-up time critical – Complex products poor candidates – Captive teams better choices for these projects• NEVER as simple as direct lower costs!
  49. 49. Assessing andCommunicating Risk
  50. 50. Speaking Risk To Executives• Buy them a copy of “Flaw of Averages”• Show them you are tracking & managing risk• Do – “We are 95% certain of hitting date x” – “With 1 week of analysis, that may drop to date y” – “We identified risk x, y & z that we will track weekly”• Don’t – Give them a date without likelihood • “February 29th 2013” – Give them a date without risk factors considered • “To do the backlog of features, February 29th, 2013”
  51. 51. **Major risk events have the predominate role in deciding where deliver actually occurs ** We spend all ourtime estimating here Plan Performance External Vendor Issues Delay 1 2 3
  52. 52. Risk likelihood changes constantly 95th Confidence Interval 1 2 3
  53. 53. Risk likelihood changes constantly 95th Confidence Interval 1 2 3
  54. 54. Risk likelihood changes constantly 95th Confidence Interval 1 2 3
  55. 55. Risk likelihood changes constantly 95th Confidence Interval 1 2 3
  56. 56. Key Points• There is no single release date forecast• Never use Average as a quoted forecast• Risk factors play a major role (not just backlog)• Data has shape: beware of Non-Normal data• Measurement → Insight → Decisions → Outcomes : Work Backwards!• Communicate Risk early with executive peers
  57. 57. Call to action• Read these books• Download the software FocusedObjective.com• Follow @AgileSimulation• Follow @LMaccherone
  58. 58. Please Submit an Eval Form! We want to learn too!
  59. 59. BEST PRACTICES
  60. 60. Sensitivity Model Test (a little) The Model Creation Cycle Monte- VisuallyCarlo Test Test
  61. 61. Make Informed BaselineDecision(s) The Experiment Cycle MakeCompare Single Results Change
  62. 62. Best Practice 1 Start simple and add ONE input condition at a time. Visually / Monte-carlo testeach input to verify it works
  63. 63. Best Practice 2 Find the likelihood of major events and estimate delay E.g. vendor dependencies,performance/memory issues, third party component failures.
  64. 64. Best Practice 3Only obtain and add detailed estimates and opinion to amodel if Sensitivity Analysis says that input is material
  65. 65. Best Practice 4Use a uniform random inputdistribution UNTIL sensitivity analysis says that input is influencing the output
  66. 66. Best Practice 5 Educate your managers’about risk. They will still want a “single” date for planning, but let them decide 75 th or 95 th confidence level(average is NEVER an option)
  67. 67. SIMULATION EXAMPLESReturn to main presentation…
  68. 68. unlikely Forecasts Return to main presentation…certain
  69. 69. unlikely Forecasts Return to main presentation… 50% 50% Possible Possible Outcomes Outcomescertain
  70. 70. Return to main presentation… Sensitivity ReportActively Ignore for theManage moment
  71. 71. Return to main presentation…Staff Skill Impact Report Explore what staff changes have the greatest impact
  72. 72. Return to mainpresentation…
  73. 73. Focused Objective• Risk Tools for Software Dev• Scrum/Agile Simulation• Kanban/Lean Simulation• Forecasting Staff, Date & Cost• Automated Sensitivity Analysis• Data Reverse Engineering• Consulting / Training• Book
  74. 74. We Use & Recommend: EasyFit• MathWave.com• Invaluable for – Analyzing data – Fitting Distributions – Generating Random Numbers – Determining Percentiles
  75. 75. Free Images: MorgueFile.com• http://www.morguefile.com/corporate/about – Calendar: http://www.opticgroove.com.au – Calculator: http://www.therising-sun.us – Dice: souldestine@cox.net
  1. Gostou de algum slide específico?

    Recortar slides é uma maneira fácil de colecionar informações para acessar mais tarde.

×