You will be wrong by…• 3x-10x when assuming Normal distribution• 2.5x-5x when assuming Poisson distribution• 7x-20x if you use Shewhart’s method Heavy tail phenomena are not incomprehensible… but they cannot be understood with traditional statistical tools. Using the wrong tools is incomprehensible. ~ Roger Cooke and Daan Nieboer
Bad application of control chart Control is an illusion, you infantileegomaniac. Nobody knows whats gonna happen next: not on a freeway, not in an airplane, not inside our own bodies andcertainly not on a racetrack with 40 other infantile egomaniacs. ~Days of Thunder
Time in Process (TIP) ChartA good alternative to control chart
Collection• Perceived cost is high• Little need for explicit collection activities• Use a 1-question NPS survey for customer and employee satisfaction• Plenty to learn in passive data from ALM and other tools• How you use the tools will drive your use of metrics from them
Summary of how to make good metric choices• Start with outcomes and use ODIM to make metrics Data visualization is like choices. photography. Impact is a• Make sure your metrics are function of perspective, balanced so you don’t illumination, and focus. over-emphasize one at the ~Larry Maccherone cost of others.• Be careful in your analysis. The TIP chart is a good alternative to control chart. Troy’s approach is excellent for forecasting. We’ve shown that there are many out there that are not so good.• Consider collection costs. Get maximal value out of passively gathered data.
Key PointForecasts should bepresented with the right amount of uncertainty
Demo: Forecasting… PAIN POINTMy Boss “Needs” a Date…
In this demo• Basic Scrum and Kanban Modeling• How to build a simple model – SimML Modeling Language – Visual checking of models – Forecasting Date and Cost – The “Law of Large Numbers”
Demo: Finding WhatMatters MostCost of Defects & Staff Analysis
Sensitivity ReportActively Ignore for theManage moment
Staff Skill Impact Report Explore what staff changes have the greatest impact
Key Point Modeling helpsfind what matters Fewer estimates required
In this demo• Finding what matters most – Manual experiments – Sensitivity Testing• Finding the next best 3 staff skill hires• Minimizing and simplifying estimation – Grouping backlog – Range Estimates – Deleting un-important model elements
Demo: Finding the Cost /Benefit of Outsourcing
Outsourcing Cost & Benefits• Outsourcing often controversial – Often fails when pursued for cost savings alone – Doesn’t always reduce local employment – An important tool to remain competitive – I.Q. has no geographic boundaries• Many models – Entire project – Augmentation of local team
Build Date & Cost Matrix 1x 1.5 x 2x Estimates Estimates Estimates1 x Staff Best Case1.5 x Staff Midpoint2 x Staff Worst CaseBenefit = (Baseline Dev Cost – New Dev Cost) - Cost of Delay + Local Staff Cost Savings
NOT LINEAR & NOT YOUR PROJECT$150,000$100,000 $50,000 1x Multiplier $- 1.5x Multiplier 1 1.5 2 2x Multiplier $(50,000)$(100,000)$(150,000)
In this demo• Model the impact of various outsourcing models
New Project Rules of Thumb…• Cost of Delay plays a significant role – High cost of delay project poor candidates – Increase staffing some compensation• Knowledge transfer and ramp-up time critical – Complex products poor candidates – Captive teams better choices for these projects• NEVER as simple as direct lower costs!
Speaking Risk To Executives• Buy them a copy of “Flaw of Averages”• Show them you are tracking & managing risk• Do – “We are 95% certain of hitting date x” – “With 1 week of analysis, that may drop to date y” – “We identified risk x, y & z that we will track weekly”• Don’t – Give them a date without likelihood • “February 29th 2013” – Give them a date without risk factors considered • “To do the backlog of features, February 29th, 2013”
**Major risk events have the predominate role in deciding where deliver actually occurs ** We spend all ourtime estimating here Plan Performance External Vendor Issues Delay 1 2 3
Key Points• There is no single release date forecast• Never use Average as a quoted forecast• Risk factors play a major role (not just backlog)• Data has shape: beware of Non-Normal data• Measurement → Insight → Decisions → Outcomes : Work Backwards!• Communicate Risk early with executive peers
Call to action• Read these books• Download the software FocusedObjective.com• Follow @AgileSimulation• Follow @LMaccherone
Please Submit an Eval Form! We want to learn too!