Download to read offline
![Safearea–nographicshere
Safearea–nographicshere
The server team has identified work they would like to
complete, the task owners, and preliminary estimates
Analytics
capability
Story 1
Mobile
support
Story 2
Owner:
John
Task 1.1
[1w,2w,3w]
Owner:
John
Task 1.2
[2w,5w,7w]
Owner:
Alice
Task 1.3
[3w,7w,11w]
Owner:
John
Task 2.1
[1w,1.5w,3w]
Owner:
Claire
Task 2.2
[2d,1w,6w]
Owner:
Claire
Task 2.3
[1w,2w,3w]
The team needs to finish its work by
the integration milestone.
Is it a good bet that it will?
Task 1.1.7
2d
5d
6w](https://image.slidesharecdn.com/reactor-roycecantorv2-16-9-130911165945-phpapp02/85/Reactor-royce-cantor-v2-16-9-13-320.jpg)
![Safearea–nographicshere
Safearea–nographicshere
Once we trust the data, we can ask what it says about the
outcome. But there are many different possible schedules,
leading to different outcomes
Valid schedules come
from
– Task dependencies
– Implied orderings due to
owners
Actual schedules
depend on when owners
finish tasks (uncertain)
– NB: This is not critical path
analysis
Schedule 1
John
Claire
Alice
Task
1.1[1w,2w,3w]
Task
1.2[2w,5w,7w]
Task
2.1[1w,1.5w,3w]
Task
2.3[1w,2w,3w]
Task
2.2[2d,1w,6w]
Task
1.3[3w,7w,11w]
Schedule 2
John
Claire
Alice
Task
1.1[1w,2w,3w]
Task
1.2[2w,5w,7w]
Task
2.1[1w,1.5w,3w]
Task
2.3[1w,2w,3w]
Task
2.2[2d,1w,6w]
Task
1.3[3w,7w,11w]](https://image.slidesharecdn.com/reactor-roycecantorv2-16-9-130911165945-phpapp02/85/Reactor-royce-cantor-v2-16-9-16-320.jpg)
![Safearea–nographicshere
Safearea–nographicshere
There are many different possible outcomes, depending
on the actual schedule and the time the tasks take
Schedule 1
John
Claire
Alice
Task
1.1[1w,2w,3w]
Task
1.2[2w,5w,7w]
Task
2.1[1w,1.5w,3w]
Task
2.3[1w,2w,3w]
Task
2.2[2d,1w,6w]
Task
1.3[3w,7w,11w]
Schedule 2
John
Claire
Alice
Task
1.1[1w,2w,3w]
Task
1.2[2w,5w,7w]
Task
2.1[1w,1.5w,3w]
Task
2.3[1w,2w,3w]
Task
2.2[2d,1w,6w]
Task
1.3[3w,7w,11w]
What are the possible outcomes?
How likely are they?
Is this set of work a good bet?](https://image.slidesharecdn.com/reactor-roycecantorv2-16-9-130911165945-phpapp02/85/Reactor-royce-cantor-v2-16-9-17-320.jpg)
![Safearea–nographicshere
Safearea–nographicshere
Is the server team likely to finish their planned work by 2/13?
[3w,7w,11w]
Task 1.3
Alice
John
Claire
[2w,5w,7w]
Task 1.2
[1w,1.5w,3w]
Task 2.1
[2w,5w,7w]
Task 2.2
[1w,2w,3w]
Task 1.1
[1w,2w,3w]
Task 2.3
Sim 1 8d
Task 2.1
10d
Task 2.3
10d
Task 1.1
10d
Task 2.2
35d
Task 1.3
25d
Task 1.2
60d
Sim 2 5d
Task 2.1
5d
Task 2.3
5d
Task 1.1
14d
Task 2.2
15d
Task 1.3
10d
Task 1.2
54d
Sim 3 15d
Task 2.1
15d
Task 2.3
15d
Task 1.1
35d
Task 2.2
55d
Task 1.3
35d
Task 1.2
170d
10/1 1/1512/1
38% 62%
For each random variable (task
duration), sample a value from its
distribution
Different values will be sampled in each simulation,
leading to different outcomes](https://image.slidesharecdn.com/reactor-roycecantorv2-16-9-130911165945-phpapp02/85/Reactor-royce-cantor-v2-16-9-18-320.jpg)
![Safearea–nographicshere
Safearea–nographicshere
Techniques for Learning From Actuals
1. Leverage actual distribution
- The actuals form a distribution
- We can sample from this distribution during Monte Carlo
simulation
2. Predicting based on relationships between estimates and
actuals (regression)
– If estimates and actuals are strongly correlated, regression
analysis can be useful in predicting actuals from estimates
2. Predicting based on learned relationships between
multiple variables and actuals
3. Bayesian Networks
- After 50 tasks complete, we have new evidence:
• DB tasks: effort = [11d, 16d, 42d], type = 20%
• Analytics tasks: effort = [4d, 16d, 20d], type = 75%
• Mobile tasks: effort = [1d, 2d, 4d], type = 5%
- We can update the conditional probabilities to reflect this learned
evidence
21
Probability
Actual Task Effort
Node 0
n: 4591; %: 100.0
Predicted: 5.5 E8
Node 0
n: 4591; %: 100.0
Predicted: 5.5 E8
Effort
Node 1
n: 1911; %: 41.6
Predicted: 5.0 E8
Node 1
n: 1911; %: 41.6
Predicted: 5.0 E8
Node 2
n: 2680; %: 58.4
Predicted: 5.9 E8
Node 2
n: 2680; %: 58.4
Predicted: 5.9 E8
Attribute: Owner
Improvement: 1.8 E15
Alice; David John; Claire
Node 3
n: 832; %: 31.1
Predicted: 5.6 E8
Node 3
n: 832; %: 31.1
Predicted: 5.6 E8
Node 4
n: 1848; %: 68.9
Predicted: 6.1 E8
Node 4
n: 1848; %: 68.9
Predicted: 6.1 E8
Attribute: Owner
Improvement: 2.7 E14
Claire John
Task
duration
Task
duration
Task
owner
Task
owner
Task
type
Task
type
Task
expected
effort
Task
expected
effort
Qualificati
on for task
Qualificati
on for task](https://image.slidesharecdn.com/reactor-roycecantorv2-16-9-130911165945-phpapp02/85/Reactor-royce-cantor-v2-16-9-21-320.jpg)
Uploaded byMurray Cantor
Reactor royce, cantor v2-16-9
The document discusses using predictive analytics to better estimate software project outcomes and make better decisions. It describes a fictional case study of a team aiming to deliver new features by a deadline. Simulation of the team's estimated task durations shows only a 50% chance of on-time delivery. Continued monitoring as the project progresses provides new data, requiring prediction updates. The key lessons are to understand and reduce uncertainty to improve outcomes, which may require changing scope, resources or timeline. Predictive analytics can facilitate better-informed discussions when making high-stakes decisions.
Reactor royce, cantor v2-16-9
- 1. Safearea–nographicshere Safearea–nographicshere Steer with PredictiveAnalytics Less Surprises, More Insight Walker Royce Murray Cantor Chief Software Economist Distinguished Engineer IBM Software Group IBM Software Group @weroyceusibmcom @murraycantor weroyce@us.ibm.com mcantor@us.ibm.com 1
- 2. Safearea–nographicshere Safearea–nographicshere 2 Deliver business criticalsoftware product release within 12 months Challenge SCHEDULE 6 MONTHS 12 MONTHS 18 MONTHS
- 3. Safearea–nographicshere Safearea–nographicshere Jerry’s probable outcome… Reality 3 LATESCRAP & REWORK SCHEDULE 6 MONTHS 12 MONTHS 18 MONTHS 0% 100% PROGRESS INTEGRATION BEGINS
- 4. Safearea–nographicshere Safearea–nographicshere 4 Program parameters arealso delivery predictions COST, SCHEDULE, EFFORT, QUALITY… 6 MONTHS 12 MONTHS 18 MONTHS PROBABILITY Insight SCHEDULE
- 5. Safearea–nographicshere Safearea–nographicshere 5 There is roughlya 50% chance of making the date 6 MONTHS 12 MONTHS 18 MONTHS PROBABILITY Coin Flip SCHEDULE
- 6. Safearea–nographicshere Safearea–nographicshere 6 Move out thedate to improve likelihood of shipping? 6 MONTHS 12 MONTHS 18 MONTHS PROBABILITY Option 1 Absolutely NOT!Absolutely NOT! SCHEDULE
- 7. Safearea–nographicshere Safearea–nographicshere 7 Decrease time estimateby sacrificing quality or content? 6 MONTHS 12 MONTHS 18 MONTHS PROBABILITY Option 2 Absolutely NOT!Absolutely NOT! SCHEDULE
- 8. Safearea–nographicshere Safearea–nographicshere 8 Reduce the varianceto improve likelihood of shipping Optimize 6 MONTHS 12 MONTHS 18 MONTHS PROBABILITY SCHEDULE
- 9. Safearea–nographicshere Safearea–nographicshere 9 Measure validated learning| Reduce the variance Uncertainty in stakeholder satisfaction Optimize Uncertainty in Plans, Scope and Design
- 10. Safearea–nographicshere Safearea–nographicshere 10 Economic Governance The moral ofthis story… Variance Reduction Smarter Analytics
- 11. Safearea–nographicshere Safearea–nographicshere A (Fictional) CaseStudy The Setting: Product planning for KillerApp V2 for US Treasury property auctions – KillerApp v1 met with unprecedented success – Since its release and success, many state governments have indicated interest in variants specific to their local laws and needs The Opportunity – Sales reports that they can close $1B in new business with a promise for features supporting state government auctions, including mobile, analytics, and social media support The Catch – An aggressive startup has plans to release a similar capacity in August, 2014 • The first to market will have a huge competitive advantage
- 12. Safearea–nographicshere Safearea–nographicshere KillerApp’s Product DevelopmentTeam Structure Alice, Team Lead Peter, Product Manager Joe, Team Lead 12 • The server team develops using hybrid process • Agile with a target date • They must manage technical risks carefully to meet challenging delivery target • Is their planned scope of work appropriate to their timeline, resources?
- 13. Safearea–nographicshere Safearea–nographicshere The server teamhas identified work they would like to complete, the task owners, and preliminary estimates Analytics capability Story 1 Mobile support Story 2 Owner: John Task 1.1 [1w,2w,3w] Owner: John Task 1.2 [2w,5w,7w] Owner: Alice Task 1.3 [3w,7w,11w] Owner: John Task 2.1 [1w,1.5w,3w] Owner: Claire Task 2.2 [2d,1w,6w] Owner: Claire Task 2.3 [1w,2w,3w] The team needs to finish its work by the integration milestone. Is it a good bet that it will? Task 1.1.7 2d 5d 6w
- 14. Safearea–nographicshere Safearea–nographicshere Is the serverteam making a good “bet”? The conversation begins... Where did these estimates come from? – Should we bet the business on this data? – Did tampering occur? User expertise – One developer’s experience – Multiple developers’ experiences Learning from prior history – This team’s prior history (Machine Learning) – Models that run over prior projects (e.g., COCOMO, SLIM, SEER) Peter, Product Manager Alice, Team Lead Predictive analytics facilitate critical conversations between stakeholders. To have value, they must be trustworthy and explainable.
- 15. Safearea–nographicshere Safearea–nographicshere Is the serverteam making a good “bet”? The conversation continues... Does it reflect realistic expectations? – What is causing the variance (uncertainty)? What actions can we take to reduce it? – Did you account for new work being identified in flight? Variance is coming from – Estimating large-scale pieces of work – Uncertainty around reusability of components 15 Peter, Product Manager Alice, Team Lead Predictive analytics have value only if the data have value. They are not magic. They require hard work to provide value.
- 16. Safearea–nographicshere Safearea–nographicshere Once we trustthe data, we can ask what it says about the outcome. But there are many different possible schedules, leading to different outcomes Valid schedules come from – Task dependencies – Implied orderings due to owners Actual schedules depend on when owners finish tasks (uncertain) – NB: This is not critical path analysis Schedule 1 John Claire Alice Task 1.1[1w,2w,3w] Task 1.2[2w,5w,7w] Task 2.1[1w,1.5w,3w] Task 2.3[1w,2w,3w] Task 2.2[2d,1w,6w] Task 1.3[3w,7w,11w] Schedule 2 John Claire Alice Task 1.1[1w,2w,3w] Task 1.2[2w,5w,7w] Task 2.1[1w,1.5w,3w] Task 2.3[1w,2w,3w] Task 2.2[2d,1w,6w] Task 1.3[3w,7w,11w]
- 17. Safearea–nographicshere Safearea–nographicshere There are manydifferent possible outcomes, depending on the actual schedule and the time the tasks take Schedule 1 John Claire Alice Task 1.1[1w,2w,3w] Task 1.2[2w,5w,7w] Task 2.1[1w,1.5w,3w] Task 2.3[1w,2w,3w] Task 2.2[2d,1w,6w] Task 1.3[3w,7w,11w] Schedule 2 John Claire Alice Task 1.1[1w,2w,3w] Task 1.2[2w,5w,7w] Task 2.1[1w,1.5w,3w] Task 2.3[1w,2w,3w] Task 2.2[2d,1w,6w] Task 1.3[3w,7w,11w] What are the possible outcomes? How likely are they? Is this set of work a good bet?
- 18. Safearea–nographicshere Safearea–nographicshere Is the serverteam likely to finish their planned work by 2/13? [3w,7w,11w] Task 1.3 Alice John Claire [2w,5w,7w] Task 1.2 [1w,1.5w,3w] Task 2.1 [2w,5w,7w] Task 2.2 [1w,2w,3w] Task 1.1 [1w,2w,3w] Task 2.3 Sim 1 8d Task 2.1 10d Task 2.3 10d Task 1.1 10d Task 2.2 35d Task 1.3 25d Task 1.2 60d Sim 2 5d Task 2.1 5d Task 2.3 5d Task 1.1 14d Task 2.2 15d Task 1.3 10d Task 1.2 54d Sim 3 15d Task 2.1 15d Task 2.3 15d Task 1.1 35d Task 2.2 55d Task 1.3 35d Task 1.2 170d 10/1 1/1512/1 38% 62% For each random variable (task duration), sample a value from its distribution Different values will be sampled in each simulation, leading to different outcomes
- 19. Safearea–nographicshere Safearea–nographicshere Server team hasbeen working for a month. Will they deliver on time? 2/13 Delivery 8/1 Start 9/1 Now PredictedDelivery DateRange Date of Prediction 2/13 1/1 3/1 2/1 4/1 • Best case: 1/1 • Most likely: 2/11 • Worst case: 2/28 • On-time delivery: 87% • Best case: ? • Most likely: ? • Worst case: ? • On-time delivery: ? 19 We have new learning: actuals, evidence of how the team is actually performing
- 20. Safearea–nographicshere Safearea–nographicshere Data mining canhelp us learn from the past to predict the future Data mining (aka machine learning, statistical inference) is about learning correlations between inputs and outcomes from sample data (actuals) – Classification prediction seeks to learn a model that predicts the classification of artifacts – Numeric prediction seeks to learn a model that predicts a numeric value for an artifact of interest 20
- 21. Safearea–nographicshere Safearea–nographicshere Techniques for LearningFrom Actuals 1. Leverage actual distribution - The actuals form a distribution - We can sample from this distribution during Monte Carlo simulation 2. Predicting based on relationships between estimates and actuals (regression) – If estimates and actuals are strongly correlated, regression analysis can be useful in predicting actuals from estimates 2. Predicting based on learned relationships between multiple variables and actuals 3. Bayesian Networks - After 50 tasks complete, we have new evidence: • DB tasks: effort = [11d, 16d, 42d], type = 20% • Analytics tasks: effort = [4d, 16d, 20d], type = 75% • Mobile tasks: effort = [1d, 2d, 4d], type = 5% - We can update the conditional probabilities to reflect this learned evidence 21 Probability Actual Task Effort Node 0 n: 4591; %: 100.0 Predicted: 5.5 E8 Node 0 n: 4591; %: 100.0 Predicted: 5.5 E8 Effort Node 1 n: 1911; %: 41.6 Predicted: 5.0 E8 Node 1 n: 1911; %: 41.6 Predicted: 5.0 E8 Node 2 n: 2680; %: 58.4 Predicted: 5.9 E8 Node 2 n: 2680; %: 58.4 Predicted: 5.9 E8 Attribute: Owner Improvement: 1.8 E15 Alice; David John; Claire Node 3 n: 832; %: 31.1 Predicted: 5.6 E8 Node 3 n: 832; %: 31.1 Predicted: 5.6 E8 Node 4 n: 1848; %: 68.9 Predicted: 6.1 E8 Node 4 n: 1848; %: 68.9 Predicted: 6.1 E8 Attribute: Owner Improvement: 2.7 E14 Claire John Task duration Task duration Task owner Task owner Task type Task type Task expected effort Task expected effort Qualificati on for task Qualificati on for task
- 22. Safearea–nographicshere Safearea–nographicshere Server team hasbeen working for a month. It is not going well for them. It is no longer a good bet that the server team will deliver the committed scope on time 2/13 Delivery 8/1 Start 9/1 Now PredictedDelivery DateRange Date of Prediction 2/13 1/1 3/1 2/1 4/1 • Best case: 1/1 • Most likely: 2/11 • Worst case: 2/28 • On-time delivery: 87% • Best case: 1/17 • Most likely: 2/23 • Worst case: 4/7 • On-time delivery: 43%
- 23. Safearea–nographicshere Safearea–nographicshere Lessons for usall: Making better bets Eliminate some of the long, red tail – Understand and remediate sources of uncertainty that are putting the deadline at risk Change the scope – Drop or postpone work – Never negotiate estimates! Change the resources available – Beware Brooks’ Law! Change the time – Negotiate a later deadline Make changes, rerun Monte Carlo simulation, check results – Repeat until the outcome is acceptable 10/1 12/1812/1 87% 13% Task 1.1.7 2d 5d 6w • Break down the task further • Do some work now to eliminate variance-inducing questions
Editor's Notes
- #2 09/11/13 21:59 Drury Design Dynamics IBM INNOVATE 2013
- #3 09/11/13 21:59 Drury Design Dynamics IBM INNOVATE 2013
- #4 09/11/13 21:59 Drury Design Dynamics IBM INNOVATE 2013
- #5 09/11/13 21:59 Drury Design Dynamics IBM INNOVATE 2013
- #6 09/11/13 21:59 Drury Design Dynamics IBM INNOVATE 2013
- #7 09/11/13 21:59 Drury Design Dynamics IBM INNOVATE 2013
- #8 09/11/13 21:59 Drury Design Dynamics IBM INNOVATE 2013
- #9 09/11/13 21:59 Drury Design Dynamics IBM INNOVATE 2013
- #10 09/11/13 21:59 Drury Design Dynamics IBM INNOVATE 2013
- #11 09/11/13 21:59 Drury Design Dynamics IBM INNOVATE 2013
- #12 The US Treasury, in order to garner more revenue, purchased KilerApp, to take advantage of modern software trends for it property auctions
- #13 The ty Rollup/interdependencies (scene 3): Are people prioritizing work to correctly address dependencies? Who depends on me within my team? On other teams? How do I weight these?
- #17 Make point: we’re not using critical path analysis here. We’re talking about dependencies Schedule depends on uncertainty around when people will get specific tasks done Add triangles to the tasks
- #23 Server team is using an enterprise hybrid process. Their response must be negotiated. What functional commitments are at risk? Technical, business case ramifications of dropping them? What delivery date would give 90% likelihood of on-time delivery? Would additional resources increase the likelihood of on-time delivery? Phase-gate processes (e.g., waterfall): Will we deliver on time? At what level of certainty? Manage uncertainty, risk by “creeping commitment” and “ survival of the fittest ” Agile processes (e.g., Scrum): What can we deliver on time? At what level of certainty? Manage uncertainty, risk with “ scope fluidity
- #24 TBD: work on the title Maybe illustrate with example showing one of the long-tailed tasks (break it down further to identify sources of uncertainty and remediate; even by breaking it down further, you’ve helped to reduce the uncertainty)