SlideShare a Scribd company logo
Using Simulation to Manage
   Software Delivery Risk

 Effective Modeling and Simulating
 Kanban and Scrum Projects using
      Monte Carlo Techniques

                Troy Magennis
      Troy.magennis@focusedobjective.com
               @AgileSimulation
Schedule Risk




Are we meeting our
  commitments?
Staff Risk




What team skillset additions
(or losses) have the biggest
          impact?
Risks




What are the top three risks
  jeopardizing delivery?
My Mission


 Arm my teams (and yours)
with the tools and techniques
 to answer these questions

      And manage risk more effectively
• Currently: Founder and CTO Focused Objective
• Previously: Vice President of Technology (Arch)
   • Travelocity and Lastminute.com
   • Director Architecture, Corbis
   • Various: Automotive, Banking




                            Contact:
                            @AgileSimulation and @t_magennis
                            FocusedObjective.com
                            Troy.Magennis@focusedobjective.com
What, when, who, why

DEFINITIONS, HISTORY AND USE
Definition: Model

 A model is a tool used to
mimic a real world process
    A tool for low-cost
     experimentation
Definition: Simulation


A technique of using a model
to determine a result given a
   set of input conditions
Monte Carlo Simulation

Performing a simulation of a
 model multiple times using
random input conditions and
 recording the frequency of
   each result occurrence
Simple to more complex model and simulation of a software project

    DEMO: VISUAL MODEL SIMULATION
    DEMO: MONTE-CARLO SIMULATION

In case of demo disaster, press here…
History




  Stan Ulam Holding
     the FERMIAC




Credits: Wikipedia
When to use Monte Carlo Simulation

   When there is no correct
  single answer (knowable in
     advance) or when the
time/effort taken to compute
an answer is beyond realistic
When to use Monte Carlo Simulation

  When a range of input
conditions can MASSIVELY
 alter the final outcome
Who Uses Monte Carlo Simulation

           High risk industries
  Natural resource exploration, insurance,
    finance, banking, pharmaceutical…


Software Development == High Risk!
Just look at our reputation, and on-time, on-budget success rate…
APPLYING MONTE CARLO
METHODS TO SOFTWARE DEV
Why? To Answer Tough Questions…

  Date and cost forecasts
 Impact of staff hire/loss
      Cost of defects
  Cost of blocking events
             …
       And my three 1:1 questions each week!
But doesn’t it require estimates?

     Yes, but very few…
MUST: Estimate major risks
SHOULD: Column cycle-times
    and story counts
We need to estimate risk events
         **Major risk events have the predominate role
          in deciding where deliver actually occurs **



     We spend all our
   time estimating here




        1                   2                    3
Is it Accurate?

  1. Gin still equals Gout
2. Doesn’t suffer from the
    “Flaw of Averages”
Flaw of Averages

50%                  50%
Possible             Possible
Outcomes             Outcomes
The average            Major issue: Race
                                            release            condition, third party
                                            date!!!             component failure…


                      25
Frequency of Result




                      20

                      15

                      10

                      5

                      1

                                                        Major Risk Event Shifts
                           Developer Estimates
                                                        Delivery Shape Right
We need to estimate risk events
                     **Major risk events have the predominate role
                      in deciding where deliver actually occurs **



              We spend all our
            time estimating here




                 1                      2                    3
See model example…
Risk likelihood changes constantly
                              95th
                          Confidence
                            Interval




      1         2         3
Risk likelihood changes constantly
                          95th
                      Confidence
                        Interval




      1         2           3
Risk likelihood changes constantly
                        95th
                    Confidence
                      Interval




      1         2                3
Risk likelihood changes constantly
                    95th
                Confidence
                  Interval




      1         2            3
DEMO: FORECASTING (DATES & COST)
    DEMO: SENSITIVITY (COST OF DEFECTS)
    DEMO: STAFF IMPACT (STAFF RISK)
In case of demo disaster or no internet, press here…
BEST PRACTICES AND TIPS
Sensitivity            Model
   Test               (a little)

          The Model
           Creation
            Cycle

 Monte-               Visually
Carlo Test              Test
Make
 Informed           Baseline
Decision(s)

               The
           Experiment
              Cycle
                     Make
Compare
                     Single
 Results
                    Change
Best Practice 1

 Start simple and add ONE
 input condition at a time.

 Visually / Monte-carlo test
each input to verify it works
Best Practice 2

 Find the likelihood of major
  events and estimate delay
  E.g. vendor dependencies,
performance/memory issues,
    third party component
            failures.
Best Practice 3

Only obtain and add detailed
 estimates and opinion to a
model if Sensitivity Analysis
 says that input is material
Best Practice 4

Use a uniform random input
distribution UNTIL sensitivity
  analysis says that input is
    influencing the output
Best Practice 5

   Educate your managers’
about risk. They will still want
 a “single” date for planning,
  but let them decide 75   th or

     95 th confidence level

(average is NEVER an option)
Q1. Are we meeting our commitments?
    Is the likelihood of the models forecast date
    increasing or decreasing?

Q2. What are the top three risks
jeopardizing on-time delivery?
    Top three items in the Sensitivity report

Q3. What skillsets do your next three
hires need to have?
    Skills applicable to the top three WIP limit increases
    that cause the biggest reduction in forecast
Call to action
• Read these books




• Download the software FocusedObjective.com
• Follow @AgileSimulation
• Learn: http://strategicdecisions.stanford.edu/
Questions?

My Contact Details and to get these slides, the
 software or the book used in this session -

            FocusedObjective.com

 Me: Troy.magennis@FocusedObjective.com

Follow: @AgileSimulation and @t_magennis
BASICS OF MODELING AND
   SIMULATION

Return to main presentation…
Manual Kanban Model & Simulation
                 2                3                4
              Design          Develop            Test
    Backlog   1 – 2 days      1 – 2 days      1 – 2 days
                                                             Deployed
1



      2


                5
                    PLUS: For this manual example, at least 1 defect,
                         blocking event and scope-creep item.
Day 1
          Design       Develop          Test
Backlog   1 – 2 days   1 – 5 days   1 – 2 days
                                                 Deployed




             1 Day picked at random
  2           for this columns cycle-
                    time range
Day 2
          Design       Develop        Test
Backlog   1 – 2 days   1 – 5 days   1 – 2 days
                                                 Deployed

             2
           1 day
Day 3
          Design       Develop        Test
Backlog   1 – 2 days   1 – 5 days   1 – 2 days
                                                 Deployed




                          2
                        1 day
Day 4
          Design       Develop        Test
Backlog   1 – 2 days   1 – 5 days   1 – 2 days
                                                 Deployed

Added
Scope




                          2
                        1 day
Day 5
          Design       Develop        Test
Backlog   1 – 2 days   1 – 5 days   1 – 2 days
                                                 Deployed


            Added
            Scope




                          2
                        1 day
Day 6
          Design       Develop        Test
Backlog   1 – 2 days   1 – 5 days   1 – 2 days
                                                 Deployed


            Added
            Scope




                          2
                        1 day
Day 7
          Design       Develop        Test
Backlog   1 – 2 days   1 – 5 days   1 – 2 days
                                                 Deployed


            Added
            Scope




                          2
                        1 day
Day 8
          Design       Develop        Test
Backlog   1 – 2 days   1 – 5 days   1 – 2 days
                                                 Deployed

                                       2
                                     1 day


                         Added
                         Scope
Day 9
          Design       Develop        Test
Backlog   1 – 2 days   1 – 5 days   1 – 2 days
                                                 Deployed




                         Added
                         Scope



                                                    2
Day 10
          Design       Develop        Test
Backlog   1 – 2 days   1 – 5 days   1 – 2 days
                                                 Deployed

                                      Added
                                      Scope




                                                    2
Day 11
          Design       Develop        Test
Backlog   1 – 2 days   1 – 5 days   1 – 2 days
                                                 Deployed



                                                        Added
                                                        Scope




                                                    2
Result versus Frequency (50 runs)

                                                                     More Often
                      25
Frequency of Result




                      20

                      15

                      10

                      5

                      1

                           10               15                  20
                                                                     Less Often
                            Result Values – For example, Days
Result versus Frequency (250 runs)

                                                                     More Often
                      25
Frequency of Result




                      20

                      15

                      10

                      5

                      1

                           10               15                  20
                                                                     Less Often
                            Result Values – For example, Days
Result versus Frequency (1000+ runs)

                                                                     More Often
                      25
Frequency of Result




                      20

                      15

                      10

                      5

                      1

                           10               15                  60
                                                                     Less Often
                            Result Values – For example, Days
Central Limit Theorum




Return to main presentation…
Flaw of Averages

 50%                              50%
 Possible                         Possible
 Outcomes                         Outcomes




Return to main presentation…
Software Development Model
                               4
                3                  Blocking            5
                                    Events    Added
                     Defects
                                              Work
     2                                                            6
                                                        Staff
           Work
                                                      Vacations

1                                                                     7
     Columns
      & WIP                        Model                    …



Return to main presentation…
Return to main
presentation…
SIMULATION EXAMPLES


Return to main presentation…
unlikely   Forecasts   Return to main presentation…




certain
unlikely     Forecasts    Return to main presentation…




           50%             50%
           Possible    Possible
           Outcomes   Outcomes




certain
Return to main presentation…


    Sensitivity Report




Actively             Ignore for the
Manage                    moment
Return to main presentation…



Staff Skill Impact Report
              Explore what staff
              changes have the
              greatest impact
Return to main
presentation…
MULTI-MODAL RESULT MODEL
Return to main presentation…
Return to main presentation…
 <setup>

  <backlog type="custom" >

    <deliverable name=“work">
      <custom count="10">Build website</custom>
    </deliverable>

    <deliverable name="performance issues, add caching" skipPercentage="50">
      <custom count="10" >Rework: Performance Issues</custom>
    </deliverable>

  </backlog>

  <columns>
    <column id="1" estimateLowBound="1" estimateHighBound="5"
            wipLimit="1">Develop</column>
    <column id="2" estimateLowBound="1" estimateHighBound="5"
           wipLimit="1">Test</column>
  </columns>

</setup>
Return to main
presentation…

More Related Content

What's hot

Analyze Phase Roadmap (Level 3)
Analyze Phase Roadmap (Level 3)Analyze Phase Roadmap (Level 3)
Analyze Phase Roadmap (Level 3)
Matt Hansen
 
Issue based work planning and hypothesis problem solving
Issue based work planning and hypothesis problem solvingIssue based work planning and hypothesis problem solving
Issue based work planning and hypothesis problem solving
STRATICX
 
Hypothesis Testing: Central Tendency – Non-Normal (Compare 1:1)
Hypothesis Testing: Central Tendency – Non-Normal (Compare 1:1)Hypothesis Testing: Central Tendency – Non-Normal (Compare 1:1)
Hypothesis Testing: Central Tendency – Non-Normal (Compare 1:1)
Matt Hansen
 
Improving Forecasts with Monte Carlo Simulations
Improving Forecasts with Monte Carlo Simulations  Improving Forecasts with Monte Carlo Simulations
Improving Forecasts with Monte Carlo Simulations
Michael Wallace
 
Agile Metrics 101
Agile Metrics 101Agile Metrics 101
Agile Metrics 101
Andrii Brylov
 
PS 130 Rev D Problem Solving
PS 130 Rev D Problem SolvingPS 130 Rev D Problem Solving
PS 130 Rev D Problem Solving
Larry Thompson, MfgT.
 
Hypothesis Testing: Relationships (Overview)
Hypothesis Testing: Relationships (Overview)Hypothesis Testing: Relationships (Overview)
Hypothesis Testing: Relationships (Overview)
Matt Hansen
 
Issue-Based Work Planning and Hypothesis Problem Solving
Issue-Based Work Planning and Hypothesis Problem SolvingIssue-Based Work Planning and Hypothesis Problem Solving
Issue-Based Work Planning and Hypothesis Problem Solving
Flevy.com Best Practices
 
Production operations management
Production operations managementProduction operations management
Production operations management
David José Atencio Boscán
 
Risk Analysis with Matt Hansen at StatStuff
Risk Analysis with Matt Hansen at StatStuffRisk Analysis with Matt Hansen at StatStuff
Risk Analysis with Matt Hansen at StatStuff
Matt Hansen
 
Beyond Churn Prediction : An Introduction to uplift modeling
Beyond Churn Prediction : An Introduction to uplift modelingBeyond Churn Prediction : An Introduction to uplift modeling
Beyond Churn Prediction : An Introduction to uplift modeling
Pierre Gutierrez
 
Technical Debt - osbridge
Technical Debt - osbridgeTechnical Debt - osbridge
Technical Debt - osbridge
enaramore
 
MN IT Symposium Products and Platforms OVER Progs, Projs, and; Processes
MN IT Symposium Products and Platforms OVER Progs, Projs, and; ProcessesMN IT Symposium Products and Platforms OVER Progs, Projs, and; Processes
MN IT Symposium Products and Platforms OVER Progs, Projs, and; Processes
DevJam
 
Jack Jager AMPEAK 2014 Presentation - 6 steps for a successful root cause ana...
Jack Jager AMPEAK 2014 Presentation - 6 steps for a successful root cause ana...Jack Jager AMPEAK 2014 Presentation - 6 steps for a successful root cause ana...
Jack Jager AMPEAK 2014 Presentation - 6 steps for a successful root cause ana...
ARMS Reliability
 

What's hot (14)

Analyze Phase Roadmap (Level 3)
Analyze Phase Roadmap (Level 3)Analyze Phase Roadmap (Level 3)
Analyze Phase Roadmap (Level 3)
 
Issue based work planning and hypothesis problem solving
Issue based work planning and hypothesis problem solvingIssue based work planning and hypothesis problem solving
Issue based work planning and hypothesis problem solving
 
Hypothesis Testing: Central Tendency – Non-Normal (Compare 1:1)
Hypothesis Testing: Central Tendency – Non-Normal (Compare 1:1)Hypothesis Testing: Central Tendency – Non-Normal (Compare 1:1)
Hypothesis Testing: Central Tendency – Non-Normal (Compare 1:1)
 
Improving Forecasts with Monte Carlo Simulations
Improving Forecasts with Monte Carlo Simulations  Improving Forecasts with Monte Carlo Simulations
Improving Forecasts with Monte Carlo Simulations
 
Agile Metrics 101
Agile Metrics 101Agile Metrics 101
Agile Metrics 101
 
PS 130 Rev D Problem Solving
PS 130 Rev D Problem SolvingPS 130 Rev D Problem Solving
PS 130 Rev D Problem Solving
 
Hypothesis Testing: Relationships (Overview)
Hypothesis Testing: Relationships (Overview)Hypothesis Testing: Relationships (Overview)
Hypothesis Testing: Relationships (Overview)
 
Issue-Based Work Planning and Hypothesis Problem Solving
Issue-Based Work Planning and Hypothesis Problem SolvingIssue-Based Work Planning and Hypothesis Problem Solving
Issue-Based Work Planning and Hypothesis Problem Solving
 
Production operations management
Production operations managementProduction operations management
Production operations management
 
Risk Analysis with Matt Hansen at StatStuff
Risk Analysis with Matt Hansen at StatStuffRisk Analysis with Matt Hansen at StatStuff
Risk Analysis with Matt Hansen at StatStuff
 
Beyond Churn Prediction : An Introduction to uplift modeling
Beyond Churn Prediction : An Introduction to uplift modelingBeyond Churn Prediction : An Introduction to uplift modeling
Beyond Churn Prediction : An Introduction to uplift modeling
 
Technical Debt - osbridge
Technical Debt - osbridgeTechnical Debt - osbridge
Technical Debt - osbridge
 
MN IT Symposium Products and Platforms OVER Progs, Projs, and; Processes
MN IT Symposium Products and Platforms OVER Progs, Projs, and; ProcessesMN IT Symposium Products and Platforms OVER Progs, Projs, and; Processes
MN IT Symposium Products and Platforms OVER Progs, Projs, and; Processes
 
Jack Jager AMPEAK 2014 Presentation - 6 steps for a successful root cause ana...
Jack Jager AMPEAK 2014 Presentation - 6 steps for a successful root cause ana...Jack Jager AMPEAK 2014 Presentation - 6 steps for a successful root cause ana...
Jack Jager AMPEAK 2014 Presentation - 6 steps for a successful root cause ana...
 

Similar to Using Simulation to Manage Software Delivery Risk

7 (+/- 2) Steps to Agility
7 (+/- 2) Steps to Agility7 (+/- 2) Steps to Agility
7 (+/- 2) Steps to Agility
Tim Gifford
 
Rethinking Risk-Based Project Management in the Emerging IT initiatives.pptx
Rethinking Risk-Based Project Management in the Emerging IT initiatives.pptxRethinking Risk-Based Project Management in the Emerging IT initiatives.pptx
Rethinking Risk-Based Project Management in the Emerging IT initiatives.pptx
Inflectra
 
Release Management for Large Enterprises
Release Management for Large EnterprisesRelease Management for Large Enterprises
Release Management for Large Enterprises
Salesforce Developers
 
201008 Software Testing Notes (part 1/2)
201008 Software Testing Notes (part 1/2)201008 Software Testing Notes (part 1/2)
201008 Software Testing Notes (part 1/2)
Javier Gonzalez-Sanchez
 
Agile Software Development in Practice - A Developer Perspective
Agile Software Development in Practice - A Developer PerspectiveAgile Software Development in Practice - A Developer Perspective
Agile Software Development in Practice - A Developer Perspective
Wee Witthawaskul
 
Value driven continuous delivery
Value driven continuous deliveryValue driven continuous delivery
Value driven continuous delivery
Gabriel Prat
 
Agile principles and practices
Agile principles and practicesAgile principles and practices
Agile principles and practices
Vipin Jose
 
Agile & Secure SDLC
Agile & Secure SDLCAgile & Secure SDLC
Agile & Secure SDLC
Paul Yang
 
Agile - Product is Progress.
Agile - Product is Progress.Agile - Product is Progress.
Agile - Product is Progress.
Brian Dreyer
 
The Agile PMP - Pillar Technology
The Agile PMP - Pillar TechnologyThe Agile PMP - Pillar Technology
The Agile PMP - Pillar Technology
Mike Cottmeyer
 
How To Handle Exploding Complexity in Product Development
How To Handle Exploding Complexity in Product DevelopmentHow To Handle Exploding Complexity in Product Development
How To Handle Exploding Complexity in Product Development
Perforce
 
How to live with agile - Aware in BugDay Bangkok 2012
How to live with agile - Aware in BugDay Bangkok 2012How to live with agile - Aware in BugDay Bangkok 2012
How to live with agile - Aware in BugDay Bangkok 2012
Prathan Dansakulcharoenkit
 
Introduction to Scrum - 1 day workshop
Introduction to Scrum - 1 day workshopIntroduction to Scrum - 1 day workshop
Introduction to Scrum - 1 day workshop
Evan Leybourn
 
Lean DevOps - Lessons Learned from Innovation-driven Companies
Lean DevOps - Lessons Learned from Innovation-driven CompaniesLean DevOps - Lessons Learned from Innovation-driven Companies
Lean DevOps - Lessons Learned from Innovation-driven Companies
Xavier Amatriain
 
Project Management - Introduction
Project Management - IntroductionProject Management - Introduction
Project Management - Introduction
SAINBAYAR Bayarsaikhan
 
Benefits of Agile Software Development for Senior Management
Benefits of Agile Software Development for Senior ManagementBenefits of Agile Software Development for Senior Management
Benefits of Agile Software Development for Senior Management
David Updike
 
Scrum Introduction
Scrum IntroductionScrum Introduction
Scrum Introduction
James Brett
 
test
testtest
test
gikrauss
 
test
testtest
test
gikrauss
 
Thezenofscrum1 090221154550-phpapp01
Thezenofscrum1 090221154550-phpapp01Thezenofscrum1 090221154550-phpapp01
Thezenofscrum1 090221154550-phpapp01
Dani Llamazares
 

Similar to Using Simulation to Manage Software Delivery Risk (20)

7 (+/- 2) Steps to Agility
7 (+/- 2) Steps to Agility7 (+/- 2) Steps to Agility
7 (+/- 2) Steps to Agility
 
Rethinking Risk-Based Project Management in the Emerging IT initiatives.pptx
Rethinking Risk-Based Project Management in the Emerging IT initiatives.pptxRethinking Risk-Based Project Management in the Emerging IT initiatives.pptx
Rethinking Risk-Based Project Management in the Emerging IT initiatives.pptx
 
Release Management for Large Enterprises
Release Management for Large EnterprisesRelease Management for Large Enterprises
Release Management for Large Enterprises
 
201008 Software Testing Notes (part 1/2)
201008 Software Testing Notes (part 1/2)201008 Software Testing Notes (part 1/2)
201008 Software Testing Notes (part 1/2)
 
Agile Software Development in Practice - A Developer Perspective
Agile Software Development in Practice - A Developer PerspectiveAgile Software Development in Practice - A Developer Perspective
Agile Software Development in Practice - A Developer Perspective
 
Value driven continuous delivery
Value driven continuous deliveryValue driven continuous delivery
Value driven continuous delivery
 
Agile principles and practices
Agile principles and practicesAgile principles and practices
Agile principles and practices
 
Agile & Secure SDLC
Agile & Secure SDLCAgile & Secure SDLC
Agile & Secure SDLC
 
Agile - Product is Progress.
Agile - Product is Progress.Agile - Product is Progress.
Agile - Product is Progress.
 
The Agile PMP - Pillar Technology
The Agile PMP - Pillar TechnologyThe Agile PMP - Pillar Technology
The Agile PMP - Pillar Technology
 
How To Handle Exploding Complexity in Product Development
How To Handle Exploding Complexity in Product DevelopmentHow To Handle Exploding Complexity in Product Development
How To Handle Exploding Complexity in Product Development
 
How to live with agile - Aware in BugDay Bangkok 2012
How to live with agile - Aware in BugDay Bangkok 2012How to live with agile - Aware in BugDay Bangkok 2012
How to live with agile - Aware in BugDay Bangkok 2012
 
Introduction to Scrum - 1 day workshop
Introduction to Scrum - 1 day workshopIntroduction to Scrum - 1 day workshop
Introduction to Scrum - 1 day workshop
 
Lean DevOps - Lessons Learned from Innovation-driven Companies
Lean DevOps - Lessons Learned from Innovation-driven CompaniesLean DevOps - Lessons Learned from Innovation-driven Companies
Lean DevOps - Lessons Learned from Innovation-driven Companies
 
Project Management - Introduction
Project Management - IntroductionProject Management - Introduction
Project Management - Introduction
 
Benefits of Agile Software Development for Senior Management
Benefits of Agile Software Development for Senior ManagementBenefits of Agile Software Development for Senior Management
Benefits of Agile Software Development for Senior Management
 
Scrum Introduction
Scrum IntroductionScrum Introduction
Scrum Introduction
 
test
testtest
test
 
test
testtest
test
 
Thezenofscrum1 090221154550-phpapp01
Thezenofscrum1 090221154550-phpapp01Thezenofscrum1 090221154550-phpapp01
Thezenofscrum1 090221154550-phpapp01
 

More from Troy Magennis

What is the story with agile data keynote agile 2018 (Magennis)
What is the story with agile data keynote   agile 2018 (Magennis)What is the story with agile data keynote   agile 2018 (Magennis)
What is the story with agile data keynote agile 2018 (Magennis)
Troy Magennis
 
I love the smell of data in the morning (getting started with data science) ...
I love the smell of data in the morning (getting started with data science)  ...I love the smell of data in the morning (getting started with data science)  ...
I love the smell of data in the morning (getting started with data science) ...
Troy Magennis
 
Forecasting using data - Deliver 2016
Forecasting using data  - Deliver 2016Forecasting using data  - Deliver 2016
Forecasting using data - Deliver 2016
Troy Magennis
 
Data driven coaching - Deliver 2016
Data driven coaching - Deliver 2016Data driven coaching - Deliver 2016
Data driven coaching - Deliver 2016
Troy Magennis
 
Data driven coaching - Agile 2016 (troy magennis)
Data driven coaching  - Agile 2016 (troy magennis)Data driven coaching  - Agile 2016 (troy magennis)
Data driven coaching - Agile 2016 (troy magennis)
Troy Magennis
 
Risk Management and Reliable Forecasting using Un-reliable Data (magennis) - ...
Risk Management and Reliable Forecasting using Un-reliable Data (magennis) - ...Risk Management and Reliable Forecasting using Un-reliable Data (magennis) - ...
Risk Management and Reliable Forecasting using Un-reliable Data (magennis) - ...
Troy Magennis
 
LKNA 2014 Risk and Impediment Analysis and Analytics - Troy Magennis
LKNA 2014 Risk and Impediment Analysis and Analytics - Troy MagennisLKNA 2014 Risk and Impediment Analysis and Analytics - Troy Magennis
LKNA 2014 Risk and Impediment Analysis and Analytics - Troy Magennis
Troy Magennis
 
Modeling, simulation & data mining: Answering Tough Executive Questions (Agil...
Modeling, simulation & data mining: Answering Tough Executive Questions (Agil...Modeling, simulation & data mining: Answering Tough Executive Questions (Agil...
Modeling, simulation & data mining: Answering Tough Executive Questions (Agil...
Troy Magennis
 

More from Troy Magennis (8)

What is the story with agile data keynote agile 2018 (Magennis)
What is the story with agile data keynote   agile 2018 (Magennis)What is the story with agile data keynote   agile 2018 (Magennis)
What is the story with agile data keynote agile 2018 (Magennis)
 
I love the smell of data in the morning (getting started with data science) ...
I love the smell of data in the morning (getting started with data science)  ...I love the smell of data in the morning (getting started with data science)  ...
I love the smell of data in the morning (getting started with data science) ...
 
Forecasting using data - Deliver 2016
Forecasting using data  - Deliver 2016Forecasting using data  - Deliver 2016
Forecasting using data - Deliver 2016
 
Data driven coaching - Deliver 2016
Data driven coaching - Deliver 2016Data driven coaching - Deliver 2016
Data driven coaching - Deliver 2016
 
Data driven coaching - Agile 2016 (troy magennis)
Data driven coaching  - Agile 2016 (troy magennis)Data driven coaching  - Agile 2016 (troy magennis)
Data driven coaching - Agile 2016 (troy magennis)
 
Risk Management and Reliable Forecasting using Un-reliable Data (magennis) - ...
Risk Management and Reliable Forecasting using Un-reliable Data (magennis) - ...Risk Management and Reliable Forecasting using Un-reliable Data (magennis) - ...
Risk Management and Reliable Forecasting using Un-reliable Data (magennis) - ...
 
LKNA 2014 Risk and Impediment Analysis and Analytics - Troy Magennis
LKNA 2014 Risk and Impediment Analysis and Analytics - Troy MagennisLKNA 2014 Risk and Impediment Analysis and Analytics - Troy Magennis
LKNA 2014 Risk and Impediment Analysis and Analytics - Troy Magennis
 
Modeling, simulation & data mining: Answering Tough Executive Questions (Agil...
Modeling, simulation & data mining: Answering Tough Executive Questions (Agil...Modeling, simulation & data mining: Answering Tough Executive Questions (Agil...
Modeling, simulation & data mining: Answering Tough Executive Questions (Agil...
 

Recently uploaded

Lukas Rycek - GreenChemForCE - project structure.pptx
Lukas Rycek - GreenChemForCE - project structure.pptxLukas Rycek - GreenChemForCE - project structure.pptx
Lukas Rycek - GreenChemForCE - project structure.pptx
pavelborek
 
CULR Spring 2024 Journal.pdf testing for duke
CULR Spring 2024 Journal.pdf testing for dukeCULR Spring 2024 Journal.pdf testing for duke
CULR Spring 2024 Journal.pdf testing for duke
ZevinAttisha
 
Discover the Beauty and Functionality of The Expert Remodeling Service
Discover the Beauty and Functionality of The Expert Remodeling ServiceDiscover the Beauty and Functionality of The Expert Remodeling Service
Discover the Beauty and Functionality of The Expert Remodeling Service
obriengroupinc04
 
Satta Matka Dpboss Kalyan Matka Results Kalyan Chart
Satta Matka Dpboss Kalyan Matka Results Kalyan ChartSatta Matka Dpboss Kalyan Matka Results Kalyan Chart
Satta Matka Dpboss Kalyan Matka Results Kalyan Chart
Satta Matka Dpboss Kalyan Matka Results
 
The Steadfast and Reliable Bull: Taurus Zodiac Sign
The Steadfast and Reliable Bull: Taurus Zodiac SignThe Steadfast and Reliable Bull: Taurus Zodiac Sign
The Steadfast and Reliable Bull: Taurus Zodiac Sign
my Pandit
 
Kalyan Chart Satta Matka Dpboss Kalyan Matka Results
Kalyan Chart Satta Matka Dpboss Kalyan Matka ResultsKalyan Chart Satta Matka Dpboss Kalyan Matka Results
Kalyan Chart Satta Matka Dpboss Kalyan Matka Results
Satta Matka Dpboss Kalyan Matka Results
 
Best Competitive Marble Pricing in Dubai - ☎ 9928909666
Best Competitive Marble Pricing in Dubai - ☎ 9928909666Best Competitive Marble Pricing in Dubai - ☎ 9928909666
Best Competitive Marble Pricing in Dubai - ☎ 9928909666
Stone Art Hub
 
Dpboss Matka Guessing Satta Matta Matka Kalyan panel Chart Indian Matka Dpbos...
Dpboss Matka Guessing Satta Matta Matka Kalyan panel Chart Indian Matka Dpbos...Dpboss Matka Guessing Satta Matta Matka Kalyan panel Chart Indian Matka Dpbos...
Dpboss Matka Guessing Satta Matta Matka Kalyan panel Chart Indian Matka Dpbos...
➒➌➎➏➑➐➋➑➐➐Dpboss Matka Guessing Satta Matka Kalyan Chart Indian Matka
 
Kirill Klip GEM Royalty TNR Gold Copper Presentation
Kirill Klip GEM Royalty TNR Gold Copper PresentationKirill Klip GEM Royalty TNR Gold Copper Presentation
Kirill Klip GEM Royalty TNR Gold Copper Presentation
Kirill Klip
 
1Q24_HYUNDAI CAPITAL SERVICES INC. AND SUBSIDIARIES
1Q24_HYUNDAI CAPITAL SERVICES INC. AND SUBSIDIARIES1Q24_HYUNDAI CAPITAL SERVICES INC. AND SUBSIDIARIES
1Q24_HYUNDAI CAPITAL SERVICES INC. AND SUBSIDIARIES
irhcs
 
IMG_20240615_091110.pdf dpboss guessing
IMG_20240615_091110.pdf dpboss  guessingIMG_20240615_091110.pdf dpboss  guessing
一比一原版(QMUE毕业证书)英国爱丁堡玛格丽特女王大学毕业证文凭如何办理
一比一原版(QMUE毕业证书)英国爱丁堡玛格丽特女王大学毕业证文凭如何办理一比一原版(QMUE毕业证书)英国爱丁堡玛格丽特女王大学毕业证文凭如何办理
一比一原版(QMUE毕业证书)英国爱丁堡玛格丽特女王大学毕业证文凭如何办理
taqyea
 
DearbornMusic-KatherineJasperFullSailUni
DearbornMusic-KatherineJasperFullSailUniDearbornMusic-KatherineJasperFullSailUni
DearbornMusic-KatherineJasperFullSailUni
katiejasper96
 
Satta Matka Dpboss Kalyan Matka Results Kalyan Chart
Satta Matka Dpboss Kalyan Matka Results Kalyan ChartSatta Matka Dpboss Kalyan Matka Results Kalyan Chart
Satta Matka Dpboss Kalyan Matka Results Kalyan Chart
Satta Matka Dpboss Kalyan Matka Results
 
NIMA2024 | De toegevoegde waarde van DEI en ESG in campagnes | Nathalie Lam |...
NIMA2024 | De toegevoegde waarde van DEI en ESG in campagnes | Nathalie Lam |...NIMA2024 | De toegevoegde waarde van DEI en ESG in campagnes | Nathalie Lam |...
NIMA2024 | De toegevoegde waarde van DEI en ESG in campagnes | Nathalie Lam |...
BBPMedia1
 
Registered-Establishment-List-in-Uttarakhand-pdf.pdf
Registered-Establishment-List-in-Uttarakhand-pdf.pdfRegistered-Establishment-List-in-Uttarakhand-pdf.pdf
Registered-Establishment-List-in-Uttarakhand-pdf.pdf
dazzjoker
 
Satta Matka Dpboss Kalyan Matka Results Kalyan Chart
Satta Matka Dpboss Kalyan Matka Results Kalyan ChartSatta Matka Dpboss Kalyan Matka Results Kalyan Chart
Satta Matka Dpboss Kalyan Matka Results Kalyan Chart
Satta Matka Dpboss Kalyan Matka Results
 
Kirill Klip GEM Royalty TNR Gold Lithium Presentation
Kirill Klip GEM Royalty TNR Gold Lithium PresentationKirill Klip GEM Royalty TNR Gold Lithium Presentation
Kirill Klip GEM Royalty TNR Gold Lithium Presentation
Kirill Klip
 
list of states and organizations .pdf
list of  states  and  organizations .pdflist of  states  and  organizations .pdf
list of states and organizations .pdf
Rbc Rbcua
 
The latest Heat Pump Manual from Newentide
The latest Heat Pump Manual from NewentideThe latest Heat Pump Manual from Newentide
The latest Heat Pump Manual from Newentide
JoeYangGreatMachiner
 

Recently uploaded (20)

Lukas Rycek - GreenChemForCE - project structure.pptx
Lukas Rycek - GreenChemForCE - project structure.pptxLukas Rycek - GreenChemForCE - project structure.pptx
Lukas Rycek - GreenChemForCE - project structure.pptx
 
CULR Spring 2024 Journal.pdf testing for duke
CULR Spring 2024 Journal.pdf testing for dukeCULR Spring 2024 Journal.pdf testing for duke
CULR Spring 2024 Journal.pdf testing for duke
 
Discover the Beauty and Functionality of The Expert Remodeling Service
Discover the Beauty and Functionality of The Expert Remodeling ServiceDiscover the Beauty and Functionality of The Expert Remodeling Service
Discover the Beauty and Functionality of The Expert Remodeling Service
 
Satta Matka Dpboss Kalyan Matka Results Kalyan Chart
Satta Matka Dpboss Kalyan Matka Results Kalyan ChartSatta Matka Dpboss Kalyan Matka Results Kalyan Chart
Satta Matka Dpboss Kalyan Matka Results Kalyan Chart
 
The Steadfast and Reliable Bull: Taurus Zodiac Sign
The Steadfast and Reliable Bull: Taurus Zodiac SignThe Steadfast and Reliable Bull: Taurus Zodiac Sign
The Steadfast and Reliable Bull: Taurus Zodiac Sign
 
Kalyan Chart Satta Matka Dpboss Kalyan Matka Results
Kalyan Chart Satta Matka Dpboss Kalyan Matka ResultsKalyan Chart Satta Matka Dpboss Kalyan Matka Results
Kalyan Chart Satta Matka Dpboss Kalyan Matka Results
 
Best Competitive Marble Pricing in Dubai - ☎ 9928909666
Best Competitive Marble Pricing in Dubai - ☎ 9928909666Best Competitive Marble Pricing in Dubai - ☎ 9928909666
Best Competitive Marble Pricing in Dubai - ☎ 9928909666
 
Dpboss Matka Guessing Satta Matta Matka Kalyan panel Chart Indian Matka Dpbos...
Dpboss Matka Guessing Satta Matta Matka Kalyan panel Chart Indian Matka Dpbos...Dpboss Matka Guessing Satta Matta Matka Kalyan panel Chart Indian Matka Dpbos...
Dpboss Matka Guessing Satta Matta Matka Kalyan panel Chart Indian Matka Dpbos...
 
Kirill Klip GEM Royalty TNR Gold Copper Presentation
Kirill Klip GEM Royalty TNR Gold Copper PresentationKirill Klip GEM Royalty TNR Gold Copper Presentation
Kirill Klip GEM Royalty TNR Gold Copper Presentation
 
1Q24_HYUNDAI CAPITAL SERVICES INC. AND SUBSIDIARIES
1Q24_HYUNDAI CAPITAL SERVICES INC. AND SUBSIDIARIES1Q24_HYUNDAI CAPITAL SERVICES INC. AND SUBSIDIARIES
1Q24_HYUNDAI CAPITAL SERVICES INC. AND SUBSIDIARIES
 
IMG_20240615_091110.pdf dpboss guessing
IMG_20240615_091110.pdf dpboss  guessingIMG_20240615_091110.pdf dpboss  guessing
IMG_20240615_091110.pdf dpboss guessing
 
一比一原版(QMUE毕业证书)英国爱丁堡玛格丽特女王大学毕业证文凭如何办理
一比一原版(QMUE毕业证书)英国爱丁堡玛格丽特女王大学毕业证文凭如何办理一比一原版(QMUE毕业证书)英国爱丁堡玛格丽特女王大学毕业证文凭如何办理
一比一原版(QMUE毕业证书)英国爱丁堡玛格丽特女王大学毕业证文凭如何办理
 
DearbornMusic-KatherineJasperFullSailUni
DearbornMusic-KatherineJasperFullSailUniDearbornMusic-KatherineJasperFullSailUni
DearbornMusic-KatherineJasperFullSailUni
 
Satta Matka Dpboss Kalyan Matka Results Kalyan Chart
Satta Matka Dpboss Kalyan Matka Results Kalyan ChartSatta Matka Dpboss Kalyan Matka Results Kalyan Chart
Satta Matka Dpboss Kalyan Matka Results Kalyan Chart
 
NIMA2024 | De toegevoegde waarde van DEI en ESG in campagnes | Nathalie Lam |...
NIMA2024 | De toegevoegde waarde van DEI en ESG in campagnes | Nathalie Lam |...NIMA2024 | De toegevoegde waarde van DEI en ESG in campagnes | Nathalie Lam |...
NIMA2024 | De toegevoegde waarde van DEI en ESG in campagnes | Nathalie Lam |...
 
Registered-Establishment-List-in-Uttarakhand-pdf.pdf
Registered-Establishment-List-in-Uttarakhand-pdf.pdfRegistered-Establishment-List-in-Uttarakhand-pdf.pdf
Registered-Establishment-List-in-Uttarakhand-pdf.pdf
 
Satta Matka Dpboss Kalyan Matka Results Kalyan Chart
Satta Matka Dpboss Kalyan Matka Results Kalyan ChartSatta Matka Dpboss Kalyan Matka Results Kalyan Chart
Satta Matka Dpboss Kalyan Matka Results Kalyan Chart
 
Kirill Klip GEM Royalty TNR Gold Lithium Presentation
Kirill Klip GEM Royalty TNR Gold Lithium PresentationKirill Klip GEM Royalty TNR Gold Lithium Presentation
Kirill Klip GEM Royalty TNR Gold Lithium Presentation
 
list of states and organizations .pdf
list of  states  and  organizations .pdflist of  states  and  organizations .pdf
list of states and organizations .pdf
 
The latest Heat Pump Manual from Newentide
The latest Heat Pump Manual from NewentideThe latest Heat Pump Manual from Newentide
The latest Heat Pump Manual from Newentide
 

Using Simulation to Manage Software Delivery Risk

  • 1. Using Simulation to Manage Software Delivery Risk Effective Modeling and Simulating Kanban and Scrum Projects using Monte Carlo Techniques Troy Magennis Troy.magennis@focusedobjective.com @AgileSimulation
  • 2. Schedule Risk Are we meeting our commitments?
  • 3. Staff Risk What team skillset additions (or losses) have the biggest impact?
  • 4. Risks What are the top three risks jeopardizing delivery?
  • 5.
  • 6. My Mission Arm my teams (and yours) with the tools and techniques to answer these questions And manage risk more effectively
  • 7. • Currently: Founder and CTO Focused Objective • Previously: Vice President of Technology (Arch) • Travelocity and Lastminute.com • Director Architecture, Corbis • Various: Automotive, Banking Contact: @AgileSimulation and @t_magennis FocusedObjective.com Troy.Magennis@focusedobjective.com
  • 8.
  • 9. What, when, who, why DEFINITIONS, HISTORY AND USE
  • 10. Definition: Model A model is a tool used to mimic a real world process A tool for low-cost experimentation
  • 11. Definition: Simulation A technique of using a model to determine a result given a set of input conditions
  • 12. Monte Carlo Simulation Performing a simulation of a model multiple times using random input conditions and recording the frequency of each result occurrence
  • 13. Simple to more complex model and simulation of a software project DEMO: VISUAL MODEL SIMULATION DEMO: MONTE-CARLO SIMULATION In case of demo disaster, press here…
  • 14. History Stan Ulam Holding the FERMIAC Credits: Wikipedia
  • 15. When to use Monte Carlo Simulation When there is no correct single answer (knowable in advance) or when the time/effort taken to compute an answer is beyond realistic
  • 16. When to use Monte Carlo Simulation When a range of input conditions can MASSIVELY alter the final outcome
  • 17.
  • 18. Who Uses Monte Carlo Simulation High risk industries Natural resource exploration, insurance, finance, banking, pharmaceutical… Software Development == High Risk! Just look at our reputation, and on-time, on-budget success rate…
  • 19. APPLYING MONTE CARLO METHODS TO SOFTWARE DEV
  • 20. Why? To Answer Tough Questions… Date and cost forecasts Impact of staff hire/loss Cost of defects Cost of blocking events … And my three 1:1 questions each week!
  • 21. But doesn’t it require estimates? Yes, but very few… MUST: Estimate major risks SHOULD: Column cycle-times and story counts
  • 22. We need to estimate risk events **Major risk events have the predominate role in deciding where deliver actually occurs ** We spend all our time estimating here 1 2 3
  • 23. Is it Accurate? 1. Gin still equals Gout 2. Doesn’t suffer from the “Flaw of Averages”
  • 24. Flaw of Averages 50% 50% Possible Possible Outcomes Outcomes
  • 25. The average Major issue: Race release condition, third party date!!! component failure… 25 Frequency of Result 20 15 10 5 1 Major Risk Event Shifts Developer Estimates Delivery Shape Right
  • 26. We need to estimate risk events **Major risk events have the predominate role in deciding where deliver actually occurs ** We spend all our time estimating here 1 2 3 See model example…
  • 27. Risk likelihood changes constantly 95th Confidence Interval 1 2 3
  • 28. Risk likelihood changes constantly 95th Confidence Interval 1 2 3
  • 29. Risk likelihood changes constantly 95th Confidence Interval 1 2 3
  • 30. Risk likelihood changes constantly 95th Confidence Interval 1 2 3
  • 31. DEMO: FORECASTING (DATES & COST) DEMO: SENSITIVITY (COST OF DEFECTS) DEMO: STAFF IMPACT (STAFF RISK) In case of demo disaster or no internet, press here…
  • 33. Sensitivity Model Test (a little) The Model Creation Cycle Monte- Visually Carlo Test Test
  • 34. Make Informed Baseline Decision(s) The Experiment Cycle Make Compare Single Results Change
  • 35. Best Practice 1 Start simple and add ONE input condition at a time. Visually / Monte-carlo test each input to verify it works
  • 36. Best Practice 2 Find the likelihood of major events and estimate delay E.g. vendor dependencies, performance/memory issues, third party component failures.
  • 37. Best Practice 3 Only obtain and add detailed estimates and opinion to a model if Sensitivity Analysis says that input is material
  • 38. Best Practice 4 Use a uniform random input distribution UNTIL sensitivity analysis says that input is influencing the output
  • 39. Best Practice 5 Educate your managers’ about risk. They will still want a “single” date for planning, but let them decide 75 th or 95 th confidence level (average is NEVER an option)
  • 40. Q1. Are we meeting our commitments? Is the likelihood of the models forecast date increasing or decreasing? Q2. What are the top three risks jeopardizing on-time delivery? Top three items in the Sensitivity report Q3. What skillsets do your next three hires need to have? Skills applicable to the top three WIP limit increases that cause the biggest reduction in forecast
  • 41. Call to action • Read these books • Download the software FocusedObjective.com • Follow @AgileSimulation • Learn: http://strategicdecisions.stanford.edu/
  • 42. Questions? My Contact Details and to get these slides, the software or the book used in this session - FocusedObjective.com Me: Troy.magennis@FocusedObjective.com Follow: @AgileSimulation and @t_magennis
  • 43.
  • 44. BASICS OF MODELING AND SIMULATION Return to main presentation…
  • 45. Manual Kanban Model & Simulation 2 3 4 Design Develop Test Backlog 1 – 2 days 1 – 2 days 1 – 2 days Deployed 1 2 5 PLUS: For this manual example, at least 1 defect, blocking event and scope-creep item.
  • 46. Day 1 Design Develop Test Backlog 1 – 2 days 1 – 5 days 1 – 2 days Deployed 1 Day picked at random 2 for this columns cycle- time range
  • 47. Day 2 Design Develop Test Backlog 1 – 2 days 1 – 5 days 1 – 2 days Deployed 2 1 day
  • 48. Day 3 Design Develop Test Backlog 1 – 2 days 1 – 5 days 1 – 2 days Deployed 2 1 day
  • 49. Day 4 Design Develop Test Backlog 1 – 2 days 1 – 5 days 1 – 2 days Deployed Added Scope 2 1 day
  • 50. Day 5 Design Develop Test Backlog 1 – 2 days 1 – 5 days 1 – 2 days Deployed Added Scope 2 1 day
  • 51. Day 6 Design Develop Test Backlog 1 – 2 days 1 – 5 days 1 – 2 days Deployed Added Scope 2 1 day
  • 52. Day 7 Design Develop Test Backlog 1 – 2 days 1 – 5 days 1 – 2 days Deployed Added Scope 2 1 day
  • 53. Day 8 Design Develop Test Backlog 1 – 2 days 1 – 5 days 1 – 2 days Deployed 2 1 day Added Scope
  • 54. Day 9 Design Develop Test Backlog 1 – 2 days 1 – 5 days 1 – 2 days Deployed Added Scope 2
  • 55. Day 10 Design Develop Test Backlog 1 – 2 days 1 – 5 days 1 – 2 days Deployed Added Scope 2
  • 56. Day 11 Design Develop Test Backlog 1 – 2 days 1 – 5 days 1 – 2 days Deployed Added Scope 2
  • 57. Result versus Frequency (50 runs) More Often 25 Frequency of Result 20 15 10 5 1 10 15 20 Less Often Result Values – For example, Days
  • 58. Result versus Frequency (250 runs) More Often 25 Frequency of Result 20 15 10 5 1 10 15 20 Less Often Result Values – For example, Days
  • 59. Result versus Frequency (1000+ runs) More Often 25 Frequency of Result 20 15 10 5 1 10 15 60 Less Often Result Values – For example, Days
  • 60. Central Limit Theorum Return to main presentation…
  • 61. Flaw of Averages 50% 50% Possible Possible Outcomes Outcomes Return to main presentation…
  • 62. Software Development Model 4 3 Blocking 5 Events Added Defects Work 2 6 Staff Work Vacations 1 7 Columns & WIP Model … Return to main presentation…
  • 64. SIMULATION EXAMPLES Return to main presentation…
  • 65. unlikely Forecasts Return to main presentation… certain
  • 66. unlikely Forecasts Return to main presentation… 50% 50% Possible Possible Outcomes Outcomes certain
  • 67. Return to main presentation… Sensitivity Report Actively Ignore for the Manage moment
  • 68. Return to main presentation… Staff Skill Impact Report Explore what staff changes have the greatest impact
  • 71. Return to main presentation…
  • 72. Return to main presentation… <setup> <backlog type="custom" > <deliverable name=“work"> <custom count="10">Build website</custom> </deliverable> <deliverable name="performance issues, add caching" skipPercentage="50"> <custom count="10" >Rework: Performance Issues</custom> </deliverable> </backlog> <columns> <column id="1" estimateLowBound="1" estimateHighBound="5" wipLimit="1">Develop</column> <column id="2" estimateLowBound="1" estimateHighBound="5" wipLimit="1">Test</column> </columns> </setup>