When Agile Estimating is not Sufficient


Published on

Agile evangelists frequently skip the realities of the world. This is especially true when it comes to estimation. It often appears as if authors and presenters live in a world in which the customer is always a deep-pocketed in-house resource, with an abundance of confidence in the development team.

The realities, however, is that whether doing in-house development or contracting, the customer expects estimates for the development work. Potential benefits have to be weighed against estimated costs.

This talk deals with why estimation is crucial also in an Agile world.

Agile projects are hardly immune to overruns, delays and bad business decisions based on poor estimates. Before one can sit down and use “Planning Poker” or similar techniques for estimating sprints or releases, it is often necessary to provide a relative accurate estimate of total project delivery schedule and costs.

The talk presents five collaborative techniques to use for estimating software projects up front:


Wideband Delphi

Unstructured groups

Statistical groups

Decision markets

All techniques have various strengths and weaknesses. They have been detailed to a great extent in the literature, but actual experiences and scientific results for use in Agile projects are scarce.

Combination techniques also supplement each other, and may be appropriate to use at different stages of a project. I.e. some techniques are more suitable for estimating projects up front (e.g. in bidding), some are good for release planning, and some are good for more detailed sprint planning.

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

When Agile Estimating is not Sufficient

  1. 1. When Agile Estimating is not Sufficient - Five Collaborative Techniques to use for Estimating Software Projects Up Front Kjetil Moløkken-Østvold – Conceptos Agile 2008, Toronto.
  2. 2. Agile development and estimation? ”-the concept of an overrun is not one the typically found in agile development processes themselves and precision themselves, estimation up front is not typically seen as a priority.” - Comment from anonymous reviewer, Agile 2007, research track.
  3. 3. Main challenge • It appears as if many books, papers and tutorials in the Agile community assumes: g y – A customer with no need for budget or schedule when starting a project project. – That minor estimates derived as you go (e.g. for sprints) are sufficient sufficient.
  4. 4. Agenda 1. Why ti ti is l important i ( 1 Wh estimation i also i t t in (most) t) Agile projects. 2. Outline of five collaborative techniques to use for estimating software projects up front: – Delphi. – Wideband Delphi. – Unstructured groups. U d – Statistical groups. – Decision D i i markets. kt 3. Discussion.
  5. 5. Why estimation is also important in (most) Agile projects.
  6. 6. Software Project Overruns – Research findings • About 70-80% of all projects encounter effort (cost) overruns¹. ( ) • The average magnitude of effort overruns is 30 40% 30-40%. • Similar results for schedule overruns. ¹ Moløkken-Østvold, Jørgensen, Tanilkan, Gallis, Lien and Hove. A Survey on Software Estimation in the Norwegian Industry, In 10th International Software Metrics Symposium (METRICS 2004).
  7. 7. Software Project Overruns – Research findings (cont.) • Similar findings in published research from several western countries ¹. • No apparent change the past 30-40 years. •IImportant: accurate numbers are difficult tt t b diffi lt to report. ¹ Moløkken-Østvold and Jørgensen. A Review of Surveys on Software Effort Estimation, In: IEEE International Symposium on Empirical Software Engineering (ISESE 2003), pp. 223-230, Rome, Italy, Italy IEEE Computer Society, 2003. Society 2003
  8. 8. Software Project Overruns – Important • P j t with overruns are not necessarily Projects ith t il failures. And vice versa… • Projects that deliver according to budget and j g g schedule may suffer from: – Lack of customer satisfaction. – Lack of functionality. – Lack of quality. q y
  9. 9. What is an estimate? • Wh t i th purpose of the estimate? What is the f th ti t ? – Budget. – Most likely estimate. y – Bid. – Price-to-win. – Planning estimate. • Who is it for? – Internal estimates. – Et External estimates. Communicated t managers/customers etc. l ti t C i t d to / t t • At what stage? – Bidding p ocess o equ a e t dd g process or equivalent. – Planning phase. – Re-estimation(s).
  10. 10. Estimating Agile projects • A il projects are h dl i Agile jt hardly immune t to overruns, delays and bad business decisions based on poor estimates. • “Planning Poker” or similar techniques for g q estimating sprints or releases are important, but not sufficient. po ta t, ot su c e t • It is often necessary to provide a relative accurate estimate of total project delivery schedule and costs at en early stage.
  11. 11. Estimating Agile projects •CConcerns regarding estimation and contracts di ti ti d tt are frequently omitted in the agile literature. • As described by Jamieson, Vinsen and Callender: – “Software can be developed in-house, but is more often obtained from vendors either as a package or through bespoke software development services.” – “…contemporary agile methods of software contemporary development do not appear to consider the role of the procurement p p process in influencing success.” g
  12. 12. Five collaborative techniques to use f estimating software for ti ti ft projects up front
  13. 13. Why combine estimates? • Obt i knowledge f Obtain k l d from various sources. i • Avoid extreme decisions. • S nchroni e perceptions abo t estimates and Synchronize about work at hand. • Create ownership of estimates estimates. • Remove irrelevant information (if using moderator). moderator) • Introduce a ”devils advocate”.
  14. 14. Pitfalls when combining estimates • Passive participants. • Depending on chosen method: p p g political p pressure (groupthink). • Requires good moderators and experts. • Time-consuming and costly (?).
  15. 15. An overview of some methods for combining estimates Method Structure Anonymity Interaction Overhead Delphi Heavy Yes No Major Wideband Moderate Limited Limited Moderate Delphi Planning Light No Yes Limited Poker Unstructured Light No Yes Limited groups Statistical Light g Yes No Limited groups Decision Heavy Yes No Moderate markets k
  16. 16. Delphi • The Delphi technique was developed by RAND as a method for eliciting and refining group judgment¹. • Three main features: – Anonymous responses (questionnaires). – Iterations and controlled feedback (several rounds, with moderator). ) – Statistical group responses (final round is verdict). • Scientific evidence is sparse – Indications that it outperforms statistical groups and unstructured interacting groups. – No conclusive evidence that it outperforms other structured group combination techniques. – No published studies from Software Engineering. ¹ http://www.rand.org/pubs/research_memoranda/RM5888/
  17. 17. Wideband Delphi • The technique is a modification of the Delphi technique, developed by Boehm and Farquhar. • Si il t th N i l G Similar to the Nominal Group t h i technique, also k l know as the estimate-talk-estimate technique. • The experts meet for group discussions both prior to to, and during, the estimation iterations. • Lack of empirical studies studies. • Inspiration for Planning Poker.
  18. 18. Unstructured groups • Various types of unstructured groups exists exists. – Basically discussions with a group decision being made at the end. – I di id l can d i th i own estimates b f Individuals derive their ti t before th di the discussion, i this is recommended in order to mitigate pressure. • Unstructured groups can outperform a Delphi group if the th motivation and i f ti ti d information sharing i adequate. ti h i is d t • In a previous study on software estimation, we found that: a – Group estimates made after an unstructured discussion were less optimistic and more realistic a statistical group. – The main driver was identification of additional activities and complexity.
  19. 19. Statistical groups • In a statistical group, there is no interaction between the group members. •CComputing th mean or median of th diff ti the di f the different individual t i di id l estimates will give us one estimate that is based on multiple estimates estimates. • Magne Jørgensen claims that taking a simple average oe often works as the bes method for co b os e best e od o combining es a es g estimates.
  20. 20. Decision markets • A decision market can be set up like a stock market: – Traders are invited to invest money in alternatives, represented by stocks (decisions), that they think will be the eventual outcome. – A trader holding a stock (decision) that becomes the actual outcome receives a fixed amount of money, prize or similar. – Through the dynamics of a market, this results in higher stock (decision) prices for the alternatives that most people think will be the outcome, which creates a lik lih d di t ib ti f th diff hi h t likelihood distribution for the different outcomes. tt • According to Surowiecki, the traders should be diverse in their backgrounds, independent of each other, and have knowledge. •GGreen, Armstrong and Graefe¹ claim that Delphi h A t d G f ¹ l i th t D l hi have advantages d t over decision markets, including: – Broader applicability. – Not vulnerable to speculation speculation. – Requires fewer experts. ¹http://forecastingprinciples.com/paperpdf/Delphi-WPlatestV.pdf p gp p pp p p p
  21. 21. Questions? •M More i f information: ti • Slides: www conceptos no/presentations/agile2008 pdf www.conceptos.no/presentations/agile2008.pdf • Paper on ”Using planning poker for combining expert Using estimates in software projects”, Moløkken-Østvold, Haugen and Benestad. http://dx.doi.org/10.1016/j.jss.2008.03.058 h //d d i /10 1016/j j 2008 03 0 8