The Impact of Methods and Techniques on Outcomes from Agile Software Development Projects


Published on

Slides about agile methods and techniques from the IFIP 8.6 conference in Manchester, 2007

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

The Impact of Methods and Techniques on Outcomes from Agile Software Development Projects

  1. 1. The Impact of Methods and Techniques on Outcomes from Agile Software Development Projects David Parsons, Hokyoung Ryu, Ramesh Lal
  2. 2. Agenda  Analysis of Scott Ambler’s 2006 survey  Deeper statistical analysis • Do agile methods have a positive effect? • Do agile methods work better in combinations? • What are the most effective methods, individually and in combination • What are the most effective techniques?
  3. 3. Ambler’s Survey  March 2006  Used mailing lists from both Dr. Dobb’s Journal and Software Development magazine  Hosted on line using SurveyMonkey.  Summary article published in Dr. Dobbs Journal (September 2006)  Raw data available for download •  4,232 respondents (good sample size!)
  4. 4. Survey Questions Please rate your skills at the following IT activities…  Number of IT people in your organization  How would you rate your knowledge of agile methodologies?  Which agile methodologies has your organization adopted?  Which agile techniques has your organization adopted? (check all that apply)  How have agile approaches affected your productivity?  How have agile approaches affected the quality of the systems produced?  How have agile approaches affected the cost of development?  How have agile approaches affected the satisfaction of your business stakeholders in the work produced?
  5. 5. Ambler’s conclusions  ‘adoption of agile processes has clearly been a resounding success’ • Productivity, satisfaction and quality all improved • (Ambler did not mention that cost also increased)  Is this true?  Can we ‘prove’ it?
  6. 6. Adopting and Adapting Agile Methods  Adoption and adaptation of agile methods is a complex process  Social, organizational, and technical factors that may influence why an agile technique may or may not be used by a particular software development team.
  7. 7. Our Observations  Many organisations are using more than one method • Is there any point?  Techniques and methods don’t seem to be used as one might expect • What is the underlying relationship?
  8. 8. WARNING!  Approach the rest of this presentation with caution  ‘There are three kinds of lies: lies, damned lies, and statistics’ • Disraeli
  9. 9. Do Different Methods Have Different Outcomes? Outcomes from Agile Methods 4.50 Very murky data! 4.00 3.50 3.00 Agile MSF Agile UP Crystal 2.50 DSDM Value FDD 2.00 Scrum XP Other Method 1.50 No Agile Method 1.00 0.50 0.00 Productivity Quality Cost Satisfaction Outcomes
  10. 10. Agile Methodologies Ag Ac ile tiv M e od st ak el e D ho riv ld en er D pa rti ev c 0 10 20 30 40 50 60 70 80 el o ip pm at io en n t( AM C od D C e D od re ) e fa re ct gr or es in si g on C te st om in m C g on o- co lo ca di tio C ng n on gu tin id uo el us in es D in at te D ab gr at at as io ab e n as re e fa re ct gr or es in si g on Agile Techniques Si Pa te ng ir st in le pr og g so ra ur m Te ci m ng in st in g D fo riv rm Vary Between Methods? en at D io n es N ig Percentage usage of techniques within methodologies o n Ag (T ile D D Te ) ch Does the Use of Techniques ni qu Very murky data! es XP FDD None Other Scrum DSDM Crystal Agile UP AgileMSF
  11. 11. The Data is Deceptive Because  Lots of ‘don’t knows’  Large variations in sample sizes • DSDM used by only a handful, Crystal Clear also not a large sample, unwise to draw too many conclusions  Lots of overlap between methods  Lack of clarity about the relationship between techniques and methods • For example ‘we are doing XP’…
  12. 12. How Many Methods?  No agile methods: 2,541  One agile method: 1,019  Two agile methods: 500  Three or more methods: 175  Does this make any difference?
  13. 13. Our Approach  Do a ‘proper’ statistical analysis • Non parametric analysis (Tukey test) to identify significantly different groupings  Remove all the ‘don’t knows’  Separate out those using zero, one, two three (four, five, seven!) methods • We only want to count them once!  Analyse techniques separately from methods
  14. 14. Some Things to be Aware of  Respondents almost certainly misunderstood Feature Driven Development and over-responded  Respondents may well have misrepresented their use of Test Driven Development and over-responded
  15. 15. Result 1: Number of Methods  Cost • no difference  Productivity • significant difference between using no method or using one  Quality and Satisfaction • significant difference between using no method or using one AND between using one method and using two!  So: • Use at least one, preferably two • No point using three or more
  16. 16. Result 2: If I Use One, Which One?  Productivity: • There is a significant different between both Crystal Clear and Scrum, (0.02) and Crystal Clear and XP (0.02)  Quality • There is a significant difference between Crystal Clear and XP (0.03)  Cost • There is a significant difference in cost between Crystal Clear and Scrum (0.02)  Satisfaction • There is a significant difference between Crystal Clear and Scrum (0.023)  So: Maybe use XP or Scrum?
  17. 17. Which Pairs are Agile Method Pairs Used Most? XP and FDD XP and Scrum Agile UP and XP Agile MSF and XP FDD and Scrum Agile UP and FDD Agile Method Pair Agile UP and Scrum Crystal Clear and XP XP and Other Agile MSF and FDD Crystal Clear and FDD 0 20 40 60 80 100 120 140 160
  18. 18. Result 3: If I Use Two, Which Two  Quality • There is a significant difference between the XP/Scrum combination and all other pairs of methods. • Only 6 pairs were considered in the sample because the sample sizes were too small for other combinations.  Productivity • There is a significant difference between the top pair (XP and Scrum) and the lowest three pairs  There was no significant difference in cost between any pairs of methods  So: Mix XP and Scrum together?
  19. 19. Agile Techniques Techniques Number of Percentage of respondents respondents Active stakeholder participation 938 22.15% Agile model driven development 260 6.14% Code refactoring 1,467 34.64% Code regression testing 1,383 32.66% Co-location 447 10.55% Common coding guidelines 1,595 37.66% Continuous integration 1,113 26.28% Database refactoring 416 9.82% Database regression testing 407 9.61% Pair programming 587 13.86% Single sourcing 241 5.69% Test driven design 959 22.64%
  20. 20. What About Techniques?  Not using any agile techniques is significantly worse in terms of productivity, satisfaction and quality  Only DB regression testing is significantly more expensive  Best techniques for productivity • Co-location, continuous integration, pair programming, test driven design  Best techniques for Quality • Test driven design, pair programming, continuous integration,  Best techniques for Satisfaction • Active stakeholder participation, agile model driven development, test driven design
  21. 21. Are We Doing XP? Agile Technique used with XP Number Percentage of Sample Active stakeholder participation 114 27.14% Code refactoring 269 64.05% Code regression testing 210 50.00% Co-location 66 15.71% Continuous integration 176 41.90% Pair programming 183 43.57% Test Driven Design (TDD) 180 42.86%
  22. 22. Does it matter? Productivity Quality Cost Satisfaction ‘Full’ XP 4.25 4.39 2.70 4.07 Partial XP 3.62 3.77 2.63 3.55
  23. 23. Best Practices?  In the XP context, the most effective techniques appear to be • Test driven design • Code refactoring • Collaborative working • Stakeholder involvement • Co-location
  24. 24. How Many XP Techniques? 5 4 Productivity Quality 3 Cost Satisfaction 2 1 0 1 2 3 4 5 6 7 Number of XP Techniques
  25. 25. Tentative Conclusions  Using agile methods and techniques does appear to be beneficial  It may be useful to combine aspects of XP and Scrum  These techniques appear to consistently contribute to better results • Co-location, pair programming, test driven design  More techniques = better?
  26. 26. Caveat Emptor  This was not our data set and we cannot guarantee its validity  However its conclusions can help us to direct our future research to refute or validate them