Downloaded 100 times
![Agile Cost & Benefit Analysis
Costs based on avg. productivity and quality
Productivity ranged from 4.7 to 5.9 LOC an hour
Costs were $588,202 and benefits were $3,930,631
18
Rico, D. F., Sayani, H. H., & Sone, S. (2009). The business value of agile software methods: Maximizing ROI with just-in-time processes and documentation.
Ft. Lauderdale, FL: J. Ross Publishing.
d1 = [ln(Benefits Costs) + (Rate + 0.5 Risk2) Years] Risk Years, d2 = d1 Risk Years
5
1i](https://image.slidesharecdn.com/rico12z-130507210748-phpapp01/85/Agile-Methods-Cost-of-Quality-Benefits-of-Testing-Early-Often-18-320.jpg)
Uploaded byDavid Rico
Agile Methods Cost of Quality: Benefits of Testing Early & Often
This document discusses the benefits of testing early and often using agile methods. It summarizes that agile testing is done continuously and automated through frequent integration, which finds defects much faster than traditional late testing. This reduces costs and risks while improving quality.
More Related Content
Similar to Agile Methods Cost of Quality: Benefits of Testing Early & Often
Agile Methods Cost of Quality: Benefits of Testing Early & Often
- 1. Agile Methods Cost ofQuality Benefits of Testing Early & Often Dr. David F. Rico, PMP, ACP, CSM Twitter: @dr_david_f_rico Website: http://www.davidfrico.com LinkedIn: http://www.linkedin.com/in/davidfrico Facebook: http://www.facebook.com/profile.php?id=1540017424 Dave’s Agile Articles: http://davidfrico.com/agile-message.doc
- 2. Author Background DoDcontractor with 28+ years of IT experience B.S. Comp. Sci., M.S. Soft. Eng., & D.M. Info. Sys. Large gov’t projects in U.S., Far/Mid-East, & Europe 2 Published six books & numerous journal articles Adjunct at George Wash, UMBC, UMUC, Argosy Agile Program Management & Lean Development Specializes in metrics, models, & cost engineering Six Sigma, CMMI, ISO 9001, DoDAF, & DoD 5000 Cloud Computing, SOA, Web Services, FOSS, etc.
- 3. Today’s Whirlwind Environment 3 Overruns Attrition Escalation Runaways Cancellation Global Competition Demanding Customers Organization Downsizing System Complexity Technology Change Vague Requirements WorkLife Imbalance Inefficiency High O&M Lower DoQ Vulnerable N-M Breach Reduced IT Budgets 81 Month Cycle Times Redundant Data Centers Lack of Interoperability Poor IT Security Overburdening Legacy Systems Obsolete Technology & Skills Pine, B. J. (1993). Mass customization: The new frontier in business competition. Boston, MA: Harvard Business School Press. Pontius, R. W. (2012). Acquisition of IT: Improving efficiency and effectiveness in IT acquisition in the DoD. Second Annual AFEI/NDIA Conference on Agile in DoD, Springfield, VA, USA.
- 4. Traditional Projects 4 Bigprojects result in poor quality and scope changes Productivity declines with long queues/wait times Large projects are unsuccessful or canceled Jones, C. (1991). Applied software measurement: Assuring productivity and quality. New York, NY: McGraw-Hill. Size vs. Quality DefectDensity 0.00 3.20 6.40 9.60 12.80 16.00 0 2 6 25 100 400 Lines of Code (Thousands) Size vs. Productivity CodeProductionRate 0.00 1.00 2.00 3.00 4.00 5.00 0 2 6 25 100 400 Lines of Code (Thousands) Size vs. Requirements Growth Percentage 0% 8% 16% 24% 32% 40% 0 2 6 25 100 400 Lines of Code (Thousands) Size vs. Success Percentage 0% 12% 24% 36% 48% 60% 0 2 6 25 100 400 Lines of Code (Thousands)
- 5. Global Project Failures 5 StandishGroup. (2010). Chaos summary 2010. Boston, MA: Author. Sessions, R. (2009). The IT complexity crisis: Danger and opportunity. Houston, TX: Object Watch. Challenged and failed projects hover at 67% Big projects fail more often, which is 5% to 10% Of $1.7T spent on IT projects, over $858B were lost 16% 53% 31% 27% 33% 40% 26% 46% 28% 28% 49% 23% 34% 51% 15% 29% 53% 18% 35% 46% 19% 32% 44% 24% 33% 41% 26% 0% 20% 40% 60% 80% 100% 1994 1996 1998 2000 2002 2004 2006 2008 2010 Year Successful Challenged Failed $0.0 $0.4 $0.7 $1.1 $1.4 $1.8 2002 2003 2004 2005 2006 2007 2008 2009 2010 Trillions(USDollars) Expenditures Failed Investments
- 6. Requirements Defects &Waste 6 Sheldon, F. T. et al. (1992). Reliability measurement: From theory to practice. IEEE Software, 9(4), 13-20 Johnson, J. (2002). ROI: It's your job. Extreme Programming 2002 Conference, Alghero, Sardinia, Italy. Requirements defects are #1 reason projects fail Traditional projects specify too many requirements More than 65% of requirements are never used at all Other 7% Requirements 47% Design 28% Implementation 18% Defects Always 7% Often 13% Sometimes 16% Rarely 19% Never 45% Waste
- 7. Network Computer Operating System Middleware Applications APIs GUI How AgileWorks Agile requirements implemented in slices vs. layers User needs with higher business value are done first Reduces cost & risk while increasing business success 7Shore, J. (2011). Evolutionary design illustrated. Norwegian Developers Conference, Oslo, Norway. Agile Traditional 1 2 3 Faster Early ROI Lower Costs Fewer Defects Manageable Risk Better Performance Smaller Attack Surface Late No Value Cost Overruns Very Poor Quality Uncontrollable Risk Slowest Performance More Security Incidents Seven Wastes 1.Rework 2.Motion 3.Waiting 4.Inventory 5.Transportation 6.Overprocessing 7.Overproduction MINIMIZES MAXIMIZES JIT, Just-enough architecture Early, in-process system V&V Fast continuous improvement Scalable to systems of systems Maximizes successful outcomes Myth of perfect architecture Late big-bang integration tests Year long improvement cycles Breaks down on large projects Undermines business success
- 8. What is AgileTesting? Traditional testing is a late, manual process Agile testing is an early and automated process The goal of agile testing is to deliver early and often 8 Rico, D. F. (2012). Agile testing resources. Retrieved Sep. 9, 2012, from http://davidfrico.com/agile-testing-resources.txt Crispin, L., & Gregory, J. (2009). Agile testing: A practical guide for testers and agile teams. Boston, MA: Addison-Wesley. Grant, T. (2005). Continuous integration using cruise control. Northern Virginia Java Users Group (Novajug), Reston, Virginia, USA. Traditional Testing Combining source files Combining software and environment Combining software and data Combining software and tests Combining developers Agile Testing Code is frequently checked in Code is automatically retrieved Compilation is done automatically Tests are done automatically Code reports are generated Developers get instant feedback Code is automatically deployed or packaged for delivery
- 9. Thousands of Tests ContinuouslyExecuted No More Late Big Bang Integration Agile Testing Model User needs designed & developed one-at-a-time Changes automatically detected, built, and tested System fully tested and deployed as changes occur 9Humble, J., & Farley, D. (2011). Continuous delivery. Boston, MA: Pearson Education. Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration. Boston, MA: Addison-Wesley. Build Integration Server Version Control Server Build Scripts UsesWatches Build Status ProvidesDeveloper A Developer B Developer C Commits Changes Commits Changes Commits Changes Builds Database Analysis Testing Reporting Documentation Deployment Early, Automated, Fast, Efficient, & Repeatable Constant Readiness State & CM Control Lean, Waste Free, Low WIP, No Deadlocked Test Queues Rapidly & Successfully Dev. Complex Systems
- 10. Agile Testing DoneEarly & Often Eliminates big-bang integration in the 11th hour Creates a repeatable and reliable testing process Evaluates system-wide changes throughout project 10Maeda, M. K. (2009). Agile testing: Early, often, and Smart. Arlington, MA: Cutter Consortium.
- 11. Agile Testing Practices Agile testing consists of seven broad practices Includes automated builds, testing, inspections, etc. Also includes reporting, documentation, deployment, etc. 11 Practice Building Database Inspections Testing Feedback Documentation Deployment Description Frequently assembling products and services to ensure delivery readiness Frequently generating/analyzing database schemas, queries, and forms Frequently performing automated static analysis of product/service quality Frequently performing automated dynamic product and service evaluation Frequently generating automated status reports/messages for all stakeholders Frequently performing automated technical/customer document generation Frequently performing automated delivery of products/services to end users Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley. Humble, J., & Farley, D. (2011). Continuous delivery. Boston, MA: Pearson Education.
- 12. Agile Testing Workflow 12 Traditionalvs. Agile Cumulative Flow Work(Story,Point,Task)orEffort(Week,Day,Hour) Time Unit (Roadmap, Release, Iteration, Month, Week, Day, Hour, etc.) Work(Story,Point,Task)orEffort(Week,Day,Hour) Time Unit (Roadmap, Release, Iteration, Month, Week, Day, Hour, etc.) Traditional Cumulative Flow Agile Cumulative Flow Late big bang integration increases WIP backlog Agile testing early and often reduces WIP backlog Improves workflow and reduces WIP and lead times Anderson, D. J. (2004). Agile management for software engineering. Upper Saddle River, NJ: Pearson Education. Anderson, D. J. (2010). Kanban: Successful evolutionary change for your technology business. Sequim, WA: Blue Hole Press.
- 13. Agile Testing Costs& Benefits Grant, T. (2005). Continuous integration using cruise control. Northern Virginia Java Users Group (Novajug), Reston, Virginia, USA. Fredrick, J. (2008). Accelerate software delivery with continuous integration and testing. Japanese Symposium on Software Testing, Tokyo, Japan. Most agile testing tools are “free” open source A build server is no more than a commodity PC 10x more efficient/effective than traditional testing 13
- 14. Agile Testing Economics Traditional testing finds a defect in about 10 hours Manual code inspections find a defect in 1 hour Agile testing finds a defect every 6 minutes 14 Rico, D. F. (2012). The Cost of Quality (CoQ) for Agile vs. Traditional Project Management. Fairfax, VA: Gantthead.Com.
- 15. Agile Cost ofQuality (CoQ) Agile testing is 10x better than code inspections Agile testing is 100x better than traditional testing Agile testing is done earlier “and” 1,000x more often 15 Rico, D. F. (2012). The Cost of Quality (CoQ) for Agile vs. Traditional Project Management. Fairfax, VA: Gantthead.Com.
- 16. Agile Testing Statistics Fewer builds leave in higher bug counts A high number of builds eliminates the defects Goal is to have as many, early builds as possible 16 Lacoste, F. J. (2009). Killing the gatekeeper: Introducing a continuous integration system. Proceedings of the Agile 2009 Conference, Chicago, Illinois, USA, 387-392.
- 17. Scaling Agile Testing Agile testing slows down with very large systems Slow testing slows integration and increases bugs Agile testing can speed back up with proper attention 17 Kokko, H. (2009). Increase productivity with large scale continuous integration. Proceedings of the Agile 2009 Conference, Chicago, Illinois, USA. Wide Impact Tuning Fast builds – less changes – more green Parallelization of test runs ClearCase to subversion Pre-installing as much as possible Removal of randomness Compilation in memory Installation starting parallel with system build Focused Impact Tuning More memory and CPUs Parallelize builds Replace 3rd party test libraries Reduce/remove timeouts in tests Select different tests Refactor code & components Tune the network & software Tune the database
- 18. Agile Cost &Benefit Analysis Costs based on avg. productivity and quality Productivity ranged from 4.7 to 5.9 LOC an hour Costs were $588,202 and benefits were $3,930,631 18 Rico, D. F., Sayani, H. H., & Sone, S. (2009). The business value of agile software methods: Maximizing ROI with just-in-time processes and documentation. Ft. Lauderdale, FL: J. Ross Publishing. d1 = [ln(Benefits Costs) + (Rate + 0.5 Risk2) Years] Risk Years, d2 = d1 Risk Years 5 1i
- 19. Benefits of AgileMethods Analysis of 23 agile vs. 7,500 traditional projects Agile projects are 54% better than traditional ones Agile has lower costs (61%) and fewer defects (93%) Mah, M. (2008). Measuring agile in the enterprise: Proceedings of the Agile 2008 Conference, Toronto, Canada. Project Cost in Millions $ 0.75 1.50 2.25 3.00 2.8 1.1 Before Agile After Agile 61% Lower Cost Total Staffing 18 11 Before Agile After Agile 39% Less Staff 5 10 15 20 Delivery Time in Months 5 10 15 20 18 13.5 Before Agile After Agile 24% Faster Cumulative Defects 625 1250 1875 2500 2270 381 Before Agile After Agile 93% Less Defects 19
- 20. Agile vs. TraditionalSuccess Traditional projects succeed at 50% industry avg. Traditional projects are challenged 20% more often Agile projects succeed 3x more and fail 3x less often Standish Group. (2012). Chaos manifesto. Boston, MA: Author. 20 Agile Traditional Success 42% Failed 9% Challenged 49% Success 14% Failed 29% Challenged 57%
- 21. Hoque, F., etal. (2007). Business technology convergence. The role of business technology convergence in innovation and adaptability and its effect on financial performance. Stamford, CT: BTM Institute. 21 Study of 15 agile vs. non-agile Fortune 500 firms Based on models to measure organizational agility Agile firms out perform non agile firms by up to 36% Benefits of Organizational Agility
- 22. Agile Recap Agilemethods DON’T mean deliver it now & fix it later Lightweight, yet disciplined approach to development Reduced cost, risk, & waste while improving quality 22 Rico, D. F. (2012). What’s really happening in agile methods: Its principles revisited? Retrieved June 6, 2012, from http://davidfrico.com/agile-principles.pdf Rico, D. F. (2012). The promises and pitfalls of agile methods. Retrieved February 6, 2013 from, http://davidfrico.com/agile-pros-cons.pdf Rico, D. F. (2012). How do lean & agile intersect? Retrieved February 6, 2013, from http://davidfrico.com/agile-concept-model-3.pdf What How Result Flexibility Use lightweight, yet disciplined processes and artifacts Low work-in-process Customer Involve customers early and often throughout development Early feedback Prioritize Identify highest-priority, value-adding business needs Focus resources Descope Descope complex programs by an order of magnitude Simplify problem Decompose Divide the remaining scope into smaller batches Manageable pieces Iterate Implement pieces one at a time over long periods of time Diffuse risk Leanness Architect and design the system one iteration at a time JIT waste-free design Swarm Implement each component in small cross-functional teams Knowledge transfer Collaborate Use frequent informal communications as often as possible Efficient data transfer Test Early Incrementally test each component as it is developed Early verification Test Often Perform system-level regression testing every few minutes Early validation Adapt Frequently identify optimal process and product solutions Improve performance
- 23. Conclusion 23 Agility isthe evolution of management thought Confluence of traditional and non-traditional ideas Improve performance by over an order of magnitude “The world of traditional methods belongs to yesterday” “Don’t waste your time using traditional methods on 21st century projects” Agile methods are … Systems development approaches New product development approaches Expertly designed to be fast and efficient Intentionally lean and free of waste (muda) Systematic highly-disciplined approaches Capable of producing high quality systems Right-sized, just-enough, and just-in-time tools Scalable to large, complex mission-critical systems Designed to maximize business value for customers Wysocki, R.F. (2010). Adaptive project framework: Managing complexity in the face of uncertainty. Boston, MA: Pearson Education.
- 24. Books on ROIof SW Methods Guides to software methods for business leaders Communicates business value of software methods Rosetta stones to unlocking ROI of software methods http://davidfrico.com/agile-book.htm (Description) http://davidfrico.com/roi-book.htm (Description) 24