Agile Methods Cost of Quality: Benefits of Testing Early & Often


Published on

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Agile Methods Cost of Quality: Benefits of Testing Early & Often

  1. 1. Agile MethodsCost of QualityBenefits of Testing Early & OftenDr. David F. Rico, PMP, ACP, CSMTwitter: @dr_david_f_ricoWebsite: http://www.davidfrico.comLinkedIn:’s Agile Articles:
  2. 2. Author Background DoD contractor with 28+ years of IT experience B.S. Comp. Sci., M.S. Soft. Eng., & D.M. Info. Sys. Large gov’t projects in U.S., Far/Mid-East, & Europe2 Published six books & numerous journal articles Adjunct at George Wash, UMBC, UMUC, Argosy Agile Program Management & Lean Development Specializes in metrics, models, & cost engineering Six Sigma, CMMI, ISO 9001, DoDAF, & DoD 5000 Cloud Computing, SOA, Web Services, FOSS, etc.
  3. 3. Today’s Whirlwind Environment3OverrunsAttritionEscalationRunawaysCancellationGlobalCompetitionDemandingCustomersOrganizationDownsizingSystemComplexityTechnologyChangeVagueRequirementsWork LifeImbalanceInefficiencyHigh O&MLower DoQVulnerableN-M BreachReducedIT Budgets81 MonthCycle TimesRedundantData CentersLack ofInteroperabilityPoorIT SecurityOverburdeningLegacy SystemsObsoleteTechnology & SkillsPine, B. J. (1993). Mass customization: The new frontier in business competition. Boston, MA: Harvard Business School Press.Pontius, R. W. (2012). Acquisition of IT: Improving efficiency and effectiveness in IT acquisition in the DoD. Second AnnualAFEI/NDIA Conference on Agile in DoD, Springfield, VA, USA.
  4. 4. Traditional Projects4 Big projects result in poor quality and scope changes Productivity declines with long queues/wait times Large projects are unsuccessful or canceledJones, C. (1991). Applied software measurement: Assuring productivity and quality. New York, NY: McGraw-Hill.Size vs. QualityDefectDensity0.003.206.409.6012.8016.000 2 6 25 100 400Lines of Code (Thousands)Size vs. ProductivityCodeProductionRate0. 2 6 25 100 400Lines of Code (Thousands)Size vs. Requirements GrowthPercentage0%8%16%24%32%40%0 2 6 25 100 400Lines of Code (Thousands)Size vs. SuccessPercentage0%12%24%36%48%60%0 2 6 25 100 400Lines of Code (Thousands)
  5. 5. Global Project Failures5Standish Group. (2010). Chaos summary 2010. Boston, MA: Author.Sessions, R. (2009). The IT complexity crisis: Danger and opportunity. Houston, TX: Object Watch. Challenged and failed projects hover at 67% Big projects fail more often, which is 5% to 10% Of $1.7T spent on IT projects, over $858B were lost16% 53% 31%27% 33% 40%26% 46% 28%28% 49% 23%34% 51% 15%29% 53% 18%35% 46% 19%32% 44% 24%33% 41% 26%0% 20% 40% 60% 80% 100%199419961998200020022004200620082010YearSuccessful Challenged Failed$0.0$0.4$0.7$1.1$1.4$1.82002 2003 2004 2005 2006 2007 2008 2009 2010Trillions(USDollars)Expenditures Failed Investments
  6. 6. Requirements Defects & Waste6Sheldon, F. T. et al. (1992). Reliability measurement: From theory to practice. IEEE Software, 9(4), 13-20Johnson, J. (2002). ROI: Its your job. Extreme Programming 2002 Conference, Alghero, Sardinia, Italy. Requirements defects are #1 reason projects fail Traditional projects specify too many requirements More than 65% of requirements are never used at allOther 7%Requirements47%Design28%Implementation18%DefectsAlways 7%Often 13%Sometimes16%Rarely19%Never45%Waste
  7. 7. NetworkComputerOperating SystemMiddlewareApplicationsAPIsGUIHow Agile Works Agile requirements implemented in slices vs. layers User needs with higher business value are done first Reduces cost & risk while increasing business success7Shore, J. (2011). Evolutionary design illustrated. Norwegian Developers Conference, Oslo, Norway.Agile Traditional1 2 3 Faster Early ROI Lower Costs Fewer Defects Manageable Risk Better Performance Smaller Attack SurfaceLate No Value Cost Overruns Very Poor Quality Uncontrollable Risk Slowest Performance More Security Incidents Seven Wastes1.Rework2.Motion3.Waiting4.Inventory5.Transportation6.Overprocessing7.OverproductionMINIMIZES MAXIMIZES JIT, Just-enough architecture Early, in-process system V&V Fast continuous improvement Scalable to systems of systems Maximizes successful outcomes Myth of perfect architecture Late big-bang integration tests Year long improvement cycles Breaks down on large projects Undermines business success
  8. 8. What is Agile Testing? Traditional testing is a late, manual process Agile testing is an early and automated process The goal of agile testing is to deliver early and often8Rico, D. F. (2012). Agile testing resources. Retrieved Sep. 9, 2012, from, L., & Gregory, J. (2009). Agile testing: A practical guide for testers and agile teams. Boston, MA: Addison-Wesley.Grant, T. (2005). Continuous integration using cruise control. Northern Virginia Java Users Group (Novajug), Reston, Virginia, USA.Traditional TestingCombining source filesCombining software and environmentCombining software and dataCombining software and testsCombining developersAgile TestingCode is frequently checked inCode is automatically retrievedCompilation is done automaticallyTests are done automaticallyCode reports are generatedDevelopers get instant feedbackCode is automatically deployed orpackaged for delivery
  9. 9. Thousands of TestsContinuously ExecutedNo More Late BigBang IntegrationAgile Testing Model User needs designed & developed one-at-a-time Changes automatically detected, built, and tested System fully tested and deployed as changes occur9Humble, J., & Farley, D. (2011). Continuous delivery. Boston, MA: Pearson Education.Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration. Boston, MA: Addison-Wesley.BuildIntegrationServerVersionControlServerBuildScriptsUsesWatchesBuildStatusProvidesDeveloper ADeveloper BDeveloper CCommitsChangesCommitsChangesCommitsChangesBuildsDatabaseAnalysisTestingReportingDocumentationDeploymentEarly, Automated, Fast,Efficient, & RepeatableConstant ReadinessState & CM ControlLean, Waste Free, Low WIP,No Deadlocked Test QueuesRapidly & SuccessfullyDev. Complex Systems
  10. 10. Agile Testing Done Early & Often Eliminates big-bang integration in the 11th hour Creates a repeatable and reliable testing process Evaluates system-wide changes throughout project10Maeda, M. K. (2009). Agile testing: Early, often, and Smart. Arlington, MA: Cutter Consortium.
  11. 11. Agile Testing Practices Agile testing consists of seven broad practices Includes automated builds, testing, inspections, etc. Also includes reporting, documentation, deployment, etc.11PracticeBuildingDatabaseInspectionsTestingFeedbackDocumentationDeploymentDescriptionFrequently assembling products and services to ensure delivery readinessFrequently generating/analyzing database schemas, queries, and formsFrequently performing automated static analysis of product/service qualityFrequently performing automated dynamic product and service evaluationFrequently generating automated status reports/messages for all stakeholdersFrequently performing automated technical/customer document generationFrequently performing automated delivery of products/services to end usersDuvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley.Humble, J., & Farley, D. (2011). Continuous delivery. Boston, MA: Pearson Education.
  12. 12. Agile Testing Workflow12Traditional vs. Agile Cumulative FlowWork(Story,Point,Task)orEffort(Week,Day,Hour)Time Unit (Roadmap, Release, Iteration, Month, Week, Day, Hour, etc.)Work(Story,Point,Task)orEffort(Week,Day,Hour)Time Unit (Roadmap, Release, Iteration, Month, Week, Day, Hour, etc.)Traditional Cumulative Flow Agile Cumulative Flow Late big bang integration increases WIP backlog Agile testing early and often reduces WIP backlog Improves workflow and reduces WIP and lead timesAnderson, D. J. (2004). Agile management for software engineering. Upper Saddle River, NJ: Pearson Education.Anderson, D. J. (2010). Kanban: Successful evolutionary change for your technology business. Sequim, WA: Blue Hole Press.
  13. 13. Agile Testing Costs & BenefitsGrant, T. (2005). Continuous integration using cruise control. Northern Virginia Java Users Group (Novajug), Reston, Virginia, USA.Fredrick, J. (2008). Accelerate software delivery with continuous integration and testing. Japanese Symposium on Software Testing, Tokyo, Japan. Most agile testing tools are “free” open source A build server is no more than a commodity PC 10x more efficient/effective than traditional testing13
  14. 14. Agile Testing Economics Traditional testing finds a defect in about 10 hours Manual code inspections find a defect in 1 hour Agile testing finds a defect every 6 minutes14Rico, D. F. (2012). The Cost of Quality (CoQ) for Agile vs. Traditional Project Management. Fairfax, VA: Gantthead.Com.
  15. 15. Agile Cost of Quality (CoQ) Agile testing is 10x better than code inspections Agile testing is 100x better than traditional testing Agile testing is done earlier “and” 1,000x more often15Rico, D. F. (2012). The Cost of Quality (CoQ) for Agile vs. Traditional Project Management. Fairfax, VA: Gantthead.Com.
  16. 16. Agile Testing Statistics Fewer builds leave in higher bug counts A high number of builds eliminates the defects Goal is to have as many, early builds as possible16Lacoste, F. J. (2009). Killing the gatekeeper: Introducing a continuous integration system. Proceedings of the Agile 2009 Conference, Chicago, Illinois, USA, 387-392. 
  17. 17. Scaling Agile Testing Agile testing slows down with very large systems Slow testing slows integration and increases bugs Agile testing can speed back up with proper attention17Kokko, H. (2009). Increase productivity with large scale continuous integration. Proceedings of the Agile 2009 Conference, Chicago, Illinois, USA.Wide Impact TuningFast builds – less changes – more greenParallelization of test runsClearCase to subversionPre-installing as much as possibleRemoval of randomnessCompilation in memoryInstallation starting parallel with systembuildFocused Impact TuningMore memory and CPUsParallelize buildsReplace 3rd party test librariesReduce/remove timeouts in testsSelect different testsRefactor code & componentsTune the network & softwareTune the database
  18. 18. Agile Cost & Benefit Analysis Costs based on avg. productivity and quality Productivity ranged from 4.7 to 5.9 LOC an hour Costs were $588,202 and benefits were $3,930,63118Rico, D. F., Sayani, H. H., & Sone, S. (2009). The business value of agile software methods: Maximizing ROI with just-in-time processes and documentation.Ft. Lauderdale, FL: J. Ross Publishing.d1 = [ln(Benefits  Costs) + (Rate + 0.5  Risk2)  Years]  Risk   Years, d2 = d1  Risk   Years 51i
  19. 19. Benefits of Agile Methods Analysis of 23 agile vs. 7,500 traditional projects Agile projects are 54% better than traditional ones Agile has lower costs (61%) and fewer defects (93%)Mah, M. (2008). Measuring agile in the enterprise: Proceedings of the Agile 2008 Conference, Toronto, Canada.Project Cost in Millions $0.751.502. AgileAfter Agile61%LowerCostTotal Staffing1811Before AgileAfter Agile39%LessStaff5101520Delivery Time in Months51015201813.5Before AgileAfter Agile24%FasterCumulative Defects6251250187525002270381Before AgileAfter Agile93%LessDefects19
  20. 20. Agile vs. Traditional Success Traditional projects succeed at 50% industry avg. Traditional projects are challenged 20% more often Agile projects succeed 3x more and fail 3x less oftenStandish Group. (2012). Chaos manifesto. Boston, MA: Author.20Agile TraditionalSuccess42%Failed9%Challenged49%Success14%Failed29%Challenged57%
  21. 21. Hoque, F., et al. (2007). Business technology convergence. The role of business technology convergence in innovationand adaptability and its effect on financial performance. Stamford, CT: BTM Institute.21 Study of 15 agile vs. non-agile Fortune 500 firms Based on models to measure organizational agility Agile firms out perform non agile firms by up to 36%Benefits of Organizational Agility
  22. 22. Agile Recap Agile methods DON’T mean deliver it now & fix it later Lightweight, yet disciplined approach to development Reduced cost, risk, & waste while improving quality22Rico, D. F. (2012). What’s really happening in agile methods: Its principles revisited? Retrieved June 6, 2012, from, D. F. (2012). The promises and pitfalls of agile methods. Retrieved February 6, 2013 from,, D. F. (2012). How do lean & agile intersect? Retrieved February 6, 2013, from How ResultFlexibility Use lightweight, yet disciplined processes and artifacts Low work-in-processCustomer Involve customers early and often throughout development Early feedbackPrioritize Identify highest-priority, value-adding business needs Focus resourcesDescope Descope complex programs by an order of magnitude Simplify problemDecompose Divide the remaining scope into smaller batches Manageable piecesIterate Implement pieces one at a time over long periods of time Diffuse riskLeanness Architect and design the system one iteration at a time JIT waste-free designSwarm Implement each component in small cross-functional teams Knowledge transferCollaborate Use frequent informal communications as often as possible Efficient data transferTest Early Incrementally test each component as it is developed Early verificationTest Often Perform system-level regression testing every few minutes Early validationAdapt Frequently identify optimal process and product solutions Improve performance
  23. 23. Conclusion23 Agility is the evolution of management thought Confluence of traditional and non-traditional ideas Improve performance by over an order of magnitude“The world of traditional methods belongs to yesterday”“Don’t waste your time using traditional methods on 21st century projects”Agile methods are …Systems development approachesNew product development approachesExpertly designed to be fast and efficientIntentionally lean and free of waste (muda)Systematic highly-disciplined approachesCapable of producing high quality systemsRight-sized, just-enough, and just-in-time tools Scalable to large, complex mission-critical systems Designed to maximize business value for customersWysocki, R.F. (2010). Adaptive project framework: Managing complexity in the face of uncertainty. Boston, MA: Pearson Education. 
  24. 24. Books on ROI of SW Methods Guides to software methods for business leaders Communicates business value of software methods Rosetta stones to unlocking ROI of software methods (Description) (Description)24