Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The Business Value of SW Quality

526 views

Published on

Доклад Gualtiero Bazzana на конференции SPM Conf-5,
6 ноября 2015 г., Минск
www.spmconf.ru

Published in: Education
  • Be the first to comment

The Business Value of SW Quality

  1. 1. The Business Value of SW Quality Minks, November 2015 G. Bazzana – ISTQB® President
  2. 2. ISTQB® 2015 2 • Quantifying the impacts on sw engineering practices on Sw Project productivity – Approach – Quantitative Data • The importance of testing – Good practices – Certification of sw testing competencies – ISTQB® Contents
  3. 3. ISTQB® 2015 3 Drivers COST QUALITY RISK AGILITY • Quicker time to market • Consumption based pricing • Increasing dependency of business from SW • Multiple platforms • SW Intensive systems • Knowledge retention • Competition • Tightening margins
  4. 4. ISTQB® 2015 4 APPLICABLE MARKETS IT Engineering Telco
  5. 5. ISTQB® 2015 5 • We need to provide evidence of quantitative benefits • In God we trust, all others must bring data Identifying «Best Practices» Metrics Benchmarks ROI Analysis
  6. 6. ISTQB® 2015 6 BASIC DEFINITIONS OF SOFTWARE QUALITY • Functional Software Quality Software that combines low defect rates and high levels Of user satisfaction. The software should also meet all user requirements and adhere to international standards. • Structural Software Quality Software that exhibits a robust architecture and can operate In a multi-tier environment without failures or degraded performance. Software has low cyclomatic complexity levels. • Aesthetic Software Quality Software with elegant and easy to use commands and Interfaces, attractive screens, and well formatted outputs.
  7. 7. ISTQB® 2015 7 ECONOMIC DEFINITIONS OF SOFTWARE QUALITY • “Technical debt” The assertion (by Ward Cunningham in 1992) that quick and careless development with poor quality leads to many years of expensive maintenance and enhancements. • Cost of Quality (COQ) The overall costs of prevention, appraisal, internal failures, and external failures. For software these mean defect prevention, pre-test defect removal, testing, and post-release defect repairs. (Consequential damages are usually not counted.) • Total Cost of Ownership (TCO) The sum of development + enhancement + maintenance + support from day 1 until application is retired. (Recalculation at 5 year intervals is recommended.)
  8. 8. ISTQB® 2015 8 World Quality Report Findings
  9. 9. ISTQB® 2015 9 • Budgets dedicated to QA&Testing are growing at warp speed, with a growth pace much higher than anticipated, reaching in average 35% of the IT budgets • Test automation is increasing, jumping in one year from 28% to 45% of test cases • Digital transformation is putting more and more emphasis on customer value and end-user experience as targets for testing • Security is the most pressing concern • Agile and DevOps adoption has become widespread and call for extreme level of speed and integration in testing practices; a lack of professionals with Agile Testing expertise is among the top-three challenges • Mobile testing is maturing, being adopted by 92% of the organizations and consuming 35% of the QA&Testing Budgets for new projects • Testing Centers of Excellence are widely adopted and are turning into more agile and hybridized forms • Test environments are being enhanced with virtualization and cloud solutions HIGHLIGHTS FROM THE 2015 WORLD SW QUALITY REPORT
  10. 10. ISTQB® 2015 10 • Software engineering and software project management are complex activities. Both software development and software management have dozens of methodologies and scores of tools available that are beneficial. In addition, there are quite a few methods and practices that have been shown to be harmful. • In order to evaluate the effectiveness or harm of various methods and practices Capers Jones has developed a scoring method. • Data collected from 1984 through 2015 – About 725 companies (150 clients in Fortune 500) – About 40 government/military groups – About 25,000 total projects – New data = monthly benchmark studies – Data collected from 27 countries – Observations during more than 17 lawsuits • The analysis is based on the author’s book Software Engineering Best Practices published by McGraw Hill in 2010. Some new data is taken from The Economics of Software Quality published by Addison Wesley in 2012. QUANTITATIVE BENCHMARKS (Based on work of Capers Jones)
  11. 11. ISTQB® 2015 11 Certified reusable components 80% Experienced development teams 65% Effective methodologies for specific project types 40% High-level programming languages 30% Use of inspections for complex systems 27% Experienced managers 25% Moderate unpaid overtime by teams 20% Impact of Critical Factors on Software Productivity . POSITIVE (1)
  12. 12. ISTQB® 2015 12 Low requirements creep 20% Logical, planned architecture for large systems 20% Model-based development 20% Due diligence on COTS acquisitions 20% Use of static analysis before testing 18% High CMMI levels 15% Low cyclomatic complexity (< 10) 15% Effective project status tracking 15% Effective defect prevention 15% Experienced test teams 12% Impact of Critical Factors on Software Productivity . POSITIVE (2)
  13. 13. ISTQB® 2015 13 Experienced clients 10% SCRUM < 1000 function points 10% 24-hour continuous development 7% Effective parametric estimating tools 7% Testing by certified test personnel 7% Annual staff training > 5 days 7% Formal mathematical test case design 6% Co-located teams 5% Impact of Critical Factors on Software Productivity . POSITIVE (3)
  14. 14. ISTQB® 2015 14 High requirements creep: poor change control -60% Inexperienced managers -50% Truncating testing to "meet schedule" -45% Concealing problems in status reports -45% Inexperienced clients -40% Chaotic, unplanned architecture for large systems -37% Inexperienced development teams -35% Inaccurate manual estimates -33% Concurrent maintenance and development tasks -30% False claims by outsource vendors -30% Impact of Critical Factors on Software Productivity – NEGATIVE (1)
  15. 15. ISTQB® 2015 15 Adding personnel to late projects -25% Low-level programming languages -25% Excessive unpaid overtime by team -23% Manual estimates > 1000 function points -23% Poor status tracking -20% Unverified, buggy COTS acquisitions -20% High cyclomatic complexity (> 25) -18% Ineffective methodologies -15% Inexperienced test teams -15% Distributed teams: poor communications -15% Impact of Critical Factors on Software Productivity – NEGATIVE (2)
  16. 16. ISTQB® 2015 16 Waterfall > 5000 function points -12% Low CMMI levels -10% Agile > 5000 function points - 9% Informal test case design - 8% No annual staff training - 7% Testing by developers only - 6% Impact of Critical Factors on Software Productivity – NEGATIVE (3)
  17. 17. ISTQB® 2015 17 CONCLUSIONS ON SOFTWARE QUALITY • No single quality method is adequate by itself. • Inspections + static analysis + formal testing > 99% efficient. • Defect prevention + pre-test removal + formal test best overall • Higher CMMI levels, TSP, RUP, Agile, XP are effective • Quality excellence has ROI > $15 for each $1 spent • High quality benefits schedules, productivity, users! • Poor quality leads to cost and schedule overruns!
  18. 18. ISTQB® 2015 18 USEFUL AND HARMFUL QUALITY METRICS • Useful quality metrics – Defect potentials using function points – Defect detection efficiency (DDE) – Defect removal efficiency (DRE) – Delivered defects per function point – Defect removal cost per function point – Cost of quality (COQ) – Cyclomatic complexity – Test coverage
  19. 19. ISTQB® 2015 19 US AVERAGE FOR SW QUALITY AS OF 2015 Defect Removal Delivered Defect Origins Potential Efficiency Defects Requirements 1.11 88.00% 0.13 Architecture 0.25 94.00% 0.02 Design 1.20 93.00% 0.08 Coding 1.30 97.00% 0.03 Documents 0.50 93.40% 0.04 Bad Fixes 0.35 85.00% 0.07 TOTAL 4.71 92.57% 0.35 (Data expressed in terms of defects per function point) (Function points show all defect sources - not just coding defects) (Code defects = 35% of total defects)
  20. 20. ISTQB® 2015 20 RANGES OF DEFECT REMOVAL EFFICIENCY Worst Median Best Defect Potentials 1,100 1,000 900 Defect Prevention 40% 50% 60% Pre-test inspections 70% 85% 90% Pre-test static analysis 40% 65% 75% Unit tests 20% 25% 35% New Function tests 25% 35% 45% Regression tests 15% 25% 35% System test 35% 45% 55% Acceptance/Beta tests 20% 30% 40% DELIVERED DEFECTS 53 18 2 REMOVAL EFFICIENCY 95% 98% 99.7%
  21. 21. ISTQB® 2015 21 ROI FROM STRUCTURED TESTING Requirement Analysis Development Test Production 5 5 % 20 % 40 % 30 % < 5 % 4 3 % 12 % 30 % 50 % 5 % 3 0 % 2 % 20 % 70 % 8 % 2 0 % 0 % 3 % 80 % 17 % 1 0 % 0 % 2 % 50 % 48% FAULT DISTRIBUTION 1 5 20 50 100 TESTMATURITY
  22. 22. ISTQB® 2015 22 ROI FROM STRUCTURED TESTING  With no structured test process, 50% of the overall issues are found during the test phase.  With a structured test it is reasonable to expect to find 80% of the issues during the test phase.  Bug fixing cost if the problem is found during the test phase: 2 md  Bug fixing cost if the problem is found after release: 4 md  A medium sized application generates overall 500 issues (both test and production).  Suppose to reserve a testing team for an effort of 180 md, then…
  23. 23. ISTQB® 2015 23 ROI FROM STRUCTURED TESTING RESULTS:  Issues in production (after release) are reduced by 60%  The overall effort (test + bug fixing) is reduced by 8% Without Structured Test With Structured Test Number of faults found during test 250 400 Effort to fix during test 500 800 Number of faults found in production 250 100 Effort to fix faults found in production 1000 400 Effort for structured testing 0 180 Overall effort 1500 1380
  24. 24. ISTQB® 2015 24 • In the following some factors related to testing are presented, with their impact in terms of reduction or increase in project work hours wrt to projects consisting of traditional application development methods such as “waterfall” development performed by organizations that do not have apply systematically sound software engineering practices INCREASING PRODUCTIVITY OF SW PROJECTS GOOD TESTING PRACTICES CAN HELP Factor Impact on productivity Experienced test teams 12% Improvement Testing by certified test personnel 7% Improvement Testing by developers only 6% Worsening Informal test case design 8% Worsening Inexperienced test teams 15% Worsening Truncating testing to "meet schedule" 45% Worsening (Capers Jones, Scoring and evaluating software methods, practices, and results, version 15.0, September 6, 2014)
  25. 25. ISTQB® 2015 25 • ISTQB®: International Software Testing Qualifications Board (www.istqb.org): • - Non-profit association – Founded in 2002 – Has its own constitution, rules and regulations – Composed of volunteer international Testing Experts – Responsible for the “ISTQB® Certified Tester” scheme worldwide • ISTQB® is the world’s leading organization for Software Testing Certification WHAT IS ISTQB®? Advancing the software testing profession
  26. 26. ISTQB® 2015 26 ISTQB® - Current Product Architecture
  27. 27. ISTQB® 2015 27 “To continually improve and advance the software testing profession by: Defining and maintaining a Body of Knowledge which allows testers to be certified based on best practices, connecting the international software testing community, and encouraging research.” ISTQB® VISION
  28. 28. ISTQB® 2015 28 ISTQB® VIDEOS • ISTQB® Videos give you insights into the ISTQB® Certified Tester scheme http://www.istqb.org/introduction-to-istqb.html
  29. 29. ISTQB® 2015 29 WORLD-WIDE COVERAGE  Countries covered by Member Boards (49 Member Boards covering 72 countries, representing over 90% of the world-wide GDP) and Global Exam Providers  Countries covered by Global Exam Providers The list of Member Boards and Global Exam Providers is available on the ISTQB® Web Site
  30. 30. ISTQB® 2015 30 More than 560.000 exams world wide! Figures as of 2015Q2 Executive Summary In 2015Q2, 18993 exams and 14036 certificates Trend YOY is OK Wrt FL+AL+EL  + 8,5% Adding-up Agile  + 14,5% Close to 410.000 certificates world wide!
  31. 31. ISTQB® 2015 31 Agile uptake VERY Good: - Almost 2000 exams - 1600 certificates - By 33 Boards - Forecast for 2015: over 4000 exams Figures as of 2015Q2 Executive Summary (2) Certificates issued in 108 countries
  32. 32. ISTQB® 2015 32 ISTQB® ECO-SYSTEM
  33. 33. ISTQB® 2015 33 • International recognition of acquired competencies and skills • Authorized to use the “Certified Tester” logo (specifying the level of certification) • Whole career path support, from Foundation to Expert level • Higher appeal in the labor market BENEFITS FOR PROFESSIONALS
  34. 34. ISTQB® 2015 34 BENEFITS FOR PROFESSIONALS - SURVEY • Would you recommend the ISTQB® Foundation Level (CTFL) certification to your colleagues?
  35. 35. ISTQB® 2014 35 BENEFITS FOR COMPANIES • ISTQB® certification can provide a competitive advantage for companies, promising a higher level of reliability of the applications being developed due to efficient and cost effective testing practices derived from the ISTQB® competencies • Consulting companies with certified staff can offer higher-level services to customers, increasing revenues and brand value • ISTQB® has defined a “Partner Program” for companies that engage a large number of certified testers
  36. 36. ISTQB® 2014 36 BENEFITS FOR ATPs (Accredited Training Providers) • Educational institutions and consulting companies may become an ISTQB® Accredited Trainer Provider (ATP) according to processes and rules defined at the international level • Accredited Training Providers ensure a high standing of training is delivered through having: • certified trainers • the content, quality and syllabus coverage of training materials checked by ISTQB® Boards • advance notice of changes to the ISTQB® Glossary and syllabi • Accredited Training Providers are entitled to use relevant logos and are listed in the ISTQB® Website
  37. 37. ISTQB® 2015 37 • The exam is based on multiple-choice questions for Foundation and Advanced Levels • For Expert Level, an essay is also required • Exams can be attempted also without having attended a training course (e.g. through self-study) EXAMS
  38. 38. ISTQB® 2015 38 • Questions are classified according to the cognitive level, the K-level (also known as level of knowledge): • K1 = Remember (recognize a term or concept) • K2 = Understand (able to explain a concept) • K3 = Apply (select correct application of concept or technique) • K4 = Analyze (can distinguish between facts and inferences for deeper understanding) • K5 = Evaluate (can make judgments based on criteria and standards) • K6 = Create (can put elements together to form a coherent or functional whole) • The number of questions for each topic is related to the length of the topic in the syllabus • For more details, see the FAQ section in the ISTQB® Website EXAM QUESTIONS – K LEVELS
  39. 39. ISTQB® 2014 39 Provides recognition to Companies that are investing in the ISTQB® scheme ISTQB® PARTNER PROGRAM
  40. 40. ISTQB® 2015 40 ISTQB® Planned evolution  ISTQB® is evaluating an evolution of its product portfolio architecture and contents in order to:  Maintain the mission and vision of ISTQB® and keep the high quality of deliverables that has marked the success of the scheme world- wide  Make the scheme more modular  Make it easier for professionals to obtain the certification they are interested in  Provide an overall framework in which all future potential modules may fit in a way which is coherent and understandable to our stakeholders  Maintain the validity of certifications already obtained
  41. 41. ISTQB® 2015 41 The customer perspective  Some people like to follow a breadth-first generalist approach (going from Foundation to the “classic” Advanced)  Whereas others prefer to follow a depth-first approach (going from Foundation to Specialist)  We have so many different professionals interested in our programme that we cannot think we can propose a “one size fits all” approach …  Indeed, it is exactly the goal of a modular approach to allow people to design their own training/ certification path and allow them to broaden their knowledge base
  42. 42. ISTQB® 2015 42 Evolution Axes  The review of the product architecture and product portfolio embraces three aspects 1. The overall framework in which the various certification modules have to fit 2. The modules that populate the framework: fitting the current modules (the existing ones and those under development) and identifying/ positioning the potential new ones 3. The preconditions/ pre-requisites/ entry criteria
  43. 43. ISTQB® 2015 43 Overall Framework/ architecture Levels and categories  The future ISTQB® Portfolio will be a Matrix characterized by:  Levels (FL, AL, EL): they identify progressively increasing LOs  Categories: they identify differing target groups of certification modules:  Core  they cover a topic in a broad, horizontal way,  valid for any technology/ methodology/ application domain  Allowing for a common understanding  Agile  So important in new developments  Specialist  they cover a specific topic/ domain in a vertical way,
  44. 44. Let’s Make a Difference… Let’s support our Customers… THANK YOU!

×