Successfully reported this slideshow.
Mag. Rudolf Ramler
                                               rudolf.ramler@scch.at
                                  ...
Overview


    The Vital Few and the Useful Many
    The Value of Priorities
    Requirements-based Test Management
    Ri...
The Vital Few
                                    and the Useful Many

      20% of his peapods yielded
      80% of the p...
Static Analysis
                                of Large Software Systems

    20% files   74% LOC (relevant lines)
    20...
Boehm & Basili
                       Software Defect Reduction Top 10



    "About 80 percent of the defects come from
 ...
Overview


    The Vital Few and the Useful Many
    The Value of Priorities
    Requirements-based Test Management
    Ri...
Value of Priorities
                          The Vital Few and the Useful Many


    Is every test cases, defect, require...
Value of Priorities
                                 ROI of ATG vs. Pareto




R. Ramler                                  ...
Setting Priorities
                                                         Approaches


1. Requirements-based testing
   ...
Requirements-based Testing
                                                      Approach


    Trace requirements to     ...
Overview


    The Vital Few and the Useful Many
    The Value of Priorities
    Requirements-based Test Management
    Ri...
Risk-based Testing
                                                                                                       ...
Risk-based Testing
                                                                                       Case Study: Web ...
Overview


    The Vital Few and the Useful Many
    The Value of Priorities
    Requirements-based Test Management
    Ri...
Test Case Selection
                                                 Research Studies




                                ...
Upcoming SlideShare
Loading in …5
×

Value-Based Software Testing

1,119 views

Published on

  • Be the first to comment

  • Be the first to like this

Value-Based Software Testing

  1. 1. Mag. Rudolf Ramler rudolf.ramler@scch.at +43 7236 3343 872 http://www.scch.at Value-Based Software Testing using Requirements and Risks Software Competence Center Hagenberg Software Research Company About 65 employees We focus on the areas Software Architectures and Technologies Automated Software Engineering Industrial Data Warehousing Knowledge-based Vision Systems Industrial Data Mining Process and Quality Engineering SCCH closes the gap between Software Research and Software Product Development 2 R. Ramler 2 1
  2. 2. Overview The Vital Few and the Useful Many The Value of Priorities Requirements-based Test Management Risk-based Test Management Test Case Selection R. Ramler 3 The Vital Few and the Useful Many Pareto, Vilfredo (1848-1923) Italian economist and sociologist, known for his theory on mass and elite interaction and his application of mathematics to economic analysis. 20% of the population controls 80% of the wealth and income … R. Ramler 4 2
  3. 3. The Vital Few and the Useful Many 20% of his peapods yielded 80% of the peas that were harvested 20% of criminals account for 80% of crime 20% of motorists account for 80% of accidents 20% of streets account for 80% of the traffic 20% of your carpet gets 80% of the wear … R. Ramler 5 SLAC Computer Networking WAN Monitoring, Dec. 1998 20% of pings 80% of delay R. Ramler 6 3
  4. 4. Static Analysis of Large Software Systems 20% files 74% LOC (relevant lines) 20% classes 82% outbound calls, 50% classes no calls R. Ramler 7 The Vital Few and the Useful Many 20% of his peapods yielded 80% of the peas that were harvested 20% of criminals account for 80% of crime 20% of motorists account for 80% of accidents 20% of streets account for 80% of the traffic 20% of your carpet gets 80% of the wear … 20% of product flaws account for 80% of problems R. Ramler 8 4
  5. 5. Boehm & Basili Software Defect Reduction Top 10 "About 80 percent of the defects come from 20 percent of the modules, and about half the modules are defect free. … About 90 percent of the downtime comes from, at most, 10 percent of the defects." Barry Boehm, Victor R. Basili: Software Defect Reduction Top 10 List. IEEE Computer, January 2001 R. Ramler 9 The Vital Few and the Useful Many 20% of his peapods yielded 80% of the peas that were harvested 20% of criminals account for 80% of crime 20% of motorists account for 80% of accidents 20% of streets account for 80% of the traffic 20% of your carpet gets 80% of the wear … 20% of product flaws account for 80% of problems 20% of clients usually account for 80% of an organization’s profits R. Ramler 10 5
  6. 6. Overview The Vital Few and the Useful Many The Value of Priorities Requirements-based Test Management Risk-based Test Management Test Case Selection R. Ramler 11 Value of Priorities Example R. Ramler 12 6
  7. 7. Value of Priorities The Vital Few and the Useful Many Is every test cases, defect, requirement equally important? Pareto principle: 20% of clients account for 80% of the profits In the example: 15 different customer billing types Type #1 accounts for about 50% of the transactions Type #1 to #3 together 60 for about 80% 50 40 30 20 10 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 R. Ramler 13 Value of Priorities 20% Customer Types 80% Business Value R. Ramler 14 7
  8. 8. Value of Priorities ROI of ATG vs. Pareto R. Ramler 15 Value of Priorities ROI of ATG vs. Pareto Pareto Testing Maximum ROI with only 40% of tests! ting G Tes AT R. Ramler 16 8
  9. 9. Setting Priorities Approaches 1. Requirements-based testing What value do the requirements have? 2. Risk-based testing What puts the value realization in danger? 3. Test case selection techniques What are your best test cases? TCS Risks Requirements R. Ramler 17 Overview The Vital Few and the Useful Many The Value of Priorities Requirements-based Test Management Risk-based Test Management Test Case Selection TCS Risks Requirements R. Ramler 18 9
  10. 10. Requirements-based Testing Approach Trace requirements to Tool support needed test cases and vice versa Value-based tracing Derive test cases from Avoid bureaucracy requirements Trace only important requirements Derive priorities from requirements Assess requirements coverage ? TC1 Notify testing on TC2 requirements changes R1 TC3 Report test results against R2 TC4 requirements R3 ? Requirements Test cases R. Ramler 19 R. Ramler 20 10
  11. 11. Overview The Vital Few and the Useful Many The Value of Priorities Requirements-based Test Management Risk-based Test Management Test Case Selection TCS Risks Requirements R. Ramler 21 Risk-based Testing Approach [Bach 1999] R. Ramler 22 11
  12. 12. Risk-based Testing How to identify your risks? Risk catalogs – common and domain specific risks Bug taxonomies – common bugs and bug patterns Quality criteria categories Starting point, e.g., customized characteristics ISO 9126 of ISO/IEC Std. 9126-1:2001 ♦ Functionality Assemble a list of quality aspects important for the product or specific in the project ♦ Reliability – What is important for the customer and end user? ♦ Usability – What is typical for the market segment, ♦ Efficiency in similar products? ♦ Maintainability – Experience, support database, expert advice, ... ♦ Portability R. Ramler 23 Risk-based Testing Case Study: Web Testing login <<extend>> view details Quality x Feature User view events <<extend>> upload new document Matrix attach document <<extend>> add event select exisitng document <<extend>> <<extend>> edit event i nvite attendees Registered User <<i nclude>> <<extend>> delete event set security properties subscribe to event Feature Set properties Delete event View events View details Add event Edit event document Attach Functionality Security aspect 3 View events View events View events View events View events View events View events General identity B B C C C C C Suitability Reliability Message content authenticity B B C C C C C Accurac y Interoperability Quality Compliance Aspects Security Maturi y Message content origin C C B B B A B Usability Fault Tolerance Recoverability Integrity C C A A A A Understandability Learnability Operability Secrecy and privacy B B C B C C C Efficiency Attracti veness Ti me Behavior Resource Utilization Accountability B B B B B R. Ramler 24 12
  13. 13. Risk-based Testing Case Study: Web Testing Feature Set properties Delete event View events View details Add event Edit event document Level of Detail in Testing Attach Quality Level Basic Operation Risk Profile Detail R. Ramler 25 Risk-based Testing Case Study: Web Testing In Kappel G. et al.: Web Engineering, Wiley 2006 From "Risk-orientation" to Risk Management at project level R. Ramler 26 13
  14. 14. Overview The Vital Few and the Useful Many The Value of Priorities Requirements-based Test Management Risk-based Test Management Test Case Selection TCS Risks Requirements R. Ramler 27 Test Case Selection Approach Various approaches based on different heuristics Change – favor modified functions, methods, or covered items (provided by version management) Coverage – statement, additional statement, branch, functions, requirements Cost – test cost estimates (e.g., execution time) History – time since a test was last run or previously detected faults Ongoing reseach … R. Ramler 28 14
  15. 15. Test Case Selection Research Studies Random (un-prioritized) Coverage (measured in terms of total number of functions) Change (measured in terms of additional modified functions covered) Optimal (upper bound on prioritization effectiveness) Rothermel G. and S. Elbaum Putting Your Best Tests Forward IEEE Software Aug./Sept. 2003 Summary What You Can Do Value-based Quality Management Risk- and Requirements-based Testing Mining Repositories for Defect Prediction Software Quality Assessment TCS Risks Requirements R. Ramler 30 15

×