SASQAG PowerPoint Slides


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Under-testing Lack of knowledge in formal testing techniques Lack of technical skills Limited domain and/or system expertise Over-testing Initial focus on finding bugs Redundant, unfocused testing Limited domain and/or system expertise
  • Increasing effective coverage efficiently (white box testing - more detail in code) Design structural tests for code coverage (> 75%) Participate in code reviews (single most effective method to prevent bugs) Understand formal testing techniques and application Automate tests effectively Domain expertise Broad system knowledge
  • SASQAG PowerPoint Slides

    1. 1. Software Testers: The Next Generation Bj Rollison, Test Architect Engineering Excellence Group Microsoft, Inc. (
    2. 2. Overview <ul><li>Today’s testers </li></ul><ul><ul><li>State of tester’s knowledge </li></ul></ul><ul><ul><li>Tester performance </li></ul></ul><ul><li>The proof (Lies, damn lies and statistics) </li></ul><ul><ul><li>Empirical evidence </li></ul></ul><ul><li>Why the current way isn’t good enough anymore </li></ul><ul><ul><li>Influences causing change </li></ul></ul><ul><li>Tomorrow’s testers </li></ul><ul><ul><li>Here's what’s happening at MS </li></ul></ul><ul><ul><li>Here's what you should/could do personally to help yourself </li></ul></ul>
    3. 3. Tester training (or lack thereof) <ul><li>“ Less than 10% of testers have formal training in test techniques” – Dorothy Graham </li></ul><ul><li>“… few testers or developers have received any training in formal methods, especially test techniques.” – Marne Hutcheson ( Software Testing Fundamentals ) </li></ul><ul><li>The average software developer reads less than one professional book per year and subscribes to no professional magazines. - Steve McConnell, DeMarco and Lister, Peopleware , 2d Ed, 1999. </li></ul>
    4. 4. Tester’s Body of Knowledge <ul><li>The average tester has read one book on software testing </li></ul><ul><ul><li>Anecdotal evidence suggests the majority of testers have NOT read more than one book on testing </li></ul></ul><ul><ul><ul><li>97% read Testing Computer Software (Kaner et. al) </li></ul></ul></ul><ul><ul><ul><li>4% read How to Break Software (Whittaker) </li></ul></ul></ul><ul><li>Less than 25% have a degree in computer science or other engineering field </li></ul><ul><li>More than 75% unable to effectively code using a modern programming language </li></ul>
    5. 5. The problem is... Universe of functionality & bugs Actual testing effort Under-testing Over-testing Average 30%
    6. 6. The venerable triangle case study “ A program reads three (3) integer values. The three values are interpreted as representing the lengths of the sides of a triangle. The program displays a message that states whether the triangle is scalene, isosceles, or equilateral.” – G. Myers <ul><li>Suggested answers </li></ul><ul><ul><li>G. Meyers – 65 </li></ul></ul><ul><ul><li>P. Jorgensen – 185 </li></ul></ul><ul><ul><li>R. Binder - 65 </li></ul></ul><ul><ul><li>K. Beck – 6 </li></ul></ul><ul><ul><li>R. Collard – 4 </li></ul></ul><ul><li>All answers are right </li></ul><ul><li>because there is no </li></ul><ul><li>real context! </li></ul>
    7. 7. Establishing the baseline if (Side A) or (Side B) or (Side C) != int16 (signed 16 bit int) if input > maxint then error message for overflow condition else if input ! = integer value then error message for format exception if Side A or Side B or Side C <= 0 then error message sides must be > 0 if (Side A + Side B <= Side C) or (Side B + Side C <= Side A) or (Side A + Side C <= Side C) then input does not equate to valid triangle if (Side A = = Side B) and (Side B = = Side C) then triangle is equilateral else if (Side A = = Side B) or (Side A = = Side C) or (Side B = = Side C) then triangle is isosceles else triangle is scalene *Aside from the obvious conditions above additional tests include 1 for max val for all 3 sides, 1 for min val for all 3 sides, and 3 to test for BO of sum of 2 sides
    8. 8. Testing the testers <ul><li>Generate a set of tests using error guessing and exploratory testing methods that will: </li></ul><ul><ul><li>adequately* evaluate the </li></ul></ul><ul><ul><li>triangle algorithm functionality </li></ul></ul><ul><ul><li>implemented in C# </li></ul></ul><ul><ul><li>against the specification </li></ul></ul><ul><li>(Time limit 15 minutes) </li></ul>
    9. 9. Case study demographics <ul><li>Total number participants = 400 </li></ul><ul><ul><li>0% ever had formal training in software testing </li></ul></ul><ul><li>Avg. years testing experience = 2¼ years </li></ul><ul><ul><li>53% – < 1 year experience (20% STE) </li></ul></ul><ul><ul><li>12% – 1 - 2 years experience (60% STE) </li></ul></ul><ul><ul><li>10% – 2 - 4 years experience (70% STE) </li></ul></ul><ul><ul><li>25% – > 4 years experience (100% STE) </li></ul></ul><ul><li>Approx. 50% C/C++/C# coding skill </li></ul>
    10. 10. Case study synopsis <ul><li>31% of tests covered baseline functionality </li></ul><ul><li>28% of the tests did not add significant value </li></ul><ul><ul><li>Redundant coverage, results would not prove or disprove anything not covered by previous tests </li></ul></ul><ul><li>14% of the tests missed the primary objective </li></ul><ul><ul><li>These tests did not focus on the specific triangle functionality of the program </li></ul></ul><ul><li>12% of the tests were incorrect assumptions </li></ul><ul><ul><li>Wrong or incorrect results expected </li></ul></ul>
    11. 11. Overall Results 31% of tests met the primary objective > 50% of test effort was redundant or not focused on primary objective
    12. 12. Results by Experience
    13. 13. STE Results by Experience
    14. 14. SDET Results by Experience
    15. 15. Technical skill comparison
    16. 16. Detailed Analysis <ul><li>40% of the tests covered only 4 tests for the triangle (equilateral, scalene, isosceles, invalid) </li></ul><ul><ul><li>STE 56% more likely to execute 4 or less tests </li></ul></ul><ul><ul><li>20% STEs (1-12 month) did not test for triangle </li></ul></ul><ul><ul><li>24% of STEs did not test for invalid triangle input </li></ul></ul><ul><li>Only 54% tested for specific boundary conditions </li></ul><ul><li>Only 5% tested for overflow conditions </li></ul><ul><ul><li>Less than 1% of STEs tested for overflow conditions </li></ul></ul>
    17. 17. Skill Comparison <ul><li>STE 4 times more likely to write an incorrect / invalid test </li></ul><ul><li>STE 1.5 times more likely to execute a redundant test </li></ul><ul><li>SDET 2 times more likely to exercise specific boundary conditions and overflow errors </li></ul><ul><li>SDET tests 2 times more likely to be an effective test </li></ul>
    18. 18. Case study summary <ul><li>Black box testing is less than 35% effective </li></ul><ul><li>Testers who lack formal training in testing techniques are most likely to under test boundaries, exception handling routines, and critical functional areas </li></ul><ul><li>More than 50% of the testing effort by untrained testers results in redundant or ineffective* tests </li></ul><ul><li>Testers without computer/programming knowledge are less capable of improving test effectiveness efficiently </li></ul>
    19. 19. So, what’s changing? <ul><li>Market pressures </li></ul><ul><ul><li>Increasing demand for higher quality </li></ul></ul><ul><ul><li>Customer’s more aware of capabilities </li></ul></ul><ul><li>Complex solutions </li></ul><ul><ul><li>Increasing complexity (esp. integration) </li></ul></ul><ul><li>Drive quality upstream </li></ul><ul><ul><li>Unit testing, test driven development, etc </li></ul></ul><ul><ul><li>“ Good Bugs harder to find </li></ul></ul><ul><li>Cost cutting </li></ul><ul><ul><li>Hot-fixes, service packs, shelf-life (10 years) </li></ul></ul><ul><ul><li>Less testers with greater skills & knowledge </li></ul></ul>
    20. 20. Changes at Microsoft <ul><li>Hiring standards </li></ul><ul><ul><li>Computer science and other engineering background </li></ul></ul><ul><li>Training </li></ul><ul><ul><li>40 hours of hands on training for new test engineers </li></ul></ul><ul><ul><li>Study groups, focus groups, etc. </li></ul></ul><ul><ul><li>External conferences and seminars </li></ul></ul><ul><li>Retention in testing discipline </li></ul><ul><ul><li>Stop the “brain-drain” </li></ul></ul><ul><ul><li>Remove the “glass ceiling” for non-management roles </li></ul></ul><ul><ul><li>Provide greater challenges and scope of influence </li></ul></ul>
    21. 21. What should I do? <ul><li>Increase professional knowledge </li></ul><ul><ul><li>Formal training, conferences, etc. </li></ul></ul><ul><ul><li>Books, magazines, industry white papers </li></ul></ul><ul><ul><ul><li>Software Testing Techniques 2 nd Ed . – Boris Beizer </li></ul></ul></ul><ul><ul><ul><li>The Art of Software Testing . – Glenford Myers </li></ul></ul></ul><ul><ul><ul><li>Testing Object Oriented Systems – Robert Binder </li></ul></ul></ul><ul><ul><ul><li>A Practitioners Guide to Software Test Design – Lee Copeland </li></ul></ul></ul><ul><li>Increase technical knowledge </li></ul><ul><ul><li>Modern programming language </li></ul></ul><ul><ul><ul><li>Automation </li></ul></ul></ul><ul><ul><li>Domain expertise </li></ul></ul><ul><ul><li>System expertise </li></ul></ul>
    22. 22. Questions? Testing is our profession; Quality is our passion! ™