Better Software Classic Testing Mistakes


Published on

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Reducing altitude, he spotted a man on the ground and descended to shouting range.. "Excuse me," he shouted. "Can you help me? I promised my friend I would meet him a half hour ago, but I don't know where I am." The man below responded: "Yes. You are in a hot air balloon, hovering approximately 30 feet above this field. You are between 40 and 42 degrees North Latitude, and between 58 and 60 degrees West Longitude." "You must be an tester," responded the balloonist. "I am," the man replied. "How did you know?" "Well," said the balloonist, "everything you have told me is technically correct, but I have no idea what to make of your information, and the fact is I am still lost." Whereupon the man on the ground responded, "You must be a manager." "That I am" replied the balloonist, "but how did you know?" "Well," said the man, "you don't know where you are, or where you're going. You have made a promise which you have no idea how to keep, and you expect me to solve your problem. The fact is, you are in the exact same position you were before we met, but now it is somehow my fault.“ Now - Why is that joke funny? Because it happens at work, every day. And all that time spent complaining about each other is time not spent building solutions – it’s productivity lost. This talk is about those productivity timesinks – and what we can do about them.
  • Better Software Classic Testing Mistakes

    1. 1. Classic Testing Mistakes: Revisited Matthew Heusser [email_address] Presented at the Better Software Conference San Francisco, CA - Sept. 21st , 2005 Contributing peer reviewers: James Bach Paul Carvalho Michael Kelly Harry Robinson
    2. 2. Organization <ul><li>Classic Mistakes: A different approach </li></ul><ul><li>The mistakes enumerated </li></ul><ul><ul><li>Test Management Mistakes </li></ul></ul><ul><ul><li>Test Automation Mistakes </li></ul></ul><ul><ul><li>Development Mistakes </li></ul></ul><ul><ul><li>Test Strategy Mistakes </li></ul></ul><ul><li>Root Causes </li></ul><ul><li>What to do tomorrow </li></ul>
    3. 3. On the shoulders of pioneers
    4. 6. Classic Mistake #1: De-humanize the test process Test Management Mistakes AKA Management by Spreadsheet, Management by Email, Management by MS Project …
    5. 7. Classic Mistake #2: Testers Responsible for Quality “ It’s strange that QA let that bug slip through” Test Management Mistakes
    6. 8. Classic Mistake #3: IV&V Determines ship date Do they really? Test Management Mistakes
    7. 9. Classic Mistake #4: Task-based status reporting <ul><li>Examples: </li></ul><ul><ul><li>Testing is “on schedule” </li></ul></ul><ul><ul><li>Testing “should be done by Tuesday” </li></ul></ul><ul><li>Consequences </li></ul><ul><ul><li>Loss of credibility </li></ul></ul><ul><ul><li>Bad information for decision makers </li></ul></ul>Test Management Mistakes
    8. 10. Classic Mistake #5: Evaluating testers by bugs found <ul><li>… and developers by number of bugs injected </li></ul><ul><li>Consequences: </li></ul><ul><ul><li>Friction </li></ul></ul><ul><ul><li>Focus on easy-to-find yet trivial bugs (usability) </li></ul></ul><ul><ul><li>Information hiding </li></ul></ul>Test Management Mistakes
    9. 11. Classic Mistakes #6 Inappropriate Models for Test Improvement <ul><li>NO </li></ul>
    10. 12. Classic Mistake #7: Lack of test training for developers <ul><li>Testing is a skill . </li></ul><ul><li>It won’t appear like magic. </li></ul>Development Testing Mistakes
    11. 13. Classic Mistake #8: Separate devs and testers <ul><li>To create friction, emphasize division </li></ul><ul><li>Anything that increases the length of the feedback loop is bad . </li></ul><ul><li>To improve get rid of waste and tighten the feedback loop. </li></ul>Development Testing Mistakes
    12. 14. Mistake #9: When late, add Test Automation <ul><li>Someone has to learn the tool </li></ul><ul><li>Someone has to record the scripts </li></ul>Test Automation Mistakes
    13. 15. Mistake #10: Mine Field Test Automation Test Automation Mistakes
    14. 16. Mistake #11: Hiring for test tool skills <ul><li>Technology skills can be taught </li></ul><ul><li>Talent can’t </li></ul><ul><li>The “Hit the Ground Running” Argument </li></ul>Test Automation Mistakes
    15. 17. Classic Mistake #12: Insufficient diversity in test strategy <ul><li>Examples: </li></ul><ul><ul><li>Only requirements based testing </li></ul></ul><ul><ul><li>Only coverage testing </li></ul></ul><ul><li>Consequence: </li></ul><ul><ul><li>Missing entire classifications of defects </li></ul></ul>Test Strategy Mistakes
    16. 18. Classic Mistake #13: Over-reliance on scripted testing All the testing we did, meticulously pulling down every menu and seeing if it worked right, didn't uncover the showstoppers that made it impossible to do what the product was intended to allow. Trying to use the product, as a customer would, found these showstoppers in a minute. - Joel Spolsky, Test Strategy Mistakes
    17. 19. Mistake #14: Untrained exploratory Testing <ul><li>“Just think creatively” </li></ul><ul><li>“Try to break it” </li></ul><ul><li>Exploratory Testing is a discipline </li></ul>Test Strategy Mistakes
    18. 20. Classic Mistake #15: Test ‘Engineers’ and ‘Executors’ Test Strategy Mistakes
    19. 21. Classic Mistake #16: Vacuous Documentation <ul><li>Examples: </li></ul><ul><ul><li>The issue resolution document </li></ul></ul><ul><ul><li>Physical signoff/check marks </li></ul></ul><ul><ul><li>Elaborate test case templates </li></ul></ul><ul><li>Consequence: </li></ul><ul><ul><li>Time spent documenting is time not spent testing </li></ul></ul>Test Strategy Mistakes
    20. 22. Mistake #17: Trying to fix things beyond your reach The Meta-Mistake
    21. 23. Don’t treat the symptoms Find & Fix the root cause!
    22. 24. Root Cause #1 Lack of Systems Thinking in Testing <ul><li>The law of unintended consequences </li></ul>Root Causes
    23. 25. Root Cause #2: Translation Problems Example: - “You need to completely test this module” Root Causes
    24. 26. Root Cause #3: Process Myopia <ul><li>Example: </li></ul><ul><ul><li>The [in]famous Issue Resolution Document </li></ul></ul><ul><ul><li>“ We don’t do things that way here” </li></ul></ul><ul><ul><li>Elevating process over skills </li></ul></ul><ul><li>Solutions: </li></ul><ul><ul><li>The ear of the king / History Lessons </li></ul></ul><ul><ul><li>If your boss doesn’t care – ignore it </li></ul></ul>Root Causes
    25. 27. Root Cause #4: Technology Myopia <ul><li>Example </li></ul><ul><ul><li>“ Use XML on the next project” </li></ul></ul><ul><ul><li>“ I just bought 5 copies of WinRunner …” </li></ul></ul><ul><li>Solution: </li></ul><ul><ul><li>If you’re technical, they need you to do it </li></ul></ul><ul><ul><li>If you’re a manager, focus on business impact and risk </li></ul></ul>Root Causes
    26. 28. Root Cause #5: Fred Taylor in the organization <ul><li>Examples: </li></ul><ul><ul><li>Factory Mentality </li></ul></ul><ul><ul><li>High Specialization </li></ul></ul><ul><ul><li>Mixing of skill sets is verboten </li></ul></ul><ul><li>Solutions: </li></ul><ul><ul><li>Peopleware , or anything by Weinberg </li></ul></ul><ul><ul><li>First Break all the rules – Jim Collins </li></ul></ul><ul><ul><li>Lead. Insulate your team. </li></ul></ul>Root Causes
    27. 29. A ‘new’ methodology <ul><li>Root Cause Analysis </li></ul><ul><li>Pareto Analysis </li></ul><ul><li>Drive out waste/tighten the feedback loop </li></ul><ul><li>Then worry about better practices </li></ul><ul><li>(Image from Rapid Development , (c) 1996 by Steve McConnell. Used with permission from the author) </li></ul>What to do tomorrow
    28. 30. Why <ul><li>New practices take permission </li></ul><ul><li>It is hard (but possible) to get more done by adding work </li></ul><ul><li>It is easy to get more done by subtracting work </li></ul><ul><li>So start by looking to remove worst practices </li></ul>
    29. 31. What to do tomorrow <ul><li>Discuss </li></ul><ul><li>Q&A </li></ul>
    30. 32. Where to go for more <ul><li>“ Classic Mistakes in Software Testing”, Brian Marick, STQE, 1997 </li></ul><ul><li>Rapid Development , Steve McConnell </li></ul><ul><li>An Introduction to general systems thinking , Gerold Weinberg, 1975 </li></ul><ul><li>Lessons Learned in Software Testing , Kaner, Bach, Pettichord </li></ul>
    31. 33. Bonus Section
    32. 34. Root Cause #6: Pressure for short-term results <ul><li>Example: </li></ul><ul><ul><li>“ Ship to make 4 th quarter numbers” </li></ul></ul><ul><li>Putting off problems instead of addressing them </li></ul><ul><li>Solution: </li></ul><ul><ul><li>Save your team </li></ul></ul><ul><ul><li>Professionalism means something </li></ul></ul><ul><ul><li>The Quake Example </li></ul></ul>Root Causes