Better Software Classic Testing Mistakes

Classic Testing Mistakes: Revisited Matthew Heusser [email_address] Presented at the Better Software Conference San Francisco, CA - Sept. 21st , 2005 Contributing peer reviewers: James Bach Paul Carvalho Michael Kelly Harry Robinson

Organization
Classic Mistakes: A different approach
The mistakes enumerated
Test Management Mistakes
Test Automation Mistakes
Development Mistakes
Test Strategy Mistakes
Root Causes
What to do tomorrow

On the shoulders of pioneers

Classic Mistake #1: De-humanize the test process Test Management Mistakes AKA Management by Spreadsheet, Management by Email, Management by MS Project …

Classic Mistake #2: Testers Responsible for Quality “ It’s strange that QA let that bug slip through” Test Management Mistakes

Classic Mistake #3: IV&V Determines ship date Do they really? Test Management Mistakes

Classic Mistake #4: Task-based status reporting
Examples:
Testing is “on schedule”
Testing “should be done by Tuesday”
Consequences
Loss of credibility
Bad information for decision makers

Classic Mistake #5: Evaluating testers by bugs found
… and developers by number of bugs injected
Consequences:
Friction
Focus on easy-to-find yet trivial bugs (usability)
Information hiding

Classic Mistakes #6 Inappropriate Models for Test Improvement
NO

Classic Mistake #7: Lack of test training for developers
Testing is a skill .
It won’t appear like magic.
Development Testing Mistakes

Classic Mistake #8: Separate devs and testers
To create friction, emphasize division
Anything that increases the length of the feedback loop is bad .
To improve get rid of waste and tighten the feedback loop.
Development Testing Mistakes

Mistake #9: When late, add Test Automation
Someone has to learn the tool
Someone has to record the scripts

Mistake #10: Mine Field Test Automation Test Automation Mistakes

Mistake #11: Hiring for test tool skills
Technology skills can be taught
Talent can’t
The “Hit the Ground Running” Argument

Classic Mistake #12: Insufficient diversity in test strategy
Examples:
Only requirements based testing
Only coverage testing
Consequence:
Missing entire classifications of defects

Classic Mistake #13: Over-reliance on scripted testing All the testing we did, meticulously pulling down every menu and seeing if it worked right, didn't uncover the showstoppers that made it impossible to do what the product was intended to allow. Trying to use the product, as a customer would, found these showstoppers in a minute. - Joel Spolsky, JoelOnSoftware.com Test Strategy Mistakes

Mistake #14: Untrained exploratory Testing
“Just think creatively”
“Try to break it”
Exploratory Testing is a discipline

Classic Mistake #15: Test ‘Engineers’ and ‘Executors’ Test Strategy Mistakes

Classic Mistake #16: Vacuous Documentation
Examples:
The issue resolution document
Physical signoff/check marks
Elaborate test case templates
Consequence:
Time spent documenting is time not spent testing

Mistake #17: Trying to fix things beyond your reach The Meta-Mistake

Don’t treat the symptoms Find & Fix the root cause!

Root Cause #1 Lack of Systems Thinking in Testing
The law of unintended consequences
Root Causes

Root Cause #2: Translation Problems Example: - “You need to completely test this module” Root Causes

Root Cause #3: Process Myopia
Example:
The [in]famous Issue Resolution Document
“ We don’t do things that way here”
Elevating process over skills
Solutions:
The ear of the king / History Lessons
If your boss doesn’t care – ignore it
Root Causes

Root Cause #4: Technology Myopia
Example
“ Use XML on the next project”
“ I just bought 5 copies of WinRunner …”
Solution:
If you’re technical, they need you to do it
If you’re a manager, focus on business impact and risk
Root Causes

Root Cause #5: Fred Taylor in the organization
Examples:
Factory Mentality
High Specialization
Mixing of skill sets is verboten
Solutions:
Peopleware , or anything by Weinberg
First Break all the rules – Jim Collins
Lead. Insulate your team.
Root Causes

A ‘new’ methodology
Root Cause Analysis
Pareto Analysis
Drive out waste/tighten the feedback loop
Then worry about better practices
(Image from Rapid Development , (c) 1996 by Steve McConnell. Used with permission from the author)
What to do tomorrow

Why
New practices take permission
It is hard (but possible) to get more done by adding work
It is easy to get more done by subtracting work
So start by looking to remove worst practices

What to do tomorrow
Discuss
Q&A

Where to go for more
“ Classic Mistakes in Software Testing”, Brian Marick, STQE, 1997
Rapid Development , Steve McConnell
An Introduction to general systems thinking , Gerold Weinberg, 1975
Lessons Learned in Software Testing , Kaner, Bach, Pettichord

Bonus Section

Root Cause #6: Pressure for short-term results
Example:
“ Ship to make 4 th quarter numbers”
Putting off problems instead of addressing them
Solution:
Save your team
Professionalism means something
The Quake Example
Root Causes

http://qade1.socialtext.net	35
https://www2.socialtext.net	6
http://www.techgig.com	4
http://borax.socialtext.net:22015	3
http://qa2.socialtext.net	2
http://qa3.socialtext.net	2
http://www.slideshare.net	2
http://qa1.socialtext.net	2
http://qa6.socialtext.net	1
http://lucite.socialtext.net	1
http://topaz.socialtext.net:22201	1
http://www.techgig.timesjobs.com	1
http://qa4.socialtext.net	1
http://qade3.socialtext.net	1
https://web.tweetdeck.com	1
http://dev3.socialtext.net	1

Better Software Classic Testing Mistakes

by nazeer pasha

Accessibility

Categories

Upload Details

Usage Rights

16 Embeds 64

Statistics

Better Software Classic Testing Mistakes Presentation Transcript