Matt Heusser from Excelon Development discusses various methods for testing software products. He describes taking an early build of a product called CityDesk home to use as a customer would and discovering serious bugs that prevented basic tasks. This highlighted limitations of their meticulous functional testing which failed to uncover "showstoppers". Heusser advocates using different types of testing beyond automation, including simulating user scenarios, considering failure modes, and using "system two thinking" to uncover hard to find "black swan" issues. A repeated focus on automation and specifications can create blind spots where important problems are missed.

1. Cool new ideas
... and some old ones, too
MATTHEW HEUSSER – EXCELON DEVELOPMENT
Matt@xndev.com
QAOrTheHighWay – February, 2014
Columbus, OH
Tuesday, February 11, 14

2. A Word of Warning
2
Tuesday, February 11, 14

3. System 1, System 2
Tuesday, February 11, 14

4. Your Brain At Work
4
Tuesday, February 11, 14

5. Black Swans
Tuesday, February 11, 14

6. Another Example
6
Tuesday, February 11, 14

7. Ten Kind of Tests
Let’s Get Beyond Testing the functions
Tuesday, February 11, 14

8. Ten Kind of Tests - II
Dog Fooding, User Scenarios, Failure Modes
A month ago, I took home a build of CityDesk (thinking it was
about 3 weeks from shipping) and tried to build a site with it.
Phew! There were a few bugs that literally made it impossible for
me to proceed, so I had to fix those before I could even continue.
All the testing we did, meticulously pulling down every menu and
seeing if it worked right, didn't uncover the showstoppers that
made it impossible to do what the product was intended to allow.
Trying to use the product, as a customer would, found these
showstoppers in a minute.
8
Tuesday, February 11, 14

9. Ten Kind of Tests - III
Quality Factors, Creative Ideas, States
6
9
Tuesday, February 11, 14

10. Ten Kind of Tests - IV
Data, Environment, White Box, Chartered Missions
10
6
7
Tuesday, February 11, 14
6
7

11. Ten Kind of Tests - V
Taxonomies, Previous Failures
11
6
7
8
9
Tuesday, February 11, 14
6
7

12. Ten Kind of Tests - V
Taxonomies, Previous Failures, Quick Attacks
12
Tuesday, February 11, 14

13. The Minefield
Tuesday, February 11, 14

14. Actual Risks - Those That Emerge
Now think about the big failures you've experienced. The Crashes
in production, the corruptions of financial transactions, the projects
that were cancelled.
Was it ever "gee if only we'd had more time to automation those
functional tests?" or is it "huh. We never thought to test that at all?"
Most of the big problems I have experienced in software delivery
were Black Swans, the kind of thing that we only could have found
by system two thinking -- and excessive focus on functional testing
means we don't have time for it.
Tuesday, February 11, 14

15. Q&A
Tuesday, February 11, 14

16. Tomorrow
Ma#
Heusser
@mheusser
-­‐
ma#@xndev.com
h#p://www.xndev.com
The risks on each build are different - so maybe we
don’t want a repeatable process.
Over-focus on automation, or deadlines, or
specifications, creates cognitive blinders that cause us
to miss black swans.
I know, continuous lean cloudy agile devlops is ‘hot’ but the real challenge of soft ware testing is cognative.
The pendulum may have swung just a little too far. Let’s
swing it back.
Thank you, and god bless you.
Tuesday, February 11, 14

17. ... and where to go for more
James Bach, The Heuristic Test Strategy Model, Specifically SFDPO
http://www.satisfice.com/tools/htsm.pdf
Roubert Sabourin, Ten Sources of Test Ideas
http://www.amibugshare.com/articles/Article_10_Sources_of_Testing_Ideas.pdf
Hans Buwalda, Soap Opera Tests
http://www.logigear.com/logi_media_dir/Documents/Soap_Opera_Testing.pdf
iOs Testing Mindmap, Bernard Shaw Lelunk
http://tuppad.com/blog/2012/09/06/ios-testing-mind-map-how-to-test-an-iphoneipadipod-by-bernardshai-lelchuk/
Moolya Software Testing Limited’s Blog, Specifically Mobile Tests
http://moolya.com/blog/2012/04/18/test-mobile-applications-with-cop-who-flung-gun/
Elisabeth Henrickson’s Test Heusser-istic Cheat Sheet
http://testobsessed.com/wp-content/uploads/2011/04/testheuristicscheatsheetv1.pdf
For an electronic copy of this with links, email Matt@xndev.com
Tuesday, February 11, 14
10
17
6
7
8
9
6
7

18. A Famous Ship
Tuesday, February 11, 14