Presented in QAOTH 2023

1. Testing Between the Buckets
Breaking Test Type Partitions to Find More Bugs
Caleb Crandall
Senior Software Engineer in Test
Beckman Coulter Life Sciences
QA or the Highway 2023

2. What Are Some Of Your Buckets?

3. Overbucketing
from an ISTQB sample exam…for the “advanced” level!!!
(credits to Aaron Hodder for originally highlighting this example on LinkedIn)

4. Overbucketing
“Automating tests of your application's logic
through the UI is an example of extreme waste.
And if you can't unit test that logic, there's a
problem with your software's architecture.”
“I’m a (manual|automation) tester.”
“We should shift left and prevent defects from
being injected instead of testing for them later.”

5. False Dichotomies!
https://xkcd.com/2592/ (I couldn’t help adding the buckets)

6. Semi-Automated Testing
Problem: We needed to test software on a
legacy centrifuge with no simulation
support, and run the tests frequently
Solution: We wrote automated scripts
that ran on actual hardware, but still
required some physical interaction*
* we didn’t have time to get the robotic arm set
up for loading rotors** and closing the door
** my favorite rotor type is the swinging bucket
rotor (bottom right)

7. Semi-Automated Testing
Problem: Testing i18n of a UI in 10+ languages
• humans are good at recognizing missing, untranslated,
or cut-off text and other visual issues
• but bad at navigating a UI in a language they can’t read
• and vice versa for automation
Solution: Write a script that
• navigates the UI without depending on English display text
• takes screenshots of all the screens/critical states
And have humans “blink test” the output screenshots

8. Semi-Automated Testing

9. Semi-Automated Testing
What if you’re interested in testing part of the software
that’s complex, difficult, or just plain boring to set up?
Let a script get you there and then take over!
(e.g. cy.pause() in Cypress)

10. End-to-End Testing
Test your “full stack” from the UI all the
way through to the data storage layer
Simple, right?

11. End-to-End Testing?
But what if your test has…
More than two “ends”?
More than one data storage location?
Multiple API’s?
Multiple UI’s?

12. End-to-End Testing?
You might still have some tests
that focus on part of the system
And target the behaviors, interactions, and
risks specific to that part of the system

13. End-to-End Testing?
You might have other tests that cover larger
swaths, but still cut out some of the system
For convenience
For speed
For clarity
For purpose

14. End-to-End Testing?
And you might have some tests that do look
closer to “end-to-end”

15. Diverse Half-Measures
“It’s better to do more different kinds of
testing to a pretty good level, than to do
one or two kinds of testing perfectly”
Kaner, Bach, and Pettichord, Lessons Learned in Software Testing, 2002

16. Takeaways
Testing categorization isn’t nearly as important as your context and
the problems and risks you’re testing to uncover
Do the testing that makes sense to address those risks and find
those problems, even if it doesn’t fit neatly in a predefined bucket
Try combining ideas from traditionally
separate buckets and see how it improves
your testing!

17. Questions?

18. Acknowledgements
• Michael Bolton
• Griffin Jones
• Sebastian Stautz
• Aaron Hodder
• Wayne Roseberry (you should totally hire him)
• Peter Nei (who pointed out to me early in a project that the system
had more than two ends)