BTDConf: Breaking Illusions: Testing is Your Most Valuable Asset
1. Breaking Illusions: Testing is
your most valuable asset!
Maaret Pyhäjärvi
Email: <maaret@iki.fi> | Twitter: maaretp
Maaret Pyhäjärvi
Nimeä | Attribution (Finland)
http://creativecommons.org/licenses/by/1.0/fi/
http://creativecommons.org/licenses/by/1.0/fi/deed.en
2. Testing is about Feedback
Checking
• Machines can do with
algorithmic decision
rules
• Focuses on specific
facts
• Cmp. “compiling”,
“generating”
Exploring
• Skilled humans can do
• Open-ended search for
information
• Cmp. “programming”
Testing not testers – but
becoming a skilled tester
requires time and focus
3. Context Matters – Expectations
Differ
• Testing in Production?
– Rolling releases: limit the extent of impact
– Product lock-in: limit the options
• Testing in the delivery chain?
– Development teams skilled in exploring (as
well as checking)
– Internal releases – Beta releases – Releases
• Available skills?
– Intentional vs. accidental production feedback
4. Testers don’t break your
code, they break your
illusions about the
code.
-- adapted from James Bach
6. The World Has Already
Changed
COMMODITY
TESTERS
• Manual checkers
• Tests are an artifact
SKILLED TESTERS
• Explorers of products
and businesses
• Testing is a performance
• DEVELOPERS IN TEST
• All development is
automating, testing is
just a domain of
development
7. Exploration Skills
Source: Adapted from James Bach, Jon Bach, Michael Bolton. Exploratory Testing
Dynamics. v.2.2. 2009
Self-
managemen
t
Developing
ideas
Examining
product
Done
To Do
Issues
Coverage
All sources available
Best use of time – effective and efficient work
Making models
Tool support – creative solutions
Risk-based testing – scientific approach
Keeping one’s eyes open
10. Culture of Feedback and
Experiments
• Illusions are meant to be broken
• Teams over individuals
– Diverse teams (personalities, skills) deliver
better results
– Social dissonance gets the team talking and
voices out uncertainties to address on time
• Experiment: try something different
11. Summary
1. TESTING is FEEDBACK and LEARNING. EMPIRICAL
supports learning over theoretical. FAST is better than
slower. Not outsourcing the cost of feedback could be better.
2. CONTEXT matters: sometimes the “best testing” happens
live – as long as you’re covering the checking part; testing !=
testers
3. BUGS not defects: anything this might bug a user, there’s
many types of illusions we have
4. COMMODITY testers and SKILLED testers are two
completely different breeds
5. SKILL to break DIVERSE illusions – skill in team, not in
individual
6. DIVERSITY brings good results: people who choose to love
testing tend to be different from those who choose to love
checking and development
Editor's Notes
This presentation is about FEEDBACK. I don’t care if you have specialized testers to help you work out the multidimensional feedback or not. Feedback is still valuable, and for all contexts, asking for all the feedback from the end users / product management just isn’t the thing to do…
I come from an organization where (some) developers strongly still believe that it would be better to hear about bugs from the end users so that we would only hear about “real bugs” and that regardless of me showing I do understand what they complain on, it would still be different (there would be less of it…)
Point of this slide: TESTING is FEEDBACK and LEARNING. EMPIRICAL supports learning over theoretical. FAST is better than slower. Not outsourcing the cost of feedback could be better.
MENTION: Testing looks into more than functionality. Other quality criteria and stakeholder views are particular relevance.
Emergent information, you do not know what you are looking for
For more info, see http://www.satisfice.com/blog/archives/856
Checking
the process of making evaluations by applying algorithmic decision rules to specific observations of a product.
Testing
the process of evaluating a product by learning about it through exploration and experimentation, which includes to some degree: questioning, study, modeling, observation, inference, etc.
Point of this slide: CONTEXT matters: sometimes the “best testing” happens live – as long as you’re covering the checking part; testing != testers
Example: Security vulnerabilities
Vulnerabilities are considered real problems
Still only the black market pays real money for reporting them
Functional bugs / missing value reporting is actual effort too, and wastes time for people
Something we as end users can do: reclamation
I tend to work with PRODUCT BUSINESS and CUSTOMER ORGANIZATIONS, not delivering bespoke software.
Inability to produce quality vs. productivity service – different stakeholder considerations, incl. “design with a change in mind’
I belong to context-driven school of testing – I know things can be different!
History of having things tested in production – wasting a lot of time.
Not every team has the most skilled developers but they can still have diverse teams delivering excellent results
Point of this slide: Testing helps with EFFECTIVENESS, as a productivity service. Your users might not really like reporting your bugs for you, as it wastes their time.
It’s co-creation. The cake example. You can’t separate the ideas, ideas feeding the ideas and implementation.
Illusion stories:
“I tested it, it works”
“reporting engine”
“no time for refactoring” / “releasing as scheduled over done”
“Need document grouping feature over others”
---
Ray Arell (Intel) giving apple example: showing you’re living with an illlusion. Tester mentality!!
----
Point of this slide: BUGS not defects: anything this might bug a user, there’s many types of illusions we have
Finding really expensive bugs: missing value, wrong features
Illusion: releasing as scheduled (sprint) vs. releasing when done (kanban)
EMPIRICAL evidence trumps speculation. Every Single Time. – testobsessed
Point of this slide: COMMODITY testers and SKILLED testers are two completely different breeds
The spider web idea.
Your ideas of testers may be outdated. You get what you ask for, and if you ask for test cases, you get commodity approaches.
Point of this slide: SKILL to break DIVERSE illusions – skill in team, not in individual
Exploratory is a MINDSET / APPROACH, not an activity.
Testing is PERFORMANCE, not ARTIFACT.
Testing is testing even if context is agile.
Point of this slide: DIVERSITY brings good results: people who choose to love testing tend to be different from those who choose to love checking and development
Not every team has the most skilled developers but they can still have diverse teams delivering excellent results
Creating culture of feedback and experiments with diversity of personalities and skills in the teams