When testing a complex system you are often faced with complex test problems. Cause and effect cannot be deduced in advance, only in retrospect. According to the Cynefin framework, the general approach to tackle complexity is probe-sense-respond. Try something, analyze the outcome, and based on that outcome, try something else. This is the basis of all my approaches to begin unraveling complex test problems. But how do I select my test scope for a specific complex test problem?

1. Approaches to unraveling a complex test problem
Disclaimer: Nothing in this article is groundbreaking or new and takes inspiration from already existing material, it is
just one of many mental model or approach I use to select a test scope for complex test problems, and may or may
not be applicable to your context
When testing a complex system you are often faced with complex test problems. Cause and
effect cannot be deduced in advance, only in retrospect. According to the Cynefin framework,
the general approach to tackle complexity is probe-sense-respond [1]. Try something, analyze
the outcome, and based on that outcome, try something else. This is the basis of all my
approaches to begin unraveling complex test problems. But how do I select my test scope for a
specific complex test problem? The different approaches listed below are not exclusive,
sometimes even overlapping, and I often use several in combination. They help me with
direction as to what I should base my scope selection on, but not exactly what to test, or how to
test it. This is not an exhaustive list, but some or the more common ones I use.
Our theoretical complex test problem:
A developer has made a code change in a part of the software system with many complicated
dependencies. The developer has designed and executed unit tests for the code change, but is
worried that the code change may have caused regressions, functional or non-functional, in
other parts of the system.
The Explorer
You know very little about (or do not take into consideration) the system you are testing, its
stakeholders or anything else, and you just start exploring, and adjusting your path as you learn
and discover new information. This approach is the basic probe-sense-respond. This approach
could be used the first time you get your hands on a new system, or if you have exhausted all
known leads and want to start over.

2. The Detective
Often the developer responsible for the code change can give you some insight into what
potential impact the change could have on the system, or at least some understanding of the
risks involved in the code change. In this approach you latch on to those leads and follow them
until you have unearthed any product risks associated with them. Developers are not perfect
oracles, so be prepared that they could be wrong.
The Sculptor
You always create a model of any system you are testing, at least in your head. Sometimes it
can be valuable to explicitly document that model to investigate it thoroughly. What capabilities
does the system have? How does information flow through the system? What are the
interactions and dependencies between different parts of the system? Are there any actions that
need to happen in sequence? Explicitly creating this model can allow you to catch details that
you might have missed if it was just in your head, but it can also be needless documentation if
you do not actually need it of course.

3. The Historian
History repeats itself. Obviously that is not always the case, but sometimes it is worth looking
into. If changes to one part of the system has historically had impact on another part of the
system, it makes sense to at least look into if that is also the case this time. This could be in the
form of previously reported bugs or test results. History also tells us how we reached a certain
point, and what outcomes different decisions had. This knowledge can help us unveil additional
product risks.
The Architect
Having insight into the architecture of the system and how different features and changes have
been implemented, is of course extremely valuable. Knowing how the system is built, and in
what programming language, can help you set the right test scope and design the right tests to
mitigate the product risks and find those regressions. Using this approach requires you to not
only have a good understanding of software engineering in general, but also extensive
knowledge of the system in question, which is why this approach is more often a dream than
reality. If you have actually built similar systems yourself, even better.

4. The Engineer
When you have explored all known product risks, one option is to fall back on different
coverage-based test techniques or heuristics [2], that all testers have learned, to try to uncover
potential risks. Equivalence class analysis, boundary testing, state transition testing and
pairwise testing are some examples of techniques that can be used. Probably used in
combination with some other approach, such as The Economist, explained below.
The Customer
Another option can be to focus on different user types and user behaviours so that you are at
least sure that the common use cases are not affected by the code change. So one part of this
approach is covering important use cases, but you might also generate some good test ideas by
approaching the system from a user perspective that you might not have come up with
otherwise. But it requires some groundwork to be done - you need understanding of the users of
your system, and their behaviour. One source of this information could be to look at videos of
user tests, to study different behaviours.

5. The Data Cruncher
If the code change has already gone live to a percentage of the users, maybe in the form of a
live tech test, then looking at the data those users generate may give valuable insights. This
approach depends a lot on if the system is built in such a way that you can gain this type of
insight. If you can understand how users are actually utilizing the system, and if you can get any
error messages the system generates, then this can greatly facilitate pinpointing potential
regressions.
The Economist
If there are no known product risks, starting with covering high value features may be one way
to go. This approach is basically about regression testing after a priority list based on the value
different features or parts of the system brings to stakeholders, and relevant business risks they
have identified. This approach is probably the one I use most in combination with others.

6. The Ghost Whisperer
Sometimes you just know what you need to test. Not everything can be explicitly stated. Tacit
knowledge and gut feeling must sometimes be allowed to lead the way. Given time you might be
able to dissect why you had this gut feeling, but we don’t always have the luxury of time.
The Blacksmith
Tools can be very valuable when used correctly. Sometimes this means building a tool that will
help you explore the complex problem more efficiently, like creating a Python script. Sometimes
“used correctly” just means unleashing tools on the system, if they can help you gain a lot of
insight at a low cost. A monkey testing tool or a performance measurement tool could be good
to have just running in the background to help you come up with new test ideas - maybe they
will trigger something and give you a lead.

7. The Paper Pusher
If you are stuck or out of test ideas, falling back to available requirements documentation or
scripted test cases may be a good palate cleanser. Maybe it will trigger a new test idea or cover
something you had forgotten about.
The Tailgater
If you have an automated test framework, analyzing the test results may give you some idea of
where to start. Even if all the tests have passed, the coverage of the automated tests could help
you set a direction for your testing. Maybe start in an area with less automated tests for
example. The same thing goes if you are not the first tester to try to tackle the complex test
problem - look at the previous tester’s result and test scope and set your scope with that in
mind.

8. The Psychologist
Everyone has cognitive biases [3]. By understanding these cognitive biases we as testers can
gain insight into where developers may have made mistakes, and this in turn can lead us to
potential regressions in the system. It also helps us avoid making similar mistakes.
The Machine
If our system is communicating with other systems, the interface between the systems may be a
good place to start our testing. Partly because it is important to make sure our system works for
third parties, but also because understanding how other systems communicate with our system
may give valuable insights into what could go wrong.

9. The Ambulance Chaser
If we have already released our change to a percentage of our users, following up with
customer support can be a quick way to get valuable information. Users may have run into
explicit bugs, but they could also have faced minor symptoms that could lead us to the source.
The Cyborg
Hardware can often have unforeseen impacts on our software system. Even something as
“simple” as a mobile application has to run on a wide range of devices. This approach focuses
on testing those areas which are impacted by different hardware configurations, since the
complexity is often even higher here.

10. The Clone
Sometimes comparing the current system version with a previous one may reveal regressions
or give you test ideas that would have been hard to find or come up with by only looking at the
current version. The smaller discrepancies which are hard to spot. Running through test
scenarios (selected by perhaps using The Economist approach) on both versions
simultaneously and comparing outcomes is one way to go about this.
The Mind Reader
Not all requirements are explicitly written down, and requirements can often be interpreted in a
multitude of ways. Sitting down and talking to the developer about his or her interpretation of the
requirements can be enlightening - it can give more insight into how something was actually
implemented and help reveal discrepancies between what is expected, and what actually is.
Knowing the developers’ starting point and mindset during implementation can help reveal
additional product risks.

11. Conclusion
So where does this leave us? When I am faced with a complex test problem, I usually bring with
me this toolkit of approaches, and start applying them as the context demands, until the problem
is solved. In a specific context different approaches are more or less valid, and this list is far
from exhaustive, but hopefully it can trigger some ideas on what to base your test scope
selection on when faced with a complex test problem.
Graphic Design: ChatGPT 4.0 & DALL-E

12. References
[1] Cynefin Framework
https://en.wikipedia.org/wiki/Cynefin_framework
[2] Test Heuristics Cheat Sheet
http://testobsessed.com/wp-content/uploads/2011/04/testheuristicscheatsheetv1.pdf
[3] Cognitive Biases
https://en.wikipedia.org/wiki/List_of_cognitive_biases