It is possible for a product to pass quality assurance tests and acceptance testing without being user-friendly. It is also too easy for those of us who build digital products to make assumptions about what our users need. As a design thinker, I strive to bring the authentic voices of complex audiences into the product lifecycle through pragmatic research. A sound design research process not only shapes digital products to be more usable, it also adds value to drive engagement.

1. Design Research for a quality experience
National Software Testing Conference 2016
Carmen Brion

2. Tech Reliability and Usability are basic
requirements in a digital product for customers
To find competitive advantage we need to understand the specific
Jobs To Be Done that solve problems for our core audience

3. Lack of Design Research, or leaving it too late,
increases the risk of failure
IMVU, Eric Ries’ first startup, was
initially a failure because the team
never undertook any research to
assess the idea and strategy.
“Listen old man, you don’t
understand. What is the
deal with this crazy
business of inviting friends
before I know if it is cool”
Source: Lean Startup by Eric Ries

4. Continuous Design Research increases likelihood
of success, even in difficult markets
Intuit became the leader in the
difficult market of personal finance
by investing in understanding the
jobs to be done for their core
audience.
Quicken was the result of:
● early generative
research
● recurrent usability
testing while building
● public beta versions
before launch
Source: The Lean Product Playbook by Dan Olsen

5. Postponing or not doing Design Research will
affect the product metrics
Acquisition
Activation
Retention
Revenue
Referral
How do customers find you?
Do they have a great first experience?
Do they come back?
How do you make money?
Do they tell others?
Becomes harder and more expensive
Reduces brand loyalty and likelihood of retention
“If you make a customer unhappy in the physical
world they might tell 6 friends, if you do so on the
internet, they can tell 6,000” Jeff Bezos
Unhappy customers are less likely to come back
Increase of costs by rework and customer support
Effects of bad experiences
Source: Lean Startup Pirate Metrics (AARRR)

6. “Your job isn’t to build more software faster: it’
s to maximize the outcome and impact you get
from what you choose to build” Jeff Patton
● QA and Usability Testing just guarantee usable software outputs
● To meet customer outcomes we need continuous and adequate
Design Research

7. Customers buy and use a product because
it does a job for them
Job To Be Done (JTBD)
Context of use
Desired OUTCOME
Product/Service
● What outcome is the customer trying to
achieve?
● What are the problems and pain points
getting in the way of the outcomes?
● How can they achieve outcomes in a
way that also allows the business to
reach its goals?
Digital
Ability
Expectations from
other solutions
JTBD
Ability

8. Who is the audience?
What are their JTBD?
What are they currently using?
How can we do the job better?
Which solutions do the job?
Which solution does the job better?
Is the solution easy to use?
How effective is the solution at
doing the job?
Which is the most efficient
variant to get the job done?
Why might one variant work
better than the other(s)?
Evaluative research
Generative research
Product Exploration
To get outcomes we need Continuous Design Research
Product Development Product Validation

9. To get outcomes we need Continuous Design Research
Ethnography
Diary studies
Contextual
inquiry
Interviews
Co-creation
workshop
Tree test
Card sorts
Click tests
Prototype testing
MVT, A/B testing
Path analysis, Heatmaps
Recorded sessions
Feedback surveys
Usability testing
Generative research
Evaluative research
Usability testing
Surveys
Benchmark
Desk research
Analytics
Evidence-based Output Outcome
Product Development Product ValidationProduct Exploration

10. Research/Design Framework
DEFINE
SOLUTION
ASSESSMENT
SOLUTION
EXPLORATION
PROBLEM
DEFINITION
PROBLEM
EXPLORATION
HYPOTHESES AND
JOBS TO BE DONE
DEFINITION
“Few ideas work on the first try. Iteration is key to innovation”
― Sebastian Thrun

11. “Design Research is
a team sport” GDS

12. 1 32 4 5 6 7 8 9
Kick off meeting
Determine the
assumptions and
questions
Our research process for prototype testing
Prototype definition
Determine and agree
what to prototype
Recruitment spec
Agree participants
profiles and start
recruitment
Sessions plan
Define all hypotheses to
test
Prototype reviews
The core team assesses
progress and discusses
any changes
Discussion guide
Determine and
triangulate tasks
and questions for
participants
Run the sessions
Team and stakeholders
practice ‘Active
Observation’
Sessions washup
Review and agree the
main themes and insights
with the core team
Insights presentation
Present insights and
hypotheses outcomes
to the core team and
stakeholders
2 weeks 1 week

13. “[Both QA and UAT] follow repetitive and set
user paths that assume a level of rational
predictability which is rarely present in human
behaviour”
Jamie Mayers, Made by Many
How can we cater for the user paths we may not have
predicted?

14. QA that goes beyond checking that technical
requirements have been met
Product Development
USABILITY TESTING and QA
UAT
● Work with UX and Development to prioritise bug fixes taking into account
potential benefits to the personas and the impact to their JTBD
● Get ideas from the following in order to define test cases:
○ using the product personas to understand the context of use, JTBD,
ability and level of expertise
○ observe usability sessions to gather inspiration on how customers
use the product and potential unusual paths/usage
Product ValidationProduct Exploration

15. UAT that goes beyond an acceptable system
USABILITY TESTING and QA
UAT
Business Analysts can collaborate with UX to:
● Define key User Journeys to assess
● Define an experience rating that includes how well the product meets:
○ the experience principles
○ the JTBD for each User Journey
Product Development Product ValidationProduct Exploration

16. But it is really unlikely that we can
predict all unusual paths that
people might take
Give the product to your customers,
let them play, see where it breaks for
them
Photo by Storem

17. Product Development
USABILITY TESTING and QA
UAT
Product Validation
Evidence-based Output Outcome
Let your customers be the product testers
MVT, A/B testing
Path analysis, Heatmaps
Recorded sessions
Usability testing
Run usability sessions and complement with a Beta test or a product
validation phase.

18. “Alone we can do so little; together
we can do so much”
- Helen Keller
Photo by Easa Shamih

19. Thank you!
You can find me @Tea_monster

20. References
Irrational realism: on QA vs Testing, by Jamie Mayes
https://medium.com/the-many/irrational-realism-on-qa-vs-testing-7fd8618f7fdb#.q9b3wxosm
It’s not just semantics: managing outcomes vs outputs, by Deborah Mills-Scofield
https://hbr.org/2012/11/its-not-just-semantics-managing-outcomes
The neglected practice of iteration, by Jeff Patton
https://www.agileconnection.com/article/neglected-practice-iteration
Early usability testing decides the product’s destiny, by Tamara Rendulic
http://www.paulolyslager.com/early-usability-testing-decides-product-destiny/?
utm_content=buffer4482d&utm_medium=social&utm_source=twitter.
com&utm_campaign=buffer