This document discusses using diary studies for evaluative research on apps. Diary studies involve having participants use an app and provide feedback on their experiences and interactions in context over a period of time. This allows researchers to see how apps are used in real-life situations and identify problems. The document provides guidance on running diary studies, including collecting feedback before, during, and after app interactions; using videos; and fitting diary studies into the product development cycle from discovery through beta testing. Diary studies help evaluate solutions, refine hypotheses, and identify needed changes based on how users interact with apps in everyday contexts and situations.

1. CB
Insights
Apps evaluative research
using diary studies
UXCamp Brighton 2018
Carmen Brion

2. CB
Why use diary studies for
apps evaluative research?

3. CB
But it’s hard to know if
that is what they will do
Participants tell us what
they think they will do
?
=
The problem with lab research

4. CB
Home
Work
Between
home and work
Use apps on the phone
while on the go.
Prefer bigger screens at
home.
Use apps on the
phone all the time.
Run research with people that uses
app or apps and mobile web
Consider what
happens in these 3

5. CBWhen it comes to apps, context is important
Situations
Space
Motion
Privacy
Length of time
Weekday or weekend
Social proof
Time Location
Weather

6. CB
CONTEXT
OF USE
Who
Why
When
Where
What
How
SITUATIONS
INTERACTIONS
JTBD

7. CB
Generative
Research
Evaluative
Research
Identify problems to
solve and use insights
to generate solutions
Assessing how a solution
to a problem meets
customer and business
outcomes.
Why not using generative research to evaluate
a solution in context?

8. CBWe’ll get to see ‘the mess of everyday life’

9. CB
How to use diary studies
for evaluative research

10. CB
Learnings Learnings
Round 1.1 Round 1.2 Round 2.1 Round 2.2 Round 3.1
Baglog Baglog
5p 6p 6p 6p 5p
T: 28p
How did we get to a version to assess with diary studies?
Discovery track
Delivery track

11. CB
M T W T F S S
Diary Studies
13 participants
30 active participants
○ Run a survey with core experience metrics at the beginning and at the end.Lookback for the
first few days, but allow those that preferred to go on until the end.
○ Feedback on Google docs/video every time they used the app.
○ Provide clear instructions about the feedback every time they interact with the app.
○ Set their expectations that during the first days you might change the guidance on how to
provide feedback and sometimes you might ask him questions if you don’t understand
something about what they are reporting.
○ Follow up if they are not filling in their feedback.
○ Co-create the study with your participants, some of them gave me good ideas that helped me
improve the way of collecting feedback.
○ Use the app and provide feedback.
○ Run a survey with core experience metrics at the beginning and at the end.
……………………………………………………………………………………………………………………………………………………………
○ Track core experience usage affecting hypotheses validation and experience metrics.
What did we do during the diary study?

12. CBProvide participants with clear guidance on
how to give feedback during the diary study
○ Define the situation -
where they are, day, time,
location
○ Define what do they want
to do and how long do they
have
○ Define their aim - what do
they want to achieve
○ Define their emotional
state - how are they feeling
○ If they can, to take pictures
of their environment
As they go about using the app:
○ What they like
○ What they dislike
○ What gets in their way
○ Anything they feel is
important we know
○ Their overall thought
about the experience
○ Did they achieved what
they aimed at the
beginning?
○ How are they feelings
about their experience
with the app and the main
reasons to feel that way
○ Overall satisfaction rating
of their intersection
Before the interaction During the interaction After the interaction

13. CBVideos are a fantastic source of insights

14. CBBenefits
For the different situations
○ Identify how and why the core functionality help the app users get the job done.
○ Discard/iterate features/functionality that was desirable in the lab but didn’t work
well in the context of real life situations.
○ Measure learnability of new features/interactions and identify how to improve it,
when necessary.
○ Identify barriers to get the job done.
○ Identify how habits affect the app experience.
○ Define experience metrics and initial success criteria.
○ Reframe hypothese with the reasons why the solution:
○ Solves app users problems
○ Meets core JTBD
○ Help to overcome barriers of use and habits that prevent app uptake
○ Effect of the experience metrics to core product KPIs

15. CBHow to fit diary studies within
the product development lifecycle
Discovery AlphaDual track concept iteration Beta
Discovery diary studies
○ Identify all the core
situations when the app is
used:
○ Identify the core JTBD in
these situations
○ Any barriers of use
○ First draft of hypotheses
to meet JTBD and solve
barriers
○ Ideate solutions
Lab iterative conceptual
research to shape the product
○ Solutions to hypotheses
○ Re-frame hypotheses as
you discard options or
uncover new app users’
needs
○ |identify experience
metrics that are
important to meet the
business KPIs
Mix of diary studies and small
quantitative analytics of usage
○ Discard features that
habit or learnability
make less desirable
than initially identified
○ Re-frame hypotheses
with learnings
○ Narrow down core
experience metrics and
start estimating success
criteria
Mix of diary studies and big
quantitative analytics of usage
○ Fine tune solution
○ Re-frame hypotheses
with learnings
○ Run correlations
between experience
metrics and core KPIs
meeting the success
criteria

16. CB
Thanks
Get in touch
carmen.brion@cbinsights.co.uk
@tea_monster