1.
End-to-end validation and optimisation
via Diary Study
Katie Phillips
User Experience Researcher -
Australia Post

2.
Agenda
2
1. Project brief
2. Methodologies
3. Recruitment & planning
4. Why Slack?
5. Key Insights & implementation
6. Can you run it?
7. Useful links and resources

3.
3
Project brief
• New application for consumers & businesses to pay for & print Australia Post
labels online
• Application live since October
• New features being deployed frequently
• “International” feature soft-released November

4.
Choosing a
research
methodology
4
Key factors
• What is your objective?
• Validating the end-to-end journey of a customer
sending internationally through this application & using
insights to enhance it
• Where are you fidelity-wise?
• In production
• How much time can you commit?
• Endless (aka 2.5 iterations)
• What’s your budget?
• Enough for professional recruitment, incentives, and a
dedicated researcher
Create
order
Get to
MyPost
Business
page
Review &
confirm
Pay Print
labels
Prepare
package
Lodge
parcel

5.
Moderated user testing
Pros
• Great insights into usability
• Defects obvious
• Can be observed/live streamed
• Remote options
• Can watch what customers are actually doing,
not just what they say they do
• Any fidelity

6.
Cons
• Moderator required (can do self-moderated as well)
• Logistics
Time requirements
• Try for at least 5 participants
• Run time: 45min-1hr each
• Analysis time: 1-3hr per participant
Moderated user testing

7.
Diary study
Pros
• Great insight into end-to-end process
• Real world factors
• Lots of raw quotes and photos
• Runs in background
• Helps team understand value of product

8.
Cons
• Dependent on what customers
say they do
• Drop outs
• Technology
• Requires very clear instructions
• Prototype must be functional
Time requirements
• Try for at least 8 participants
• Run time: 1-2 weeks
• Analysis time: 2-4 hours per participant
Diary study

9.
Timeline
Prepared
stakeholders
Plan & recruit
Analysis &
Report
Diary study &
in-flight
activities

10.
Diary study
planning &
recruitment
• Recruited 12 small business
participants for a 3 week
study
Tips
• Be tight on recruitment
• Be clear on what they need to do
to receive incentive
• Incentives at the end (or segment)
• Find an easy platform
Recruiting specs
• Demographics (age, gender, location)
• Behavioural (online habits, social, travel)
• Exclusions (colleagues, language
restrictions)

11.
Why Slack?
• Free!
• Great desktop and mobile apps
• Private channels
• Easily invite others from your
team to observe
• Keep the team open for as long as
you need
• Group channels (help, participant
chat)
• Bots (Anonymous feedback, Polly
surveys, To Do, SlackBot)
• Easy to moderate & set up
notification preferences

12.
Setting up Slack
• Write clear instructions for participants BEFORE the start date
• Set up a private channel for each participant
• Set up a group help channel
• Be prepared to moderate at all hours
• Provide an alternative

13.
Setting up Slack

14.
Setting up Slack

15.
Key insights & resolutions

16.
16
“If you could let it accept .3kg and
auto add the zero that would be
great - adding the leading zero
stuffs me up each time.”
Insights & resolutions
Resolutions
• Development team allocated 1/2 - 1
iteration to incorporate critical feedback
during study
• Optimisation backlog was defined and
prioritised based on insights
• Instructions page re-designed
Insights
• Confirmation of value
• Form fixes needed (12 usability issues)
• No one reads instructions
• It’s easy to learn
“All of my friends who run online
businesses will be jumping for joy”

17.
17
Insights & resolutions
Resolutions
• Non-interface issues raised with
international product team
• Non-application issues raised with segment
team
• Conducted additional user testing around
language
• Input into website IA
• Insights remained with our team for future
features and projects
Insights
• Plastic pockets aren’t easy to get or use!
• Trouble navigating to application
• Awareness issues
• Language confusion
“Oh, there it is! It's like hidden! That's way
too hidden”

18.
Can you run
it?
18
Key factors
• Is it the right method for your product?
• $$ for incentives
• $150-$250 per participant for a 2 week diary-study
• Plus $100-$200 per participant for recruitment
services
• your time if you choose to recruit on your own
• Time
• time requirements as mentioned previously
• development time to incorporate feedback and
changes

19.
19
The real question:
Can you afford NOT to run something?
“If I wasn't participating in this
there is no way in heck I would
even know this is available”
“I'm just assuming I would take this
into my local post office and have
them scan and hopefully they don't tell
me that I can't use them.”
“ It does not like the price I have
entered for the shoes to the US yet it is
the price she paid for them. I have tried
varying the amount but it still remains
highlighted in red no matter what I do”

20.
Nielsen Norman Group on choosing methodologies
ethn.io's Remote Research tools guide
Frog’s Design Research bookshelf
IDEO’s Method Cards
20
Useful
resources

21.
Thanks for
your time!
Katie Phillips
User Experience Researcher -
Australia Post