SlideShare a Scribd company logo
1 of 21
End-to-end validation and optimisation
via Diary Study
Katie Phillips
User Experience Researcher -
Australia Post
Agenda
2
1. Project brief
2. Methodologies
3. Recruitment & planning
4. Why Slack?
5. Key Insights & implementation
6. Can you run it?
7. Useful links and resources
3
Project brief
• New application for consumers & businesses to pay for & print Australia Post
labels online
• Application live since October
• New features being deployed frequently
• “International” feature soft-released November
Choosing a
research
methodology
4
Key factors
• What is your objective?
• Validating the end-to-end journey of a customer
sending internationally through this application & using
insights to enhance it
• Where are you fidelity-wise?
• In production
• How much time can you commit?
• Endless (aka 2.5 iterations)
• What’s your budget?
• Enough for professional recruitment, incentives, and a
dedicated researcher
Create
order
Get to
MyPost
Business
page
Review &
confirm
Pay Print
labels
Prepare
package
Lodge
parcel
Moderated user testing
Pros
• Great insights into usability
• Defects obvious
• Can be observed/live streamed
• Remote options
• Can watch what customers are actually doing,
not just what they say they do
• Any fidelity
Cons
• Moderator required (can do self-moderated as well)
• Logistics
Time requirements
• Try for at least 5 participants
• Run time: 45min-1hr each
• Analysis time: 1-3hr per participant
Moderated user testing
Diary study
Pros
• Great insight into end-to-end process
• Real world factors
• Lots of raw quotes and photos
• Runs in background
• Helps team understand value of product
Cons
• Dependent on what customers
say they do
• Drop outs
• Technology
• Requires very clear instructions
• Prototype must be functional
Time requirements
• Try for at least 8 participants
• Run time: 1-2 weeks
• Analysis time: 2-4 hours per participant
Diary study
Timeline
Prepared
stakeholders
Plan & recruit
Analysis &
Report
Diary study &
in-flight
activities
Diary study
planning &
recruitment
• Recruited 12 small business
participants for a 3 week
study
Tips
• Be tight on recruitment
• Be clear on what they need to do
to receive incentive
• Incentives at the end (or segment)
• Find an easy platform
Recruiting specs
• Demographics (age, gender, location)
• Behavioural (online habits, social, travel)
• Exclusions (colleagues, language
restrictions)
Why Slack?
• Free!
• Great desktop and mobile apps
• Private channels
• Easily invite others from your
team to observe
• Keep the team open for as long as
you need
• Group channels (help, participant
chat)
• Bots (Anonymous feedback, Polly
surveys, To Do, SlackBot)
• Easy to moderate & set up
notification preferences
Setting up Slack
• Write clear instructions for participants BEFORE the start date
• Set up a private channel for each participant
• Set up a group help channel
• Be prepared to moderate at all hours
• Provide an alternative
Setting up Slack
Setting up Slack
Key insights & resolutions
16
“If you could let it accept .3kg and
auto add the zero that would be
great - adding the leading zero
stuffs me up each time.”
Insights & resolutions
Resolutions
• Development team allocated 1/2 - 1
iteration to incorporate critical feedback
during study
• Optimisation backlog was defined and
prioritised based on insights
• Instructions page re-designed
Insights
• Confirmation of value
• Form fixes needed (12 usability issues)
• No one reads instructions
• It’s easy to learn
“All of my friends who run online
businesses will be jumping for joy”
17
Insights & resolutions
Resolutions
• Non-interface issues raised with
international product team
• Non-application issues raised with segment
team
• Conducted additional user testing around
language
• Input into website IA
• Insights remained with our team for future
features and projects
Insights
• Plastic pockets aren’t easy to get or use!
• Trouble navigating to application
• Awareness issues
• Language confusion
“Oh, there it is! It's like hidden! That's way
too hidden”
Can you run
it?
18
Key factors
• Is it the right method for your product?
• $$ for incentives
• $150-$250 per participant for a 2 week diary-study
• Plus $100-$200 per participant for recruitment
services
• your time if you choose to recruit on your own
• Time
• time requirements as mentioned previously
• development time to incorporate feedback and
changes
19
The real question:
Can you afford NOT to run something?
“If I wasn't participating in this
there is no way in heck I would
even know this is available”
“I'm just assuming I would take this
into my local post office and have
them scan and hopefully they don't tell
me that I can't use them.”
“ It does not like the price I have
entered for the shoes to the US yet it is
the price she paid for them. I have tried
varying the amount but it still remains
highlighted in red no matter what I do”
Nielsen Norman Group on choosing methodologies
ethn.io's Remote Research tools guide
Frog’s Design Research bookshelf
IDEO’s Method Cards
20
Useful
resources
Thanks for
your time!
Katie Phillips
User Experience Researcher -
Australia Post

More Related Content

Similar to Learning from your customers - A diary study with Slack

How to Effectively Lead Focus Groups: Presented at Product School NYC
How to Effectively Lead Focus Groups: Presented at Product School NYC How to Effectively Lead Focus Groups: Presented at Product School NYC
How to Effectively Lead Focus Groups: Presented at Product School NYC
Tremis Skeete
 
Kcic bootcamp webinar_aug_2011
Kcic bootcamp webinar_aug_2011Kcic bootcamp webinar_aug_2011
Kcic bootcamp webinar_aug_2011
Hack the Hood
 
Kcic bootcamp webinar_aug_2011
Kcic bootcamp webinar_aug_2011Kcic bootcamp webinar_aug_2011
Kcic bootcamp webinar_aug_2011
Hack the Hood
 
How to do lantern entries 2012 presentation for pral nola (condensed version)
How to do lantern entries   2012 presentation for pral nola (condensed version)How to do lantern entries   2012 presentation for pral nola (condensed version)
How to do lantern entries 2012 presentation for pral nola (condensed version)
bconra2
 

Similar to Learning from your customers - A diary study with Slack (20)

How to Effectively Lead Focus Groups: Presented at Product School NYC
How to Effectively Lead Focus Groups: Presented at Product School NYC How to Effectively Lead Focus Groups: Presented at Product School NYC
How to Effectively Lead Focus Groups: Presented at Product School NYC
 
Intro to Lean UX with UserTesting
Intro to Lean UX with UserTestingIntro to Lean UX with UserTesting
Intro to Lean UX with UserTesting
 
Kcic bootcamp webinar_aug_2011
Kcic bootcamp webinar_aug_2011Kcic bootcamp webinar_aug_2011
Kcic bootcamp webinar_aug_2011
 
Kcic bootcamp webinar_aug_2011
Kcic bootcamp webinar_aug_2011Kcic bootcamp webinar_aug_2011
Kcic bootcamp webinar_aug_2011
 
UX Field Research Toolkit - A Workshop at Big Design - 2017
UX Field Research Toolkit - A Workshop at Big Design - 2017UX Field Research Toolkit - A Workshop at Big Design - 2017
UX Field Research Toolkit - A Workshop at Big Design - 2017
 
User Research for the Web and Applications
User Research for the Web and ApplicationsUser Research for the Web and Applications
User Research for the Web and Applications
 
Proyectos Investigación y Desarrollo
Proyectos Investigación y DesarrolloProyectos Investigación y Desarrollo
Proyectos Investigación y Desarrollo
 
Institutional Team January 2013
Institutional Team January 2013Institutional Team January 2013
Institutional Team January 2013
 
Learning from our grant writing mistakes
Learning from our grant writing mistakesLearning from our grant writing mistakes
Learning from our grant writing mistakes
 
Creating a great content strategy
Creating a great content strategyCreating a great content strategy
Creating a great content strategy
 
User Experience Service showcase lightning talks - December 2018
User Experience Service showcase lightning talks - December 2018User Experience Service showcase lightning talks - December 2018
User Experience Service showcase lightning talks - December 2018
 
Project Outcome Webinar
Project Outcome WebinarProject Outcome Webinar
Project Outcome Webinar
 
Kcic boot camp oct 2011 idea to implementation 2011
Kcic boot camp oct 2011 idea to implementation 2011Kcic boot camp oct 2011 idea to implementation 2011
Kcic boot camp oct 2011 idea to implementation 2011
 
Improving success with Distributed Teams
Improving success with Distributed TeamsImproving success with Distributed Teams
Improving success with Distributed Teams
 
Learn Learning + Prototype Testing
Learn Learning + Prototype TestingLearn Learning + Prototype Testing
Learn Learning + Prototype Testing
 
體驗劇場_1050524_W14_易用性測試_楊政達
體驗劇場_1050524_W14_易用性測試_楊政達體驗劇場_1050524_W14_易用性測試_楊政達
體驗劇場_1050524_W14_易用性測試_楊政達
 
How to do lantern entries 2012 presentation for pral nola (condensed version)
How to do lantern entries   2012 presentation for pral nola (condensed version)How to do lantern entries   2012 presentation for pral nola (condensed version)
How to do lantern entries 2012 presentation for pral nola (condensed version)
 
More Than Usability
More Than UsabilityMore Than Usability
More Than Usability
 
Personas: Where Product Meets User Needs
Personas: Where Product Meets User NeedsPersonas: Where Product Meets User Needs
Personas: Where Product Meets User Needs
 
Remote Moderated Usability Testing & Tools
Remote Moderated Usability Testing & ToolsRemote Moderated Usability Testing & Tools
Remote Moderated Usability Testing & Tools
 

More from Product Anonymous

More from Product Anonymous (20)

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
Helping Product Teams Do Product - Product Anonymous
Helping Product Teams Do Product - Product AnonymousHelping Product Teams Do Product - Product Anonymous
Helping Product Teams Do Product - Product Anonymous
 
Making Retention Fun - How Games Get Players Coming Back For More.pdf
Making Retention Fun - How Games Get Players Coming Back For More.pdfMaking Retention Fun - How Games Get Players Coming Back For More.pdf
Making Retention Fun - How Games Get Players Coming Back For More.pdf
 
Product-Led In Practice
Product-Led In PracticeProduct-Led In Practice
Product-Led In Practice
 
Product Anonymous - Level Up with a Bespoke Product Management Capability Fra...
Product Anonymous - Level Up with a Bespoke Product Management Capability Fra...Product Anonymous - Level Up with a Bespoke Product Management Capability Fra...
Product Anonymous - Level Up with a Bespoke Product Management Capability Fra...
 
Moving into product leadership
Moving into product leadershipMoving into product leadership
Moving into product leadership
 
Products don’t last forever by Ana Rowe
Products don’t last forever by Ana RoweProducts don’t last forever by Ana Rowe
Products don’t last forever by Ana Rowe
 
Product Led Transformation - Steve Bauer
Product Led Transformation - Steve BauerProduct Led Transformation - Steve Bauer
Product Led Transformation - Steve Bauer
 
Stakeholder engagement - Eadaoin Doherty
Stakeholder engagement - Eadaoin Doherty Stakeholder engagement - Eadaoin Doherty
Stakeholder engagement - Eadaoin Doherty
 
Borrowing from Wardley: how PM's can leverage this tool - Craig Brown
Borrowing from Wardley: how PM's can leverage this tool - Craig BrownBorrowing from Wardley: how PM's can leverage this tool - Craig Brown
Borrowing from Wardley: how PM's can leverage this tool - Craig Brown
 
Managing folk in a workshop - Nosh Dabari
Managing folk in a workshop - Nosh DabariManaging folk in a workshop - Nosh Dabari
Managing folk in a workshop - Nosh Dabari
 
Product manager interview tips - Jen Leibhart
Product manager interview tips - Jen LeibhartProduct manager interview tips - Jen Leibhart
Product manager interview tips - Jen Leibhart
 
How to design powerful experiments - Ying Zhang
How to design powerful experiments - Ying ZhangHow to design powerful experiments - Ying Zhang
How to design powerful experiments - Ying Zhang
 
Learnings from a lockdown hobby - Ambily Menon
Learnings from a lockdown hobby - Ambily MenonLearnings from a lockdown hobby - Ambily Menon
Learnings from a lockdown hobby - Ambily Menon
 
Customer feedback: practical advice
Customer feedback: practical adviceCustomer feedback: practical advice
Customer feedback: practical advice
 
Using Continuous Discovery IRL
Using Continuous Discovery IRLUsing Continuous Discovery IRL
Using Continuous Discovery IRL
 
How to overcome floccinaucinihilipilification
How to overcome floccinaucinihilipilificationHow to overcome floccinaucinihilipilification
How to overcome floccinaucinihilipilification
 
Prioritisation: the ultimate hamster wheel
Prioritisation: the ultimate hamster wheel Prioritisation: the ultimate hamster wheel
Prioritisation: the ultimate hamster wheel
 
Product Anon - Marketing does USP's part II
Product Anon - Marketing does USP's part IIProduct Anon - Marketing does USP's part II
Product Anon - Marketing does USP's part II
 
Product Anon - Marketing does USPs part I
Product Anon - Marketing does USPs part IProduct Anon - Marketing does USPs part I
Product Anon - Marketing does USPs part I
 

Recently uploaded

audience research (emma) 1.pptxkkkkkkkkkkkkkkkkk
audience research (emma) 1.pptxkkkkkkkkkkkkkkkkkaudience research (emma) 1.pptxkkkkkkkkkkkkkkkkk
audience research (emma) 1.pptxkkkkkkkkkkkkkkkkk
lolsDocherty
 
Production 2024 sunderland culture final - Copy.pptx
Production 2024 sunderland culture final - Copy.pptxProduction 2024 sunderland culture final - Copy.pptx
Production 2024 sunderland culture final - Copy.pptx
ChloeMeadows1
 

Recently uploaded (17)

audience research (emma) 1.pptxkkkkkkkkkkkkkkkkk
audience research (emma) 1.pptxkkkkkkkkkkkkkkkkkaudience research (emma) 1.pptxkkkkkkkkkkkkkkkkk
audience research (emma) 1.pptxkkkkkkkkkkkkkkkkk
 
Reggie miller choke t shirtsReggie miller choke t shirts
Reggie miller choke t shirtsReggie miller choke t shirtsReggie miller choke t shirtsReggie miller choke t shirts
Reggie miller choke t shirtsReggie miller choke t shirts
 
Development Lifecycle.pptx for the secure development of apps
Development Lifecycle.pptx for the secure development of appsDevelopment Lifecycle.pptx for the secure development of apps
Development Lifecycle.pptx for the secure development of apps
 
Statistical Analysis of DNS Latencies.pdf
Statistical Analysis of DNS Latencies.pdfStatistical Analysis of DNS Latencies.pdf
Statistical Analysis of DNS Latencies.pdf
 
Production 2024 sunderland culture final - Copy.pptx
Production 2024 sunderland culture final - Copy.pptxProduction 2024 sunderland culture final - Copy.pptx
Production 2024 sunderland culture final - Copy.pptx
 
Cyber Security Services Unveiled: Strategies to Secure Your Digital Presence
Cyber Security Services Unveiled: Strategies to Secure Your Digital PresenceCyber Security Services Unveiled: Strategies to Secure Your Digital Presence
Cyber Security Services Unveiled: Strategies to Secure Your Digital Presence
 
Thank You Luv I’ll Never Walk Alone Again T shirts
Thank You Luv I’ll Never Walk Alone Again T shirtsThank You Luv I’ll Never Walk Alone Again T shirts
Thank You Luv I’ll Never Walk Alone Again T shirts
 
AI Generated 3D Models | AI 3D Model Generator
AI Generated 3D Models | AI 3D Model GeneratorAI Generated 3D Models | AI 3D Model Generator
AI Generated 3D Models | AI 3D Model Generator
 
I’ll See Y’All Motherfuckers In Game 7 Shirt
I’ll See Y’All Motherfuckers In Game 7 ShirtI’ll See Y’All Motherfuckers In Game 7 Shirt
I’ll See Y’All Motherfuckers In Game 7 Shirt
 
Registry Data Accuracy Improvements, presented by Chimi Dorji at SANOG 41 / I...
Registry Data Accuracy Improvements, presented by Chimi Dorji at SANOG 41 / I...Registry Data Accuracy Improvements, presented by Chimi Dorji at SANOG 41 / I...
Registry Data Accuracy Improvements, presented by Chimi Dorji at SANOG 41 / I...
 
Bug Bounty Blueprint : A Beginner's Guide
Bug Bounty Blueprint : A Beginner's GuideBug Bounty Blueprint : A Beginner's Guide
Bug Bounty Blueprint : A Beginner's Guide
 
Free scottie t shirts Free scottie t shirts
Free scottie t shirts Free scottie t shirtsFree scottie t shirts Free scottie t shirts
Free scottie t shirts Free scottie t shirts
 
iThome_CYBERSEC2024_Drive_Into_the_DarkWeb
iThome_CYBERSEC2024_Drive_Into_the_DarkWebiThome_CYBERSEC2024_Drive_Into_the_DarkWeb
iThome_CYBERSEC2024_Drive_Into_the_DarkWeb
 
GOOGLE Io 2024 At takes center stage.pdf
GOOGLE Io 2024 At takes center stage.pdfGOOGLE Io 2024 At takes center stage.pdf
GOOGLE Io 2024 At takes center stage.pdf
 
Premier Mobile App Development Agency in USA.pdf
Premier Mobile App Development Agency in USA.pdfPremier Mobile App Development Agency in USA.pdf
Premier Mobile App Development Agency in USA.pdf
 
The Rise of Subscription-Based Digital Services.pdf
The Rise of Subscription-Based Digital Services.pdfThe Rise of Subscription-Based Digital Services.pdf
The Rise of Subscription-Based Digital Services.pdf
 
TORTOGEL TELAH MENJADI SALAH SATU PLATFORM PERMAINAN PALING FAVORIT.
TORTOGEL TELAH MENJADI SALAH SATU PLATFORM PERMAINAN PALING FAVORIT.TORTOGEL TELAH MENJADI SALAH SATU PLATFORM PERMAINAN PALING FAVORIT.
TORTOGEL TELAH MENJADI SALAH SATU PLATFORM PERMAINAN PALING FAVORIT.
 

Learning from your customers - A diary study with Slack

  • 1. End-to-end validation and optimisation via Diary Study Katie Phillips User Experience Researcher - Australia Post
  • 2. Agenda 2 1. Project brief 2. Methodologies 3. Recruitment & planning 4. Why Slack? 5. Key Insights & implementation 6. Can you run it? 7. Useful links and resources
  • 3. 3 Project brief • New application for consumers & businesses to pay for & print Australia Post labels online • Application live since October • New features being deployed frequently • “International” feature soft-released November
  • 4. Choosing a research methodology 4 Key factors • What is your objective? • Validating the end-to-end journey of a customer sending internationally through this application & using insights to enhance it • Where are you fidelity-wise? • In production • How much time can you commit? • Endless (aka 2.5 iterations) • What’s your budget? • Enough for professional recruitment, incentives, and a dedicated researcher Create order Get to MyPost Business page Review & confirm Pay Print labels Prepare package Lodge parcel
  • 5. Moderated user testing Pros • Great insights into usability • Defects obvious • Can be observed/live streamed • Remote options • Can watch what customers are actually doing, not just what they say they do • Any fidelity
  • 6. Cons • Moderator required (can do self-moderated as well) • Logistics Time requirements • Try for at least 5 participants • Run time: 45min-1hr each • Analysis time: 1-3hr per participant Moderated user testing
  • 7. Diary study Pros • Great insight into end-to-end process • Real world factors • Lots of raw quotes and photos • Runs in background • Helps team understand value of product
  • 8. Cons • Dependent on what customers say they do • Drop outs • Technology • Requires very clear instructions • Prototype must be functional Time requirements • Try for at least 8 participants • Run time: 1-2 weeks • Analysis time: 2-4 hours per participant Diary study
  • 9. Timeline Prepared stakeholders Plan & recruit Analysis & Report Diary study & in-flight activities
  • 10. Diary study planning & recruitment • Recruited 12 small business participants for a 3 week study Tips • Be tight on recruitment • Be clear on what they need to do to receive incentive • Incentives at the end (or segment) • Find an easy platform Recruiting specs • Demographics (age, gender, location) • Behavioural (online habits, social, travel) • Exclusions (colleagues, language restrictions)
  • 11. Why Slack? • Free! • Great desktop and mobile apps • Private channels • Easily invite others from your team to observe • Keep the team open for as long as you need • Group channels (help, participant chat) • Bots (Anonymous feedback, Polly surveys, To Do, SlackBot) • Easy to moderate & set up notification preferences
  • 12. Setting up Slack • Write clear instructions for participants BEFORE the start date • Set up a private channel for each participant • Set up a group help channel • Be prepared to moderate at all hours • Provide an alternative
  • 15. Key insights & resolutions
  • 16. 16 “If you could let it accept .3kg and auto add the zero that would be great - adding the leading zero stuffs me up each time.” Insights & resolutions Resolutions • Development team allocated 1/2 - 1 iteration to incorporate critical feedback during study • Optimisation backlog was defined and prioritised based on insights • Instructions page re-designed Insights • Confirmation of value • Form fixes needed (12 usability issues) • No one reads instructions • It’s easy to learn “All of my friends who run online businesses will be jumping for joy”
  • 17. 17 Insights & resolutions Resolutions • Non-interface issues raised with international product team • Non-application issues raised with segment team • Conducted additional user testing around language • Input into website IA • Insights remained with our team for future features and projects Insights • Plastic pockets aren’t easy to get or use! • Trouble navigating to application • Awareness issues • Language confusion “Oh, there it is! It's like hidden! That's way too hidden”
  • 18. Can you run it? 18 Key factors • Is it the right method for your product? • $$ for incentives • $150-$250 per participant for a 2 week diary-study • Plus $100-$200 per participant for recruitment services • your time if you choose to recruit on your own • Time • time requirements as mentioned previously • development time to incorporate feedback and changes
  • 19. 19 The real question: Can you afford NOT to run something? “If I wasn't participating in this there is no way in heck I would even know this is available” “I'm just assuming I would take this into my local post office and have them scan and hopefully they don't tell me that I can't use them.” “ It does not like the price I have entered for the shoes to the US yet it is the price she paid for them. I have tried varying the amount but it still remains highlighted in red no matter what I do”
  • 20. Nielsen Norman Group on choosing methodologies ethn.io's Remote Research tools guide Frog’s Design Research bookshelf IDEO’s Method Cards 20 Useful resources
  • 21. Thanks for your time! Katie Phillips User Experience Researcher - Australia Post