Voice usability testing with WOZ methodology - UX SCOT 2019
•
1 like•147 views
This is a presentation of the tutorial that Steven Fullerton and I ran which took participants through the end-to-end process for setting up and running user testing sessions for voice interfaces such as Alexa using the Wizard of Oz testing method.
Recommended
Recommended
More Related Content
Similar to Voice usability testing with WOZ methodology - UX SCOT 2019
Similar to Voice usability testing with WOZ methodology - UX SCOT 2019 (20)
More from Abi Reynolds
Recently uploaded
Recently uploaded (20)
Voice usability testing with WOZ methodology - UX SCOT 2019
- 1. Voice recognition in Scotland? 1Source :https://giphy.com
- 2. Designing Voice application and testing with Wizard of Oz Abi Reynolds Steven Fullerton 2
- 3. 3 Abi Reynolds Principal UX consultant @AbiJReynolds Steven Fullerton UX consultant @StevenFullerton Who are we?
- 4. Why are you interested in Voice? 4
- 5. Our Role in Conversation Design today. 5Image source: unsplash photo @bradneathery
- 6. Good and bad implementation 6
- 7. Good and bad implementation 7
- 8. Why Voice? 8
- 9. Why Voice? 9
- 10. What do we mean by voice? Google Home Amazon Echo 10Image sources: unsplash photos @andresurena and @@benceboros
- 11. Designing conversations 11 Keeping the user at the centre What could a user centred design process for Voice look like?
- 13. Intents = Purpose 13Source :https://giphy.com
- 14. What does a smart speaker hear? 14 Alexa, Ask Tile to find my Keys
- 15. What does a smart speaker hear? 15 Alexa, Ask Tile to find my Keys Utterance
- 16. What does a smart speaker hear? 16 Alexa, Ask Tile to find my Keys Wake Word Utterance
- 17. What does a smart speaker hear? 17 Alexa, Ask Tile to find my Keys Wake Word Invocation Utterance
- 18. What does a smart speaker hear? 18 Alexa, Ask Tile to find my Keys Wake Word Invocation Intent Slot Utterance
- 19. 19 “Design for how people talk, rather than how you wish them to talk.” - Cathy Pearl
- 20. Amy the Hotel Concierge ▪ Our business objective: To cut down on calls to reception (save costs). ▪ Our customer experience objective: We think Amy will provide an innovative and useful service that will add to the overall the guest experience. A voice app should be easy to use hands free within hotel room whilst guests are doing other things. 20
- 21. Activity – 10 Minutes Start the conversation! Write down: • A two way conversation between a customer who wants to know when checkout is and the smart assistant concierge. • The concierge can upsell late checkout at a cost of £10, IF the customer wants it. Create a short script based on the scenario above and work in pairs or small groups… 21 Activity – 10 Minutes Start the conversation! Write down: • A two way conversation between a customer who wants to know when checkout is and the smart assistant concierge. • The concierge can upsell late checkout at a cost of £10, IF the customer wants it. Create a short script based on the scenario above and work in pairs or small groups… 22
- 22. Feedback How did it go? How many different utterances did we get? Lets act out your script How could the flow be improved or made to feel more natural? 22
- 24. 24Source NNG Wireflow https://www.nngroup.com/articles/wireflows/
- 25. Dialogue flows gives you an overview of all directions the conversations can go (based on intents). It plots how you plan to respond to questions and guide the user towards their goal. 25
- 27. Activity – 10 Minutes Complete the extended check-out flow The hotel offers extended check out at 3pm for an extra 10 pounds, which can be added to the room bill Work in groups and using the sheets provided 1. Complete the steps in the path until the user reaches their goal 27
- 28. 28
- 29. Remember 1. Core conversation - Y/N proceed. Until end of task 2. Prompt to go forward or finish task 3. Think how this could go wrong and how Amy could recover the conversation 29
- 30. Feedback How did it go? How many steps did you come up with? What error messages did you come up with… Here is one we did earlier… 30
- 31. Think about… (when starting to write conversations..) ▪ Ask clarifying questions at start ▪ Make assumptions and build for there, make clarifications throughout ▪ Think about how to present choices (and cater for human’s limited working memory) ▪ Avoid overloading the skill with instructions ▪ Think personality and tone and vary responses ▪ Keep the conversation moving forward and bring user closer to their goal ▪ Consider how to repair flow if user goes off happy path – how to get user back on track or fail gracefully. 31
- 33. WOZ Testing Testing voice app using WOZ Testing 33
- 34. Wizard of Oz (WOz) Approach In the WOz test, participants interact with a VUI such as Alexa, whom they believe to be ‘live’, but which is actually being operated by unseen researcher in an another room.
- 35. what the user sees the wizard Wizard of Oz in Voice usability testing
- 36. WOZ Testing 1. Understand main use cases/tasks - user intents 2. Construct sample dialogue flows and responses for each task 3. Create a moderator script including scenarios and metrics for gathering feedback 4. Create audio files for each flow include messages for error recovery etc. (can use Amazon’s Polly or prototype tool such as Sayspring) 5. Connect audio files soundboard (can use soundboard lite) 6. Set up WOZ test 7. Record session (can use audacity, Evernote etc.) 36
- 37. Running a Voice usability session 37 Research brief Test stimulus Test plan/moderator script Set up and record the session
- 39. Test Stimulus 39 1 2 3
- 40. 40 Welcome to UX Scotland Image source: Unsplash photo @ugsdreamer Polly system Speech into text from Amazon Web Services
- 41. Moderator script 41 Very difficult Difficult Slightly difficult Neutral Slightly easy Easy Very easy 1 2 3 4 5 6 7
- 42. Run and record session 42 Amy How can I help?
- 43. What did you think? Tasks/Flows? Language used? Amy’s Personality? Anything else? 43
- 45. Finally… Some tips when testing Voice apps ▪ Test to see if your conversational flows match user intents ▪ It’s important not to mention command words or ask leading questions in TEST as we want to find our more what language and synonyms participants use in testing ▪ Check how your app handles feedback, contextual help and errors ▪ Collect feedback on Persona of app ▪ Use metrics such as ease of use and satisfaction to measure user experience ▪ Agile methodology and quick to iterate. *Remember it may be agile and quick to do but its important to follow a robust research process and have clear research aims and objectives 45