Businesses turn to intelligent assistants to provide 24/7 support for their customers and to increase efficiency. When intelligent assistants are built well, you can foster customer loyalty and support internal processes by automating simple use cases. It’s a win-win for both customers and businesses. However, when interactions with intelligent assistants become frustrating it can become a liability. The key to delivering an effective intelligent assistant is user testing. Join Inge De Bleecker, Senior Director of UX and Conversational AI for Applause, as she breaks down the role user testing plays in the development and growth of intelligent assistants. Learn how to plan and execute a user testing strategy, and use those results to create a highly-capable intelligent assistant.

1. Boost Your Intelligent
Assistants with UX Testing
Inge De Bleecker
Senior Director of CX/UX and
Conversational AI, Applause

2. Presenters of This Session
2
Inge De Bleecker
Senior Director of
CX/UX and
Conversational AI

3. Today’s Agenda
3
1
3
2
4
The importance of UX in
intelligent assistants
Stages of planning and
performing UX testing
What does user experience
testing uncover?
Best practices for designing
UX tests

4. From the company’s point of view:
• Increase customer loyalty through personalized
support, immediate always-on 24/7 service
• Increase operational efficiency by automating simple
use cases and have human agents cover the more
difficult questions
Why Use Intelligent Assistants in Customer Service
4
A great customer experience for voice and chat will result in satisfied customers who use
and continue to use the automated system and stay loyal to your brand
From the user’s point of view:
• Customers want to use intelligent assistants if
there is a perceived value and it is useful to
them
• In a customer service scenario, speed is crucial;
customers want to use an assistant if this can
get them to a resolution faster

5. 3 Steps To Build Great Automated Customer Service Experiences
5
1
Define what the assistant covers.
Common topics that are less complex
are great candidates
2
Build the conversation and populate
intents with utterances based on
prior transcripts or training data
3
Conduct UX testing with
representative actual users to
uncover important issues

6. WHAT DOES UX TESTING UNCOVER
IN INTELLIGENT ASSISTANTS?

7. Request Is Not Recognized
1. User Experience Testing Uncovers That Users Can Not Complete Their Task
7
Hello – I have a flight tomorrow. Is there
anyway I can upgrade my ticket?
I’m happy to help you change your flight. Did
ACME contact you about a flight change?
Answers Are Not Relevant
A
I want to know if there is any tickets for my
account
Okay, so you would like to inquire about your
account subscriptions like active plans and
add-ons, contract duration and expiry, right?A
I’m sorry, I didn’t understand that.
A
I didn’t catch that. Can you rephrase your
query?A
Issue
Customers will abandon the automated
system and trust in the brand may be
compromised

8. 2. User Experience Testing Uncovers Conversation Design Issues
8
A number of conversation design issues can impact the experience. Some of the common ones are shown below.
Unclear What the Assistant Can
Help With
Too Many Options Result In
Choice Paralysis and Timeouts Unnatural Conversations
Issue
Tasks are cumbersome and take longer
than expected leading customers to
abandon the automated system
How may I help you?A
“Welcome to ACME.You can
discover current deals, check
the status of your order, and
get store hours and locations.
Or, for frequently asked
questions and tips, you can say
things like "how to set up a
default printer” or “help with
my store credit card.”
What can I help you with
today?A
Okay, do you have the ticket
reference number you would like to
inquire about?
46654053
I didn’t get that.
Okay, do you have the ticket
reference number you would like to
inquire about?
Please reply withYes or No
A
A
A
A

9. Handoff Can Be Unexpected to the Customer
3. User Experience Testing Uncovers Challenges In Handoff To Live
Agent
9
Issue
Handoff is not explained to the customer or was not
expected. This may lead to confusion, the user doubting
themselves and what they did wrong, or decreased
confidence in the brand
Do you also have prepaid mobile plans?
In order to better fulfill your request, we will
be transferring your chat to the agent, please
stay on the line.A
b2b products
I didn’t catch that. Can you rephrase your
query?
A
b2b products
You can contact a Specialist anytime, just click
on “Send message” below.
A
Send message No, thanks

10. STAGES OF PLANNING AND
PERFORMING UX TESTING

11. An iterative approach to ensure your conversational experiences are intuitive and easy to use.
7 Steps of User Experience Testing
11
1
Goal
2
Product
3
Participants
4
UserTasks
& Questions
5
Execution
6
Results
7
Improve
experience
GOAL
What are you looking to learn?
PARTICIPANTS
Participants in the user experience testing
should resemble your target user, e.g. in age,
gender, location, language variation, existing
customers, ….
PRODUCT UNDER TESTING
Are you validating a concept, gathering
feedback on a couple of intents in a
prototype, gathering feedback on a live
interaction?
USER TASKS & QUESTIONS
Craft user tasks to mimic typical tasks a
user would execute.Create questions for
each task.
SESSIONS
Participants complete the session
providing deep and relevant insights.
RESULTS
Analyze the user feedback, summarize
and create actionable recommendations
to improve the user experience.
IMPROVE EXPERIENCE
Recommendations and insights are
reviewed by the relevant party, e.g.
content designer, front-end
developer, training data gathering
team.
User
Experience
Testing

12. Make Customer-Centric Decisions at Every Phase of Development
12
Note: all the questions asked in
the previous stages apply.
DEFINE > DESIGN > BUILD > DEPLOY
ConceptTesting
Competitive Research
Is it clear?
Is it useful?
What do
others do?
Iterative Conversation and RobustnessChecks
Does it work?
Is the conversation natural?
Do changes based on prior feedback
improve the experience?
Continued Feedback &
Analysis

13. BEST PRACTICES FOR USER
EXPERIENCE TESTING

14. Key
Takeaways
14
Test early to start improving the product as soon as possible
Test regularly to continue improving the product
Just a handful of users will provide a great deal of feedback
Focus on the user feedback to understand how to improve the
experience
1
2
3
4

15. 15