How to Design (and Test) Conversational User Interfaces

HOWTO DESIGN
(ANDTEST)
CONVERSATIONAL
USER INTERFACES
Half-DayWorkshop
Facilitator: Stacey Seronick
UXLx – Lisbon, Portugal – May, 2018

Agenda
@esteeayceeewhy | sseronick@gmail.com 2
Activity Time estimate
Introductions (Self, Group,Topic) 15 minutes
Lecture: Creating for speaking/listening, versus for typing/reading 15 minutes
Lecture:Where do I begin?! 15 minutes
Breakout Group activity: Given a specific use case, understand these folks’ real-world situation(s) 45 minutes
Break 30 minutes
Lecture: How to look for patterns to take an initial stab at what kind of experience and/or intents and entities are needed 15 minutes
Breakout Group activity: Create buckets for potential groups of intents and entities by looking at the data 50 minutes
Lecture - How to test prototyped conversations 10 minutes
Group activity - Regroup, share, questions 15 minutes

Introduction –Who is Stacey Seronick?
■ Designing, writing, strategizing for products, services, and brands for over 20 years
■ Recently consulted on a six-month engagement to help expand the capabilities of a
robot with a spoken and visual UI
■ Current day job encompasses design and product management of NLP-enabled
chatbots
■ Spare time: paint, enjoy looking at the beach, compulsively eavesdrop

Introduction – Group
■ Name
■ Where you call home, live now
■ Functional job title
■ Best vacation you ever took – where did you go?

Introduction –What is this all about?
■ Is AI the new UI?
■ What’s the difference between conversational UIs and chatbots? How does that
difference matter to me?
■ So how you do you test intent?
■ Where do I begin?!
■ My brain is starting to ache a bit – am I going to need lots more coffee or something
stronger?
■ Questions before we dive in?

CREATING FOR
SPEAKING/LISTENING,
NOT FOR
TYPING/READING
Short lecture #1 – 15 minutes

Reading vs Listening
Reading Listening
!
!
^
^
^

Speaking vsTyping
Speaking
■ What do you want?
■ What was that?!
■ Did you eat yet? No, did you?
Typing
■ Whaddyawan?
■ Wa wuz that?!
■ Jeet jet? No, d’ju?

CUIs for Chatbots vs Robots (Smart Objects)
Chatbots
■ Relatively “dumb” – can only give
answers it has specifically been told
to give
■ May or may not sit on top of a
learning model
■ Interface may or may not be spoken
Smart Objects
■ Still requires human moderation and
training, but can make educated
guesses which become more
accurate with time
■ Interface may or may not be spoken,
but lends itself to spoken UI more
than bots do
As of today, there is no such thing as productionalized NLG – Natural Language Generation.
No, not even Google Duplex.

Two things to keep in mind
Importance of JTBD
■ When you go off-track, remind
yourself what job is the bot being
hired to do?
■ There may be several – overt and
symptomatic, and those may vary
between users or relationship of
someone else important to the user,
to user
Sticky-notes, conceptually
■ Sticky-notes, digital and analog,
should be thought of as conversation
parts
■ These parts will start broad and will
eventually get as granular as
sentences and questions with
specific language
• Set reasonable expectations with test users
• Choose your words carefully

Two ImportantTerms to Remember
■ Intent=
“Basically, what you’re saying is…”
■ “It’s pouring out! Where’s
Bloomie’s?”
– So, basically what you’re sating
is…”where can I buy (#WhereIs_Store
intent)” an umbrella (#WhereCanIDryOff
intent)?
■ (Or are you looking for somewhere dry to
entertain yourself while the rain clears
up?)
■ Entities=
a means to “super-charge” intents
with more specific details
■ “It’s pouring out!Where’s
Bloomie’s?”
– “Are you looking for a nearby store
(@MyFavoriteStores_NearbyMe entity) to
window shop or do you want to buy
an umbrella (@Raingear_General entity) or
raincoat (@Raingear_Tops entity)?”

WHERE DO I BEGIN?!
Short Lecture #2 – 15 minutes

Task: Create a CUI for a robot
1. Pick a very specific use case
2. Understand people’s real-world situation(s)
3. Take initial assessment to determine best experience possible given Business, Market,
and Human needs
4. If a CUI is truly best, take initial assessment of research for conversation components
(intents and entities)
5. Determine what outside data is needed to answer questions
6. Create a rough prototype (script) and test
7. Iterate

1. Pick a very, very specific use case
■ You work for ConferenceBot, a start-up whose mission is to provide an integratable
bot dedicated to wayfinding bathrooms in cities, at conventions and conferences, etc.
■ BathroomBot plans to release the first version of its physical robot, dedicated to
conference and convention use, in about 10 months and while there is an approved
physical design for the robot, the conversational interface needs to be built
■ Your team is tasked with developing the Non-EmergencyConversations – the set of
conversations which will help users who are seeking a restroom for reasons other than
a dire physical need
– Pick one (best mirrors for makeup/hair, best lounge area to sit down, best wifi connection)
to start with and build/test from there

OBSERVE AND
UNDERSTANDTHE
REAL-WORLD
SITUATION(S)
BreakoutGroup activity #1 – 45 minutes

2. Understand people’s real-world
situations
■ Direct observation, in-situ
■ Ethnographic studies
■ Diary studies
■ Existing artifacts (IM logs, IVR logs, etc)
■ Open-ended individual interviews
■ Co-creation workshops
■ Focus groups
■ Eavesdropping

BREAK
Break – 30 minutes

HOWTO LOOK FOR
PATTERNS IN PURSUIT OF
BUILDINGTHE RIGHT
THING,THE RIGHTWAY

3.Take initial assessment of your data and
observations to determine what kind of
experience is best and/or how the intents
and entities may get categorized
■ Forget technology and consider
what kind of experience will best
serve these needs
■ Try to pick out themes from research
■ If project is in later in the Discovery
or anytime in the Design phase, be
aware of confirmation bias – let the
research tell you what it will, and
don’t try to fit it into what you
want/need/expect to hear

4. Assess research for initial stab at
intents and entities for a CUI
■ Digitize all notes and put individual observations and quotes into a spreadsheet
■ Categorize them by type
– Informational (What is…? How do I…?)
– Transactional (I want to do…)
– Agentive (Act on my behalf/Help me to do…)
■ Create a column labeled “Basically, …” and in that cell for each data point for which
this makes sense, take a stab at what the intent is behind the note (e.g. “Basically, I
want to know the fastest route to the closest restroom”)
■ Create a column labeled Related Possible Entities and fort each row that has a value
for the Basically,… column, note what entities may be needed (e.g. Location_To,
Room_Restroom, Location_Current)

CREATE BUCKETS FOR
POTENTIAL GROUPS OF
INTENTS AND ENTITIES
BY LOOKING ATTHE DATA
BreakoutGroupActivity #2 – 50 minutes

4. Assess research for initial stab at
intents and entities for a CUI
■ Digitize all notes and put individual observations and quotes into a spreadsheet
■ Categorize them by type
– Informational (What is? How do I?)
– Transactional (I want to do…)
– Agentive (Act on my behalf/Help me to do…)
■ Create a column labeled “Basically, …” and in that cell for each data point for which
this makes sense, take a stab at what the intent is behind the note (e.g. “Basically, I
want to know the fastest route to the closest restroom”)
■ Create a column labeled Entities and fort each row that has a value for the Basically,…
column, note what entities may be needed (e.g. Location_To, Room_Restroom,
Location_Current)

HOWTOTEST
PROTOYPED
CONVERSATIONS

Create prototype
■ Create prototype on paper or sticky notes

Test, and test, and test again
■ With as many different people as you have time for
■ Watch their body language
■ Listen for hesitation
■ One tester, one “user,” several note-takers/observers

Create a very lo-fi prototype of your
physical interface
■ Use found objects, markers, paper, tape, scissors, pens - create an object to be your
robot or smart object for testing purposes
■ Consider using a plush animal if you are not a “maker”

Testing conversations with a prototype
of any kind
In-Person
■ “Wizard-of-Oz” with object
■ Tester-controlled tablet
■ Video recordings
Remote
■ One-way video call “Wizard-of-Oz”
■ Phone call ”Wizard-of-Oz”
■ Use user’s video camera

REGROUP AND SHARE
Group activity – 15 minutes

What was that like?
What did you learn?
■ Share with larger group any a-ha moments

QUESTIONS?
Contact me!
SSERONICK@GMAIL.COM | STACEY.SERONICK@WELLSFARGO.COM

TOOLS: MURAL AND
TWINE

(Free)Tools – Mural andTwine
Mural
■ Mural.com
– Free trial; individual, enterprise, and
educational licenses available
– Online-based version only
– Intended as a collaborative
whiteboard tool
– Use for early user research
■ Good for testing conversation structure
and how it fits/may fit in with other
conversations
Twine
■ Twinery.org
– Open-source
– Online-based version and local
versions available
– Intended for text-based “choose-your-
own-adventure” games
– Use for mid-stage user testing
■ Good for testing intents and some
specific language

Testing conversations in Mural
In-Person
■ “Wizard-of-Oz”
■ Video recordings
Remote
■ Video call “Wizard-of-Oz”
■ Phone call

Example of Mural conversation map
– Link to example shown here

Example ofTwine conversation map

Testing:Twine + “Wizard of Oz” Robot
■ You may need to relabel links for yourself/the tester
■ Explain the role of the found object being used in lieu of a real robot
■ Be consistent in language
■ Have 2 note-takers
– one noting any inconsistencies in your language
– one noting body language and facial expressions of user if no video camera is available or
desired/allowed
Remember!
Don’t create link labels that reflect the exact words you expect anyone to use – it’s
the INTENT that’s important, primarily, and secondarily, it’s important to capture
the variety of ways people verbalize the same intent

Twine tips
■ Move revised Mural conversation intoTwine structure
■ How to create links []
■ How to label links [say this|but go here]
www.twinery.org
■ Don’t create link labels that reflect the exact words you expect anyone to use – it’s the
INTENT that’s important since people will use a wide variety of actual words in their
responses
■ The content that you read, as the robot, should be consistent in test to test, so choose
specific language for your script and intent-based language for link labels representing user
answers

IterateTwine conversation
■ Iterate as needed
– What confused users?
– What words/terms weren’t clear?
– Were unexpected questions asked?
– (Any discoveries that aren’t part of your use case, but should be added to the backlog for
consideration?)
■ Test again as time permits

Other thoughts, tips
■ Proto-personas and empathy maps from your user tests – using them every day
■ Eavesdropping is a good habit to acquire – but make sure you do so with the “right” people
■ What do I do with theseTwine files? My developers can’t use these, can they? Do I even
need developers?
– Api.ai
– Kasisto.com
– Clinc.com
– Facebook messenger API
■ Consider if this is a learning model or not – that will determine the level of complexity and
use of APIs (or not) in the conversation architecture when you go to actually create this bot

Two things to keep in mind
Importance of JTBD
■ When you go off-track, remind
yourself what job is the robot or
smart object being hired to do?
■ There may be several – overt and
symptomatic, and those may vary
between users or relationship of
someone else important to the user,
to user
Sticky-notes, conceptually
■ Sticky-notes, digital and analog,
should be thought of as conversation
parts
■ These parts will start broad and will
eventually get as granular as
sentences and questions with
specific language
• Set reasonable expectations with test users
• Choose your words carefully

Example of “Basically…” spreadsheet

How to Design (and Test) Conversational User Interfaces

Recommended

Recommended

More Related Content

Similar to How to Design (and Test) Conversational User Interfaces

Similar to How to Design (and Test) Conversational User Interfaces (20)

Recently uploaded

Recently uploaded (20)

How to Design (and Test) Conversational User Interfaces

Editor's Notes