Next-generation
intelligent 

applications require
cognitive design
John Whalen, Ph.D.
Chief Experience Officer
10Pearls
Presented at O’Reilly AI Conference, San Francisco, 2017
Brief Introductions
PhD Cognitive Science
Johns Hopkins University
Cognitive Neuroscience
Linguistics
Neural Networks/ML
Vision Science
Post Doc. at UCLA
during dot.com
boom
Professor in
Psychology 

Univ. Delaware
Biometrics
Numerical Cognition
Founder, UX Lead
Brilliant Experience
User Insights
Digital Strategy
UX / CX
CXO

10Pearls
User Insights
Digital Strategy
AI + UX
John Whalen
According to Google Images, I am User Experience!
“Intelligent Experiences”
Business reimagined

- Significant development 

- Data driven / AI-based
Experience reimagined

- Research, psychology 

- Innovative UX/CX
+
An end-to-end digital experience and enterprise
software application partner.
Digital Experience &
Enterprise Mobile
Partner
Supplier of the Year:
Building Digital
Marketplace
50 on Fire – Hottest
Companies Award
Special thanks:
Max Kalicka, Researcher
Imran Aftab, CEO
Jen Foster, Director
Agenda
1. Why AI UX is so different

2. Designing for how people think

3. Introducing studies

4. Examples & results breakdown

5. Summary & implications
1. Why AI UX is so different
Challenge 1:
What is a usability success?
“Okay Google,
show me a
waterfall.”
@johnwhalen #TheAIConf
“Okay Google,
order me a pizza.”



“Here are some
nearby pizza
restaurants…”
@johnwhalen #TheAIConf
Challenge 2: 

Humanized experiences
“Ask Alexa to play some music.”

“Okay, I’ll ask her.”
Pronouns given
It It It
Her Him Her
@johnwhalen #TheAIConf
“Hey Siri,
do you love me?”
[ouch!]
2. Design for how people think
Language Wayfinding
Vision /
Attention
MemoryEmotion
Decision
Making
Design for how people think
@johnwhalen #TheAIConf
Language Wayfinding
Vision /
Attention
MemoryEmotion
Decision
Making
@johnwhalen #TheAIConf
Use brain science to build intelligent experiences
What words and
word order were in the
command?
Could user
understand their choices and
where they are in the
system?
Did the AI match
expectations?
Did the AI generate

a positive emotional

response?
Did the answer help
to solve a problem?
Did AI focus the user 

on the answer?
3. Case in point: AI experiences
Prompt:
Chicago, second
tallest building
Participant:



“Alexa, what is the
second tallest
building in
Chicago?”
@johnwhalen #TheAIConf
Task types and AI assistants studied
Siri
Study1

Personal
Cortana Google

Assistant
Alexa
Google

Home
Alexa SiriHound
Study2

Personal+

Business
Ages
Younger Older
Experience with AI
AI experience No AI experience
Careers
Techie Non-techie
English fluency
Native English speaker
Non-native English
speaker
Types of requests
1. Simple facts

2. Complex facts

3. Commands

4. Games/Jokes

5. Business needs
Captured:
1. Request uttered

2. AI response

3. Facial expression

4. Heart rate

5. GSR (sweat)
4a. Study 1 Results: Personal Tasks
Hotel near MOMA in NYC
Population of country with Eiffel Tower
24hr grocery store near me
Do aliens exist?
Do you like me?
Answer accuracy by age
63% 68% 66% 65%
@johnwhalen #TheAIConf
Note: Command style variance
Answer accuracy by assistant
76% 68% 61% 61%
@johnwhalen #TheAIConf
Siri CortanaGoogle

Assistant
Alexa
Note: The most accurate AI tools 

are not the most preferred.
Emotion matters!
@johnwhalen #TheAIConf
Accuracy and preference by assistant
76% 61% 68% 61%
@johnwhalen #TheAIConf
PREFERREDACCURACY
38% 33% 5% 5%
Siri CortanaGoogle

Assistant
Alexa
Preferred Google Assistant:
“Annoying when they are
being human. I don't feel
anything with them.”
“It is weird that they
have a personality and
can already tell a joke.”
“Just give me the answer”
“I want general information
and answers fast.”
@johnwhalen #TheAIConf
Preferred Alexa:
“Alexa feels like it is addressing
me back. Feels like I am
interacting with a person.”
“Prefer Alexa because
it tells me its not sure.”
“I like when it answers the
question the way I worded it.”
“Responses were direct.
Fun to banter back and
forth.”
@johnwhalen #TheAIConf
4b. Study 2 Results: 

Personal + Business Tasks
Task types & AI assistants
Google

Home
Alexa SiriHound
Study2

Personal+

Business
News on Johnny Depp
Flight status United 753, tomorrow
Find 24-hour grocery store
Microsoft’s last closing stock price
When to leave, arrive Dulles airport 10pm
May housing starts, up or down?
Text Max Kalicka “Meet me on July 10”
Answer accuracy by assistant
77% 74% 59% 50%
@johnwhalen #TheAIConf
Hound
Google

Assistant
Siri Alexa
Answer accuracy by task type
72% 58%
@johnwhalen #TheAIConf
Personal
Tasks
Business
Tasks
Accuracy and preference by assistant
50% 74% 59% 77%
@johnwhalen #TheAIConf
PREFERREDACCURACY
35% 21% 21% 14%
Google
Assistant
SiriAlexa Hound
Preferred Hound:
 “I do love that when you ask
it a question it provides you
an answer in two options.”
“It remembered what I
was talking about.”
“I liked the way it asked more
than one question to get to
the answer.”
“It was better at the 

back and forth.”
@johnwhalen #TheAIConf
Biometric signature of preferred tool
Heart

Rate GSR Positive 

Affect
@johnwhalen #TheAIConf
Product designers:

We can detect which AI assistant

is preferred and if your assistant is
improving over time.
@johnwhalen #TheAIConf
5. Summary & implications
@johnwhalen #TheAIConf
1. The most accurate tools are not the most preferred.
50% 74% 59% 77%
@johnwhalen #TheAIConf
FAVORITEACCURACY
35% 21% 21% 14%
Google
Assistant
SiriAlexa Hound
2. Biometrics can detect emotional preference.
Heart

Rate GSR Positive 

Affect
@johnwhalen #TheAIConf
3. There is an opportunity for business assistants.

72% 58%
@johnwhalen #TheAIConf
Personal
Tasks
Business
Tasks
4. Design for how 

people think.
1. Conversational

cadence

2. Context relevance

3. Emotional response
@johnwhalen #TheAIConf
Language Wayfinding
Vision /
Attention
MemoryEmotion
Decision
Making
Thank you!
John Whalen
@johnwhalen
linkedin.com/in/johnwhalen
John Whalen
Designing
for How
People Think
USING BRAIN SCIENCE TO BUILD BETTER PRODUCTS
Summary of Findings
1. The most accurate tools are not the most preferred.
Emotion matters!

2. Biometrics can detect emotional preference. 

3. There is an opportunity for business assistants.

4. Humanize the experience. We need to design for how
people think.
@johnwhalen #TheAIConf

Next-generation intelligent applications require cognitive design