Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Next-generation
intelligent 

applications require
cognitive design
John Whalen, Ph.D.
Chief Experience Officer
10Pearls
P...
Brief Introductions
PhD Cognitive Science
Johns Hopkins University
Cognitive Neuroscience
Linguistics
Neural Networks/ML
Vision Science
Post D...
According to Google Images, I am User Experience!
“Intelligent Experiences”
Business reimagined

- Significant development 

- Data driven / AI-based
Experience reimagined
...
An end-to-end digital experience and enterprise
software application partner.
Digital Experience &
Enterprise Mobile
Partn...
Special thanks:
Max Kalicka, Researcher
Imran Aftab, CEO
Jen Foster, Director
Agenda
1. Why AI UX is so different

2. Designing for how people think

3. Introducing studies

4. Examples & results brea...
1. Why AI UX is so different
Challenge 1:
What is a usability success?
“Okay Google,
show me a
waterfall.”
@johnwhalen #TheAIConf
“Okay Google,
order me a pizza.”



“Here are some
nearby pizza
restaurants…”
@johnwhalen #TheAIConf
Challenge 2: 

Humanized experiences
“Ask Alexa to play some music.”

“Okay, I’ll ask her.”
Pronouns given
It It It
Her Him Her
@johnwhalen #TheAIConf
“Hey Siri,
do you love me?”
[ouch!]
2. Design for how people think
Language Wayfinding
Vision /
Attention
MemoryEmotion
Decision
Making
Design for how people think
@johnwhalen #TheAIConf
Language Wayfinding
Vision /
Attention
MemoryEmotion
Decision
Making
@johnwhalen #TheAIConf
Use brain science to build inte...
3. Case in point: AI experiences
Prompt:
Chicago, second
tallest building
Participant:



“Alexa, what is the
second tallest
building in
Chicago?”
@johnwha...
Task types and AI assistants studied
Siri
Study1

Personal
Cortana Google

Assistant
Alexa
Google

Home
Alexa SiriHound
St...
Ages
Younger Older
Experience with AI
AI experience No AI experience
Careers
Techie Non-techie
English fluency
Native English speaker
Non-native English
speaker
Types of requests
1. Simple facts

2. Complex facts

3. Commands

4. Games/Jokes

5. Business needs
Captured:
1. Request uttered

2. AI response

3. Facial expression

4. Heart rate

5. GSR (sweat)
4a. Study 1 Results: Personal Tasks
Hotel near MOMA in NYC
Population of country with Eiffel Tower
24hr grocery store near me
Do aliens exist?
Do you like me?
Answer accuracy by age
63% 68% 66% 65%
@johnwhalen #TheAIConf
Note: Command style variance
Answer accuracy by assistant
76% 68% 61% 61%
@johnwhalen #TheAIConf
Siri CortanaGoogle

Assistant
Alexa
Note: The most accurate AI tools 

are not the most preferred.
Emotion matters!
@johnwhalen #TheAIConf
Accuracy and preference by assistant
76% 61% 68% 61%
@johnwhalen #TheAIConf
PREFERREDACCURACY
38% 33% 5% 5%
Siri CortanaGo...
Preferred Google Assistant:
“Annoying when they are
being human. I don't feel
anything with them.”
“It is weird that they
...
Preferred Alexa:
“Alexa feels like it is addressing
me back. Feels like I am
interacting with a person.”
“Prefer Alexa bec...
4b. Study 2 Results: 

Personal + Business Tasks
Task types & AI assistants
Google

Home
Alexa SiriHound
Study2

Personal+

Business
News on Johnny Depp
Flight status United 753, tomorrow
Find 24-hour grocery store
Microsoft’s last closing stock price
When to leave, arrive Dulles airport 10pm
May housing starts, up or down?
Text Max Kalicka “Meet me on July 10”
Answer accuracy by assistant
77% 74% 59% 50%
@johnwhalen #TheAIConf
Hound
Google

Assistant
Siri Alexa
Answer accuracy by task type
72% 58%
@johnwhalen #TheAIConf
Personal
Tasks
Business
Tasks
Accuracy and preference by assistant
50% 74% 59% 77%
@johnwhalen #TheAIConf
PREFERREDACCURACY
35% 21% 21% 14%
Google
Assis...
Preferred Hound:
 “I do love that when you ask
it a question it provides you
an answer in two options.”
“It remembered wha...
Biometric signature of preferred tool
Heart

Rate GSR Positive 

Affect
@johnwhalen #TheAIConf
Product designers:

We can detect which AI assistant

is preferred and if your assistant is
improving over time.
@johnwhal...
5. Summary & implications
@johnwhalen #TheAIConf
1. The most accurate tools are not the most preferred.
50% 74% 59% 77%
@johnwhalen #TheAIConf
FAVORITEACCURACY
35% 21% 21%...
2. Biometrics can detect emotional preference.
Heart

Rate GSR Positive 

Affect
@johnwhalen #TheAIConf
3. There is an opportunity for business assistants.

72% 58%
@johnwhalen #TheAIConf
Personal
Tasks
Business
Tasks
4. Design for how 

people think.
1. Conversational

cadence

2. Context relevance

3. Emotional response
@johnwhalen #The...
Thank you!
John Whalen
@johnwhalen
linkedin.com/in/johnwhalen
John Whalen
Designing
for How
People Think
USING BRAIN SCIEN...
Summary of Findings
1. The most accurate tools are not the most preferred.
Emotion matters!

2. Biometrics can detect emot...
Upcoming SlideShare
Loading in …5
×

Next-generation intelligent applications require cognitive design

1,007 views

Published on

- Learn how to "usability test" AI interactions with humans and measure success
- Understand the two distinct ways that humans construct commands to AI systems and how, using physiological measurements, you can measure the human response to the AI system responses

Description

John Whalen explores the concept of cognitive design, describing how humans structure their commands to AI systems (syntax, word usage, prosody) and how to measure human reactions to AI responses using biometrics (facial emotion recognition, heart rate, GSR). Along the way, John shares insights into how to optimally architect the customer experience.

John offers an overview of the results of an evaluation of four major AI systems (Siri, Cortana, Alexa, and Google Assistant), tested by the young and old, those new to AI systems and those that use these tools every day, native and non-native speakers, and techies and non-techies. Each were asked to interact with the systems to request facts, complex information, jokes, commands, and calendar information while the evaluators recorded their commands, the AI response, and the human’s physiological response to the AI response (facial emotion, heart rate, and GSR).

There were several intriguing findings:

- There were two distinct ways humans constructed commands for the AI systems.
- The testers’ favorite AI systems were not always the ones that performed the best in terms of giving correct answers.
- There was a distinct physiological signature associated with a positive experience.

John explains how these findings can help you determine how you should measure the success of your AI system or chatbot and suggests new ways to predict market success that go beyond AI answer accuracy.

Published in: Engineering
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website! https://vk.cc/818RFv
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Next-generation intelligent applications require cognitive design

  1. 1. Next-generation intelligent 
 applications require cognitive design John Whalen, Ph.D. Chief Experience Officer 10Pearls Presented at O’Reilly AI Conference, San Francisco, 2017
  2. 2. Brief Introductions
  3. 3. PhD Cognitive Science Johns Hopkins University Cognitive Neuroscience Linguistics Neural Networks/ML Vision Science Post Doc. at UCLA during dot.com boom Professor in Psychology 
 Univ. Delaware Biometrics Numerical Cognition Founder, UX Lead Brilliant Experience User Insights Digital Strategy UX / CX CXO
 10Pearls User Insights Digital Strategy AI + UX John Whalen
  4. 4. According to Google Images, I am User Experience!
  5. 5. “Intelligent Experiences” Business reimagined
 - Significant development 
 - Data driven / AI-based Experience reimagined
 - Research, psychology 
 - Innovative UX/CX +
  6. 6. An end-to-end digital experience and enterprise software application partner. Digital Experience & Enterprise Mobile Partner Supplier of the Year: Building Digital Marketplace 50 on Fire – Hottest Companies Award
  7. 7. Special thanks: Max Kalicka, Researcher Imran Aftab, CEO Jen Foster, Director
  8. 8. Agenda 1. Why AI UX is so different
 2. Designing for how people think
 3. Introducing studies
 4. Examples & results breakdown
 5. Summary & implications
  9. 9. 1. Why AI UX is so different
  10. 10. Challenge 1: What is a usability success?
  11. 11. “Okay Google, show me a waterfall.” @johnwhalen #TheAIConf
  12. 12. “Okay Google, order me a pizza.”
 
 “Here are some nearby pizza restaurants…” @johnwhalen #TheAIConf
  13. 13. Challenge 2: 
 Humanized experiences
  14. 14. “Ask Alexa to play some music.”
 “Okay, I’ll ask her.”
  15. 15. Pronouns given It It It Her Him Her @johnwhalen #TheAIConf
  16. 16. “Hey Siri, do you love me?” [ouch!]
  17. 17. 2. Design for how people think
  18. 18. Language Wayfinding Vision / Attention MemoryEmotion Decision Making Design for how people think @johnwhalen #TheAIConf
  19. 19. Language Wayfinding Vision / Attention MemoryEmotion Decision Making @johnwhalen #TheAIConf Use brain science to build intelligent experiences What words and word order were in the command? Could user understand their choices and where they are in the system? Did the AI match expectations? Did the AI generate
 a positive emotional
 response? Did the answer help to solve a problem? Did AI focus the user 
 on the answer?
  20. 20. 3. Case in point: AI experiences
  21. 21. Prompt: Chicago, second tallest building Participant:
 
 “Alexa, what is the second tallest building in Chicago?” @johnwhalen #TheAIConf
  22. 22. Task types and AI assistants studied Siri Study1
 Personal Cortana Google
 Assistant Alexa Google
 Home Alexa SiriHound Study2
 Personal+
 Business
  23. 23. Ages Younger Older
  24. 24. Experience with AI AI experience No AI experience
  25. 25. Careers Techie Non-techie
  26. 26. English fluency Native English speaker Non-native English speaker
  27. 27. Types of requests 1. Simple facts
 2. Complex facts
 3. Commands
 4. Games/Jokes
 5. Business needs
  28. 28. Captured: 1. Request uttered
 2. AI response
 3. Facial expression
 4. Heart rate
 5. GSR (sweat)
  29. 29. 4a. Study 1 Results: Personal Tasks
  30. 30. Hotel near MOMA in NYC
  31. 31. Population of country with Eiffel Tower
  32. 32. 24hr grocery store near me
  33. 33. Do aliens exist?
  34. 34. Do you like me?
  35. 35. Answer accuracy by age 63% 68% 66% 65% @johnwhalen #TheAIConf Note: Command style variance
  36. 36. Answer accuracy by assistant 76% 68% 61% 61% @johnwhalen #TheAIConf Siri CortanaGoogle
 Assistant Alexa
  37. 37. Note: The most accurate AI tools 
 are not the most preferred. Emotion matters! @johnwhalen #TheAIConf
  38. 38. Accuracy and preference by assistant 76% 61% 68% 61% @johnwhalen #TheAIConf PREFERREDACCURACY 38% 33% 5% 5% Siri CortanaGoogle
 Assistant Alexa
  39. 39. Preferred Google Assistant: “Annoying when they are being human. I don't feel anything with them.” “It is weird that they have a personality and can already tell a joke.” “Just give me the answer” “I want general information and answers fast.” @johnwhalen #TheAIConf
  40. 40. Preferred Alexa: “Alexa feels like it is addressing me back. Feels like I am interacting with a person.” “Prefer Alexa because it tells me its not sure.” “I like when it answers the question the way I worded it.” “Responses were direct. Fun to banter back and forth.” @johnwhalen #TheAIConf
  41. 41. 4b. Study 2 Results: 
 Personal + Business Tasks
  42. 42. Task types & AI assistants Google
 Home Alexa SiriHound Study2
 Personal+
 Business
  43. 43. News on Johnny Depp
  44. 44. Flight status United 753, tomorrow
  45. 45. Find 24-hour grocery store
  46. 46. Microsoft’s last closing stock price
  47. 47. When to leave, arrive Dulles airport 10pm
  48. 48. May housing starts, up or down?
  49. 49. Text Max Kalicka “Meet me on July 10”
  50. 50. Answer accuracy by assistant 77% 74% 59% 50% @johnwhalen #TheAIConf Hound Google
 Assistant Siri Alexa
  51. 51. Answer accuracy by task type 72% 58% @johnwhalen #TheAIConf Personal Tasks Business Tasks
  52. 52. Accuracy and preference by assistant 50% 74% 59% 77% @johnwhalen #TheAIConf PREFERREDACCURACY 35% 21% 21% 14% Google Assistant SiriAlexa Hound
  53. 53. Preferred Hound:  “I do love that when you ask it a question it provides you an answer in two options.” “It remembered what I was talking about.” “I liked the way it asked more than one question to get to the answer.” “It was better at the 
 back and forth.” @johnwhalen #TheAIConf
  54. 54. Biometric signature of preferred tool Heart
 Rate GSR Positive 
 Affect @johnwhalen #TheAIConf
  55. 55. Product designers:
 We can detect which AI assistant
 is preferred and if your assistant is improving over time. @johnwhalen #TheAIConf
  56. 56. 5. Summary & implications @johnwhalen #TheAIConf
  57. 57. 1. The most accurate tools are not the most preferred. 50% 74% 59% 77% @johnwhalen #TheAIConf FAVORITEACCURACY 35% 21% 21% 14% Google Assistant SiriAlexa Hound
  58. 58. 2. Biometrics can detect emotional preference. Heart
 Rate GSR Positive 
 Affect @johnwhalen #TheAIConf
  59. 59. 3. There is an opportunity for business assistants.
 72% 58% @johnwhalen #TheAIConf Personal Tasks Business Tasks
  60. 60. 4. Design for how 
 people think. 1. Conversational
 cadence
 2. Context relevance
 3. Emotional response @johnwhalen #TheAIConf Language Wayfinding Vision / Attention MemoryEmotion Decision Making
  61. 61. Thank you! John Whalen @johnwhalen linkedin.com/in/johnwhalen John Whalen Designing for How People Think USING BRAIN SCIENCE TO BUILD BETTER PRODUCTS
  62. 62. Summary of Findings 1. The most accurate tools are not the most preferred. Emotion matters!
 2. Biometrics can detect emotional preference. 
 3. There is an opportunity for business assistants.
 4. Humanize the experience. We need to design for how people think. @johnwhalen #TheAIConf

×