Using smartphones to recognize facial expressions, designers at TurboTax are identifying new challenges, tools and practices to build experiences that learn and adapt to emotion in real time.

1. GRAPHING CUSTOMER
EMOTION IN TURBOTAX
IN REAL TIME
1

2. 2
We’ve known for years that TurboTax can
be a rough emotional ride. Wild swings in
the refund amounts, missed expectations,
dramatic changes year-to-year. We’ve
actually done research…

3. activity that correspond to seven core emotions – are studied on a
second-by-second basis.
Pursuant to US Patent #7,113,916
3
In 2013 we recorded customers using TurboTax then analyzed the video frame-by-frame to read
emotions according to a scale, then graphed those emotions to try and determine how customers are
feeling. This is the W2 and on the right the graph associated with it. Manual, labor intensive and not
real time.

4. Why now?
In the last 3-5 years emotion recognition technology has dramatically advanced.
Companies like Affectiva, Realsense, Emotient and others have created algorithms
that can reliably recognize human emotions in milliseconds - faster than humans.
Combine these algorithms with high-resolution mobile cameras, web api’s, mobile
SDKs and we have an extraordinary opportunity.
4

5. 5
Gathering emotion response data
will allow us to see our customers
in a completely new way. It will
unlock the unique individual
behind the faceless demographic.

6. This would allow us to…
6
Serve up different versions our experiences according to emotional responses
Capture a completely new dimension of data in analytics
Change marketing offers based on emotion response
Better facilitate customer care (especially agent interaction)

7. Baseline requirements
7
real time
fast
accurate
seamless integration

8. Technology Highlights
Valence, Engagement, and other emotions
SDK runs native/local on mobile + desktop platforms
Real-time, reads every 250 ms
Transient — no images are stored
8

9. Experiments, insights and
customer benefits
9

10. Experiment A positive reaction
(valence >0)
negative reaction
(valence <0)
AFFECTIVA
measure
page
1
page
2A
page
2B

11. 11
WHAT THE
CUSTOMER SEES
WHAT THE
CAMERA SEES
Experiment A

12. 12
negative
valence
positive
valence
1. integrate the emotion software with a mobile device
2. track emotions in real time
3. see that emotions correspond to facial responses
4. change the experience based on emotion response (positive or negative
5. store the emotions with a bespoke analytics tool

13. 13
-48
+35
We were able to correlate emotion with the URL and by observing customers in the lab,
confirm that we can see that the customer’s emotional reaction correlated to the content

14. 14
Noisy Data

15. we needed a controlled test on a specific event
at a specific time
15

16. We measured emotions between 3 points for a 3 second interval
number starts rolling number stops rolling 2 seconds after number
stops rolling
THE TIME INTERVAL
0 1 3

17. Stt
17
Clean Data

18. noisy data
- wide range of deviation on a single URL giving low
confidence on what the customer is experiencing
event driven data
With data tied to intervals around
events we get an accurate
reflection of emotion response.
18
Clean Data

19. 19
We now know how to
best to capture what a
customer feels and that
we can respond
accurately…
We can know that
person…

20. Customer benefits
we can understand our customers on a deeper level
we can respond to a negative or positive outcome in a way that builds confidence
we can provide customers with support when they need it most
we can and help in ways we never did before
we can identify areas of concern that are otherwise invisible
20

21. Conclusions / insights
We can gather a completely new understanding of the end-to-end
customer journey AND key moments in the customer experience
We can gather valuable, actionable analytics giving us a radically new data
dimension, creating a strategic advantage that we have never had before.
21

22. Risks and principles
22

23. Risks
People are very wary of sharing personal data
Biometric data is more intimate than social media data
There are profound risks to our brand if this technology is misused
23

24. Principles
Always be up front about capturing biometric data - never hide it in legalese
State clearly how the data will be used and what happens to it aftewards
Give customers a simple and direct way of opting out
Since only a subset of customers are interested, explore the technology with a
willing few.
24

25. 25
‘There is no art/
to find the man's
construction in the face’
William Shakespeare, Macbeth