Understanding emotions is becoming more important as technology is expected to respond to each individual based on their tastes. AI is the technology that is powering this expectation. We will talk about how, using emotional research and design methodologies, it is possible to gather not only what people think about using a system, but also how they feel. Doing emotional research to gain insights and catalogue them is one of the first steps. From there designers can leverage these findings and translate the feelings into design conventions. These conventions can then provide the machine learning with the signal it can use to generate more refined and meaningful results based on a person's preferences. These emotionally based features can then be quantifiably measured to prove out the effectiveness of the process. By using this process with machine learning technologies we can create systems that go from being simply useful to something that is a joy to use. by Rob Strati and Jesse Schifano (Co-Founders, ECHO) part of NYAI #18: AI & UX on Tues, 27 Feb 2018 at Capital One Labs. nyai.co

2. Echo
Echo is an emotional research and design agency.
We focus on the emotional and motivational forces that
drive decision making. Unpacking these feelings and
identifying the reasons people are having them can
change a system from being simply useful to something
that is a joy to use.

3. AMAZON
GOOGLE
UBER
IBM WATSON
AI VENTURES
X.AI
INTEL
Case Study

4. Getting Design in the mix
Here are the four areas of developing machine
learning where it is important for designers to
Contribute:
• Gathering Data
• Preparing Data
• Training the Model
• Creating an Interface

5. The power of love
The first version of MightyTV was designed to capture two
emotional preferences:
● Like
● Dislike
During the emotionally attuned testing we saw there were always a
few titles where people demonstrated strong emotion indicators:
● Eyes would open wide
● Big smiles
● Pointing excitedly
● “Oh, I love this movie.” or “This movie is one of my
favorites”
Our findings indicated the need for a fifth emotion / preference:
● LOVE
We designed a slow swiping convention that would build a “like”
into a “love”, allowing people to express that preference.
Our recommendation was not only to satisfy the desire people had
to express their love for a movie or tv show. We understood that the
machine learning powering the system could use that point of
signal to improve the results were getting.

6. Trusting emotions
Using emotionally attuned testing we pay attention to any
emotional indication people make and from that can identify
emotional triggers.
These triggers can be problems to solve or opportunities to
enhance an experience. Through our testing at MightyTV one of
these triggers emerged in the form of “Brainy” the cupcake.
People would have emotional reactions including smiles or
expressions. There were a number of expressions centered on a
feeling of security, with one user expressing, “He makes me feel
safe.”
We cataloged this as a positive emotional trigger and continued
to build out the product.

7. Why are you showing me this?
As the development process and subsequent user testing
continued, the issue of a “lack of confidence” was being
expressed by the users.
People were being presented with a movie they knew nothing
about. This was an important part of finding something new to
watch, but if a title was too unfamiliar the user didn’t understand
why it was being shown to them and they didn’t have the
confidence in the machine learning to explore more about it in the
app.
In exploring the feeling of a “lack of confidence” it was
understood to be tied to a feeling of “trust”.

8. Turning hesitation into trust
The emotional research around the feelings associated with
Brainy was then leveraged. Brainy triggered the feeling of safety
and so could be leveraged to enhance the trust in what was being
shown to users.
At this point, the Brainy cupcake icon was brought to the surface
level of the title and presented with small factoids about the
titles.

9. HAPPINESS
A Measurable Difference
Another example of the power of emotional research is in the
methodology Echo used to develop a “Happiness Score”, which was
then incorporated into the machine learning algorithm to increase the
personalization of the results users were getting in the app.
People’s confidence and happiness with the app varied greatly
depending exactly on how the likes and dislikes were distributed, even
when the like and dislike counts remained consistent. While this
feeling was still subjective, Echo ran tests to quantify this emotion and
properly track it.
The research focused on what made users happy instead of solely
relying on traditional metrics, such as how choices affected the
number of sessions or time spent using the app. More specifically, the
tests were designed to quantify the user’s happiness as an additional
metric to put alongside traditional ones.

10. Listening
User testing sessions were conducted to measure how well the machine
learning algorithm understood the users. Each participant was asked to
swipe through 200 titles in the app. Rather than the focus being on
functionality or what people thought about the app, close attention
was paid to when users indicated emotional responses either verbally
or through facial or other expressions.
Graphing
In addition to this feedback, at the conclusion of each test, people were
asked to draw out two graphs. One was to show how well they thought
the app was learning about them and the other to indicate how happy
they felt while using the app with each swipe.
METHODS

11. Quantitative Analysis
All 200 choices for each tester - Like, Dislike, Love, Skip, or Add to
Watchlist was recorded. Echo then mapped various changes in
happiness to particular choice sequences.
It was observed that the action of adding a title to the watchlist was an
extremely positive reaction and was able to nullify a large amount of
past dislikes in the experience. Furthermore, rules were inferred
regarding both how particular choices and their order affect user
happiness and ultimately modify user behavior within the app.
Once it was understood how the relationship between personalized
recommendations were presented to each user and that user’s
happiness, Echo then worked with the machine learning team to
optimize the system’s presentation strategy.
Using standard nonlinear optimization techniques, the machine
learning team was able to locate the range of recommendations that
produced the best happiness score.
THE NUMBERS

12. Quantitative improvements
The emotional research and design conventions we created as a
result of the testing were tied to key business goals.
The resulting quantitative analytics for the improvements made
with the Love swipe and the addition of Brainy to the factoids
showed:
Higher engagement
● 12% more swipes per session (11 swipes per person)
● 18% increase in discovery - titles added to a person’s
watchlist (2 titles per person)
● Average of 45 sec more time spent on the app per user
● 7% increase in user visits
These increases in engagement showed an improvement in the
overall experience of the app related to trust, discovery and
enjoyment.
They also show measurable improvements to the machine learning
powering the recommendations.

13. Qualitative improvements
Our follow up qualitative testing showed:
● When people first experienced the “Love”, slow swipe in
the tutorial and build up from a “like”, response were of
excitement and verbally saying things like, “I love that”
● People were oriented to the idea that they would find
things they love by using the app
● Consistent feedback that people really enjoyed seeing
Brainy and the factoids because it helped them
understand why something was being shown
● If they didn’t know the movie they could get a quick
sense of it
● When people spoke of this new feature they would
identify it not as a factoid, but as the “little cupcake guy”
or the “cupcake at the top of the screen”
● This emphasized the power of the icon and tied back to
the emotional resonance associated with Brainy in earlier
testing

14. What happened
The power of emotions cannot be underestimated. Looking the emotional
research done by the Echo team on MightyTV is strong evidence that
understanding how users feel, analysing emotional research findings and
integrating it into the design and product development processes can garner
measurable results.
Successes:
● #1 Best New App in the iTunes App Store - April 22, 2016
● Featured in the “Apps we Love” section of the App store
● Fast Company Design Innovation Award Finalist
● Acquired by Spotify in March 2017
Emotional research was a key factor in achieving these accomplishments.

15. Workshop
Emotional Design Techniques for AI / ML Systems
Friday, March 23
go to Eventbrite.com for more details

16. Real people, real insights