PERSONALIZATION
10 LESSONS LEARNED from NETFLIX SUCCESS
(and how to apply them to your video services)

Personalization
10 Lessons Learned from Netflix’s Success
(and how to apply them to video services)

Greetings!
Kauser Kanji Pancrazio Auteri
Managing Editor, VOD Professional CTO, Contentwise

Today’s objectives
2 1
Share the findings of observing how Netflix uses
personalization as a competitive advantage in engaging
and retaining users and planning content acquisition
Show how you can connect the dots and take advantage
of those findings for your online video services

So what is personalization?

ph. Thinkstock
Not like this Personalized like this

It’s about the pleasure of choice and the abundance of options
It’s understanding needs and narrowing the options
to make choosing a pleasant experience

Personalization can go wrong

It’s not just high-tech
Brian & Doreen remember customers’ taste, curate and organize shelves,
plan promotions, update the display windows,
listen to requests and recommend products and let customers browse
their shop in Somerset, UK

Netflix
Hulu
Amazon
Home page (web)
Pure S-VOD
S-VOD Prime
Free, ad-based VOD
Upsell S-VOD Hulu+
T-VOD

So what’s the problem?
Broad range of user types and tastes
Fragmentation of content sources and applications: too many places to look at
Content availability can be very dynamic over time
Most UI solutions rely on drill-down and user efforts
User’s attention span and screen real estate are very limited
You name it!

See the opportunity?
Consumption increases
Habits formation occurs
Users feel more engaged
Things can happen when
people have a truly
personalized experience
Your service becomes a destination for unified discovery
People talk about your brand with passion
Subscribers perceive the value and the abundance of what you offer

Let’s start!

10 lessons learned from Netflix
The information, interpretations, advice and recommendations contained in this presentation are not
endorsed in any way by Netflix and are based on information publicly shared by Netflix or its employees.

10 lessons learned from Netflix
1. Set objectives and pick metrics

10 lessons learned from Netflix
1. Set objectives and pick metrics
Netflix: maximize member satisfaction and
month-to-month subscription retention
Example Metrics

10 lessons learned from Netflix
1. Set objectives and pick metrics
Example Metrics
Canceled subscriptions per month
Interactive sessions resulting in a playback
Played minutes per user per month
Fully watched playbacks
Interaction time before starting a playback
Returning users

10 lessons learned from Netflix
2. Consider UX as mission-critical

10 lessons learned from Netflix
2. Consider UX as mission-critical
secret sauce
best practices
innovative functionalities
well-tuned business rules
UX Engine Users
Content offer
Audiovisual quality
Delivery infrastructure
Editorial curation
Customer support

10 lessons learned from Netflix
2. Consider UX as mission-critical
UX Engine =
Code Users
Rules
Widgets
Behaviors
Configurations
Management tools

10 lessons learned from Netflix
2. Consider UX as mission-critical
Changes in UI behavior can have a dramatic impact on key metrics.
Multiscreen? Make sure behavior is consistent across devices
Pay special attention to
cross-screen consistency of
Welcome screens
Frequent user actions
User “lost” actions
Leverage the UX Engine
to control UI behavior
across all screens

10 lessons learned from Netflix
3. Personalize UX as much as possible

10 lessons learned from Netflix
3. Personalize UX as much as possible
At Netflix, more than 75% of views
come from some sort of
recommendation or personalized ranking
Views %
from personalized ranking
source: Netflix
0 25 50 75 100

10 lessons learned from Netflix
3. Personalize UX as much as possible
User’s attention span is very limited
The first 8-12 seconds are critical
Weinreich et al. - ACM 2008
Desired outcomes
- find something to watch
- engage in some sort of exploration

10 lessons learned from Netflix
3. Personalize UX as much as possible
Screen real estate is very limited too
Ideally user should
find relevant content
in the first screen

10 lessons learned from Netflix
3. Personalize UX as much as possible
A
Ineffective sort criteria Effective criteria
B
Alphabetic
C
By release year
Personalized order
By ingestion order
D
Canned categories
Even “computed” lists such as
E
Most popular
Most viewed
F
Recently added
and My list!

10 lessons learned from Netflix
3. Personalize UX as much as possible
There are two “folds”
Netflix personalizes in both directions
1. ranking of items in a carousel
2. ranking of carousels in the layout
2
Real estate
“above-the-fold”
1

10 lessons learned from Netflix
3. Personalize UX as much as possible
Featured content
Resume play + My list
{
Most likely actions
Popular
Top picks for you
Recently added
Main genres
Pseudo-genres
Because you watched…
Friends watching…
Watch again
Displayed in a
personalized order"
"
Some of them disappear for
a while if never “touched” "
or because of A/B Testing

10 lessons learned from Netflix
3. Personalize UX as much as possible
Personalized reordering may be
disorienting for some people
Indeed some Netflix users
complain about this;
but it seems to be effective
and we’ll show you
how to handle it

Aha!
My Aha! moment with NETFLIX

10 lessons learned from Netflix
4. Understand user’s lifestyle and context

10 lessons learned from Netflix
4. Understand user’s lifestyle and context
Netflix mines usage data to extract behavior patterns
Personalization may be affected by
context elements
Device type
Time of the day
Day of the week
Season of the year
User is at-home or out-of-home
Geo-location (traveling, commuting, weekend-home…)
Local weather
Popular news
Other users in close proximity (phones/wearables)

10 lessons learned from Netflix
5. Use interaction data then ask for feedback
Priority on high-value usage events
Playback start/stop/resume
View asset details
Add to personal list
Other interactions
Trick-play control
Search
Sharing
Navigation paths…
Ask for feedback
5-stars
Like
Dislike
Love it!

10 lessons learned from Netflix
6. Let users know how the service is adapting
to their tastes

10 lessons learned from Netflix
6. Let users know how the service is adapting
to their tastes
Promote trust in the system
Encourage users to give feedback
Better personalization

10 lessons learned from Netflix
6. Let users know how the service is adapting
to their tastes
Use meaningful labels
referring to past behavior
user can recognize
Because you watched Breaking Bad
Because of your interest for Time Travel
Because you loved Kill Bill Vol.1

10 lessons learned from Netflix
7. Ensure metadata captures content nuances
and is consistent

10 lessons learned from Netflix
7. Ensure metadata captures content nuances
and is consistent
Actors, Directors, Writers
Genres
Synopsis Release Year
Duration
Country Studio
Language
Characters Topics Themes Moods
Locations Time Periods
Keywords - Microtags
“NETFLIX QUANTUM THEORY”
A set of best practices for manual
micro-tagging of video content
Social acceptability of the lead character

10 lessons learned from Netflix
7. Ensure metadata captures content nuances
and is consistent
Let users search for content you don’t have

10 lessons learned from Netflix
7. Ensure metadata captures content nuances
and is consistent
With richer content metadata
you can use analytics to
understand content performance
and drive content acquisition
(or even original production)
And add meaning to user profiles

Movie Iron Man 3
Data from Gracenote-TMS

TV Series Breaking Bad
Data from Gracenote-TMS

10 lessons learned from Netflix
8. Give reasons to come back often
Refresh catalog frequently - OR - Let the UX Engine do it for you (virtually)

10 lessons learned from Netflix
8. Give reasons to come back often
Re-shuffle top items to periodically
change the ones above-the-fold
Items outside the first screen are
still highly relevant for the user
User perceives novelty and will
be keen to return more often

10 lessons learned from Netflix
9. Run frequent UI experiments
There is no “perfect way” and there are many types of users:
experiments and adaptation seem to be the most effective ways
Identify the UI elements on the path to the key goals
Roll-out the variations and look at 2-5 metrics
Run the experiments for two weeks or until statistical validity
Design and plan experiments not to interfere with each other
Experiments consume interaction events:
make sure there is enough activity to feed all of the active variations

10 lessons learned from Netflix
10. Close the loop, base decisions upon data

10. Close the loop, base decisions upon data
Netflix was the only network that
said “We believe in you. We’ve run
our data, and it tells us that our
audience would watch this series.
We don’t need you to do a pilot”
Kevin Spacey, actor and producer
Listen to Kevin saying this (video)

10 lessons learned from Netflix
10. Close the loop, base decisions upon data
Netflix uses analytics to heavily influence the
content acquisition policy
Netflix proved to be agile and effective in rolling out variations and
track several metrics across hundreds of client platforms
Netflix team is very disciplined on reporting UI events.
This enables full visibility in analytics and higher ROI
Yes. At Netflix they go nuts for analytics!
And they look to be right

10 Lessons from Netflix - Recap
1. Set objectives, pick metrics and share them with the team
2. Consider UX as mission-critical
3. Personalize UX as much as possible
4. Understand user’s lifestyle and context
5. Use interaction data then ask for feedback
6. Let users know your service is adapting to their tastes
7. Ensure metadata captures content nuances and is consistent
8. Give reasons to come back often
9. Run frequent UI experiments
10. Close the loop and base your decisions upon data

Netflix solutions are applicable (and applied) at… Netflix
Other services may include S-VOD
as well as Linear TV, DVR,
Transactional VOD, Pay TV, Pay-per view,
music videos, sports highlights,
Advertising or User-generated Content…
We need a way to turn these lessons into practice
touching all the stakeholders in our projects

"
UIDO
A set of checklists to guide
you while introducing
personalization in your
video service

What you deliver How you start
User Experience Integrator Experience
UX IX
DX OX
Developer Experience Operator Experience
How you build it Tools to manage "
UIDO

UX User Experience What you deliver

UX User Experience What you deliver
Content types ✓ Movies
Aggregates ✓ Collections
✓ Series
✓ Episodes
✓ Extras
✓ Music videos
✓ Playlists
✓ News
✓ Sports events
✓ Sports highlights
✓ Scheduled programs
✓ Channels
✓ _____________________
✓ Seasons
✓ Channel bundles
✓ Movie bundles
✓ Sports Team bundles
✓ Sports League bundles
✓ __________________

UX User Experience What you deliver
Key UX features ✓ Manually curated collections
✓ Search results
✓ Search suggestions while you type (single/multi-type)
✓ Search refine with smart filters (facets)
✓ Similar content
✓ Personalized picks for user
✓ Critics-based feed (Rotten Tomatoes, Metacritic…)
✓ Series you watch (with next-episode)
✓ VOD bookmarking (resume playback)
✓ User’s list
✓ Predictive browsing (surfacing folders)
✓ Personalized pseudo-genres

Reference UI
"
Showing most of the
personalization use cases
supported by ContentWise

UX User Experience What you deliver
Key UX features
(cont’d)
✓ Social graph (e.g. friends, followers)
✓ Sharing actions
✓ Content can be embedded
✓ Co-watching (blended profiles)
✓ Profile explanation with content metadata
✓ User can rate content (stars, like, dislike, love, etc.)

UX User Experience What you deliver
For kids ✓ Parental ratings
✓ Kids mode
✓ Specialized metadata (e.g. Commonsense)
✓ Editorial curation
✓ Curation by parents
✓ Analytics for parents

UX User Experience What you deliver
Content sources ✓ Linear schedule (line-ups)
✓ Start-over TV system
✓ VOD Catalog
✓ Local DVR
✓ Network DVR
✓ Reverse EPG (catch-up)
✓ ______________

UX User Experience What you deliver
Device types ✓ Phone
✓ Tablet
✓ PC
✓ TV
✓ Watch
Access models ✓ S-VOD
✓ T-VOD
✓ Ad-VOD
✓ Free-Linear
✓ Pay-Linear
✓ PPV
Profile types ✓ Personal
✓ Household
✓ Main account powers
✓ Blended
✓ Personas templates
✓ Personal on device
Access locations ✓ At-home, OOH
✓ On-net, off-net

UX User Experience What you deliver
Entitlements ✓ S-VOD packages
✓ Rented movies
✓ Purchased movies
✓ Purchased seasons
✓ Purchased episodes
✓ Subscribed channels
✓ Subscribed bundles (e.g. Channel + S-VOD)
✓ ______________________

Explaining a recommendation
Because you liked
these other movies
Affinity between the
user’s taste and the
recommended movie
(using the tag structure)
ContentWise Reference UI

OX Operator Experience How to manage

OX Operator Experience How to manage
✓ Managing UI Elements with UX Engine
✓ Creating and updating editorial lists
✓ Generating and curating pseudo-genres
✓ Accessing analytics
✓ Content planning using analytics
✓ Managing variations and experiments for A/B Testing
✓ Understanding the impact of business rules on key metrics

Personalized pseudo-genres
INTENSE ACTION MOVIES
mood genre type
2000s AUSTRALIAN THRILLER MOVIES
release prod
genre type
year
country
AMERICAN DRAMA MOVIES STARRING TOM HANKS
ContentWise Reference UI

The magic of richer metadata
MOVIES FROM FEMALE DIRECTORS
type person role
MOVIES STARRING A ROCKSTAR
type
gender
from semantic
enrichment
looking
into actors
person role
from semantic
enrichment
ContentWise Reference UI

Curation of Pseudo-genres Metadata fields
considered for
labels
Status of the
pseudo-genre
Type:
Editorial
or
Computed
ContentWise Management Console

Driving from the UX Engine
Rendered by UI code
Configured by UX Engine
ContentWise Management Console

UX Engine - Personalized Order of Carousels

Content planning - Choosing items to retire
Find movies with a small
number of “estimated”
residual views
and are “expiring”
Automatically create a
business rule
The rule can be used in A/B Testing
to anticipate the impact of removing
these movies from the catalog.
ContentWise Management Console

A/B/C Testing
Biz rule #1
Biz rule #N
Variation A
Biz rule #1
Biz rule #N
Variation B
Experiment
Group A
Group B
Control Group
Results Metrics
Normal
behavior

Launching an experiment
This is an experiment
Treated user base
Variations

Monitoring experiment results
Variations Affected users Metrics
ContentWise Management Console

DX Developer Experience How to build

DX Developer Experience How to build
UI Element ✓ Carousel
✓ Group of carousel
✓ Item
✓ Item attribute
✓ __________

DX Developer Experience How to build
UI Events
to be reported
✓ Item displayed
✓ Item selected
✓ Item details accessed
✓ User rating submitted
✓ Explanation displayed
✓ Tuned-in
✓ Tuned-out
✓ Playback started
✓ Playback resumed
✓ Playback paused
✓ Playback completed
✓ Playback stopped
✓ Item saved to list
✓ Item removed from list
✓ Preview playback started

IX Integrator Experience How to start

IX Integrator Experience How to start
✓ Content model map
✓ Event model map
✓ User ID map
✓ Data refresh policy
✓ Bulk ingestion automation
✓ Delta updates automation
✓ Client applications map
✓ UI elements to be managed from UX Engine

Thank you!
For more information, please visit our website or contact us
pancrazio.kauser.kanji@vodprofessional.com auteri@contentwise.tv
Digital TV. Personalized
www.vodprofessional.com www.contentwise.tv