Abstract:- At Netflix we strive to go beyond the user’s expectations for a streaming experience. A key component to achieving this is ensuring that the best quality of digital assets is made available. These assets are video, audio, subtitle and closed captioning files that collectively contribute towards the viewing experience. Having a rich catalog with the freshest content becomes inconsequential if a user experiences issues like the timing of the audio and video being off, or if the subtitles are positioned poorly. Moreover, asset quality can have a direct effect on member satisfaction and ultimately retention. Netflix sets a high bar on content quality, and has a thorough Quality Control (QC) process in place to ensure that this bar is met. Our recent global launch has necessitated having a broad catalog with a wide variety of audio and subtitle languages across countries. In order to retain a lean operation as we scale, we based our QC process on a supervised model that predicts the likelihood of an asset having an issue. We then perform a QC only on assets that are predicted with a likelihood beyond a threshold. Since we intelligently select what to QC and do not check every asset that goes on our service, there may be instances when a bad asset slips through. We needed a facility on the back end that could catch these undetected issues. Our member feedback channels were an obvious area to tap. Member feedback comes in two forms: explicit and implicit. Explicit feedback is received from sources like the “Report a Problem” section on the site, social media (Twitter, Facebook) and customer service calls. Implicit feedback can be derived from user viewing behavior, such as sharp drop-offs at certain points during the playback. This talk will focus on • How we mine explicit member feedback, in particular from the “Report a Problem” section on our site • The challenges posed with identifying what is relevant due to the variety of context in the feedback obtained • The analysis framework to monitor feedback and manage workflow • Areas for improvement / future work Bio:- Nishant Hegde is a Senior Analytics Engineer at Netflix. He focuses on data engineering, analysis and data visualization in the Digital Supply Chain Analytics team. Nishant was previously an analyst and managed teams in the Forensic Analytics practices at Deloitte and Price Waterhouse Coopers.

1. Mining Member
Feedback to
Improve
The Customer
Experience
Nishant Hegde, Sr. Analytics Engineer

2. analytics @ Netflix
Data Engineering And Analytics

4. The “Content Quality” Problem
Buying Sourcing

5. Assets
Studio
Content
Partners
Encoding
Package
How is
our
Content
Quality?
FAIL
PASS

6. Encoding Video Hit Audio Distortion Hit
Some Examples of Content Quality Issues

7. ● Value
● Easily Integrated
v1

8. v1
Event Producer
Ingest Pipeline

9. v1
Week 1 Bojack ugh
Week 1 Bojack Bojack is awesome
Week 1 Bojack No audio
Week
2
Bojack When is season 4
coming out?
Week
2
Bojack ...
Random Forest Classifier
Week 1 Bojack ugh 0.7
Week 1 Bojack Bojack is
awesome
0.3
Week 1 Bojack No audio 0.9
Week 2 Bojack When is
season 4
coming
out?
0.5
Week 2 Bojack ...
Act / Do Not Act
Retrain
Spreadsheets
Assign likelihood that comment
needs action
Collect comments per show on a
weekly basis

10. v1
Outcome
● Accuracy = 25% ● signal : noise
○ “ i didnt watch this piece of crap dumb
people call movie.”
○ “I DID NOT WATCH THIS! THEREFORE,
MY NETFLIX ACCOUNT HAS BEEN
HACKED....AGAIN!!! WOULD YOU PLEASE
DO SOMETHING ABOUT THIS???”
○ “Someone else is accessing Netflix on my
account, probably my EX wife, I'd like to
stop any other use then myself and have
tried to change my account email and
password but they still have access, this
movie was not ordered by me nor the one
previous.”
○ “Frame appears to be windowboxed
4:3-in-16:9-in-4:3 with some sort of
white line border artifact on the right
side. All other episodes are normal.”
● Class Imbalance
● Buy-in

11. ● Language Specific Models
● Daily Processing
● Integrate Twitter
v2

12. Local Language +
English Translated Models +
Separate Model for Tweets
v2
Day 1 Bojack ugh 0.7
Day 1 Bojack Bojack is
awesome
0.3
Day 1 Bojack pas de hd 0.6
Day 2 Bojack Nao fica em hd
de forma
alguma
0.5
Day 2 Bojack ...
Expanded to foreign languages
Act / Do Not Act
Cloud App
Assign likelihood that comment
needs action
Collect comments per
show on a daily basis
Retrain
Day 1 Bojack ugh
Day 1 Bojack Bojack is
awesome
Day 1 Bojack pas de hd
Day 2 Bojack Nao fica
em hd de
forma
alguma
Day 2 Bojack ...
Measure
Performance

13. Measuring Performance...
v2
● Recall = TP / (TP + FN)
● Precision = TP / (TP + FP)
● Compliance = (TP + FP) / (TP + FN + FP + TN)
● % looked at = # acted on / # served
● % accurate = # issues found/ # acted on
● % thrash = # non-issues found / # acted on
1. Models
2. Workflow

14. v2

15. Russian Dialogue IssueSome Big Finds...

16. Dubbed Audio Issue

17. Out of Sync Subtitle Issue

18. The Future ...
● Abandonment Time in Movie
● # of Seek Operations
● Average Time between Sessions
● Unsupervised Learning
Problem?!

19. plata o plomo?