Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Mining Member Feedback to Improve the Customer Experience by Nishant Hegde of Netflix


At Netflix we strive to go beyond the user’s expectations for a streaming experience. A key component to achieving this is ensuring that the best quality of digital assets is made available. These assets are video, audio, subtitle and closed captioning files that collectively contribute towards the viewing experience. Having a rich catalog with the freshest content becomes inconsequential if a user experiences issues like the timing of the audio and video being off, or if the subtitles are positioned poorly. Moreover, asset quality can have a direct effect on member satisfaction and ultimately retention.

Netflix sets a high bar on content quality, and has a thorough Quality Control (QC) process in place to ensure that this bar is met. Our recent global launch has necessitated having a broad catalog with a wide variety of audio and subtitle languages across countries. In order to retain a lean operation as we scale, we based our QC process on a supervised model that predicts the likelihood of an asset having an issue. We then perform a QC only on assets that are predicted with a likelihood beyond a threshold.

Since we intelligently select what to QC and do not check every asset that goes on our service, there may be instances when a bad asset slips through. We needed a facility on the back end that could catch these undetected issues. Our member feedback channels were an obvious area to tap. Member feedback comes in two forms: explicit and implicit. Explicit feedback is received from sources like the “Report a Problem” section on the site, social media (Twitter, Facebook) and customer service calls. Implicit feedback can be derived from user viewing behavior, such as sharp drop-offs at certain points during the playback.

This talk will focus on

• How we mine explicit member feedback, in particular from the “Report a Problem” section on our site

• The challenges posed with identifying what is relevant due to the variety of context in the feedback obtained

• The analysis framework to monitor feedback and manage workflow

• Areas for improvement / future work


Nishant Hegde is a Senior Analytics Engineer at Netflix. He focuses on data engineering, analysis and data visualization in the Digital Supply Chain Analytics team. Nishant was previously an analyst and managed teams in the Forensic Analytics practices at Deloitte and Price Waterhouse Coopers.

  • Login to see the comments

Mining Member Feedback to Improve the Customer Experience by Nishant Hegde of Netflix

  1. 1. Mining Member Feedback to Improve The Customer Experience Nishant Hegde, Sr. Analytics Engineer
  2. 2. analytics @ Netflix Data Engineering And Analytics
  3. 3. The “Content Quality” Problem Buying Sourcing
  4. 4. Assets Studio Content Partners Encoding Package How is our Content Quality? FAIL PASS
  5. 5. Encoding Video Hit Audio Distortion Hit Some Examples of Content Quality Issues
  6. 6. ● Value ● Easily Integrated v1
  7. 7. v1 Event Producer Ingest Pipeline
  8. 8. v1 Week 1 Bojack ugh Week 1 Bojack Bojack is awesome Week 1 Bojack No audio Week 2 Bojack When is season 4 coming out? Week 2 Bojack ... Random Forest Classifier Week 1 Bojack ugh 0.7 Week 1 Bojack Bojack is awesome 0.3 Week 1 Bojack No audio 0.9 Week 2 Bojack When is season 4 coming out? 0.5 Week 2 Bojack ... Act / Do Not Act Retrain Spreadsheets Assign likelihood that comment needs action Collect comments per show on a weekly basis
  9. 9. v1 Outcome ● Accuracy = 25% ● signal : noise ○ “ i didnt watch this piece of crap dumb people call movie.” ○ “I DID NOT WATCH THIS! THEREFORE, MY NETFLIX ACCOUNT HAS BEEN HACKED....AGAIN!!! WOULD YOU PLEASE DO SOMETHING ABOUT THIS???” ○ “Someone else is accessing Netflix on my account, probably my EX wife, I'd like to stop any other use then myself and have tried to change my account email and password but they still have access, this movie was not ordered by me nor the one previous.” ○ “Frame appears to be windowboxed 4:3-in-16:9-in-4:3 with some sort of white line border artifact on the right side. All other episodes are normal.” ● Class Imbalance ● Buy-in
  10. 10. ● Language Specific Models ● Daily Processing ● Integrate Twitter v2
  11. 11. Local Language + English Translated Models + Separate Model for Tweets v2 Day 1 Bojack ugh 0.7 Day 1 Bojack Bojack is awesome 0.3 Day 1 Bojack pas de hd 0.6 Day 2 Bojack Nao fica em hd de forma alguma 0.5 Day 2 Bojack ... Expanded to foreign languages Act / Do Not Act Cloud App Assign likelihood that comment needs action Collect comments per show on a daily basis Retrain Day 1 Bojack ugh Day 1 Bojack Bojack is awesome Day 1 Bojack pas de hd Day 2 Bojack Nao fica em hd de forma alguma Day 2 Bojack ... Measure Performance
  12. 12. Measuring Performance... v2 ● Recall = TP / (TP + FN) ● Precision = TP / (TP + FP) ● Compliance = (TP + FP) / (TP + FN + FP + TN) ● % looked at = # acted on / # served ● % accurate = # issues found/ # acted on ● % thrash = # non-issues found / # acted on 1. Models 2. Workflow
  13. 13. v2
  14. 14. Russian Dialogue IssueSome Big Finds...
  15. 15. Dubbed Audio Issue
  16. 16. Out of Sync Subtitle Issue
  17. 17. The Future ... ● Abandonment Time in Movie ● # of Seek Operations ● Average Time between Sessions ● Unsupervised Learning Problem?!
  18. 18. plata o plomo?