At Netflix we strive to go beyond the user’s expectations for a streaming experience. A key component to achieving this is ensuring that the best quality of digital assets is made available. These assets are video, audio, subtitle and closed captioning files that collectively contribute towards the viewing experience. Having a rich catalog with the freshest content becomes inconsequential if a user experiences issues like the timing of the audio and video being off, or if the subtitles are positioned poorly. Moreover, asset quality can have a direct effect on member satisfaction and ultimately retention.
Netflix sets a high bar on content quality, and has a thorough Quality Control (QC) process in place to ensure that this bar is met. Our recent global launch has necessitated having a broad catalog with a wide variety of audio and subtitle languages across countries. In order to retain a lean operation as we scale, we based our QC process on a supervised model that predicts the likelihood of an asset having an issue. We then perform a QC only on assets that are predicted with a likelihood beyond a threshold.
Since we intelligently select what to QC and do not check every asset that goes on our service, there may be instances when a bad asset slips through. We needed a facility on the back end that could catch these undetected issues. Our member feedback channels were an obvious area to tap. Member feedback comes in two forms: explicit and implicit. Explicit feedback is received from sources like the “Report a Problem” section on the site, social media (Twitter, Facebook) and customer service calls. Implicit feedback can be derived from user viewing behavior, such as sharp drop-offs at certain points during the playback.
This talk will focus on
• How we mine explicit member feedback, in particular from the “Report a Problem” section on our site
• The challenges posed with identifying what is relevant due to the variety of context in the feedback obtained
• The analysis framework to monitor feedback and manage workflow
• Areas for improvement / future work
Nishant Hegde is a Senior Analytics Engineer at Netflix. He focuses on data engineering, analysis and data visualization in the Digital Supply Chain Analytics team. Nishant was previously an analyst and managed teams in the Forensic Analytics practices at Deloitte and Price Waterhouse Coopers.