Your SlideShare is downloading. ×
0
Experimentation Platform at Netflix
Experimentation Platform at Netflix
Experimentation Platform at Netflix
Experimentation Platform at Netflix
Experimentation Platform at Netflix
Experimentation Platform at Netflix
Experimentation Platform at Netflix
Experimentation Platform at Netflix
Experimentation Platform at Netflix
Experimentation Platform at Netflix
Experimentation Platform at Netflix
Experimentation Platform at Netflix
Experimentation Platform at Netflix
Experimentation Platform at Netflix
Experimentation Platform at Netflix
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Experimentation Platform at Netflix

2,606

Published on

0 Comments
7 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,606
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
22
Comments
0
Likes
7
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Experimentation Platform at Netflix experimentation@netflix.com
  • 2. What is Experimentation? • It’s the process of randomly dividing users into groups – Control Group, existing experience/behavior – Variant 1…n, one or more new experiences – Gather behavior and core metrics for all variants – Analyze and evaluate hypothesis by p-value • Also known as AB Testing • Examples – User interface changes, new product features, changes personalization algorithms etc.
  • 3. Random allocation to experiments Netflix Customers Experiment 1, Variant A Experiment 2, Variant E Experiment ‘n’, VariantM Invariant: Control ……… ………
  • 4. Why? • Data driven product innovation – http://www.quora.com/What-types-of-things-does-Netflix-A-B-test-aside-from-member- sign-up – http://www.hakkalabs.co/articles/hive-controlled-experimentation-2 • Iteration of ideas in a controlled way – Validate features and presentation based on data – Fail fast • It’s the culture
  • 5. User interface Experiment Variant 2 Variant 3
  • 6. New feature (Profiles) Experiment Variant 2 Variant 3 Variant 4
  • 7. Personalization Algorithm Experiment No user interface change Variant 1: Control - No change Variant 2: Movies and TV shows Variant 3: TV shows only
  • 8. Cassandra Magma UI Experiments & Analytics Segmentation rules Stratification Allocation engine Event stream Hadoop/Hive Experimentation Platform H T T P memcached Clients Apache Spark
  • 9. Allocation engine • Responsible for assigning customers to a variant in the experiment • Randomly distribute customers across variants • Stratified Sampling – Bias or variance reduction – Varying sample size across platforms for e.g. Game console vs Blu-ray players vs Laptops • Start, Stop and Track allocations in near real-time
  • 10. Segmentation • Divide a broad target population into subsets with similar properties for e.g. customers who have not used a Tablet to access Netflix in the last ‘n’ days • Have enough data to execute real-time • Achieve scale and maintain low latency • Clean data for analysis
  • 11. Analysis • Gather core metrics and behavior data – Near-time (~15 mins) data from some data sources – Batch (~few hours) data from all data sources • Petabytes of multi-dimensional data – Pre vs post compute/aggregate processing – Adjust for biases and/or incidents – Interactive analysis within given constraints • Explore for patterns • Scale to support fast growing business and the big datasets
  • 12. Magma UI Experiments & Analytics • Experiment lifecycle management – Hypothesis, experiment variants – Real-time rule-based segmentation • Define rules and conditions to identify the right population for experiments • Rules are dynamic and applied real-time – Scheduling and forecasting for categories of experiments • Insights & dashboards – Near-time (< 1 hr) insights – Offline (few hrs) insights and trends – Experiment dashboards • Analysis – Interactive analysis of petabytes of data • Filters on core dimensions, behavior, videos, allocations etc. – Data visualization of trends, insight data over time
  • 13. Some Challenges • How do we efficiently scale the segmentation system • How do we continue to operate at low latencies, given – Customer segmentation is based on real-time activity – Real-time allocations have broad applications – Data distributed across multiple clusters • Big data (In petabytes) processing – Billions of rows of data from various sources where data joins can create exponential increase in number of rows – Efficient management of these datasets to support interactive analysis, dashboards etc – Rich and flexible filtering to support interactive analysis • Rich forecasting and insights • Resiliency • And more…
  • 14. Large scale and growing
  • 15. We’re hiring • Work on large scale, big data and distributed systems – Distributed systems – Server-side engineers • Data Structures & Algorithms • Concurrency, Multi-threading, Caching – Data systems engineering – Server-side engineers • Data Structures & Algorithms • Distributed data stores, Experience processing very large (petabytes) datasets – Data systems engineering – UI & Data visualization engineers • HTML5/JavaScript/CSS – Angular/Ember, D3 would be a plus • Previous experience in data visualization • Contact Us – experimentation@netflix.com – bit.ly/ExperimentationAtNetflix – Netflix has unique culture. Read about it here.

×