Maintaining the Front Door to Netflix : The Netflix API

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Maintaining the Front Door to Netflix
Daniel Jacobson
@daniel_jacobson
http://www.linkedin.com/in/danieljacobson
http://ww...

There are copious notes attached
to each slide in this presentation.
Please read those notes to get
the full context of th...

Global Streaming Video
for TV Shows and Movies

More than 44 Million Subscribers
More than 40 Countries

Netflix Accounts for ~33% of Peak
Internet Traffic in North America
Netflix subscribers are watching more than 1 billion h...

Team Focus:
Build the Best Global Streaming Product
Three aspects of the Streaming Product:
• Non-Member
• Discovery
• Str...

Key Responsibilities
• Broker data between services and UIs
• Maintain a resilient front-door
• Scale the system verticall...

But Before Streaming…

Monolithic Application
In Netflix Data Centers

The bigger the ship…
the slower it turns

Distributed Architecture

1000+ Device Types

Personaliz
ation
Engine
User Info
Movie
Metadata
Movie
Ratings
Similar
Movies
Reviews
A/B Test
Engine
Dozens of Dependenci...

Personaliz
ation
Engine
User Info
Movie
Metadata
Movie
Ratings
Similar
Movies
API
Reviews
A/B Test
Engine

Dependency Relationships

2,000,000,000
Requests Per Day to the
Netflix API

30
Distinct Dependent
Services for the Netflix API

~500
Dependency jars Slurped
into the Netflix API

14,000,000,000
Netflix API Calls Per Day to
those Dependent Services

0
Dependent Services with
100% SLA

99.99% = 99.7%30
0.3% of 2B = 6M failures per day
2+ Hours of Downtime
Per Month

99.99% = 99.7%30
0.3% of 2B = 6M failures per day
2+ Hours of Downtime
Per Month

99.9% = 97%30
3% of 2B = 60M failures per day
20+ Hours of Downtime
Per Month

Personaliz
ation
Engine
User Info
Movie
Metadata
Movie
Ratings
Similar
Movies
API
Reviews
A/B Test
Engine

Personaliz
ation
Engine
User Info
Movie
Metadata
Movie
Ratings
Similar
Movies
API
Reviews
A/B Test
Engine

Personaliz
ation
Engine
User Info
Movie
Metadata
Movie
Ratings
Similar
Movies
API
Reviews
A/B Test
Engine

Personaliz
ation
Engine
User Info
Movie
Metadata
Movie
Ratings
Similar
Movies
API
Reviews
A/B Test
Engine

Personaliz
ation
Engine
User Info
Movie
Metadata
Movie
Ratings
Similar
Movies
API
Reviews
A/B Test
Engine

Circuit Breaker Dashboard

Call Volume and Health / Last 10 Seconds

Call Volume / Last 2 Minutes

Successful Requests

Successful, But Slower Than Expected

Short-Circuited Requests, Delivering Fallbacks

Timeouts, Delivering Fallbacks

Thread Pool & Task Queue Full, Delivering Fallbacks

Exceptions, Delivering Fallbacks

Error Rate
# + # + # + # / (# + # + # + # + #) = Error Rate

Status of Fallback Circuit

Requests per Second, Over Last 10 Seconds

SLA Information

Personaliz
ation
Engine
User Info
Movie
Metadata
Movie
Ratings
Similar
Movies
API
Reviews
A/B Test
Engine

Personaliz
ation
Engine
User Info
Movie
Metadata
Movie
Ratings
Similar
Movies
API
Reviews
A/B Test
Engine

Personaliz
ation
Engine
User Info
Movie
Metadata
Movie
Ratings
Similar
Movies
API
Reviews
A/B Test
Engine

Personaliz
ation
Engine
User Info
Movie
Metadata
Movie
Ratings
Similar
Movies
API
Reviews
A/B Test
Engine
Fallback

Personaliz
ation
Engine
User Info
Movie
Metadata
Movie
Ratings
Similar
Movies
API
Reviews
A/B Test
Engine
Fallback

Scaling the Distributed System

AWS Cloud

Autoscaling

Autoscaling

Amazon Auto Scaling Limitations
• Hard to fit policies to variable traffic patterns
(weekday vs weekend)
• Limited control...

The Impact of AAS Limitations
• Traffic drop can lead to scale downs during
outage
• Performance degradation between new
i...

Scryer : Predictive Auto Scaling
Not yet…

Typical Traffic Patterns Over Five Days

Predicted RPS Compared to Actual RPS

Scaling Plan for Predicted Workload

What is Scryer Doing?
• Evaluating needs based on historical data
– Week over week, month over month metrics
• Adjusts ins...

Results

Results : Load Average
Reactive
Predictive

Results : Response Latencies
Reactive
Predictive

Results : Outage Recovery

Results : Outage Recovery

Results : AWS Costs

Scaling Globally

More than 44 Million Subscribers
More than 40 Countries

Zuul
Gatekeeper for the Netflix Streaming Application

Zuul *
• Multi-Region
Resiliency
• Insights
• Stress Testing
• Canary Testing
• Dynamic Routing
• Load Shedding
• Security...

Isthmus

All of these approaches are
designed to prevent failures…

But sometimes the best way to
prevent failures is to force them!

I randomly
terminate instances
in production to
identify dormant
failures.
Chaos
Monkey

Chaos
Gorilla
I simulate an
outage of an
entire Amazon
availability zone.

I simulate an
outage in an AWS
region.
Chaos
Kong

I find instances that
don’t adhere to
best practices.
Conformity
Monkey

I extend Conformity
Monkey to find
security violations.
Security
Monkey

I detect unhealthy
instances and
remove them
from service.
Doctor
Monkey

I clean up the
clutter and waste
that runs in the
cloud.
Janitor
Monkey

I induce artificial
delays and errors into
services to determine
how upstream services
will respond.
Latency
Monkey

Deployments in the Cloud

Dependency Relationships

Testing Philosophy:
Act Fast, React Fast

That Doesn’t Mean We Don’t Test

Automated Delivery Pipeline

Cloud-Based Deployment Techniques

Current Code
In Production
API Requests from
the Internet

Single Canary Instance
To Test New Code with Production Traffic
(around 1% or less of traffic)
Current Code
In Production
...

Canary Analysis Automation

Single Canary Instance
To Test New Code with Production Traffic
(around 1% or less of traffic)
Current Code
In Production
...

Current Code
In Production
API Requests from
the Internet

Current Code
In Production
API Requests from
the Internet

Current Code
In Production
API Requests from
the Internet
Perfect!

Stress Test with Zuul

Current Code
In Production
API Requests from
the Internet
New Code
Getting Prepared for Production

Current Code
In Production
API Requests from
the Internet
New Code
Getting Prepared for Production

Error!
Current Code
In Production
API Requests from
the Internet
New Code
Getting Prepared for Production

Current Code
In Production
API Requests from
the Internet
New Code
Getting Prepared for Production

Current Code
In Production
API Requests from
the Internet
Perfect!

Stress Test with Zuul

Current Code
In Production
API Requests from
the Internet
New Code
Getting Prepared for Production

Current Code
In Production
API Requests from
the Internet
New Code
Getting Prepared for Production

API Requests from
the Internet
New Code
Getting Prepared for Production

Brokering Data to
1,000+ Device Types

Screen Real Estate

Controller

Technical Capabilities

One-Size-Fits-All
API
Request
Request
Request

Courtesy of South Florida Classical Review

Resource-Based API
vs.
Experience-Based API

Resource-Based Requests
• /users/<id>/ratings/title
• /users/<id>/queues
• /users/<id>/queues/instant
• /users/<id>/recomm...

REST API
RECOMME
NDATIONS
MOVIE
DATA
SIMILAR
MOVIES
AUTH
MEMBER
DATA
A/B
TESTS
START-
UP
RATINGS
Network Border Network Bo...

RECOMME
NDATIONS
MOVIE
DATA
SIMILAR
MOVIES
AUTH
MEMBER
DATA
A/B
TESTS
START-
UP
RATINGS
OSFA API
Network Border Network Bo...

RECOMME
NDATIONS
MOVIE
DATA
SIMILAR
MOVIES
AUTH
MEMBER
DATA
A/B
TESTS
START-
UP
RATINGS
OSFA API
Network Border Network Bo...

Experience-Based Requests
• /ps3/homescreen

JAVA API
Network Border Network Border
RECOMME
NDATIONS
MOVIE
DATA
SIMILAR
MOVIES
AUTH
MEMBER
DATA
A/B
TESTS
START-
UP
RAT...

RECOMME
NDATIONSA
ZXSXX C
CCC
MOVIE
DATA
SIMILAR
MOVIES
AUTH
MEMBER
DATA
A/B
TESTS
START-
UP
RATINGS
JAVA API
SERVER CODE
...

RECOMME
NDATIONSA
ZXSXX C
CCC
MOVIE
DATA
SIMILAR
MOVIES
AUTH
MEMBER
DATA
A/B
TESTS
START-
UP
RATINGS
JAVA API
DATA GATHERI...

https://www.github.com/Netflix

Maintaining the Front Door to Netflix
Daniel Jacobson
@daniel_jacobson
http://www.linkedin.com/in/danieljacobson
http://ww...

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

API Revolutions : Netflix's API Red... by Daniel Jacobson 35841 views
Netflix Edge Engineering Open House... by Daniel Jacobson 4821 views
Maintaining the Netflix Front Door ... by Daniel Jacobson 5413 views
Netflix API - Separation of Concerns by Daniel Jacobson 10791 views
Set Your Content Free! : Case Studi... by Daniel Jacobson 35260 views
RESTful API Design, Second Edition by Apigee | Google C... 180530 views

View on Slideshare

1 of 138 Ad

View on Slideshare

1 of 138 Ad