Evolution of the Netflix API
QCon San Francisco - November 2013
Ben Christensen

Software Engineer – Edge & Playback Servi...
Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/netflix-api-evolution

InfoQ....
Presented at QCon San Francisco
www.qconsf.com
Purpose of QCon
- to empower software development by facilitating the sprea...
More than 40 million Subscribers
in 50+ Countries and Territories
Netflix accounts for 33% of Peak Downstream
Internet Traffic in North America

Netflix subscribers are watching
more than ...
API traffic has grown from
~20 million/day in 2010 to >2 billion/day
millions of API requests per day

2000

1500

1000

5...
Discovery

Streaming
Netflix API

Streaming
At the start …

millions of API requests per day

2000

1500

1000

500

0
2008

2009

2010

2011

2012

Today
2008 API Launch
Targeted 100% at External Developers
Open API

Netflix Devices
2008 API Launch
Targeted 100% at External Developers

Purpose
A “Thousand Flowers” of
3rd party innovation
!

Audience
Ext...
Pre-Cloud Architecture
API

Service

Service

Service

Oracle

Service
In 2011 …

millions of API requests per day

2000

1500

1000

500

0
2008

2009

2010

2011

2012

Today
99.9% Netflix Devices
Open API

Netflix Devices
Targeted at Internal Developers
Open API

Netflix Devices
Targeted at Internal Developers

Purpose
Enable Netflix Experience
!

Audience
Netflix Device and UI Teams
Scale & Resilience
!

Performance & Innovation
Scale & Resilience
!

Performance & Innovation
Netflix API

Dependency A

Dependency B

Dependency D

Dependency C

Dependency E

Dependency G

Dependency F

Dependency H...
Netflix API

Dependency A

Dependency B

Dependency D

Dependency C

Dependency E

Dependency G

Dependency F

Dependency H...
AWS
Availability Zone

AWS
Availability Zone

AWS
Availability Zone
User Request

Dependency A

Dependency B

Dependency D

Dependency C

Dependency E

Dependency G

Dependency F

Dependency...
User Request

Dependency A

Dependency B

Dependency D

Dependency C

Dependency E

Dependency G

Dependency F

Dependency...
Dependency A

Dependency B

Dependency D

Dependency C

Dependency E

Dependency G

Dependency F

Dependency H

Dependency...
!

Dozens of dependencies.
!

One going bad takes everything down.
!

30
99.99%

= 99.7% uptime
!

0.3% of 1 billion = 3,0...
CONSTRAINTS

Speed of Iteration
!

Client Libraries
!

Mixed Environment
CONSTRAINTS

Speed of Iteration
!

Client Libraries
!

Mixed Environment
CONSTRAINTS

Speed of Iteration
!

Client Libraries
!

Mixed Environment
CONSTRAINTS

Speed of Iteration
!

Client Libraries
!

Mixed Environment
Dependency A

Dependency B

Dependency D

Dependency C

Dependency E

Dependency G

Dependency F

Dependency H

Dependency...
User Request

User Request

User Request

Logic - argument validation, caches, metrics, logging,
multivariate testing, rou...
> 80% of requests rejected

Median
Latency
[Sat Jun 30 04:01:37 2012] [error] proxy: HTTP: disabled connection for (127.0....
User Request

Dependency A

Dependency B

Dependency D

Dependency C

Dependency E

Dependency G

Dependency F

Dependency...
User Request

Dependency A

Dependency B

Dependency D

Dependency C

Dependency E

Dependency G

Dependency F

Dependency...
User Request

Dependency A

Dependency B

Dependency D

Dependency C

Dependency E

Dependency G

Dependency F

Dependency...
User Request

Dependency A

Dependency B

Dependency D

Dependency C

Dependency E

Dependency G

Dependency F

Dependency...
circle color and size represent
health and traffic volume
circle color and size represent
health and traffic volume

2 minutes of request rate to
show relative changes in traffic
circle color and size represent
health and traffic volume

2 minutes of request rate to
show relative changes in traffic
hos...
circle color and size represent
health and traffic volume

2 minutes of request rate to
show relative changes in traffic
hos...
circle color and size represent
health and traffic volume

2 minutes of request rate to
show relative changes in traffic
hos...
circle color and size represent
health and traffic volume
Request rate
2 minutes of request rate to
show relative changes i...
circle color and size represent
health and traffic volume

Error percentage of
last 10 seconds
Request rate

2 minutes of r...
Error percentage of
last 10 seconds

circle color and size represent
health and traffic volume

Request rate
2 minutes of r...
User Request

Dependency A

Dependency B

Dependency D

Dependency C

Dependency E

Dependency G

Dependency F

Dependency...
Zuul Routing Layer

Canary vs Baseline
Squeeze

"Coalmine"

Production
Zuul Routing Layer

Canary vs Baseline
Squeeze

"Coalmine"

Production
Zuul Routing Layer

Canary vs Baseline
Squeeze

"Coalmine"

Production
Zuul Routing Layer

Canary vs Baseline
Squeeze

"Coalmine"

Production
Zuul Routing Layer

Canary vs Baseline
Squeeze

"Coalmine"

Production
User Request

Dependency A

Dependency B

Dependency D

Dependency C

Dependency E

Dependency G

Dependency F

Dependency...
Zuul Routing Layer

Canary vs Baseline
Squeeze

"Coalmine"

Production
Zuul Routing Layer

Canary vs Baseline
Squeeze

"Coalmine"

Production
Predictive + Reactive Auto-Scaling
Scale & Resilience
!

Performance & Innovation
Netflix API

Dependency A

Dependency B

Dependency D

Dependency C

Dependency E

Dependency G

Dependency F

Dependency H...
Netflix API

Dependency A

Dependency B

Dependency D

Dependency C

Dependency E

Dependency G

Dependency F

Dependency H...
One Size Fits All RESTful API

millions of API requests per day

2000

1500

1000

500

0
2010

2011

2012

Today
One Size Fits All RESTful API

millions of API requests per day

2000

1500

1000

500

0
2010

2011

2012

Today
One Size Fits All RESTful API
1000+ Devices
millions of API requests per day

2000

1500

1000

500

0
2010

2011

2012

T...
We wanted to re-architecture our call patterns ...
... to collapse network traffic into coarse API calls ...

nested, conditional, concurrent execution
... and we wanted to allow
anybody to create
endpoints, not just the
“API Team”
Concurrency without each engineer
reading and re-reading this →
!

(awesome book ... everybody isn’t going to - or
should ...
Owner of api should retain control
of concurrency behavior.
Owner of api should retain control
of concurrency behavior.
public	
  Data	
  getData();
What if the implementation needs ...
public	
  Data	
  getData();
public	
  void	
  getData(Callback<T>	
  c);	
  
!

public	
  Future<T>	
  getData();	
  
!

...
Reactive Programming with Rx
Iterable
pull
T next()
throws Exception
returns;

Observable
push
onNext(T)
onError(Exception...
(Functional) Reactive Programming with RxJava
Iterable
pull
T next()
throws Exception
returns;

Observable
push
onNext(T)
...
(Functional) Reactive Programming with RxJava
Iterable
pull
T next()
throws Exception
returns;

Observable
push
onNext(T)
...
(Functional) Reactive Programming with RxJava
Iterable
pull
T next()
throws Exception
returns;

Observable
push
onNext(T)
...
Iterable
pull
T next()
throws Exception
returns;
!
	
  //	
  Iterable<String>	
  	
  
	
  //	
  that	
  contains	
  75	
  ...
Iterable
pull
T next()
throws Exception
returns;
!
	
  //	
  Iterable<String>	
  	
  
	
  //	
  that	
  contains	
  75	
  ...
Single

Multiple

Sync

T getData()

Iterable<T> getData()

Async

Future<T> getData()

Observable<T> getData()
Single

Multiple

Sync

T getData()

Iterable<T> getData()

Async

Future<T> getData()

Observable<T> getData()

String s ...
Single

Multiple

Sync

T getData()

Iterable<T> getData()

Async

Future<T> getData()

Observable<T> getData()

Iterable<...
Single

Multiple

Sync

T getData()

Iterable<T> getData()

Async

Future<T> getData()

Observable<T> getData()

Future<St...
Single

Multiple

Sync

T getData()

Iterable<T> getData()

Async

Future<T> getData()

Observable<T> getData()

Future<St...
Single

Multiple

Sync

T getData()

Iterable<T> getData()

Async

Future<T> getData()

Observable<T> getData()

Future<St...
Single

Multiple

Sync

T getData()

Iterable<T> getData()

Async

Future<T> getData()

Observable<T> getData()

Future<St...
Single

Multiple

Sync

T getData()

Iterable<T> getData()

Async

Future<T> getData()

Observable<T> getData()

Future<St...
Single

Multiple

Sync

T getData()

Iterable<T> getData()

Async

Future<T> getData()

Observable<T> getData()

Completab...
Single

Multiple

Sync

T getData()

Iterable<T> getData()

Async

Future<T> getData()

Observable<T> getData()

Completab...
Single

Multiple

Sync

T getData()

Iterable<T> getData()

Async

Future<T> getData()

Observable<T> getData()

Future<St...
Single

Multiple

Sync

T getData()

Iterable<T> getData()

Async

Future<T> getData()

Observable<T> getData()

Future<St...
Single

Multiple

Sync

T getData()

Iterable<T> getData()

Async

Future<T> getData()

Observable<T> getData()

Future<St...
Single

Multiple

Sync

T getData()

Iterable<T> getData()

Async

Future<T> getData()

Observable<T> getData()

Observabl...
Single

Multiple

Sync

T getData()

Iterable<T> getData()

Async

Future<T> getData()

Observable<T> getData()

Observabl...
Single

Multiple

Sync

T getData()

Iterable<T> getData()

Async

Future<T> getData()

Observable<T> getData()

Observabl...
instead of a blocking api ...
class	
  VideoService	
  {	
  
	
  	
  	
  def	
  VideoList	
  getPersonalizedListOfMovies(u...
RxJava
http://github.com/Netflix/RxJava

“a library for composing
asynchronous and event-based
programs using observable
se...
client code treats all interactions
with the api as asynchronous
!
!

the api implementation chooses
whether something is
...
Netflix API

Server

Device

Optimize for Each Device. Leverage the Server
Code
SDK for API Web Service Development
SDK for API Web Service Development
SDK for API Web Service Development
api.servletResponse.writer.print("Hello")
api.servletResponse.writer.print("Hello")

$runScript.py -e PROD sample.groovy
Hello
#####################################...
api.servletResponse.writer.print("Hello")

$runScript.py -e PROD sample.groovy
Hello
#####################################...
api.servletResponse.writer.print("Hello")

$runScript.py -e PROD sample.groovy
Hello
#####################################...
api.servletResponse.writer.print("Hello")

$runScript.py -e PROD sample.groovy
Hello
#####################################...
api.servletResponse.writer.print("Hello")

$runScript.py -e PROD sample.groovy
Hello
#####################################...
api.servletResponse.writer.print("Hello")

$runScript.py -e PROD sample.groovy
Hello
#####################################...
public class HelloEndpoint extends APIEndpoint {	
@Override	
public void execute(APIRequest api) throws Throwable {	
// ge...
public class HelloEndpoint extends APIEndpoint {	
@Override	
public void execute(APIRequest api) throws Throwable {	
// ge...
public class HelloEndpoint extends APIEndpoint {	
@Override	
public void execute(APIRequest api) throws Throwable {	
// ge...
public class HelloEndpoint extends APIEndpoint {	
@Override	
public void execute(APIRequest api) throws Throwable {	
// ge...
public class HelloEndpoint extends APIEndpoint {	
@Override	
public void execute(APIRequest api) throws Throwable {	
// ge...
public class HelloEndpoint extends APIEndpoint {	
@Override	
public void execute(APIRequest api) throws Throwable {	
// ge...
public class HelloEndpoint extends APIEndpoint {	
@Override	
public void execute(APIRequest api) throws Throwable {	
// ge...
public class HelloEndpoint extends APIEndpoint {	
@Override	
public void execute(APIRequest api) throws Throwable {	
// ge...
public Observable getVideoSummary(APIVideo video) {	
// get id, title	
def seed = [id: video.id, title : video.getTitle(AP...
public Observable getVideoSummary(APIVideo video) {	
// get id, title	
def seed = [id: video.id, title : video.getTitle(AP...
public Observable getVideoSummary(APIVideo video) {	
// get id, title	
def seed = [id: video.id, title : video.getTitle(AP...
public Observable getVideoSummary(APIVideo video) {	
// get id, title	
def seed = [id: video.id, title : video.getTitle(AP...
public Observable getVideoSummary(APIVideo video) {	
// get id, title	
def seed = [id: video.id, title : video.getTitle(AP...
public Observable getVideoSummary(APIVideo video) {	
// get id, title	
def seed = [id: video.id, title : video.getTitle(AP...
public Observable getVideoSummary(APIVideo video) {	
// get id, title	
def seed = [id: video.id, title : video.getTitle(AP...
$ uploadScript.py -e TEST /test/hello sample.groovy
{
"active": false,
"allocationPercentage": 0,
"creationDate": "Mon Nov...
$ uploadScript.py -e TEST /test/hello sample.groovy
{
"active": false,
"allocationPercentage": 0,
"creationDate": "Mon Nov...
$ uploadScript.py -e TEST /test/hello sample.groovy
{
"active": false,
"allocationPercentage": 0,
"creationDate": "Mon Nov...
$ uploadScript.py -e TEST /test/hello sample.groovy
{
"active": false,
"allocationPercentage": 0,
"creationDate": "Mon Nov...
$ uploadScript.py -e TEST /test/hello sample.groovy
{
"active": false,
"allocationPercentage": 0,
"creationDate": "Mon Nov...
$ uploadScript.py -e TEST /test/hello sample.groovy
{
"active": false,
"allocationPercentage": 0,
"creationDate": "Mon Nov...
$ uploadScript.py -e TEST /test/hello sample.groovy
{
"active": false,
"allocationPercentage": 0,
"creationDate": "Mon Nov...
$ uploadScript.py -e TEST /test/hello sample.groovy
{
"active": false,
"allocationPercentage": 0,
"creationDate": "Mon Nov...
Future
millions of API requests per day

2000

1500

1000

500

0
2008

2009

2010

2011

2012

Today
/tv/home

/android/home

/ps3/home

Functional Reactive Dynamic Endpoints

Asynchronous Java API

Dependency A
10 Threads
...
/tv/home

/android/home

/ps3/home

Hystrix

Functional Reactive Dynamic Endpoints

fault-isolation layer

Asynchronous Ja...
+
Observable<User>	
  u	
  =	
  new	
  GetUserCommand(id).observe();	
  
Observable<Geo>	
  g	
  =	
  new	
  GetGeoCommand...
Ben Christensen
@benjchristensen
http://www.linkedin.com/in/benjchristensen

jobs.netflix.com

!

Functional Reactive in th...
Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/netflixapi-evolution
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Evolution of the Netflix API
Upcoming SlideShare
Loading in...5
×

Evolution of the Netflix API

1,105

Published on

Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1bJsakO.

Ben Christensen describes how the Netflix API evolved from a typical one-size-fits-all RESTful API designed to support public developers into a web service platform optimized to handle the diversity and variability of each device and user experience. The talk will also address the challenges involving operations, deployment, performance, fault-tolerance, and rate of innovation at massive scale. Filmed at qconsf.com.

Ben Christensen is a software engineer on the Netflix Edge Services Platform team responsible for fault tolerance, performance, architecture and scale while enabling millions of customers to access the Netflix experience across more than 1,000 different device types. Prior to Netflix, Ben was at Apple making iOS apps and media available on the iTunes store.

Published in: Technology, Education
1 Comment
5 Likes
Statistics
Notes
  • download free link: https://app.box.com/s/olzwnk240vfm2ir8yfdw
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
1,105
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
0
Comments
1
Likes
5
Embeds 0
No embeds

No notes for slide

Evolution of the Netflix API

  1. 1. Evolution of the Netflix API QCon San Francisco - November 2013 Ben Christensen Software Engineer – Edge & Playback Services at Netflix @benjchristensen ! ! ! ! http://techblog.netflix.com/
  2. 2. Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations /netflix-api-evolution InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month
  3. 3. Presented at QCon San Francisco www.qconsf.com Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide
  4. 4. More than 40 million Subscribers in 50+ Countries and Territories
  5. 5. Netflix accounts for 33% of Peak Downstream Internet Traffic in North America Netflix subscribers are watching more than 1 billion hours a month
  6. 6. API traffic has grown from ~20 million/day in 2010 to >2 billion/day millions of API requests per day 2000 1500 1000 500 0 2010 2011 2012 Today
  7. 7. Discovery Streaming
  8. 8. Netflix API Streaming
  9. 9. At the start … millions of API requests per day 2000 1500 1000 500 0 2008 2009 2010 2011 2012 Today
  10. 10. 2008 API Launch Targeted 100% at External Developers Open API Netflix Devices
  11. 11. 2008 API Launch Targeted 100% at External Developers Purpose A “Thousand Flowers” of 3rd party innovation ! Audience External Developers
  12. 12. Pre-Cloud Architecture API Service Service Service Oracle Service
  13. 13. In 2011 … millions of API requests per day 2000 1500 1000 500 0 2008 2009 2010 2011 2012 Today
  14. 14. 99.9% Netflix Devices Open API Netflix Devices
  15. 15. Targeted at Internal Developers Open API Netflix Devices
  16. 16. Targeted at Internal Developers Purpose Enable Netflix Experience ! Audience Netflix Device and UI Teams
  17. 17. Scale & Resilience ! Performance & Innovation
  18. 18. Scale & Resilience ! Performance & Innovation
  19. 19. Netflix API Dependency A Dependency B Dependency D Dependency C Dependency E Dependency G Dependency F Dependency H Dependency J Dependency M Dependency P Dependency I Dependency K Dependency N Dependency Q Dependency L Dependency O Dependency R
  20. 20. Netflix API Dependency A Dependency B Dependency D Dependency C Dependency E Dependency G Dependency F Dependency H Dependency J Dependency M Dependency P Dependency I Dependency K Dependency N Dependency Q Dependency L Dependency O Dependency R
  21. 21. AWS Availability Zone AWS Availability Zone AWS Availability Zone
  22. 22. User Request Dependency A Dependency B Dependency D Dependency C Dependency E Dependency G Dependency F Dependency H Dependency J Dependency M Dependency P Dependency I Dependency K Dependency N Dependency Q Dependency L Dependency O Dependency R
  23. 23. User Request Dependency A Dependency B Dependency D Dependency C Dependency E Dependency G Dependency F Dependency H Dependency J Dependency M Dependency P Dependency I Dependency K Dependency N Dependency Q Dependency L Dependency O Dependency R User request blocked by latency in single network call
  24. 24. Dependency A Dependency B Dependency D Dependency C Dependency E Dependency G Dependency F Dependency H Dependency J Dependency M Dependency P Dependency I Dependency K Dependency N Dependency Q Dependency L Dependency O Dependency R User Request User Request User Request User Request User Request User Request User Request ............................................. At high volume all request threads can block in seconds
  25. 25. ! Dozens of dependencies. ! One going bad takes everything down. ! 30 99.99% = 99.7% uptime ! 0.3% of 1 billion = 3,000,000 failures ! 2+ hours downtime/month ! ! ! Reality is generally worse.
  26. 26. CONSTRAINTS Speed of Iteration ! Client Libraries ! Mixed Environment
  27. 27. CONSTRAINTS Speed of Iteration ! Client Libraries ! Mixed Environment
  28. 28. CONSTRAINTS Speed of Iteration ! Client Libraries ! Mixed Environment
  29. 29. CONSTRAINTS Speed of Iteration ! Client Libraries ! Mixed Environment
  30. 30. Dependency A Dependency B Dependency D Dependency C Dependency E Dependency G Dependency F Dependency H Dependency J Dependency M Dependency P Dependency I Dependency K Dependency N Dependency Q Dependency L Dependency O Dependency R User Request User Request User Request User Request User Request User Request User Request .............................................
  31. 31. User Request User Request User Request Logic - argument validation, caches, metrics, logging, multivariate testing, routing, etc Serialization - URL and/or body generation Network Request - TCP/HTTP, latency, 4xx, 5xx, etc ............................................. Deserialization - JSON/XML/Thrift/Protobuf/etc Dependency B cy D Dependency C Dependency E dency G ependency J Dependency M Logic - validation, decoration, object model, caching, metrics, logging, etc Dependency F Dependency H Dependency K Dependency N Dependency I Dependency L Dependency O
  32. 32. > 80% of requests rejected Median Latency [Sat Jun 30 04:01:37 2012] [error] proxy: HTTP: disabled connection for (127.0.0.1) "Timeout guard" daemon prio=10 tid=0x00002aaacd5e5000 nid=0x3aac runnable [0x00002aaac388f000] java.lang.Thread.State: RUNNABLE! at java.net.PlainSocketImpl.socketConnect(Native Method)! at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)! - locked <0x000000055c7e8bd8> (a java.net.SocksSocketImpl)! at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)! at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)! at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)! at java.net.Socket.connect(Socket.java:579)! at java.net.Socket.connect(Socket.java:528)! at java.net.Socket.(Socket.java:425)! at java.net.Socket.(Socket.java:280)! at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80)! at org.apache.commons.httpclient.protocol.ControllerThreadSocketFactory$1.doit(ControllerThreadSocketFactory.java:91)! at org.apache.commons.httpclient.protocol.ControllerThreadSocketFactory$SocketTask.run(ControllerThreadSocketFactory.java:158) at java.lang.Thread.run(Thread.java:722)
  33. 33. User Request Dependency A Dependency B Dependency D Dependency C Dependency E Dependency G Dependency F Dependency H Dependency J Dependency M Dependency P Dependency I Dependency K Dependency N Dependency Q Dependency L Dependency O Dependency R
  34. 34. User Request Dependency A Dependency B Dependency D Dependency C Dependency E Dependency G Dependency F Dependency H Dependency J Dependency M Dependency P Dependency I Dependency K Dependency N Dependency Q Dependency L Dependency O Dependency R
  35. 35. User Request Dependency A Dependency B Dependency D Dependency C Dependency E Dependency G Dependency F Dependency H Dependency J Dependency M Dependency P Dependency I Dependency K Dependency N Dependency Q Dependency L Dependency O Dependency R
  36. 36. User Request Dependency A Dependency B Dependency D Dependency C Dependency E Dependency G Dependency F Dependency H Dependency J Dependency M Dependency P Dependency I Dependency K Dependency N Dependency Q Dependency L Dependency O Dependency R
  37. 37. circle color and size represent health and traffic volume
  38. 38. circle color and size represent health and traffic volume 2 minutes of request rate to show relative changes in traffic
  39. 39. circle color and size represent health and traffic volume 2 minutes of request rate to show relative changes in traffic hosts reporting from cluster
  40. 40. circle color and size represent health and traffic volume 2 minutes of request rate to show relative changes in traffic hosts reporting from cluster last minute latency percentiles
  41. 41. circle color and size represent health and traffic volume 2 minutes of request rate to show relative changes in traffic hosts reporting from cluster Circuit-breaker status last minute latency percentiles
  42. 42. circle color and size represent health and traffic volume Request rate 2 minutes of request rate to show relative changes in traffic hosts reporting from cluster Circuit-breaker status last minute latency percentiles
  43. 43. circle color and size represent health and traffic volume Error percentage of last 10 seconds Request rate 2 minutes of request rate to show relative changes in traffic hosts reporting from cluster Circuit-breaker status last minute latency percentiles
  44. 44. Error percentage of last 10 seconds circle color and size represent health and traffic volume Request rate 2 minutes of request rate to show relative changes in traffic Circuit-breaker status hosts reporting from cluster last minute latency percentiles Rolling 10 second counters with 1 second granularity Successes Short-circuited (rejected) Thread timeouts Thread-pool Rejections Failures/Exceptions
  45. 45. User Request Dependency A Dependency B Dependency D Dependency C Dependency E Dependency G Dependency F Dependency H Dependency J Dependency M Dependency P Dependency I Dependency K Dependency N Dependency Q Dependency L Dependency O Dependency R
  46. 46. Zuul Routing Layer Canary vs Baseline Squeeze "Coalmine" Production
  47. 47. Zuul Routing Layer Canary vs Baseline Squeeze "Coalmine" Production
  48. 48. Zuul Routing Layer Canary vs Baseline Squeeze "Coalmine" Production
  49. 49. Zuul Routing Layer Canary vs Baseline Squeeze "Coalmine" Production
  50. 50. Zuul Routing Layer Canary vs Baseline Squeeze "Coalmine" Production
  51. 51. User Request Dependency A Dependency B Dependency D Dependency C Dependency E Dependency G Dependency F Dependency H Dependency J Dependency M Dependency P Dependency I Dependency K Dependency N Dependency Q Dependency L Dependency O Dependency R System Relationship Over Network without Bulkhead
  52. 52. Zuul Routing Layer Canary vs Baseline Squeeze "Coalmine" Production
  53. 53. Zuul Routing Layer Canary vs Baseline Squeeze "Coalmine" Production
  54. 54. Predictive + Reactive Auto-Scaling
  55. 55. Scale & Resilience ! Performance & Innovation
  56. 56. Netflix API Dependency A Dependency B Dependency D Dependency C Dependency E Dependency G Dependency F Dependency H Dependency J Dependency M Dependency P Dependency I Dependency K Dependency N Dependency Q Dependency L Dependency O Dependency R
  57. 57. Netflix API Dependency A Dependency B Dependency D Dependency C Dependency E Dependency G Dependency F Dependency H Dependency J Dependency M Dependency P Dependency I Dependency K Dependency N Dependency Q Dependency L Dependency O Dependency R
  58. 58. One Size Fits All RESTful API millions of API requests per day 2000 1500 1000 500 0 2010 2011 2012 Today
  59. 59. One Size Fits All RESTful API millions of API requests per day 2000 1500 1000 500 0 2010 2011 2012 Today
  60. 60. One Size Fits All RESTful API 1000+ Devices millions of API requests per day 2000 1500 1000 500 0 2010 2011 2012 Today
  61. 61. We wanted to re-architecture our call patterns ...
  62. 62. ... to collapse network traffic into coarse API calls ... nested, conditional, concurrent execution
  63. 63. ... and we wanted to allow anybody to create endpoints, not just the “API Team”
  64. 64. Concurrency without each engineer reading and re-reading this → ! (awesome book ... everybody isn’t going to - or should have to - read it though, that’s the point)
  65. 65. Owner of api should retain control of concurrency behavior.
  66. 66. Owner of api should retain control of concurrency behavior. public  Data  getData(); What if the implementation needs to change from synchronous to asynchronous? ! How should the client execute that method without blocking? spawn a thread?
  67. 67. public  Data  getData(); public  void  getData(Callback<T>  c);   ! public  Future<T>  getData();   ! public  Future<List<Future<T>>>  getData();   ! ! other options ... ?
  68. 68. Reactive Programming with Rx Iterable pull T next() throws Exception returns; Observable push onNext(T) onError(Exception) onCompleted()
  69. 69. (Functional) Reactive Programming with RxJava Iterable pull T next() throws Exception returns; Observable push onNext(T) onError(Exception) onCompleted()
  70. 70. (Functional) Reactive Programming with RxJava Iterable pull T next() throws Exception returns; Observable push onNext(T) onError(Exception) onCompleted()
  71. 71. (Functional) Reactive Programming with RxJava Iterable pull T next() throws Exception returns; Observable push onNext(T) onError(Exception) onCompleted()
  72. 72. Iterable pull T next() throws Exception returns; !  //  Iterable<String>      //  that  contains  75  Strings    getDataFromLocalMemory()      .skip(10)      .take(5)      .map({  s  -­‐>          return  s  +  "_transformed"})      .forEach(            {  println  "next  =>  "  +  it})   Observable push onNext(T) onError(Exception) onCompleted() !  //  Observable<String>      //  that  emits  75  Strings    getDataFromNetwork()      .skip(10)      .take(5)      .map({  s  -­‐>          return  s  +  "_transformed"})      .subscribe(            {  println  "onNext  =>  "  +  it})  
  73. 73. Iterable pull T next() throws Exception returns; !  //  Iterable<String>      //  that  contains  75  Strings    getDataFromLocalMemory()      .skip(10)      .take(5)      .map({  s  -­‐>          return  s  +  "_transformed"})      .forEach(            {  println  "onNext  =>  "  +  it})   Observable push onNext(T) onError(Exception) onCompleted() !  //  Observable<String>      //  that  emits  75  Strings    getDataFromNetwork()      .skip(10)      .take(5)      .map({  s  -­‐>          return  s  +  "_transformed"})      .subscribe(            {  println  "onNext  =>  "  +  it})  
  74. 74. Single Multiple Sync T getData() Iterable<T> getData() Async Future<T> getData() Observable<T> getData()
  75. 75. Single Multiple Sync T getData() Iterable<T> getData() Async Future<T> getData() Observable<T> getData() String s = getData(args); if (s.equals(x)) { // do something } else { // do something else }
  76. 76. Single Multiple Sync T getData() Iterable<T> getData() Async Future<T> getData() Observable<T> getData() Iterable<String> values = getData(args); for (String s : values) { if (s.equals(x)) { // do something } else { // do something else } }
  77. 77. Single Multiple Sync T getData() Iterable<T> getData() Async Future<T> getData() Observable<T> getData() Future<String> s = getData(args); if (s.get().equals(x)) { // do something } else { // do something else }
  78. 78. Single Multiple Sync T getData() Iterable<T> getData() Async Future<T> getData() Observable<T> getData() Future<String> s = getData(args); if (s.get().equals(x)) { // do something } else { // do something else }
  79. 79. Single Multiple Sync T getData() Iterable<T> getData() Async Future<T> getData() Observable<T> getData() Future<String> s = getData(args); Futures.addCallback(s, new FutureCallback<String> { public void onSuccess(String s) { if (s.equals(x)) { // do something } else { // do something else } } }, executor);
  80. 80. Single Multiple Sync T getData() Iterable<T> getData() Async Future<T> getData() Observable<T> getData() Future<String> s = getData(args); Futures.addCallback(s, new FutureCallback<String> { public void onSuccess(String s) { if (s.equals(x)) { // do something } else { // do something else } } }, executor);
  81. 81. Single Multiple Sync T getData() Iterable<T> getData() Async Future<T> getData() Observable<T> getData() Future<String> s = getData(args); Futures.addCallback(s, new FutureCallback<String> { public void onSuccess(String s) { if (s.equals(x)) { // do something } else { // do something else } } }, executor);
  82. 82. Single Multiple Sync T getData() Iterable<T> getData() Async Future<T> getData() Observable<T> getData() CompletableFuture<String> s = getData(args); s.thenApply((v) -> { if (v.equals(x)) { // do something } else { // do something else } });
  83. 83. Single Multiple Sync T getData() Iterable<T> getData() Async Future<T> getData() Observable<T> getData() CompletableFuture<String> s = getData(args); s.thenApply((v) -> { if (v.equals(x)) { // do something } else { // do something else } });
  84. 84. Single Multiple Sync T getData() Iterable<T> getData() Async Future<T> getData() Observable<T> getData() Future<String> s = getData(args); s.map({ s -> if (s.equals(x)) { // do something } else { // do something else } });
  85. 85. Single Multiple Sync T getData() Iterable<T> getData() Async Future<T> getData() Observable<T> getData() Future<String> s = getData(args); s.map({ s -> if (s.equals(x)) { // do something } else { // do something else } });
  86. 86. Single Multiple Sync T getData() Iterable<T> getData() Async Future<T> getData() Observable<T> getData() Future<String> s = getData(args); s.map({ s -> if (s.equals(x)) { // do something } else { // do something else } });
  87. 87. Single Multiple Sync T getData() Iterable<T> getData() Async Future<T> getData() Observable<T> getData() Observable<String> s = getData(args); s.map({ s -> if (s.equals(x)) { // do something } else { // do something else } });
  88. 88. Single Multiple Sync T getData() Iterable<T> getData() Async Future<T> getData() Observable<T> getData() Observable<String> s = getData(args); s.map({ s -> if (s.equals(x)) { // do something } else { // do something else } });
  89. 89. Single Multiple Sync T getData() Iterable<T> getData() Async Future<T> getData() Observable<T> getData() Observable<String> s = getData(args); s.map({ s -> if (s.equals(x)) { // do something } else { // do something else } });
  90. 90. instead of a blocking api ... class  VideoService  {        def  VideoList  getPersonalizedListOfMovies(userId);        def  VideoBookmark  getBookmark(userId,  videoId);        def  VideoRating  getRating(userId,  videoId);        def  VideoMetadata  getMetadata(videoId);   } ... create an observable api: class  VideoService  {        def  Observable<VideoList>  getPersonalizedListOfMovies(userId);        def  Observable<VideoBookmark>  getBookmark(userId,  videoId);        def  Observable<VideoRating>  getRating(userId,  videoId);        def  Observable<VideoMetadata>  getMetadata(videoId);   }
  91. 91. RxJava http://github.com/Netflix/RxJava “a library for composing asynchronous and event-based programs using observable sequences for the Java VM” A Java port of Rx (Reactive Extensions) https://rx.codeplex.com (.Net and Javascript by Microsoft)
  92. 92. client code treats all interactions with the api as asynchronous ! ! the api implementation chooses whether something is blocking or non-blocking and what resources it uses
  93. 93. Netflix API Server Device Optimize for Each Device. Leverage the Server
  94. 94. Code
  95. 95. SDK for API Web Service Development
  96. 96. SDK for API Web Service Development
  97. 97. SDK for API Web Service Development
  98. 98. api.servletResponse.writer.print("Hello")
  99. 99. api.servletResponse.writer.print("Hello") $runScript.py -e PROD sample.groovy Hello ################################################################################################ ####### Script Logger (enabled via debug=true request parameter) ################################################################################################ [0ms] INFO: User => NULL ################################################################################################ ####### Script Logger (enabled via debug=true request parameter) ################################################################################################
  100. 100. api.servletResponse.writer.print("Hello") $runScript.py -e PROD sample.groovy Hello ################################################################################################ ####### Script Logger (enabled via debug=true request parameter) ################################################################################################ [0ms] INFO: User => NULL ################################################################################################ ####### Script Logger (enabled via debug=true request parameter) ################################################################################################
  101. 101. api.servletResponse.writer.print("Hello") $runScript.py -e PROD sample.groovy Hello ################################################################################################ ####### Script Logger (enabled via debug=true request parameter) ################################################################################################ [0ms] INFO: User => NULL ################################################################################################ ####### Script Logger (enabled via debug=true request parameter) ################################################################################################
  102. 102. api.servletResponse.writer.print("Hello") $runScript.py -e PROD sample.groovy Hello ################################################################################################ ####### Script Logger (enabled via debug=true request parameter) ################################################################################################ [0ms] INFO: User => NULL ################################################################################################ ####### Script Logger (enabled via debug=true request parameter) ################################################################################################
  103. 103. api.servletResponse.writer.print("Hello") $runScript.py -e PROD sample.groovy Hello ################################################################################################ ####### Script Logger (enabled via debug=true request parameter) ################################################################################################ [0ms] INFO: User => NULL ################################################################################################ ####### Script Logger (enabled via debug=true request parameter) ################################################################################################
  104. 104. api.servletResponse.writer.print("Hello") $runScript.py -e PROD sample.groovy Hello ################################################################################################ ####### Script Logger (enabled via debug=true request parameter) ################################################################################################ [0ms] INFO: User => NULL ################################################################################################ ####### Script Logger (enabled via debug=true request parameter) ################################################################################################
  105. 105. public class HelloEndpoint extends APIEndpoint { @Override public void execute(APIRequest api) throws Throwable { // get a request parameter from servlet request String alias = api.servletRequest.getParameter("name"); ! // set content type and write something to servlet response api.servletResponse.setContentType("application/json"); api.servletResponse.writer.print( JsonUtility.toJson( [ alias : alias, name : api.user.firstName ] ) ) ! } }
  106. 106. public class HelloEndpoint extends APIEndpoint { @Override public void execute(APIRequest api) throws Throwable { // get a request parameter from servlet request String alias = api.servletRequest.getParameter("name"); ! // set content type and write something to servlet response api.servletResponse.setContentType("application/json"); api.servletResponse.writer.print( JsonUtility.toJson( [ alias : alias, name : api.user.firstName ] ) ) ! } } runScript.py -e TEST --userId 1189658154 --args "name=bob" HelloEndpoint.groovy
  107. 107. public class HelloEndpoint extends APIEndpoint { @Override public void execute(APIRequest api) throws Throwable { // get a request parameter from servlet request String alias = api.servletRequest.getParameter("name"); ! // set content type and write something to servlet response api.servletResponse.setContentType("application/json"); api.servletResponse.writer.print( JsonUtility.toJson( [ alias : alias, name : api.user.firstName ] ) ) ! } } runScript.py -e TEST --userId 1189658154 --args "name=bob" HelloEndpoint.groovy {"alias":"bob","name":"Old"}
  108. 108. public class HelloEndpoint extends APIEndpoint { @Override public void execute(APIRequest api) throws Throwable { // get a request parameter from servlet request String alias = api.servletRequest.getParameter("name"); ! // set content type and write something to servlet response api.servletResponse.setContentType("application/json"); api.servletResponse.writer.print( JsonUtility.toJson( [ alias : alias, name : api.user.firstName ] ) ) ! } } runScript.py -e TEST --userId 1189658154 --args "name=bob" HelloEndpoint.groovy {"alias":"bob","name":"Old"}
  109. 109. public class HelloEndpoint extends APIEndpoint { @Override public void execute(APIRequest api) throws Throwable { // get a request parameter from servlet request String alias = api.servletRequest.getParameter("name"); ! // set content type and write something to servlet response api.servletResponse.setContentType("application/json"); api.servletResponse.writer.print( JsonUtility.toJson( [ alias : alias, name : api.user.firstName ] ) ) ! } } runScript.py -e TEST --userId 1189658154 --args "name=bob" HelloEndpoint.groovy {"alias":"bob","name":"Old"}
  110. 110. public class HelloEndpoint extends APIEndpoint { @Override public void execute(APIRequest api) throws Throwable { // get a request parameter from servlet request String alias = api.servletRequest.getParameter("name"); ! // set content type and write something to servlet response api.servletResponse.setContentType("application/json"); api.servletResponse.writer.print( JsonUtility.toJson( [ alias : alias, name : api.user.firstName ] ) ) ! } } runScript.py -e TEST --userId 1189658154 --args "name=bob" HelloEndpoint.groovy {"alias":"bob","name":"Old"}
  111. 111. public class HelloEndpoint extends APIEndpoint { @Override public void execute(APIRequest api) throws Throwable { // get a request parameter from servlet request String alias = api.servletRequest.getParameter("name"); ! // set content type and write something to servlet response api.servletResponse.setContentType("application/json"); api.servletResponse.writer.print( JsonUtility.toJson( [ alias : alias, name : api.user.firstName ] ) ) ! } } runScript.py -e TEST --userId 1189658154 --args "name=bob" HelloEndpoint.groovy {"alias":"bob","name":"Old"}
  112. 112. public class HelloEndpoint extends APIEndpoint { @Override public void execute(APIRequest api) throws Throwable { // get a request parameter from servlet request String alias = api.servletRequest.getParameter("name"); ! // set content type and write something to servlet response api.servletResponse.setContentType("application/json"); api.servletResponse.writer.print( JsonUtility.toJson( [ alias : alias, name : api.user.firstName ] ) ) ! } } runScript.py -e TEST --userId 1189658154 --args "name=bob" HelloEndpoint.groovy {"alias":"bob","name":"Old"}
  113. 113. public Observable getVideoSummary(APIVideo video) { // get id, title def seed = [id: video.id, title : video.getTitle(APIVideo.TitleType.REGULAR)] ! // get bookmark def bookmarkObservable = getBookmark(video) ! // get artwork def artworkObservable = getArtworkImageUrl(video) ! // merge them and accumulate into the seed. return Observable.merge(bookmarkObservable, artworkObservable) .reduce(seed, { aggregate, current -> aggregate << current}) .map({ [ (video.id.toString()) : it]}) }
  114. 114. public Observable getVideoSummary(APIVideo video) { // get id, title def seed = [id: video.id, title : video.getTitle(APIVideo.TitleType.REGULAR)] ! // get bookmark def bookmarkObservable = getBookmark(video) ! // get artwork def artworkObservable = getArtworkImageUrl(video) ! // merge them and accumulate into the seed. return Observable.merge(bookmarkObservable, artworkObservable) .reduce(seed, { aggregate, current -> aggregate << current}) .map({ [ (video.id.toString()) : it]}) }
  115. 115. public Observable getVideoSummary(APIVideo video) { // get id, title def seed = [id: video.id, title : video.getTitle(APIVideo.TitleType.REGULAR)] ! // get bookmark def bookmarkObservable = getBookmark(video) ! // get artwork def artworkObservable = getArtworkImageUrl(video) ! // merge them and accumulate into the seed. return Observable.merge(bookmarkObservable, artworkObservable) .reduce(seed, { aggregate, current -> aggregate << current}) .map({ [ (video.id.toString()) : it]}) }
  116. 116. public Observable getVideoSummary(APIVideo video) { // get id, title def seed = [id: video.id, title : video.getTitle(APIVideo.TitleType.REGULAR)] ! // get bookmark def bookmarkObservable = getBookmark(video) ! // get artwork def artworkObservable = getArtworkImageUrl(video) ! // merge them and accumulate into the seed. return Observable.merge(bookmarkObservable, artworkObservable) .reduce(seed, { aggregate, current -> aggregate << current}) .map({ [ (video.id.toString()) : it]}) }
  117. 117. public Observable getVideoSummary(APIVideo video) { // get id, title def seed = [id: video.id, title : video.getTitle(APIVideo.TitleType.REGULAR)] ! // get bookmark def bookmarkObservable = getBookmark(video) ! // get artwork def artworkObservable = getArtworkImageUrl(video) ! // merge them and accumulate into the seed. return Observable.merge(bookmarkObservable, artworkObservable) .reduce(seed, { aggregate, current -> aggregate << current}) .map({ [ (video.id.toString()) : it]}) }
  118. 118. public Observable getVideoSummary(APIVideo video) { // get id, title def seed = [id: video.id, title : video.getTitle(APIVideo.TitleType.REGULAR)] ! // get bookmark def bookmarkObservable = getBookmark(video) ! // get artwork def artworkObservable = getArtworkImageUrl(video) ! // merge them and accumulate into the seed. return Observable.merge(bookmarkObservable, artworkObservable) .reduce(seed, { aggregate, current -> aggregate << current}) .map({ [ (video.id.toString()) : it]}) }
  119. 119. public Observable getVideoSummary(APIVideo video) { // get id, title def seed = [id: video.id, title : video.getTitle(APIVideo.TitleType.REGULAR)] ! // get bookmark def bookmarkObservable = getBookmark(video) ! // get artwork def artworkObservable = getArtworkImageUrl(video) ! // merge them and accumulate into the seed. return Observable.merge(bookmarkObservable, artworkObservable) .reduce(seed, { aggregate, current -> aggregate << current}) .map({ [ (video.id.toString()) : it]}) }
  120. 120. $ uploadScript.py -e TEST /test/hello sample.groovy { "active": false, "allocationPercentage": 0, "creationDate": "Mon Nov 11 05:38:51 UTC 2013", "revision": 1, "userAuthorizationRequired": true, "userAuthorizationType": "https" }
  121. 121. $ uploadScript.py -e TEST /test/hello sample.groovy { "active": false, "allocationPercentage": 0, "creationDate": "Mon Nov 11 05:38:51 UTC 2013", "revision": 1, "userAuthorizationRequired": true, "userAuthorizationType": "https" }
  122. 122. $ uploadScript.py -e TEST /test/hello sample.groovy { "active": false, "allocationPercentage": 0, "creationDate": "Mon Nov 11 05:40:26 UTC 2013", "revision": 2, "userAuthorizationRequired": true, "userAuthorizationType": "https" }
  123. 123. $ uploadScript.py -e TEST /test/hello sample.groovy { "active": false, "allocationPercentage": 0, "creationDate": "Mon Nov 11 05:40:26 UTC 2013", "revision": 2, "userAuthorizationRequired": true, "userAuthorizationType": "https" } $ activateScript.py -e TEST --revision 2 /test/hello { "active": true, "allocationPercentage": 100, "creationDate": "Mon Nov 11 05:40:26 UTC 2013", "previousRevision": null, "revision": 2, "userAuthorizationRequired": true, "userAuthorizationType": "https" }
  124. 124. $ uploadScript.py -e TEST /test/hello sample.groovy { "active": false, "allocationPercentage": 0, "creationDate": "Mon Nov 11 05:40:26 UTC 2013", "revision": 2, "userAuthorizationRequired": true, "userAuthorizationType": "https" } $ activateScript.py -e TEST --revision 2 /test/hello { "active": true, "allocationPercentage": 100, "creationDate": "Mon Nov 11 05:40:26 UTC 2013", "previousRevision": null, "revision": 2, "userAuthorizationRequired": true, "userAuthorizationType": "https" }
  125. 125. $ uploadScript.py -e TEST /test/hello sample.groovy { "active": false, "allocationPercentage": 0, "creationDate": "Mon Nov 11 05:40:26 UTC 2013", "revision": 2, "userAuthorizationRequired": true, "userAuthorizationType": "https" } $ activateScript.py -e TEST --revision 2 /test/hello { "active": true, "allocationPercentage": 100, "creationDate": "Mon Nov 11 05:40:26 UTC 2013", "previousRevision": null, "revision": 2, "userAuthorizationRequired": true, "userAuthorizationType": "https" }
  126. 126. $ uploadScript.py -e TEST /test/hello sample.groovy { "active": false, "allocationPercentage": 0, "creationDate": "Mon Nov 11 05:40:26 UTC 2013", "revision": 2, "userAuthorizationRequired": true, "userAuthorizationType": "https" } $ activateScript.py -e TEST --revision 2 /test/hello { "active": true, "allocationPercentage": 100, "creationDate": "Mon Nov 11 05:40:26 UTC 2013", "previousRevision": null, "revision": 2, "userAuthorizationRequired": true, "userAuthorizationType": "https" }
  127. 127. $ uploadScript.py -e TEST /test/hello sample.groovy { "active": false, "allocationPercentage": 0, "creationDate": "Mon Nov 11 05:40:26 UTC 2013", "revision": 2, "userAuthorizationRequired": true, "userAuthorizationType": "https" } $ activateScript.py -e TEST --revision 3 /test/hello { "active": true, "allocationPercentage": 100, "creationDate": "Mon Nov 11 05:42:05 UTC 2013", "previousRevision": 2, "revision": 3, "userAuthorizationRequired": true, "userAuthorizationType": "https" }
  128. 128. Future
  129. 129. millions of API requests per day 2000 1500 1000 500 0 2008 2009 2010 2011 2012 Today
  130. 130. /tv/home /android/home /ps3/home Functional Reactive Dynamic Endpoints Asynchronous Java API Dependency A 10 Threads Dependency F 10 Threads Dependency K 15 Threads Dependency P 10 Threads Dependency B 8 Threads Dependency G 10 Threads Dependency L 4 Threads Dependency Q 8 Threads Dependency C 10 Threads Dependency H 10 Threads Dependency M 5 Threads Dependency R 10 Threads Dependency D 15 Threads Dependency I 5 Threads Dependency N 10 Threads Dependency S 8 Threads Dependency E 5 Threads Dependency J 8 Threads Dependency O 10 Threads Dependency T 10 Threads
  131. 131. /tv/home /android/home /ps3/home Hystrix Functional Reactive Dynamic Endpoints fault-isolation layer Asynchronous Java API Dependency A 10 Threads Dependency F 10 Threads Dependency K 15 Threads Dependency P 10 Threads Dependency B 8 Threads Dependency G 10 Threads Dependency L 4 Threads Dependency Q 8 Threads Dependency C 10 Threads Dependency H 10 Threads Dependency M 5 Threads Dependency R 10 Threads Dependency D 15 Threads Dependency I 5 Threads Dependency N 10 Threads Dependency S 8 Threads Dependency E 5 Threads Dependency J 8 Threads Dependency O 10 Threads Dependency T 10 Threads
  132. 132. + Observable<User>  u  =  new  GetUserCommand(id).observe();   Observable<Geo>  g  =  new  GetGeoCommand(request).observe();   ! Observable.zip(u,  g,  {user,  geo  -­‐>                    return  [username:  user.getUsername(),                                    currentLocation:  geo.getCounty()]         }) RxJava in Hystrix 1.3+ https://github.com/Netflix/Hystrix
  133. 133. Ben Christensen @benjchristensen http://www.linkedin.com/in/benjchristensen jobs.netflix.com ! Functional Reactive in the Netflix API with RxJava http://techblog.netflix.com/2013/02/rxjava-netflix-api.html ! Optimizing the Netflix API http://techblog.netflix.com/2013/01/optimizing-netflix-api.html ! Application Resilience in a Service-oriented Architecture http://programming.oreilly.com/2013/06/application-resilience-in-a-service-oriented-architecture.html ! Fault Tolerance in a High Volume, Distributed System http://techblog.netflix.com/2012/02/fault-tolerance-in-high-volume.html ! Making the Netflix API More Resilient http://techblog.netflix.com/2011/12/making-netflix-api-more-resilient.html
  134. 134. Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations/netflixapi-evolution

×