Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Netflix API - Presentation to PayPal

30,213 views

Published on

I gave this presentation to the engineering team at PayPal. This presentation discusses the history and future of the Netflix API. It also goes into API design principles as well as concepts behind system scalability and resiliency.

Published in: Technology, Business

Netflix API - Presentation to PayPal

  1. API Strategy : Know Your Audience Daniel Jacobson Director of Engineering, Netflix API djacobson@netflix.com @daniel_jacobsonhttp://www.linkedin.com/in/danieljacobson http://www.slideshare.net/danieljacobson
  2. Quick NoteHere is the basic outline for this presenation. Feel free tobounce to the sections that are most interesting toyou, especially if you have seen some of this before:• Introduction (slides 3 - 9)• Netflix API background (10 – 23)• Netflix Product Engineering Organization (24 – 27)• Netflix API Resiliency (28 – 52)• Netflix API Scaling (53 – 66)• Supporting Innovation and API Design Principles (67 – 100)• Netflix API Team is Hiring – Contact Information (101)
  3. Who Am I?• Director of Engineering, Netflix API• Previously: Director of App Dev, NPR – Responsible for and built custom CMS and API among other things• Co-Author of O’Reilly’s “APIs: A Strategy Guide”
  4. Who is Netflix?• Focus is to become the best at streaming TV shows and movies internationally• Netflix is responsible for 30+% of US internet traffic at peak times• Netflix streaming can be enjoyed on more than 800 different device types
  5. Key to API Strategy…
  6. Know your audience!
  7. Understanding Your Target Audience(s) should be the #1Influence to API Design Decisions
  8. The Netflix API - Background
  9. Netflix API More than 80,000 flowers
  10. Netflix API Requests by Audience At Launch In 2008 Netflix Devices Open API Developers
  11. NetflixMore than 800 APIunique device types More than 80,000 flowers
  12. Netflix API Requests by Audience From 2011 Netflix Devices Open API Developers
  13. Current Emphasis of Netflix API Netflix Devices
  14. Public API Internal API
  15. API Focus:Build the Best Global Streaming Product Two aspects of the Streaming Product: • Discovery • Streaming
  16. Discovery
  17. Discovery
  18. Streaming
  19. Netflix API Powers Discovery on Virtually All 800+ Devices
  20. 800+ Device Types
  21. Dozens of DependenciesPersonaliz Movie Movie Similar A/B Test ation User Info Reviews Engine Metadata Ratings Movies Engine
  22. APIPersonaliz Movie Movie Similar A/B Test ation User Info Reviews Engine Metadata Ratings Movies Engine
  23. Netflix API Goals for Internal Audience• Resiliency in a Distributed Environment• Scaling with the Demands of the System• Support Device and UI Innovation
  24. Netflix API Goals for Internal Audience• Resiliency in a Distributed Environment• Scaling with the Demands of the System• Support Device and UI Innovation
  25. APIPersonaliz Movie Movie Similar A/B Test ation User Info Reviews Engine Metadata Ratings Movies Engine
  26. APIPersonaliz Movie Movie Similar A/B Test ation User Info Reviews Engine Metadata Ratings Movies Engine
  27. APIPersonaliz Movie Movie Similar A/B Test ation User Info Reviews Engine Metadata Ratings Movies Engine
  28. APIPersonaliz Movie Movie Similar A/B Test ation User Info Reviews Engine Metadata Ratings Movies Engine
  29. Circuit Breaker Dashboard
  30. Call Volume and Health / Last 10 Seconds
  31. Call Volume / Last 2 Minutes
  32. Successful Requests
  33. Successful, But Slower Than Expected
  34. Short-Circuited Requests, Delivering Fallbacks
  35. Timeouts, Delivering Fallbacks
  36. Thread Pool & Task Queue Full, Delivering Fallbacks
  37. Exceptions, Delivering Fallbacks
  38. # + # + # + # / (# + # + # + # + #) = Error Rate Error Rate
  39. Status of Fallback Circuit
  40. Requests per Second, Over Last 10 Seconds
  41. SLA Information
  42. APIPersonaliz Movie Movie Similar A/B Test ation User Info Reviews Engine Metadata Ratings Movies Engine
  43. APIPersonaliz Movie Movie Similar A/B Test ation User Info Reviews Engine Metadata Ratings Movies Engine
  44. APIPersonaliz Movie Movie Similar A/B Test ation User Info Reviews Engine Metadata Ratings Movies Engine
  45. API FallbackPersonaliz Movie Movie Similar A/B Test ation User Info Reviews Engine Metadata Ratings Movies Engine
  46. API FallbackPersonaliz Movie Movie Similar A/B Test ation User Info Reviews Engine Metadata Ratings Movies Engine
  47. Netflix API Goals for Internal Audience• Resiliency in a Distributed Environment• Scaling with the Demands of the System• Support Device and UI Innovation
  48. 800+ Device Types
  49. Discovery
  50. Growth of Netflix API Requests 45 41.7 40 70x growth in two years! 35Request in Billions 30 25 20.7 20 15 10 5 0.6 - Jan-10 Jan-11 Jan-12
  51. APIPersonaliz Movie Movie Similar A/B Test ation User Info Reviews Engine Metadata Ratings Movies Engine Each API request translates into 5-6 dependency calls on average That is about 8-9 billion outgoing calls per day for the API
  52. AWS Cloud
  53. Autoscaling
  54. Autoscaling
  55. Netflix API : Requests Per Month 35 30 25Requests in Billions 20 15 10 5 -
  56. Netflix API : Requests Per Month 35 30 25Requests in Billions 20 15 10 5 -
  57. Netflix API Goals for Internal Audience• Resiliency in a Distributed Environment• Scaling with the Demands of the System• Support Device and UI Innovation
  58. Netflix device teams are constantly running multivariate tests andexperimenting with new UIs
  59. Netflix API must supportmetadata needs for innovation on all devices and UIs
  60. So, what does this mean for API design? And what does this mean for themore popular debates in API design?
  61. Debate: XML vs. JSON
  62. Debate: XML vs. JSON
  63. XML vs. JSON - Who Cares?!?!
  64. Debate: XML vs. JSON• This debate is over-simplified! Increased variability in device capabilities and requirements means increased approaches to interface with them – Some devices perform better with hierarchical JSON and others with flat object models – Different devices may require different XML schemas – Some devices prefer full document delivery and others prefer streaming bits – Etc.
  65. Debate: XML vs. JSON• Moreover, this should be a non-issue from an architectural perspective – Design the API to separate out the concepts of data gathering from data formatting and delivery – Rendering new output formats should be easy – This will help tremendously with device proliferation
  66. Debate: OAuth vs. Other• Oauth is the de facto standard – Excellent for unknown developers – Somewhat difficult to deal with – Chatty – (Netflix’s original approach)• Cookie-based auth – Excellent for device implementations under your control – Significantly less chatty and complicated – Not a good option for unknown developers – (Netflix’s new approach)• Query-based auth – Easy to implement – Easy to use – Not secure• Etc…
  67. OAuth vs. Other - Who Cares?!?!
  68. Debate: REST vs. SOAP
  69. REST vs. SOAP - Who Cares?!?!
  70. Debate: Versioning• Directory structure for version number – developer.paypal.com/v1/getPaymentOptions• Query string for version number – developer.paypal.com/getPaymentOptions?version=1• Versionless API
  71. Who Cares?!?!
  72. Well, actually, I care about this one
  73. Average Life of a TV : About 7-10 Years
  74. Versioning for APIs 1.0 1.5 2.0 3.0? 4.0? 5.0? Today2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020
  75. Eliminate Versioning? 1.0 1.5 2.0 New API Today2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020
  76. Benefit to Thinking Versionless• If you can achieve it, maintenance will be MUCH simpler• If you cannot, it instills better practices – Reduces lazy programming – Results in fewer versions – Results in a cleaner, less brittle system• And keep in mind, adding new features typically does not require a new version… – Schematic or structural changes, however, do
  77. So, why don’t I care about most of these debates?
  78. Traditional REST-ful ModelREST-ful REST-ful REST-ful REST-ful REST-fulEndpoint Endpoint Endpoint Endpoint Endpoint API Application Data Repository
  79. Comparing just these two devices…• XBox has Kinect features like voice commands and gestures (iPhone does not)• iPhone has touch features (XBox does not)• iPhone has small real estate, XBox has a full sized TV – different metadata needs• XBox has more powerful hard drive with better memory capacity than the iPhone• iPhone has different connection rates, XBox is connected to stable home wi-fi• Etc…
  80. Growth of Netflix API Requests 45 41.7 40 70x growth in two years! 35Request in Billions 30 25 20.7 20 15 10 5 0.6 - Jan-10 Jan-11 Jan-12
  81. What if the API request growth rate looks like this??? 160 140 Is this good for the long run??? 120Request is Billions 100 80 60 40 20 -
  82. Growth of the Netflix API 1.5 billion requests per day Exploding out to 8-9 billion outgoing calls per day
  83. <catalog_titles> <number_of_results>1140</number_of_results> <start_index>0</start_index> <results_per_page>10</results_per_page> <catalog_title> <id>http://api.netflix.com/catalog/titles/movies/60021896</id><title short="Star" regular="Star"></title> <box_art small="http://alien2.netflix.com/us/boxshots/tiny/60021896.jpg" medium="http://alien2.netflix.com/us/boxshots/small/60021896.jpg" large="http://alien2.netflix.com/us/boxshots/large/60021896.jpg"></box_art> <link href="http://api.netflix.com/catalog/titles/movies/60021896/synopsis" rel="http://schemas.netflix.com/catalog/titles/synopsis" title="synopsis"></link> <release_year>2001</release_year> <category scheme="http://api.netflix.com/catalog/titles/mpaa_ratings" label="NR"></category> <category scheme="http://api.netflix.com/categories/genres" label="Foreign"></category> <link href="http://api.netflix.com/catalog/titles/movies/60021896/cast" rel="http://schemas.netflix.com/catalog/people.cast" title="cast"></link><link href="http://api.netflix.com/catalog/titles/movies/60021896/screen_formats" rel="http://schemas.netflix.com/catalog/titles/screen_formats" title="screenformats"></link <link href="http://api.netflix.com/catalog/titles/movies/60021896/languages_and_audio" rel="http://schemas.netflix.com/catalog/titles/languages_and_audio"title="languages and audio"></link> <average_rating>1.9</average_rating> <link href="http://api.netflix.com/catalog/titles/movies/60021896/similars" rel="http://schemas.netflix.com/catalog/titles.similars" title="similars"></link> <link href="http://www.netflix.com/Movie/Star/60021896" rel="alternate" title="webpage"></link> </catalog_title> <catalog_title> <id>http://api.netflix.com/catalog/titles/movies/17985448</id><title short="Lone Star" regular="Lone Star"></title> <box_art small="http://alien2.netflix.com/us/boxshots/tiny/17985448.jpg" medium="http://alien2.netflix.com/us/boxshots/small/17985448.jpg" large=""></box_art> <link href="http://api.netflix.com/catalog/titles/movies/17985448/synopsis" rel="http://schemas.netflix.com/catalog/titles/synopsis" title="synopsis"></link> <release_year>1996</release_year> <category scheme="http://api.netflix.com/catalog/titles/mpaa_ratings" label="R"></category> <category scheme="http://api.netflix.com/categories/genres" label="Drama"></category><link href="http://api.netflix.com/catalog/titles/movies/17985448/awards" rel="http://schemas.netflix.com/catalog/titles/awards" title="awards"></link> <link href="http://api.netflix.com/catalog/titles/movies/17985448/format_availability" rel="http://schemas.netflix.com/catalog/titles/format_availability"title="formats"></link> <link href="http://api.netflix.com/catalog/titles/movies/17985448/screen_formats" rel="http://schemas.netflix.com/catalog/titles/screen_formats" title="screenformats"></link> <link href="http://api.netflix.com/catalog/titles/movies/17985448/languages_and_audio" rel="http://schemas.netflix.com/catalog/titles/languages_and_audio"title="languages and audio"></link> <average_rating>3.7</average_rating> <link href="http://api.netflix.com/catalog/titles/movies/17985448/previews" rel="http://schemas.netflix.com/catalog/titles/previews" title="previews"></link> <link href="http://api.netflix.com/catalog/titles/movies/17985448/similars" rel="http://schemas.netflix.com/catalog/titles.similars" title="similars"></link> <link href="http://www.netflix.com/Movie/Lone_Star/17985448" rel="alternate" title="webpage"></link> </catalog_title></catalog_titles>
  84. {"catalog_title":{"id":"http://api.netflix.com/catalog/titles/movies/60034967","title":{"title_short":"Rosencrantz and Guildenstern Are Dead","regular":"Rosencrantz and Guildenstern Are Dead"},"maturity_level":60,"release_year":"1990","average_rating":3.7,"box_art":{"284pix_w":"http://cdn-7.nflximg.com/en_US/boxshots/ghd/60034967.jpg","110pix_w":"http://cdn-7.nflximg.com/en_US/boxshots/large/60034967.jpg","38pix_w":"http://cdn-7.nflximg.com/en_US/boxshots/tiny/60034967.jpg","64pix_w":"http://cdn-7.nflximg.com/en_US/boxshots/small/60034967.jpg","150pix_w":"http://cdn-7.nflximg.com/en_US/boxshots/150/60034967.jpg","88pix_w":"http://cdn-7.nflximg.com/en_US/boxshots/88/60034967.jpg","124pix_w":"http://cdn-7.nflximg.com/en_US/boxshots/124/60034967.jpg"},"language":"en","web_page":"http://www.netflix.com/Movie/Rosencrantz_and_Guildenstern_Are_Dead/60034967","tiny_url":"http://movi.es/ApUP9"},"meta":{"expand":["@directors","@bonus_materials","@cast","@awards","@short_synopsis","@synopsis","@box_art","@screen_formats","@"links":{"id":"http://api.netflix.com/catalog/titles/movies/60034967","languages_and_audio":"http://api.netflix.com/catalog/titles/movies/60034967/languages_and_audio","title":"http://api.netflix.com/catalog/titles/movies/60034967/title","screen_formats":"http://api.netflix.com/catalog/titles/movies/60034967/screen_formats","cast":"http://api.netflix.com/catalog/titles/movies/60034967/cast","awards":"http://api.netflix.com/catalog/titles/movies/60034967/awards","short_synopsis":"http://api.netflix.com/catalog/titles/movies/60034967/short_synopsis","box_art":"http://api.netflix.com/catalog/titles/movies/60034967/box_art","synopsis":"http://api.netflix.com/catalog/titles/movies/60034967/synopsis","directors":"http://api.netflix.com/catalog/titles/movies/60034967/directors","similars":"http://api.netflix.com/catalog/titles/movies/60034967/similars","format_availability":"http://api.netflix.com/catalog/titles/movies/60034967/format_availability"}}}
  85. Improve Efficiency of API RequestsCould it have been 300 million requests per day? Or less? (Assuming everything else remained the same)
  86. There is a better tool for the job than the One-Size-Fits-All REST API
  87. SQL-Like APIs+ Much more flexibility than traditional RESTmodels+ Allows for a one-size-fits-all dev approach for API- Still has the server teams dictating the rules (although the rules are much more flexible)----------------------------------------------------------------= Substantially better for the device world= May hit scaling problems for increasing numberof device types (due to the increasing variabilityacross devices)
  88. We Are Hiring!If you are interested in helping us solvethese problems, you can contact me at: djacobson@netflix.com @daniel_jacobson http://www.linkedin.com/in/danieljacobson http://www.slideshare.net/danieljacobson

×