Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Top 10 Lessons Learned from the Netflix API - OSCON 2014

23,725 views

Published on

Published in: Technology
  • Great presentation! Show real life experience and I hope many theoretical architects will read it
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Top 10 Lessons Learned from the Netflix API - OSCON 2014

  1. 1. Daniel Jacobson • @daniel_jacobson • http://www.linkedin.com/in/danieljacobson • http://www.slideshare.net/danieljacobson Sangeeta Narayanan • @sangeetan • http://www.linkedin.com/in/sangeetanarayanan/
  2. 2. I have added more detail in the notes field for each slide to provide additional context
  3. 3. Strategy Lessons Implementation Lessons
  4. 4. Know Your Audience Target Audience Dictates Everything Else
  5. 5. The target audience should be the single biggest influence on your API design
  6. 6. Small Set of Known Developers SSKDs
  7. 7. Large Set of Unknown Developers LSUDs
  8. 8. Both SSKDs and LSUDs
  9. 9. No matter what… Figure this out first!
  10. 10. Target Audience Influence • Team Identity • Staffing Decisions • System Architecture • SLAs • Development Velocity • Security Needs
  11. 11. Netflix API : Key Responsibilities 2008 • Broker data between internal services and public developers • Grow community of public developers • Optimize design for reusability
  12. 12. Evangelists Partner Engagement and Support API Engineers Technical Writer QA Specialists
  13. 13. Private API Public API < 0.3% of total API traffic * * 11 years worth of public API requests = one day of private API requests
  14. 14. Netflix API : Key Responsibilities Today • Broker data between services and devices • System resiliency • Scaling the system • High velocity development • Insights
  15. 15. The consumers of the API are now Netflix subscribers We are now responsible for ensuring subscribers can stream
  16. 16. Application Engineers Platform Engineers Technical Writer Tools and Automation Engineers
  17. 17. Team is now 6x its size from 2010
  18. 18. Separation of Concerns
  19. 19. Primary Responsibilities of APIs • Data Gathering – Retrieving the requested data from one or many local or remote data sources • Data Formatting – Preparing a structured payload to the requesting agent • Data Delivery – Delivering the structured payload to the requesting agent
  20. 20. There are two players in APIs
  21. 21. API Provider API Consumer
  22. 22. API Provider PROVIDES API Consumer CONSUMES Traditional API Interactions
  23. 23. API Provider PROVIDES EVERYTHING API Consumer CONSUMES Everything means, API Provider does: • Data Gathering • Data Formatting • Data Delivery • (among other things) Traditional API Interactions
  24. 24. Why do most API providers provide everything? • Many APIs have a large set of unknown and external developers • Generic API design tends to be easier for teams closer to the source • Centralized API functions makes them easier to support
  25. 25. Why do most API providers provide everything? • Many APIs have a large set of unknown and external developers • Generic API design tends to be easier for teams closer to the source • Centralized API functions makes them easier to support
  26. 26. Data Gathering Data Formatting Data Delivery API Consumer Don’t care how data is gathered, as long as it is gathered Each consumer cares a lot about the format for that specific use Each consumer cares a lot about how payload is delivered API Provider Care a lot about how the data is gathered Only cares about the format to the extent it is easy to support Only cares that the delivery method is easy to support Separation of Concerns To be a better provider, the API should address the separation of concerns of the three core functions
  27. 27. One Size Doesn’t Fit All Embrace the Differences
  28. 28. Data Gathering Data Formatting Data Delivery API Consumer Don’t care how data is gathered, as long as it is gathered Each consumer cares a lot about the format for that specific use Each consumer cares a lot about how payload is delivered API Provider Care a lot about how the data is gathered Only cares about the format to the extent it is easy to support Only cares that the delivery method is easy to support Separation of Concerns
  29. 29. Screen Real Estate Controllers Technical Capabilities
  30. 30. Resource-Based API vs. Experience-Based API
  31. 31. Resource-Based Requests • /users/<id>/ratings/title • /users/<id>/queues • /users/<id>/queues/instant • /users/<id>/recommendations • /catalog/titles/movie • /catalog/titles/series • /catalog/people
  32. 32. REST API RECOMME NDATIONS MOVIE DATA SIMILAR MOVIES AUTH MEMBER DATA A/B TESTS START- UP RATINGS Network Border Network Border
  33. 33. Experience-Based Requests • /ps3/homescreen
  34. 34. JAVA API Network Border Network Border RECOMME NDATIONS MOVIE DATA SIMILAR MOVIES AUTH MEMBER DATA A/B TESTS START- UP RATINGS Client Adapter Code
  35. 35. Be Pragmatic, Not Dogmatic
  36. 36. Common API Debates • XML / JSON • REST / SOAP • OAuth / Other • Versioning • Hypermedia
  37. 37. Who Cares!?!?
  38. 38. Just Solve Problems for your Audience
  39. 39. Embrace Change Impermanence and Versionless APIs
  40. 40. v1.0 v1.5 v2.0
  41. 41. Versioning for APIs 1.0 1.5 2.0 3.0? 4.0? 5.0? 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020
  42. 42. Eliminate Versioning? 1.0 1.5 2.0 New Architecture 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020
  43. 43. JAVA API Network Border Network Border RECOMME NDATIONS MOVIE DATA SIMILAR MOVIES AUTH MEMBER DATA A/B TESTS START- UP RATINGS
  44. 44. Act Fast, React Fast Favor Velocity Over Completeness
  45. 45. Delivery Using Buckets
  46. 46. Testing
  47. 47. Production Traffic Old Code (Baseline) New Code (Canary) ~1% Traffic
  48. 48. Deployments
  49. 49. Old Code New Code Production Traffic
  50. 50. Enable Others to Act Fast, React Fast
  51. 51. JAVA API Network Border Network Border RECOMME NDATIONS MOVIE DATA SIMILAR MOVIES AUTH MEMBER DATA A/B TESTS START- UP RATINGS
  52. 52. Dynamically deployed endpoints
  53. 53. Statically deployed libraries Dynamically deployed endpoints
  54. 54. Dependency Canaries
  55. 55. Personalization Service API • Build • Test • Deploy Service • Release Lib Pers. Lib • Integrate Lib • Build • Test • Deploy Service UI Script Iterations in Hours or Days Access Data
  56. 56. Personalization Service API • Build • Test • Deploy Service • Release Lib • Publish to API Pers. Lib UI Script Iterations in Minutes? Access Data • Integrate Lib • Build • Test • Deploy Service
  57. 57. Internal Developers Need Engagement Too
  58. 58. Documentation
  59. 59. Tools REPL:
  60. 60. Trainings
  61. 61. Failure is Inevitable
  62. 62. ~5,000,000,000 Requests per day ~35 Dependencies ~600 Libraries
  63. 63. Things will break!
  64. 64. Scale at All Costs
  65. 65. - 10 20 30 40 50 60 June, 2010 June, 2011 June, 2012 RequestsinBillions API Requests Per Month
  66. 66. Incoming Traffic
  67. 67. Predictive Auto Scaling
  68. 68. Predicted vs. Actual RPS
  69. 69. Reactive + Predictive Autoscaling No. of instances
  70. 70. 1. Know Your Audience 2. Separation of Concerns 3. One Size Doesn’t Fit All 4. Be Pragmatic, Not Dogmatic 5. Embrace Change 1. Act Fast, React Fast 2. Enable Others to Act Fast, React Fast 3. Internal Developers Need Engagement Too 4. Failure is Inevitable 5. Scale at All Costs
  71. 71. http://github.com/Netflix
  72. 72. Daniel Jacobson • @daniel_jacobson • http://www.linkedin.com/in/danieljacobson • http://www.slideshare.net/danieljacobson Sangeeta Narayanan • @sangeetan • http://www.linkedin.com/in/sangeetanarayanan/

×