Your SlideShare is downloading. ×
0
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

OSCON 2014 - API Ecosystem with Scala, Scalatra, and Swagger at Netflix

508

Published on

In this talk I’d like to introduce the Scala-based API stack at Partner Innovation Group at Netflix. After seeing a massive growth in the business model and the device ecosystem, we needed a system …

In this talk I’d like to introduce the Scala-based API stack at Partner Innovation Group at Netflix. After seeing a massive growth in the business model and the device ecosystem, we needed a system that could scale and be flexible at the same time. Scala provided the answer and we started with a basic set of APIs which, since then, has evolved towards complex but flexible business flows. Supporting metadata for over hundreds of brands and thousands of devices, the API development has followed a well thought-out, test-driven approach, git-flow, and what most API developers dread – documentation. I will talk about the architecture of the RESTful APIs, and the development + deployment process. We use Netflix-OSS components heavily in the architecture and cloud deployment, so I will cover them as well.

Swagger is what we used for type-safe documentation, which is really easy to use and integrate. I will briefly talk about customizations we’ve done to Swagger in order to make it far more usable at Netflix.

Throughout this effort there were lessons to be learnt, and plenty of best practices and recommendations for anyone starting out to build RESTful APIs, regardless of the platform or stack of choice. It’d be a great opportunity for me to walk through the architecture, and talk about the various components, technologies, and practices that are seeing increasing adoption in the modern, API driven landscape.

Published in: Technology
1 Comment
5 Likes
Statistics
Notes
No Downloads
Views
Total Views
508
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
12
Comments
1
Likes
5
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  1. API ecosystem with Scala, Scalatra, and Swagger at Netflix Manish Pandit @lobster1234
  2. Manish Pandit Engineering Manager, Streaming Platforms Netflix @lobster1234
  3. Thank you, OSCON!
  4. 50 M+ Streaming Members 40 countries and counting Thousands of device types
  5. Power the Consumer Electronics Partner Portal Enable Certification of Netflix Ready Devices Source of truth for all Device Data at Netflix Correlate Streaming Quality Metrics Our APIs
  6. Devices Smart devices + Certification = Lots of Device Metadata!  
  7. Model Firmware Screen Resolution Subtitle Support 3D DRM Remote Control Netflix SDK 4K HD …   Everything matters..
  8. 58 Resources ~10 methods per resource
  9. Background Netflix OSS Components Development Deployment/Delivery Open Floor Architecture HTTP  Layer,  and  Manager  Layer   Cassandra EVCache   Crowd/SSO RDS Astyanax Netflix OSS Cloud Components
  10. Manager Layer = Business Code Protocol Independent Package-able as an artifact
  11. HTTP Layer = Protocol Wrapper Map HTTP methods to Manager functions Transform JSON <=> Case Classes
  12. The road trip from your IDE to a Production Node
  13. Yes, it can be fun!
  14. Background Netflix OSS Components Development Deployment/Delivery Open Floor
  15. Background Netflix OSS Components Development Deployment/Delivery Open Floor Open Source Infrastructure Components Libraries Tools Frameworks PaaS with AWS IaaS
  16. Background Netflix OSS Components Development Deployment/Delivery Open Floor
  17. Scala Background Netflix OSS Components Development Deployment/Delivery Open Floor
  18. Immutable by default Concise, yet expressive Type inference Multi paradigm Java ecosystem Functional
  19. /** * Map results from a java.sql.ResultSet to a list of objects of another type * @param rs The result set * @param f Function that takes the result set and returns an instance of T * @tparam A The type * @return Sequence of instances of type A */ def mapResults[A](rs: ResultSet, f: ResultSet => A) : Seq[A] = { val array = ArrayBuffer[A]() while (rs.next) { array += f(rs) } array.toSeq }
  20. /** * Function to return a cached value, or fetch it per the parameter f * @param key The key to look up in cache, or set against this key if not found * @param f The function that fetches the instance from some persistent store * @tparam A The type * @return None if not found in cache and persistent store, else Some[A] */ def withCache[A](key: String)(f: => Option[A]): Option[A] = { EVCacheGateway.get[A](key) match { case None => val data = f if (!data.isEmpty) EVCacheGateway.set(key, data.get) data case Some(x) => Some(x) } }
  21. /** * Get a user given an ID * @param id The User ID * @return None if user is not found, else the user */ def getById(id: Long): Option[User] = { withCache[User](s"User$id") { val query = "select * from users where user_id = ?" DBHelper.queryWith(query, extract, id).headOption } }
  22. Scalatra Background Netflix OSS Components Development Deployment/Delivery Open Floor
  23. Lightweight HTTP wrapper .war based deployment Swagger support ScalatraSpec
  24. /** * Get all the chipsets ordered by name. */ get("/providers", operation(getChipsetProviders)) { Ok( ChipsetManager.getProviders(0,20) ) }
  25. ScalaTest Background Netflix OSS Components Development Deployment/Delivery Open Floor
  26. Background Netflix OSS Components Development Deployment/Delivery Open Floor Simple Intuitive English-like Rich Promotes BDD
  27. it should "return the chipset provider partner orgs" in { val data = ChipsetManager.getProviders(0,20) data.records.isEmpty should be (false) data.records.size should be (20) data.total should be >= 21l }
  28. ScalatraSpec Background Netflix OSS Components Development Deployment/Delivery Open Floor Traits to take ScalaTest to the next level Helpers for JSON parsing Plenty of wrappers (body, headers..)
  29. addServlet(new ChipsetService, "/*") it should "get a list of chipset providers" in { getWithToken("/providers")(TestUserTokens.testToken) { status should equal (200) body should include (""""start":0,"end":19,"count":20""") } }
  30. Swagger Background Netflix OSS Components Development Deployment/Delivery Open Floor
  31. Excellent support within Scalatra Type-safe documentation Contract First API Sandbox Machine Readable
  32. get("/providers", operation(getAllChipsetProviders)) { //code } case class PartnerList(data: Seq[PartnerOrg], start: Int, end: Int, count: Long, total: Long) def getAllChipsetProviders = authApiOperation[PartnerList]("getChipsetProviders", "Get all chipset providers") .parameter(queryParam[Option[Int]]("start").description("Starting record count, defaults to 0")) .parameter(queryParam[Option[Int]]("count").description("The number of records requested, defaults to 20"))
  33. Swagger
  34. Deployment Background Netflix OSS Components Development Deployment/Delivery Open Floor Push to dev Jenkins runs dev build, tests, merges to master Jenkins runs master build, makes a .deb Aminator bakes an AMI from the .deb Asgard deploys the AMI in staging cloud
  35. Production? Background Netflix OSS Components Development Deployment/Delivery Open Floor
  36. lgml-mpandit:nrd-portal-api mpandit$ git status On branch dev Your branch is up-to-date with 'origin/dev'. Changes not staged for commit: (use "git add <file>..." to update what will be committed) (use "git checkout -- <file>..." to discard changes in working directory) modified: api/src/main/scala/com/netflix/nrdportal/http/ChipsetService.scala modified: api/src/test/scala/com/netflix/nrdportal/http/ChipsetServiceSpec.scala modified: api/src/main/scala/com/netflix/nrdportal/manager/ChipsetManager.scala modified: api/src/test/scala/com/netflix/nrdportal/manager/ChipsetManagerSpec.scala no changes added to commit (use "git add" and/or "git commit -a") Check in
  37. [info] Passed: Total 1386, Failed 0, Errors 0, Passed 1386 [success] Total time: 1273 s, completed Jul 19, 2014 8:39:54 PM Build step 'Build using sbt' changed build result to SUCCESS Pushing HEAD to branch master of origin repository Returning node parameter for ssh-dynaslave-1291624c Triggering a new build of PPD-NRD-PORTAL-API-MASTER #1172 Notifying upstream projects of job completion Finished: SUCCESS Dev Build
  38. Background Netflix OSS Components Development Deployment/Delivery Open Floor Integration Tests Hitting Real DB Refreshed every night with Production
  39. Build Artifacts: nrd-portal-api_1.0-h1172.880e187_all.deb 127.71 MB [fingerprint] Changes: Added an endpoint to return a list of chipset provider partner orgs. NFLX-5514 (detail) Started by upstream project PPD-NRD-PORTAL-API-DEV build number 1923 originally caused by: Started by an SCM change Revision: 880e1873fef63d278b3180b49af7434e234dee40 origin/master Master Build
  40. Finally!
  41. “A node, once deployed, cannot be changed.” Immutable Deployments
  42. Easy rollbacks Consistency Predictability Immutable Deployments
  43. Summary Background Netflix OSS Components Development Deployment/Delivery Open Floor Bridge the gap between dev and deploy No such thing as too many tests Automate everything Document your APIs Best solutions are implementation agnostic
  44. Netflix OSS - http://netflix.github.io/#repo Scalatra - http://www.scalatra.org/ Swagger - https://helloreverb.com/developers/swagger ScalaTest - http://www.scalatest.org/
  45. Session Feedback and Rating slideshare.net/lobster1234 speakerdeck.com/mpandit

×