Evolving IGN’s New APIs with Scala


Published on

Slides from my talk at JavaOne 2012 on how we evolved IGN's new APIs on Scala and Frameworks related to Scala.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Evolving IGN’s New APIs with Scala

  1. 1. Evolving IGN’s new APIs with Scala Manish Pandit Oct 4th, 2012
  2. 2. About me Manish Pandit Director of Engineering, IGN @lobster1234 linkedin.com/in/mpandit
  3. 3. About IGNWe are a leading online media and services companyobsessed with gaming and entertainment.56MM Monthly Unique Views737MM Monthly Page Views20MM Monthly Video Views
  4. 4. My ObjectiveIs:To get you excited about building APIs in ScalaIsnt:To incite a functional vs. imperative, or Scala vs.____ debate
  5. 5. Agenda• APIs at IGN• API Evolution• Why Scala• API Development with Scala• Monitoring, Maintaining, Evolving• Performance Management
  6. 6. APIs at IGN• Offer SOA advantages• Isolate the back end systems from the FE• IGN is going where the audience is, fast – Mobile Devices, Consoles, TVs, Aggregators• APIs are a specialized skillset• Expand the development to outside developers
  7. 7. APIs at IGN : Numbers• ~5 Billion requests a month• Average Response time of under 20ms on cache misses• 17 APIs between Social, OAuth, and Content• Hardest hit APIs doing about 25K RPM at peak
  8. 8. Evolution
  9. 9. The API Evolution : V1 [2006]• The CMS and the desktop site calling Java Services, which wrapped Oracle DB calls behind Hibernate.• Caching with Memcache – Near-far Cache Architecture to address hot spots• Not all content types were supported• Schema was still too complex
  10. 10. The API Evolution : V2 [2009]• Rails based Content API, Java based Social API – Standards around request/response – Automated Testing – MongoDB NoSQL storage to simplify the schema – Solr• Video and Blogs served by V2 API• Games and Articles still used V1• Mixed-infrastructure with Rails and Java echosystem• Performance sucked royally
  11. 11. The API Evolution : V3 [2012]• A fresh look at the APIs and IGN’s direction• First attempt to – Integrate all content types – Open up the APIs to external consumers – Evolve the CMS, traditionally a monolithic system – ElasticSearch – Varnish Response Caching• A learning opportunity for traditional Java stack engineers
  12. 12. API Development• MVC Model for APIs – View, which marshals/unmarshals request and response data – Controller, which picks the right model to serve the request – Model, which has the necessary logic to talk to the database, or other APIs to fetch data
  13. 13. Why Scala• Performance tied to cores than processor speed, i.e. concurrency• Ideal for API development – (relatively) simple to handle concurrency and immutability• JVM based = Performance• Re-use Java toolkits and libraries• Concise, expressive code (1:2.5 with Java)• Actor model makes the difficult parts easy
  14. 14. Why Scala• Richer Collections API• Traits enabling Mix-Ins and behavior re-use• Statically Typed w/Type inference• Functional• Growing developer community• REPL• Culture – learning, hacking and achieving
  15. 15. The Biggest Reason• Just makes sense
  16. 16. The Biggest Reason• Just makes sense – Pattern matching – Implicit conversions – Control abstractions for collection mapping, filtering, iterating – Guaranteed Singletons at the classloader level – Almost non-existent boilerplate ….Productive, and fun!
  17. 17. Picking up Scala : Week 1• Scala for Java Developers fast-tracked the learning• Syntax Familiarity• Java without semicolons? def total(nums: List[Int]) :Int = { var sum:Int=0 for(num<-nums) { sum+=num } sum }
  18. 18. Picking up Scala : Week 4• Reduce the lines of code, embrace immutability, use the functional paradigms built into the language• Unmystrify the implicits magic nums.foldLeft(0)((n,c) => n+c)
  19. 19. Picking up Scala : Today• Focus on immutability• Think functional – with the knowledge gained from using the functional aspects of the language, build control abstractions• Keep learning and applying
  20. 20. API Components• Scalatra (View, Controller)• Lift-MongoRecord for MongoDB (Model)• Casbah for MongoDB• Play! Anorm for MySQL• Actors for tasks like syndication• Maven for Builds• MongoDB for persistance
  21. 21. Other API Components• Varnish Cache• Elasticsearch• Yammer Metrics (ping, healthcheck)• Swagger (self-documenting RESTful APIs)• 3Scale for Partner APIs• IntelliJ IDEA and Eclipse with ScalaIDE
  22. 22. Challenges with Scala• Learning Curve depends on the culture • TPS Cover Sheets, vs. Hack Nights• Scala is what you make of it
  23. 23. Awesome..?http://scalaz.github.com/scalaz/scalaz-2.9.1-6.0.4/doc.sxr/scalaz/BKTree.scala.html
  24. 24. Challenges with Scala• Scala is what you make of it – Middle ground – simple is beautiful• Tooling not as rich as Java – Changing with Typesafe investing in ScalaIDE
  25. 25. Challenges with Scala• Slow compilation based on the source – How well the code matches up to the JVM constructs• No (binary) compatibility of dependencies for the major releases (2.8, 2.9, 2.10(?))• Monitoring – No native instrumentation agents as they exist for Java
  26. 26. API Development : Deploying• Continuous Deployment • Jenkins hooks to github • Pushes artifact(s) to Nexus • Fabric to push code to production (to be automated)
  27. 27. API Performance Management• As far as Scala goes, the tips are not much different than Java – GC Tuning – Avoid shared state – Parallel collections
  28. 28. API Performance Management• For the overall APIs, the performance tuning has many knobs to work around with – Stateless services – Async calls with Akka Agents – Response level cache• Keep an eye on the database and network – Control the payload size – Use reasonable defaults – Allow field filtering – Paginate your response
  29. 29. Monitoring and Maintenance• Newrelic – JVM stats, SQL trace – Supports Scala via annotations, but not as rich as Java• Typesafe Console for Akka
  30. 30. Future of APIs at IGN• Opening up the APIs• Tests, tests, tests..• Move towards simplifying the stack – sbt vs. maven – Typesafe stack
  31. 31. The IGN Tech Stack
  32. 32. Resources• Do NOT use scaladoc as your 1st reference, at least initially• StackOverflow• Twitter’s Scala School• Online blogs and tutorials• Book: Scala for the Impatient• Scala Meetup Groups in your area• Coursera course on FP by Martin Odersky
  33. 33. Questions