Evolving IGN’s New APIs with Scala

  • 1,072 views
Uploaded on

Slides from my talk at JavaOne 2012 on how we evolved IGN's new APIs on Scala and Frameworks related to Scala.

Slides from my talk at JavaOne 2012 on how we evolved IGN's new APIs on Scala and Frameworks related to Scala.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
1,072
On Slideshare
0
From Embeds
0
Number of Embeds
2

Actions

Shares
Downloads
12
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Evolving IGN’s new APIs with Scala Manish Pandit Oct 4th, 2012
  • 2. About me Manish Pandit Director of Engineering, IGN @lobster1234 linkedin.com/in/mpandit
  • 3. About IGNWe are a leading online media and services companyobsessed with gaming and entertainment.56MM Monthly Unique Views737MM Monthly Page Views20MM Monthly Video Views
  • 4. My ObjectiveIs:To get you excited about building APIs in ScalaIsnt:To incite a functional vs. imperative, or Scala vs.____ debate
  • 5. Agenda• APIs at IGN• API Evolution• Why Scala• API Development with Scala• Monitoring, Maintaining, Evolving• Performance Management
  • 6. APIs at IGN• Offer SOA advantages• Isolate the back end systems from the FE• IGN is going where the audience is, fast – Mobile Devices, Consoles, TVs, Aggregators• APIs are a specialized skillset• Expand the development to outside developers
  • 7. APIs at IGN : Numbers• ~5 Billion requests a month• Average Response time of under 20ms on cache misses• 17 APIs between Social, OAuth, and Content• Hardest hit APIs doing about 25K RPM at peak
  • 8. Evolution
  • 9. The API Evolution : V1 [2006]• The CMS and the desktop site calling Java Services, which wrapped Oracle DB calls behind Hibernate.• Caching with Memcache – Near-far Cache Architecture to address hot spots• Not all content types were supported• Schema was still too complex
  • 10. The API Evolution : V2 [2009]• Rails based Content API, Java based Social API – Standards around request/response – Automated Testing – MongoDB NoSQL storage to simplify the schema – Solr• Video and Blogs served by V2 API• Games and Articles still used V1• Mixed-infrastructure with Rails and Java echosystem• Performance sucked royally
  • 11. The API Evolution : V3 [2012]• A fresh look at the APIs and IGN’s direction• First attempt to – Integrate all content types – Open up the APIs to external consumers – Evolve the CMS, traditionally a monolithic system – ElasticSearch – Varnish Response Caching• A learning opportunity for traditional Java stack engineers
  • 12. API Development• MVC Model for APIs – View, which marshals/unmarshals request and response data – Controller, which picks the right model to serve the request – Model, which has the necessary logic to talk to the database, or other APIs to fetch data
  • 13. Why Scala• Performance tied to cores than processor speed, i.e. concurrency• Ideal for API development – (relatively) simple to handle concurrency and immutability• JVM based = Performance• Re-use Java toolkits and libraries• Concise, expressive code (1:2.5 with Java)• Actor model makes the difficult parts easy
  • 14. Why Scala• Richer Collections API• Traits enabling Mix-Ins and behavior re-use• Statically Typed w/Type inference• Functional• Growing developer community• REPL• Culture – learning, hacking and achieving
  • 15. The Biggest Reason• Just makes sense
  • 16. The Biggest Reason• Just makes sense – Pattern matching – Implicit conversions – Control abstractions for collection mapping, filtering, iterating – Guaranteed Singletons at the classloader level – Almost non-existent boilerplate ….Productive, and fun!
  • 17. Picking up Scala : Week 1• Scala for Java Developers fast-tracked the learning• Syntax Familiarity• Java without semicolons? def total(nums: List[Int]) :Int = { var sum:Int=0 for(num<-nums) { sum+=num } sum }
  • 18. Picking up Scala : Week 4• Reduce the lines of code, embrace immutability, use the functional paradigms built into the language• Unmystrify the implicits magic nums.foldLeft(0)((n,c) => n+c)
  • 19. Picking up Scala : Today• Focus on immutability• Think functional – with the knowledge gained from using the functional aspects of the language, build control abstractions• Keep learning and applying
  • 20. API Components• Scalatra (View, Controller)• Lift-MongoRecord for MongoDB (Model)• Casbah for MongoDB• Play! Anorm for MySQL• Actors for tasks like syndication• Maven for Builds• MongoDB for persistance
  • 21. Other API Components• Varnish Cache• Elasticsearch• Yammer Metrics (ping, healthcheck)• Swagger (self-documenting RESTful APIs)• 3Scale for Partner APIs• IntelliJ IDEA and Eclipse with ScalaIDE
  • 22. Challenges with Scala• Learning Curve depends on the culture • TPS Cover Sheets, vs. Hack Nights• Scala is what you make of it
  • 23. Awesome..?http://scalaz.github.com/scalaz/scalaz-2.9.1-6.0.4/doc.sxr/scalaz/BKTree.scala.html
  • 24. Challenges with Scala• Scala is what you make of it – Middle ground – simple is beautiful• Tooling not as rich as Java – Changing with Typesafe investing in ScalaIDE
  • 25. Challenges with Scala• Slow compilation based on the source – How well the code matches up to the JVM constructs• No (binary) compatibility of dependencies for the major releases (2.8, 2.9, 2.10(?))• Monitoring – No native instrumentation agents as they exist for Java
  • 26. API Development : Deploying• Continuous Deployment • Jenkins hooks to github • Pushes artifact(s) to Nexus • Fabric to push code to production (to be automated)
  • 27. API Performance Management• As far as Scala goes, the tips are not much different than Java – GC Tuning – Avoid shared state – Parallel collections
  • 28. API Performance Management• For the overall APIs, the performance tuning has many knobs to work around with – Stateless services – Async calls with Akka Agents – Response level cache• Keep an eye on the database and network – Control the payload size – Use reasonable defaults – Allow field filtering – Paginate your response
  • 29. Monitoring and Maintenance• Newrelic – JVM stats, SQL trace – Supports Scala via annotations, but not as rich as Java• Typesafe Console for Akka
  • 30. Future of APIs at IGN• Opening up the APIs• Tests, tests, tests..• Move towards simplifying the stack – sbt vs. maven – Typesafe stack
  • 31. The IGN Tech Stack
  • 32. Resources• Do NOT use scaladoc as your 1st reference, at least initially• StackOverflow• Twitter’s Scala School• Online blogs and tutorials• Book: Scala for the Impatient• Scala Meetup Groups in your area• Coursera course on FP by Martin Odersky
  • 33. Questions