From Ruby on Rails to Java: The Gory Details (OSCON Java 2011)

OSCON JAVA 2011 JulY 23, 2011From RAILS to tHE JVM

LESSONS Decompose Future Proof

2006 Lamp StackEverything InlineQuick and Dirty It worked!

2007 Async worker daemonsStarling: our own message queue memcache protocol ruby marshalled objects text

2008 Wanted new tools JVMCan’t sell Java to a room of Ruby programmers Enter Scala

ThriftPick any messaging system that can grow Protocol Buffers Avro ASN.1

JVM Languages we use Java Scala Pig Clojure

Experiment #1Wanted more density out of message queue Rewrote Starling in Scala: Kestrel Similar code size All Tweets Flow Through Kestrel

Experiment #2 Render public_timeline in Scala Quickly written, low latency Handled all queries in 1 machine Was ultimately a failureUnderlying systems changed without coordination

Experiment #3 Streaming API Wrote Scala prototype in 2 weeksToday’s version serves millions of users

DO WORK Experiment time is overSocial Graph Databases were falling over MySQL with packed arraysBuilt Flock: our Scala social graph store

Flock ThriftStateless Scala servers MySQL Embedded Kestrel

Namesearch Thrift Stateless Scala Merge Servers Stateful Scala Index Servers Scatter/GatherBuilt on top of a MapReduce index

Namesearch Part 2 Rewritten by Search Team Pure Java Same MapReduce Index Same Thrift Interface Drop-in Replacement! Better results

MapReduceTwitter uses Hadoop and MapReduce extensively Pig for high-level MapReductions Java for low-level MapReductions

Plain JavaPlain Java entered with our acquisition of the Summize search engine company. Java is a first class citizen Lots of new high-quality Java code Scala/Java interop

Clojure backtype acquisition Strong teams aren’t shy of new languagesExpect to see more Clojure and Scala startups

DecompositionClearly defined boundaries allow experimentation Kestrel HDFS Thrift HTTP

From Ruby on Rails to Java: The Gory Details (OSCON Java 2011)

by saladwithsteve on Nov 06, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

Statistics

From Ruby on Rails to Java: The Gory Details (OSCON Java 2011) — Presentation Transcript