rogue:a scala dsl for mongodbCS442 - 5/24/2011Jorge Ortiz (@jorgeortiz85)
what is foursquare?location-based social network - “check-in” to bars,restaurants, museums, parks, etc  friend-finder (whe...
foursquare: the numbers>9M users~3M checkins/day>15M venues>300k merchants>60 employees
foursquare: the tech Nginx, HAProxy Scala, Lift MongoDB, PostgreSQL (legacy) (Kestrel, Munin, Ganglia, Python, Memcache, ....
what is mongodb?fast, schema-less document storeindexes & rich queries on any attributesharding, auto-balancingreplication...
mongodb: our numbers8 clusters  some sharded, some not  some master/slave, some replica set~40 machines (68.4GB, m2.4xl on...
mongodb: bsonjson++binary wire formatmore built-in types:  objectid  date  regex  bina
mongodb: bson example{    “_id” :      { “oid” : “4ddbd194686148110d5c1ccc” },    “venuename” : “Starbucks”,    “mayorid” ...
mongodb: query example{    “mayorid” : { “$lte” : 100 },    “venuename” : { “$eq” : “Starbucks” },    “tags” : { “$contain...
mongodb: query exampleval query =  (BasicDBOBjectBuilder    .start    .push(“mayorid”)      .add(“$lte”, 100)    .pop    ....
rogue: a scala dsl for mongo type-safe all mongo query features logging & validation hooks pagination index-aware cursors ...
rogue: schema exampleclass Venue extends MongoRecord[Venue] {  object _id extends ObjectIdField(this)  object venuename ex...
rogue: code exampleval vs: List[Venue] =  (Venue where (_.mayorid <= 100)          and (_.venuename eqs “Starbucks”)      ...
rogue: BaseQueryclass BaseQuery[M <: MongoRecord[M],                R, Ord, Sel, Lim, Sk](...) {    def where[F](clause: M...
rogue: QueryFieldclass QueryField[V, M <: MongoRecord[M]]                (val field: Field[V, M]) {  def eqs(v: V) = new E...
rogue: implicitsField[F, M] => QueryField[F, M]Field[LatLong, M] => GeoQueryField[M]Field[List[F], M] => ListQueryField[F,...
rogue: selectval vs: List[(Long, Int)] =  (Venue where (_.venuename eqs “Starbucks”)        select (_.mayorid, _.mayorChec...
rogue: selectclass BaseQuery[M <: MongoRecord[M],                R, Ord, Sel, Lim, Sk](...) {    def select[F1, F2](      ...
rogue: select problemsval vs: List[???] =  (Venue where (_.venuename eqs “Starbucks”)        select (_.mayorid, _.mayorChe...
rogue: order problemsval vs: List[Venue] =  (Venue where (_.venuename eqs “Starbucks”)     orderDesc (_._id)      orderAsc...
rogue: limit problemsval vs: List[Venue] =  (Venue where (_.venuename eqs “Starbucks”)         limit (5)         limit (10...
rogue: phantom typesSel =:= Selected, UnselectedOrd =:= Ordered, UnorderedLim =:= Limited, UnlimitedSk =:= Skipped, Unskip...
rogue: select phantomclass BaseQuery[M <: MongoRecord[M],                R, Sel, Ord, Lim, Sk](...) {    def select[F1, F2...
rogue: select problemsval vs: List[???] =  (Venue where (_.venuename eqs “Starbucks”)        select (_.mayorid, _.mayorChe...
rogue: logging & validation logging:   slf4j   Tracer validation:   radius, $in size   index checks
rogue: paginationval query: Query[Venue] = ...val vs: List[Venue] =  (query    .countPerPage(20)    .setPage(5)    .fetch())
rogue: cursorsval query: Query[Checkin] = ...for (checkin <- query) {  ... f(checkin) ...}
rogue: index-awareval vs: List[Checkin] =  (Checkin    where (_.userid eqs 646)      and (_.venueid eqs vid)    fetch ())
rogue: index-awareval vs: List[Checkin] =  (Checkin    where (_.userid eqs 646)      and (_.venueid eqs vid) // hidden sca...
rogue: index-awareval vs: List[Checkin] =  (Checkin    where (_.userid eqs 646) // known index     scan (_.venueid eqs vid...
rogue: future directions iteratees for cursors compile-time index checking select partial objects generate full javascript...
we’re hiring    (nyc & sf)http://foursquare.com/jobs  jorge@foursquare.com        @jorgeortiz85
Upcoming SlideShare
Loading in …5
×

CS442 - Rogue: A Scala DSL for MongoDB

5,157 views

Published on

Talk at Stanford's CS442 (High Productivity and Performance with Domain Specific Languages in Scala http://www.stanford.edu/class/cs442/), on Rogue. 5/24/2011

0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
5,157
On SlideShare
0
From Embeds
0
Number of Embeds
35
Actions
Shares
0
Downloads
79
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • CS442 - Rogue: A Scala DSL for MongoDB

    1. 1. rogue:a scala dsl for mongodbCS442 - 5/24/2011Jorge Ortiz (@jorgeortiz85)
    2. 2. what is foursquare?location-based social network - “check-in” to bars,restaurants, museums, parks, etc friend-finder (where are my friends right now?) virtual game (badges, points, mayorships) city guide (local, personalized recommendations) location diary + stats engine (where was I a year ago?) specials (get rewards at your favorite restaurant)
    3. 3. foursquare: the numbers>9M users~3M checkins/day>15M venues>300k merchants>60 employees
    4. 4. foursquare: the tech Nginx, HAProxy Scala, Lift MongoDB, PostgreSQL (legacy) (Kestrel, Munin, Ganglia, Python, Memcache, ...) All on EC2
    5. 5. what is mongodb?fast, schema-less document storeindexes & rich queries on any attributesharding, auto-balancingreplicationgeo-indexes
    6. 6. mongodb: our numbers8 clusters some sharded, some not some master/slave, some replica set~40 machines (68.4GB, m2.4xl on EC2)2.3 billion records~15k QPS
    7. 7. mongodb: bsonjson++binary wire formatmore built-in types: objectid date regex bina
    8. 8. mongodb: bson example{ “_id” : { “oid” : “4ddbd194686148110d5c1ccc” }, “venuename” : “Starbucks”, “mayorid” : 464, “tags” : [“coffee”, “wifi”, “snacks”], “latlng” : [39.0, -74.0]}
    9. 9. mongodb: query example{ “mayorid” : { “$lte” : 100 }, “venuename” : { “$eq” : “Starbucks” }, “tags” : { “$contains” : “wifi” }, “latlng” : { “$near” : [39.0, -74.0] }}{ “_id” : -1 }
    10. 10. mongodb: query exampleval query = (BasicDBOBjectBuilder .start .push(“mayorid”) .add(“$lte”, 100) .pop .push(“veneuname”) .add(“$eq”, “Starbucks”) .pop .get)
    11. 11. rogue: a scala dsl for mongo type-safe all mongo query features logging & validation hooks pagination index-aware cursors http://github.com/foursquare/rogue
    12. 12. rogue: schema exampleclass Venue extends MongoRecord[Venue] { object _id extends ObjectIdField(this) object venuename extends StringField(this) object mayorid extends LongField(this) object tags extends ListField[String](this) object latlng extends LatLngField(this)}
    13. 13. rogue: code exampleval vs: List[Venue] = (Venue where (_.mayorid <= 100) and (_.venuename eqs “Starbucks”) and (_.tags contains “wifi”) and (_.latlng near (39.0, -74.0, Degrees(0.2)) orderDesc (_._id) fetch (5))
    14. 14. rogue: BaseQueryclass BaseQuery[M <: MongoRecord[M], R, Ord, Sel, Lim, Sk](...) { def where[F](clause: M => QueryClause[F]): ... ...}
    15. 15. rogue: QueryFieldclass QueryField[V, M <: MongoRecord[M]] (val field: Field[V, M]) { def eqs(v: V) = new EqClause(...) def neqs(v: V) = ... def in[L <% List[V]](vs: L) = ... def nin[L <% List[V]](vs: L) = ... def exists(b: Boolean) = ...}
    16. 16. rogue: implicitsField[F, M] => QueryField[F, M]Field[LatLong, M] => GeoQueryField[M]Field[List[F], M] => ListQueryField[F, M]Field[String, M] => StringQueryField[M]Field[Int, M] => NumericQueryField[Int, M]Field[Double, M] => NumericQueryField[Double, M]...
    17. 17. rogue: selectval vs: List[(Long, Int)] = (Venue where (_.venuename eqs “Starbucks”) select (_.mayorid, _.mayorCheckins) fetch ())
    18. 18. rogue: selectclass BaseQuery[M <: MongoRecord[M], R, Ord, Sel, Lim, Sk](...) { def select[F1, F2]( f1: M => SelectField[F1, M], f2: M => SelectField[F2, M]): BaseQuery[M, (F1, F2), ...] = ... ...}
    19. 19. rogue: select problemsval vs: List[???] = (Venue where (_.venuename eqs “Starbucks”) select (_.mayorid, _.mayorCheckins) select (_.venuename) fetch ())
    20. 20. rogue: order problemsval vs: List[Venue] = (Venue where (_.venuename eqs “Starbucks”) orderDesc (_._id) orderAsc (_.mayorid) // (???) fetch ())
    21. 21. rogue: limit problemsval vs: List[Venue] = (Venue where (_.venuename eqs “Starbucks”) limit (5) limit (100) // (???) fetch ())
    22. 22. rogue: phantom typesSel =:= Selected, UnselectedOrd =:= Ordered, UnorderedLim =:= Limited, UnlimitedSk =:= Skipped, Unskipped
    23. 23. rogue: select phantomclass BaseQuery[M <: MongoRecord[M], R, Sel, Ord, Lim, Sk](...) { def select[F1, F2]( f1: M => SelectField[F1, M], f2: M => SelectField[F2, M]) (implicit ev: Sel =:= Unselected): BaseQuery[M, (F1, F2), Selected, ...] = ... ...}
    24. 24. rogue: select problemsval vs: List[???] = (Venue where (_.venuename eqs “Starbucks”) select (_.mayorid, _.mayorCheckins) select (_.venuename) fetch ())// won’t compile!
    25. 25. rogue: logging & validation logging: slf4j Tracer validation: radius, $in size index checks
    26. 26. rogue: paginationval query: Query[Venue] = ...val vs: List[Venue] = (query .countPerPage(20) .setPage(5) .fetch())
    27. 27. rogue: cursorsval query: Query[Checkin] = ...for (checkin <- query) { ... f(checkin) ...}
    28. 28. rogue: index-awareval vs: List[Checkin] = (Checkin where (_.userid eqs 646) and (_.venueid eqs vid) fetch ())
    29. 29. rogue: index-awareval vs: List[Checkin] = (Checkin where (_.userid eqs 646) and (_.venueid eqs vid) // hidden scan! fetch ())
    30. 30. rogue: index-awareval vs: List[Checkin] = (Checkin where (_.userid eqs 646) // known index scan (_.venueid eqs vid) // known scan fetch ())
    31. 31. rogue: future directions iteratees for cursors compile-time index checking select partial objects generate full javascript for mapreduce
    32. 32. we’re hiring (nyc & sf)http://foursquare.com/jobs jorge@foursquare.com @jorgeortiz85

    ×