Building a Mongo DSL in Scala at Hot Potato (Lincoln Hochberg)

3,083 views

Published on

Published in: Technology

Building a Mongo DSL in Scala at Hot Potato (Lincoln Hochberg)

  1. 1. MongoSF April 30, 2010 1
  2. 2. Hot Potato Overview • What: Hot Potato connects audiences around shared interests • When: Founded early 2009 / PoC: Dec 2009 / Beta: 2 months ago • Where: Williamsburg, Brooklyn! • Who: Lincoln, Justin, Saadiq, Jeremy, Jace, Matt, Mike, Will 2
  3. 3. hotpotato.com 3
  4. 4. Hot Potato on the iPhone 4
  5. 5. Technology Stack Design principles: Simple and stateless API • Scala / MongoDB API Clients • Web application (Python/Django) • iPhone (Obj-C) 5
  6. 6. Hot Potato API - Why MongoDB? • Good documentation and excellent support • Fully Featured (like an RDBMS) but with a migration path to scale- out • Replication • Easy administration / scripting • Fast • Auto-sharding coming soon 6
  7. 7. Hot Potato API - Why MongoDB? 7
  8. 8. Hot Potato API - Why MongoDB? RDBMS Optimization process 7
  9. 9. Hot Potato API - Why MongoDB? RDBMS Optimization process 1. too many reads -> add cache 7
  10. 10. Hot Potato API - Why MongoDB? RDBMS Optimization process 1. too many reads -> add cache 2. too many joins -> de-normalize your data 7
  11. 11. Hot Potato API - Why MongoDB? RDBMS Optimization process 1. too many reads -> add cache 2. too many joins -> de-normalize your data 3. too many writes -> scale up hardware, CPU, RAM, IO 7
  12. 12. Hot Potato API - Why MongoDB? RDBMS Optimization process 1. too many reads -> add cache 2. too many joins -> de-normalize your data 3. too many writes -> scale up hardware, CPU, RAM, IO 4. too much db load -> eliminate triggers, stored procedures 7
  13. 13. Hot Potato API - Why MongoDB? RDBMS Optimization process 1. too many reads -> add cache 2. too many joins -> de-normalize your data 3. too many writes -> scale up hardware, CPU, RAM, IO 4. too much db load -> eliminate triggers, stored procedures 5. too much db load -> pre-materialize complex queries 7
  14. 14. Hot Potato API - Why MongoDB? RDBMS Optimization process 1. too many reads -> add cache 2. too many joins -> de-normalize your data 3. too many writes -> scale up hardware, CPU, RAM, IO 4. too much db load -> eliminate triggers, stored procedures 5. too much db load -> pre-materialize complex queries 6. writes bottleneck -> drop secondary indexes 7
  15. 15. Hot Potato API - Why MongoDB? RDBMS Optimization process 1. too many reads -> add cache 2. too many joins -> de-normalize your data 3. too many writes -> scale up hardware, CPU, RAM, IO 4. too much db load -> eliminate triggers, stored procedures 5. too much db load -> pre-materialize complex queries 6. writes bottleneck -> drop secondary indexes At this point you are using the RDBMS as a KV store. The full- featured nature of the RDBMS is merely getting in the way of scaling. 7
  16. 16. Hot Potato API - Why MongoDB? 8
  17. 17. Hot Potato API - Why MongoDB? Mongo comparison 8
  18. 18. Hot Potato API - Why MongoDB? Mongo comparison 1. reads start out fast, can be externally cached, can also be horizontally scaled 8
  19. 19. Hot Potato API - Why MongoDB? Mongo comparison 1. reads start out fast, can be externally cached, can also be horizontally scaled 2. mongo doesn’t support joins - data starts de-normalized 8
  20. 20. Hot Potato API - Why MongoDB? Mongo comparison 1. reads start out fast, can be externally cached, can also be horizontally scaled 2. mongo doesn’t support joins - data starts de-normalized 3. writes start out fast, can be horizontally scaled 8
  21. 21. Hot Potato API - Why MongoDB? Mongo comparison 1. reads start out fast, can be externally cached, can also be horizontally scaled 2. mongo doesn’t support joins - data starts de-normalized 3. writes start out fast, can be horizontally scaled 4. much less opportunity for logic in the db 8
  22. 22. Hot Potato API - Why MongoDB? Mongo comparison 1. reads start out fast, can be externally cached, can also be horizontally scaled 2. mongo doesn’t support joins - data starts de-normalized 3. writes start out fast, can be horizontally scaled 4. much less opportunity for logic in the db 5. map/reduce for pre-materialization/aggregation, which can be horizontally scaled 8
  23. 23. Hot Potato API - Why MongoDB? Mongo comparison 1. reads start out fast, can be externally cached, can also be horizontally scaled 2. mongo doesn’t support joins - data starts de-normalized 3. writes start out fast, can be horizontally scaled 4. much less opportunity for logic in the db 5. map/reduce for pre-materialization/aggregation, which can be horizontally scaled 6. indexes can be removed over time as usage becomes more KV-like 8
  24. 24. Hot Potato API - Why MongoDB? Mongo comparison 1. reads start out fast, can be externally cached, can also be horizontally scaled 2. mongo doesn’t support joins - data starts de-normalized 3. writes start out fast, can be horizontally scaled 4. much less opportunity for logic in the db 5. map/reduce for pre-materialization/aggregation, which can be horizontally scaled 6. indexes can be removed over time as usage becomes more KV-like The gradual scaling process is more natural with Mongo. Horizontal scaling is not an after-thought or bolt on addition. Mongo works perfectly well as a KV store if you ever get to that point. 8
  25. 25. Hot Potato API - Why Scala? • Runs on JVM (Stable, Fast) • Access to Java’s many libraries • Language benefits • Terse • Supports Immutability • Functional • Concurrent • Easy to write DSLs 9
  26. 26. Three key Scala features • Pattern Matching / Case classes • Implicit conversions • Actors 10
  27. 27. Building a DSL for Mongo Documents are a flexible building block: • Insertion • Updates • Queries • Sorting • parts of Map / Reduce • Indexes 11
  28. 28. Building a DSL for Mongo Goals • Stay close to the MongoDB Java API • Keep it flexible • Focus on document creation • New documents, queries, updates • Protect against mis-named fields Key classes and objects • Collection - wraps MongoDB DBCollection • MongoAST - defines the types for building Mongo documents • MongoDSL - defines DSL syntax 12
  29. 29. Usage Patterns Asynchronous atomic updates • Simple observer pattern implemented with Lift Actors • HpActor, HpActorPool, Notifier • Main action object with a Notifier • Example: checkins • Listeners that use MongoDB atomic updates • Example: EventAdjuster 13

×