A Guide to the Post Relational Revolution

394 views

Published on

Presentation held at Scandinavian Developer Conference, April 2012

Published in: Technology, Sports
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
394
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
9
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

A Guide to the Post Relational Revolution

  1. 1. A GUIDE TO THEPOST RELATIONAL REVOLUTION @iconara
  2. 2. speakerdeck.com/u/iconara (real time!)
  3. 3. Theo / @iconara
  4. 4. Chief Architect atCo-organizer of the local Ruby, Scala and JavaScript user groups More rep on StackOverflow than both Jeff & Joel
  5. 5. THE WORLD ISN’T FLAT
  6. 6. OUT IS THE NEW UP when scaling up you’reconstrained by Moore’s Law
  7. 7. DISTRIBUTED SYSTEMS AREABOUT TRADEOFFS
  8. 8. WHO NEEDSACID, ANYWAY? banks, perhaps
  9. 9. JOINS ARE A CRUTCH why split up your data, if all you’re goingto do is assemble it over and over again?
  10. 10. OBJECTS DON’T FIT IN TABLEScan you say “impedance mismatch”?
  11. 11. 40 YEARS IS A LONG TIMEyou didn’t have 256 gigabytes of RAM in 1970
  12. 12. THE RELATIONAL MODEL ISN’T AGOLDEN HAMMER the existence of object relational mappers should be proof enough
  13. 13. WELCOME TO THEPOST RELATIONAL REVOLUTION
  14. 14. POST RELATIONAL STORAGE
  15. 15. KEY/VALUE STORESthe simplest possible database, not exactly a new idea
  16. 16. OPAQUE KEY VALUE Riak, Voldemort, LevelDB,Tokyo Cabinet, Berkeley DB
  17. 17. STRUCTUREDKEY/VALUE STORES sometimes you need just a little bit more
  18. 18. SORTED + TIMESTAMP COLUMN KEY COLUMN KEY ROW KEY VALUE VALUE the Bigtable model, “column oriented”,“sparse tables” found in Cassandra and HBase
  19. 19. LIST OR SETKEY VALUE VALUE VALUE SORTED SET OR HASH KEY KEY KEYKEY VALUE VALUE VALUE INCREMENT APPEND, SLICE, CAS ,KEY VALUE“datastructure server”, e.g. Redis
  20. 20. DOCUMENTDATABASESobject databases, but for hipsters
  21. 21. { "firstName": "John", "lastName": "Smith", "age": 25, "address": { "streetAddress": "21 2nd Street", "city": "New York", "state": "NY", "postalCode": "10021" }, "phoneNumber": [ { "type": "home", "number": "212 555-1234" }, { "type": "cell", "number": "646 555-4567" } ] } complex objects with lists, numbers, strings secondary indexes* and partial updates,MongoDB, CouchDB, RavenDB, Lotus Notes * subject to availability
  22. 22. GRAPHDATABASES relational, for real
  23. 23. NAME + PROPERTIES NODE NODE NODE NODE NODE NAMEtraversal algorithms, extreme data complexity, Neo4j, AllegroGraph, FlockDB
  24. 24. DIVERSITY I haven’t even mentioned search & indexing systemslike Solr and Elastic Search, or distributed filesystems
  25. 25. SOMETIMES TABLES ARE GREAT, TOObut mostly when you rely heavily on GROUP BY, SUM, AVG, etc. and can’t precompute
  26. 26. POST RELATIONAL SCALING
  27. 27. CAP
  28. 28. CONSISTENCY AVAILABILITYPARTITION TOLERANCE (choose any two)
  29. 29. OK?
  30. 30. PARTITIONTOLERANCE ISN’T OPTIONAL
  31. 31. CONSISTENCYVS. AVAILABILITY(but in reality, it’s not even that simple)
  32. 32. CONSISTENCYyou can always read what you just wrote, but keys may become unavailable
  33. 33. AVAILABILITY you can always read and write,but you may not always get the latest value
  34. 34. NOT EITHER OR most databases let you choose on a query-by-query basis
  35. 35. SHARDINGscaling writes in a consistent system
  36. 36. DIVIDED BY DA A SIZE T KEYSPACE SHARD SHARD SHARDA Z REPLICA REPLICA REPLICA REPLICA REPLICA REPLICA REPLICA REPLICA REPLICA divide the keyspace into shards, or regions (and store each one redundantly)
  37. 37. KEYSPACE SHARD SHARD SHARD SHARDA Z REPLICA REPLICA REPLICA REPLICA REPLICA REPLICA REPLICA REPLICA REPLICA REPLICA REPLICA REPLICA SPLIT split a shard when it grows too big, move one of the new shards onto a new node
  38. 38. KEYSPACE SHARD SHARD SHARD SHARDA Z REPLICA REPLICA REPLICA REPLICA REPLICA REPLICA REPLICA REPLICA REPLICA REPLICA REPLICA REPLICAin reality there’s chunks, tablets or “virtual shards” that are distributed over physical shards
  39. 39. HBASE, MONGODB sharding is easy in theory, hard in practice,lots data needs to be moved when adding nodes
  40. 40. CONSISTENT HASHINGscaling writes in an available system
  41. 41. 2n 0 NODEhash(key) replication KEYSPACE NODE NODE NODE each node is responsible for a range of the keyspace, keys are hashed and mapped to the first following node, (optionally) replicated to subsequent nodes
  42. 42. 2n 0 NEW NODE NODE NODE KEYSPACE NODE NODE NODEwhen a new node is added, only part of the keyspace needs to be moved
  43. 43. 2n 0 NODE NODE KEYSPACE NODE NODE NODE in practice, “virtual nodes” are evenly distributed overthe keyspace, and then mapped onto physical nodes
  44. 44. CASSANDRA, RIAK perfect balance, in theory, but rings may still need rebalancing
  45. 45. GOSSIP HINTED HANDOFF , , LOG STRUCTUREDSTORAGE, COMPACTION, VECTOR CLOCKS, READ REPAIR, JOURNALING, QUORUMS, EVENTUALCONSISTENCY, DYNAMO, MAP/REDUCE, 2PC a few of the things I haven’t mentioned, look them up
  46. 46. LESSONS LEARNED
  47. 47. EVERYTHING THEYalmost TAUGHT YOU ABOUT DATABASES AT UNIVERSITY IS WRONG
  48. 48. THINK ABOUT YOUR QUERIES FIRST don’t optimize for insertion, denormalize heavily, disk is cheap, this ain’t 1970
  49. 49. GIVE A LOT OFTHOUGHT TO YOUR PRIMARY KEYS range queries over cleverly designed primary keys can be very powerful, good keys required for efficient sharding
  50. 50. M04L7NOC5NQSM04L7O05MIU2M04NX42YFUCRM04NYR7VWKJCM04NZA8MJOOAM04NZB88CT14M04NZPOCE8DMM04NZQ9G2T0SM04NZQE7E5VXM04NZSK4V3JNM04NZTRG661RM04NZTSUITJ7M04NZUAILUS5M04NZUG4DTXNM04NZWB9VV0CM04NZWW52T8NM04NZX2JEVO9M04NZX7WD77WM04NZXGOLDEXM04NZXKNQWB3M04NZXLGJ3M6M04NZY7GO39GM04NZZ2SQF1IM04O013HN9L9M04O014DASE6M04O02PE8AD3M04O02PGJBR1M04O03UPTRWGM04O04833ZTLM04O04GH21JFM04O04JQ8B57M04O04UHK3U4M04O056QBNBHM04O05E8XO8NM04O069O8CDKM04O06MG47WKM04O07BHELVDM04O07F30WYXM04O0B39DGEA
  51. 51. M04NZW B9VV0C timestamp 2012-02-28 23:59:56 UTC random number 681 731 004
  52. 52. B9VV0C M04NZWrandom number 681 731 004 timestamp 2012-02-28 23:59:56 UTC
  53. 53. CONSISTENCYIS OVERRATED when you need it you need it, but most of the time you don’t
  54. 54. DELETING DATA IS NOT TRIVIALsometimes delete operations can be more costly than inserts, design your cleaning process early
  55. 55. REDIS MONGODBCASSANDRA our current toolbox
  56. 56. REDISswiss army knife, we use it for “virtual memory”, counters and even messaging
  57. 57. REDISnot distributed (yet), no automatic failover
  58. 58. MONGODB a very good replacement for MySQL,replication and automatic failover is fantastic
  59. 59. MONGODBglobal write lock kills performance, easily fragmented, sharding is complex and (has been) very buggy
  60. 60. MONGODBwe use it for precomputing and storing metrics for our reporting app
  61. 61. MONGODB we’re currently pushing around 5K updates/s over threereplica sets, each update incrementing up to 20 numbers
  62. 62. CASSANDRAlow level building blocks, no single point of failure, great horizontal scalability, TTL on values
  63. 63. CASSANDRAwe use it to store data about website visits, indexing it to support complex queries
  64. 64. CASSANDRA millions of rows, some with millions ofcolumns, adding ~1K new every second
  65. 65. one million writes per second
  66. 66. LEARN SOMETHING NEW TODAY nosql.mypopescu.com highscalability.com nosqltapes.com
  67. 67. KTHXBAI twitter.com/iconaraspeakerdeck.com/u/iconara architecturalatrocities.com burtcorp.com

×