• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
A Guide to the Post Relational Revolution
 

A Guide to the Post Relational Revolution

on

  • 272 views

Presentation held at Scandinavian Developer Conference, April 2012

Presentation held at Scandinavian Developer Conference, April 2012

Statistics

Views

Total Views
272
Views on SlideShare
272
Embed Views
0

Actions

Likes
0
Downloads
6
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    A Guide to the Post Relational Revolution A Guide to the Post Relational Revolution Presentation Transcript

    • A GUIDE TO THEPOST RELATIONAL REVOLUTION @iconara
    • speakerdeck.com/u/iconara (real time!)
    • Theo / @iconara
    • Chief Architect atCo-organizer of the local Ruby, Scala and JavaScript user groups More rep on StackOverflow than both Jeff & Joel
    • THE WORLD ISN’T FLAT
    • OUT IS THE NEW UP when scaling up you’reconstrained by Moore’s Law
    • DISTRIBUTED SYSTEMS AREABOUT TRADEOFFS
    • WHO NEEDSACID, ANYWAY? banks, perhaps
    • JOINS ARE A CRUTCH why split up your data, if all you’re goingto do is assemble it over and over again?
    • OBJECTS DON’T FIT IN TABLEScan you say “impedance mismatch”?
    • 40 YEARS IS A LONG TIMEyou didn’t have 256 gigabytes of RAM in 1970
    • THE RELATIONAL MODEL ISN’T AGOLDEN HAMMER the existence of object relational mappers should be proof enough
    • WELCOME TO THEPOST RELATIONAL REVOLUTION
    • POST RELATIONAL STORAGE
    • KEY/VALUE STORESthe simplest possible database, not exactly a new idea
    • OPAQUE KEY VALUE Riak, Voldemort, LevelDB,Tokyo Cabinet, Berkeley DB
    • STRUCTUREDKEY/VALUE STORES sometimes you need just a little bit more
    • SORTED + TIMESTAMP COLUMN KEY COLUMN KEY ROW KEY VALUE VALUE the Bigtable model, “column oriented”,“sparse tables” found in Cassandra and HBase
    • LIST OR SETKEY VALUE VALUE VALUE SORTED SET OR HASH KEY KEY KEYKEY VALUE VALUE VALUE INCREMENT APPEND, SLICE, CAS ,KEY VALUE“datastructure server”, e.g. Redis
    • DOCUMENTDATABASESobject databases, but for hipsters
    • { "firstName": "John", "lastName": "Smith", "age": 25, "address": { "streetAddress": "21 2nd Street", "city": "New York", "state": "NY", "postalCode": "10021" }, "phoneNumber": [ { "type": "home", "number": "212 555-1234" }, { "type": "cell", "number": "646 555-4567" } ] } complex objects with lists, numbers, strings secondary indexes* and partial updates,MongoDB, CouchDB, RavenDB, Lotus Notes * subject to availability
    • GRAPHDATABASES relational, for real
    • NAME + PROPERTIES NODE NODE NODE NODE NODE NAMEtraversal algorithms, extreme data complexity, Neo4j, AllegroGraph, FlockDB
    • DIVERSITY I haven’t even mentioned search & indexing systemslike Solr and Elastic Search, or distributed filesystems
    • SOMETIMES TABLES ARE GREAT, TOObut mostly when you rely heavily on GROUP BY, SUM, AVG, etc. and can’t precompute
    • POST RELATIONAL SCALING
    • CAP
    • CONSISTENCY AVAILABILITYPARTITION TOLERANCE (choose any two)
    • OK?
    • PARTITIONTOLERANCE ISN’T OPTIONAL
    • CONSISTENCYVS. AVAILABILITY(but in reality, it’s not even that simple)
    • CONSISTENCYyou can always read what you just wrote, but keys may become unavailable
    • AVAILABILITY you can always read and write,but you may not always get the latest value
    • NOT EITHER OR most databases let you choose on a query-by-query basis
    • SHARDINGscaling writes in a consistent system
    • DIVIDED BY DA A SIZE T KEYSPACE SHARD SHARD SHARDA Z REPLICA REPLICA REPLICA REPLICA REPLICA REPLICA REPLICA REPLICA REPLICA divide the keyspace into shards, or regions (and store each one redundantly)
    • KEYSPACE SHARD SHARD SHARD SHARDA Z REPLICA REPLICA REPLICA REPLICA REPLICA REPLICA REPLICA REPLICA REPLICA REPLICA REPLICA REPLICA SPLIT split a shard when it grows too big, move one of the new shards onto a new node
    • KEYSPACE SHARD SHARD SHARD SHARDA Z REPLICA REPLICA REPLICA REPLICA REPLICA REPLICA REPLICA REPLICA REPLICA REPLICA REPLICA REPLICAin reality there’s chunks, tablets or “virtual shards” that are distributed over physical shards
    • HBASE, MONGODB sharding is easy in theory, hard in practice,lots data needs to be moved when adding nodes
    • CONSISTENT HASHINGscaling writes in an available system
    • 2n 0 NODEhash(key) replication KEYSPACE NODE NODE NODE each node is responsible for a range of the keyspace, keys are hashed and mapped to the first following node, (optionally) replicated to subsequent nodes
    • 2n 0 NEW NODE NODE NODE KEYSPACE NODE NODE NODEwhen a new node is added, only part of the keyspace needs to be moved
    • 2n 0 NODE NODE KEYSPACE NODE NODE NODE in practice, “virtual nodes” are evenly distributed overthe keyspace, and then mapped onto physical nodes
    • CASSANDRA, RIAK perfect balance, in theory, but rings may still need rebalancing
    • GOSSIP HINTED HANDOFF , , LOG STRUCTUREDSTORAGE, COMPACTION, VECTOR CLOCKS, READ REPAIR, JOURNALING, QUORUMS, EVENTUALCONSISTENCY, DYNAMO, MAP/REDUCE, 2PC a few of the things I haven’t mentioned, look them up
    • LESSONS LEARNED
    • EVERYTHING THEYalmost TAUGHT YOU ABOUT DATABASES AT UNIVERSITY IS WRONG
    • THINK ABOUT YOUR QUERIES FIRST don’t optimize for insertion, denormalize heavily, disk is cheap, this ain’t 1970
    • GIVE A LOT OFTHOUGHT TO YOUR PRIMARY KEYS range queries over cleverly designed primary keys can be very powerful, good keys required for efficient sharding
    • M04L7NOC5NQSM04L7O05MIU2M04NX42YFUCRM04NYR7VWKJCM04NZA8MJOOAM04NZB88CT14M04NZPOCE8DMM04NZQ9G2T0SM04NZQE7E5VXM04NZSK4V3JNM04NZTRG661RM04NZTSUITJ7M04NZUAILUS5M04NZUG4DTXNM04NZWB9VV0CM04NZWW52T8NM04NZX2JEVO9M04NZX7WD77WM04NZXGOLDEXM04NZXKNQWB3M04NZXLGJ3M6M04NZY7GO39GM04NZZ2SQF1IM04O013HN9L9M04O014DASE6M04O02PE8AD3M04O02PGJBR1M04O03UPTRWGM04O04833ZTLM04O04GH21JFM04O04JQ8B57M04O04UHK3U4M04O056QBNBHM04O05E8XO8NM04O069O8CDKM04O06MG47WKM04O07BHELVDM04O07F30WYXM04O0B39DGEA
    • M04NZW B9VV0C timestamp 2012-02-28 23:59:56 UTC random number 681 731 004
    • B9VV0C M04NZWrandom number 681 731 004 timestamp 2012-02-28 23:59:56 UTC
    • CONSISTENCYIS OVERRATED when you need it you need it, but most of the time you don’t
    • DELETING DATA IS NOT TRIVIALsometimes delete operations can be more costly than inserts, design your cleaning process early
    • REDIS MONGODBCASSANDRA our current toolbox
    • REDISswiss army knife, we use it for “virtual memory”, counters and even messaging
    • REDISnot distributed (yet), no automatic failover
    • MONGODB a very good replacement for MySQL,replication and automatic failover is fantastic
    • MONGODBglobal write lock kills performance, easily fragmented, sharding is complex and (has been) very buggy
    • MONGODBwe use it for precomputing and storing metrics for our reporting app
    • MONGODB we’re currently pushing around 5K updates/s over threereplica sets, each update incrementing up to 20 numbers
    • CASSANDRAlow level building blocks, no single point of failure, great horizontal scalability, TTL on values
    • CASSANDRAwe use it to store data about website visits, indexing it to support complex queries
    • CASSANDRA millions of rows, some with millions ofcolumns, adding ~1K new every second
    • one million writes per second
    • LEARN SOMETHING NEW TODAY nosql.mypopescu.com highscalability.com nosqltapes.com
    • KTHXBAI twitter.com/iconaraspeakerdeck.com/u/iconara architecturalatrocities.com burtcorp.com