"An Introduction to NoSQL" DevNexus talk 3/21/2011

1. An Introduction to NoSQL
Brad Anderson - DevNexus
March 21, 2011

2. Me
‘boorad’ most places (twitter, github, etc.)
Erlang Programmer
Cloudant BigCouch, Ericsson Monaco, Verdeeco
Java, Python, D, Javascript, Common Lisp
NoSQL East - October 2009
Data Warehousing / Big Data
pre-lunch talks... always.

3. Agenda
NoSQL is BULLSHIT
You Don’t Need It
You Can’t Query It

4. The Name
Play on MySQL (Eric Evans, Rackspace)
Not Only SQL (Emil Eifrem)
Broad Umbrella
Shitty Marketing Term and we’re stuck with it

5. Why do you need NoSQL?

6. Why do you need NoSQL?
YOU DON’T!

7. Seriously, you don’t...
Vastly different performance characteristics
Immature APIs and tools / ecosystems
Bugs, most are actively being developed
Your situation doesn’t warrant it

8. Why do they exist?
Every one of these new data storage systems
came from a particular pain someone was
having.
Each system was created to specifically solve
the pain point the authors were experiencing.
This pain usually involves a metric shit-tonne of
data and distributed processing is required.
Schema-free

9. Prediction: Pain

10. Examples
Google - index Internet (mapreduce/bigtable)
Yahoo - keep up with Google (Hadoop)
Amazon - shopping cart (Dynamo)
Facebook - inbox search (Cassandra)
Lotus - Notes legacy restrictions (CouchDB)
Cloudant - physics research (BigCouch)
Basho - CRM product (Riak)
Neo - graph traversal (Neo4J)

11. Pain of Scaling
Scale Reads with master-slave replication
Scale Writes with master-master replication
Partitioning Vertically (by functional groups)
Partitioning Horizontally (by key, i.e. ‘date’)
Caching works, kinda

12. What to do?
Distribute both data and processing
horizontal scaling
Organize data differently
Use appropriate on-disk storage

13. Sorting Hat Says...
Distribution Model
Data Model
Disk Data Structure

14. Distribution Model
Embedded (no distribution)
Replication / Sharding
Chord - peer to peer
Dynamo
consistent hashing, vnodes, vector clocks

15. No Distribution
BDB
Neo4J

16. Replication / Sharding
Distribution
MongoDB
CouchDB
Redis

17. Dynamo Distribution
BigCouch
Riak
Voldemort
Cassandra
no vnodes
no vector clocks
Hibari ?

18. Dynamo - how does it work?
N=3
W=2
Node 1
26 No
de A B C D de
No B
2
C
B C
A D
Z E
C N
od
e
D 3
E
F
D
No
de
E
4
F
G
17

19. Dynamo - how does it work?
PUT http://boorad.cloudant.com/dbname/blah?w=2
N=3
W=2
Node 1
26 No
de A B C D de
No B
2
C
B C
A D
Z E
C N
od
e
D 3
E
F
D
No
de
E
4
F
G
17

20. Dynamo - how does it work?
PUT http://boorad.cloudant.com/dbname/blah?w=2
N=3
W=2
Node 1
26 No
de A B C D de
No B
2
C
B C
A D
Z E
C N
od
e
D 3
E
F
D
No
de
E
4
F
G
17

21. Dynamo - how does it work?
PUT http://boorad.cloudant.com/dbname/blah?w=2
N=3
W=2
Node 1
26 No
de A B C D de
No B
2
C
B C
A D
Z hash(blah) E
C N
od
e
D 3
E
F
D
No
de
E
4
F
G
17

22. Dynamo - how does it work?
PUT http://boorad.cloudant.com/dbname/blah?w=2
N=3
W=2
Node 1
26 No
de A B C D de
No B
2
C
B C
A D
Z hash(blah) E
C N
od
e
D 3
E
F
D
No
de
E
4
F
G
17

23. CAP Theorem
Pick Two (at any given time)
Consistency
Availability
Partition Tolerance
CP refuses requests, AP eventually consistent
Must Read: http://codahale.com/you-cant-
sacrifice-partition-tolerance/

24. Data Model
Key/Value
Document
Column
Graph

25. Key / Value
BDB
Riak
Voldemort
Redis
Hibari

26. Document
CouchDB
MongoDB
SimpleDB

27. Column Stores
HBase
Cassandra
Hypertable

28. Graph Databases
Neo4J
AllegroGraph
FlockDB

29. Disk Data Structure
btree - many different kinds
mmap - compact bson
memtable/sstable or log structured merge tree
log-structured linear hashing
adjacency lists / adjacency matrices

30. Querying NoSQL
Key Lookups
fast, easy, limiting
Secondary Indexes
Immature part of most systems
Roll your own
MapReduce
Mongo query language

31. Polyglot Persistence
RDBMS
batch processes
Cache
Raw
Hadoop NoSQL Apps
Data
NoSQL

32. Drivers
Spring
commons, hadoop, kv, document, graph
membase, hbase, cassandra coming
Serialization
Thrift, Protocol Buffers, Avro
Native
Cassandra, Hadoop, Voldemort
JInterface to Erlang?

33. Good Luck! You’ll Need It.

34. Questions?