Slides from my RailsConf 2010 talk, Beyond No(SQL). Conceptual tools for evaluating both SQL and NoSQL databases.

1. Beyond (No)SQL
Sarah Mei
Pivotal Labs
@sarahmei
sarah@pivotallabs.com

2. ?

3. Fsck CS.
-DHH

4. Fsck CS.
-DHH (paraphrased)

5. Photo by jasonwg (http://www.flickr.com/photos/jasonwg/1382036808)

6. Photo by jasonwg (http://www.flickr.com/photos/jasonwg/1382036808)

7. Agenda

8. Agenda
• The data storage landscape

9. Agenda
• The data storage landscape
• Relational model & SQL

10. Agenda
• The data storage landscape
• Relational model & SQL
RELATIONAL ALGEBRA

11. Agenda
• The data storage landscape
• Relational model & SQL
RELATIONAL ALGEBRA
• Evaluating data stores

12. Photo by TANAKA Juuyoh http://www.flickr.com/photos/tanaka_juuyoh/3121538767/

13. Photo by TANAKA Juuyoh http://www.flickr.com/photos/tanaka_juuyoh/3121538767/

14. Photo by TANAKA Juuyoh http://www.flickr.com/photos/tanaka_juuyoh/3121538767/

15. Photo by TANAKA Juuyoh http://www.flickr.com/photos/tanaka_juuyoh/3121538767/

16. Photo by TANAKA Juuyoh http://www.flickr.com/photos/tanaka_juuyoh/3121538767/

17. “NoSQL”

18. “NoSQL”
MongoDB

19. “NoSQL”
MongoDB
Memcache

20. “NoSQL”
MongoDB
Memcache
Bigtable

21. “NoSQL”
MongoDB
Memcache
Bigtable
CouchDB

22. “NoSQL”
MongoDB
Memcache
Bigtable
CouchDB
Cassandra

23. “NoSQL”
MongoDB
Memcache
Bigtable
CouchDB
Cassandra
Project Voldemort

24. “NoSQL”
MongoDB
Memcache
Bigtable
CouchDB
Cassandra
Project Voldemort
Hbase

25. “NoSQL”
MongoDB
Memcache
Bigtable
CouchDB
Cassandra
Project Voldemort
Hbase
Riak

26. “NoSQL”
MongoDB
Memcache
Bigtable
CouchDB
Cassandra
Project Voldemort
Hbase
Riak
Redis

27. “NoSQL”
MongoDB
Memcache
Bigtable
CouchDB
Cassandra
Project Voldemort
Hbase
Riak
Redis
Tokyo Cabinet

28. “NoSQL”
MongoDB
Memcache Key-value
Bigtable
CouchDB
Cassandra
Project Voldemort
Hbase
Riak
Redis
Tokyo Cabinet

29. “NoSQL”
MongoDB
Memcache Key-value
Bigtable
CouchDB
Document
Cassandra
Project Voldemort
Hbase
Riak
Redis
Tokyo Cabinet

30. “NoSQL”
MongoDB
Memcache Key-value
Bigtable
CouchDB
Document
Cassandra
Project Voldemort
Hbase
Riak
Redis
Tokyo Cabinet

31. “NoSQL”
MongoDB
Memcache Key-value
Bigtable
CouchDB
Document
Cassandra Other
Project Voldemort
Hbase
Riak
Redis
Tokyo Cabinet

32. Photo from mga (http://www.flickr.com/photos/mgiraldo/420642350)

33. NO.

34. The Relational Model
Diagram by Wikipedia user AutumnSnow (http://en.wikipedia.org/wiki/File:Relational_model_concepts.png)

35. Sets

36. Sets
• No duplicates

37. Sets
• No duplicates
• Unordered

38. More sets

39. More sets
• TABLE: a set of columns and a set of rows

40. More sets
• TABLE: a set of columns and a set of rows
• COLUMN: a unique name and a type

41. More sets
• TABLE: a set of columns and a set of rows
• COLUMN: a unique name and a type
• ROW: a set of name-value pairs

42. Relational Algebra
Operations you can do on tables (“relations”)

43. Operations
• projection: subset of available columns
• selection: subset of available rows
• cartesian product
• set union
• set intersection
• rename

44. Operations
• projection: subset of available columns
• selection: subset of available rows
• cartesian product
• set union
• set intersection
• rename

45. A Join
select * from foo, bar where foo.ipsum = bar.lorem
First you take the cartesian product....
foo: bat ipsum bar: cat lorem
me 5 X us 5
you 4 them 4
bat ipsum cat lorem
me 5 us 5
= me 5 them 4
you 4 us 5
you 4 them 4

46. A Join
select * from foo, bar where foo.ipsum = bar.lorem
Then you select the rows that satisfy the join condition:
bat ipsum cat lorem bat ipsum cat lorem
me 5 us 5 me 5 us 5
me 5 them 4 => you 4 them 4
you 4 us 5
you 4 them 4

47. A Join
select * from foo, bar where foo.ipsum = bar.lorem
First you take the cartesian product....
foo: bat ipsum bar: cat lorem
me 5 X us 5
you 4 them 4
bat ipsum cat lorem
me 5 us 5
= me 5 them 4
you 4 us 5
you 4 them 4

48. A Join
select * from foo, bar where foo.ipsum = bar.lorem
First you take the cartesian product....
foo: bat ipsum bar: cat lorem
me 5 X us 5
8,000,000
you 4 them 4
bat ipsum cat lorem
me 5 us 5
= me 5 them 4
you 4 us 5
you 4 them 4

49. A Join
select * from foo, bar where foo.ipsum = bar.lorem
First you take the cartesian product....
foo: bat ipsum bar: cat lorem
me 5 X us 5
8,000,000 100,000
you 4 them 4
bat ipsum cat lorem
me 5 us 5
= me 5 them 4
you 4 us 5
you 4 them 4

50. A Join
select * from foo, bar where foo.ipsum = bar.lorem
First you take the cartesian product....
foo: bat ipsum bar: cat lorem
me 5 X us 5
8,000,000 100,000
you 4 them 4
bat ipsum cat lorem
me 5 us 5
= me 5 them 4
800,000,000,000
you 4 us 5
you 4 them 4

51. Your choices

52. Your choices
• Scale the database

53. Your choices
• Scale the database
• Try a different approach

54. What SQL Gets You

55. What SQL Gets You
• Speed

56. What SQL Gets You
• Speed
(when data is highly structured and small enough)

57. What SQL Gets You
• Speed
(when data is highly structured and small enough)
• Aggregation

58. What SQL Gets You
• Speed
(when data is highly structured and small enough)
• Aggregation
• Relational searching

59. What SQL Gets You
• Speed
(when data is highly structured and small enough)
• Aggregation
• Relational searching
• ACID - guaranteed full consistency

60. Image by captcreate (http://www.flickr.com/photos/27845211@N02/2662264721)

61. What SQL Gets You

62. What SQL Gets You
• Speed

63. What SQL Gets You
• Speed
(when data is highly structured and small enough)

64. What SQL Gets You
• Speed
(when data is highly structured and small enough)
• Aggregation

65. What SQL Gets You
• Speed
(when data is highly structured and small enough)
• Aggregation
• Relational searching

66. What SQL Gets You
• Speed
(when data is highly structured and small enough)
• Aggregation
• Relational searching
• ACID - guaranteed full consistency

67. What if you gave up
data aggregation?

68. What if you gave up
data aggregation?
Document databases:
MongoDB
CouchDB
Riak

69. What if you gave up
where clauses?

70. What if you gave up
where clauses?
Key-value stores:
Memcache
Project Voldemort
Redis
Tokyo Cabinet

71. What if you gave up
consistency?

72. What if you gave up
consistency?
Then things get interesting.

73. ACID & BASE

74. ACID & BASE
• Atomicity

75. ACID & BASE
• Atomicity
• Consistency

76. ACID & BASE
• Atomicity
• Consistency
• Isolation

77. ACID & BASE
• Atomicity
• Consistency
• Isolation
• Durability

78. ACID & BASE
• Atomicity
• Consistency
• Isolation
• Durability

79. ACID & BASE
• Atomicity • Basically Available
• Consistency
• Isolation
• Durability

80. ACID & BASE
• Atomicity • Basically Available
• Consistency • Soft state
• Isolation
• Durability

81. ACID & BASE
• Atomicity • Basically Available
• Consistency • Soft state
• Isolation • Eventually consistent
• Durability

82. Photo by Marcus Vegas (http://www.flickr.com/photos/vegas/413159909)

83. Fully ACID Fully BASE
Photo by Marcus Vegas (http://www.flickr.com/photos/vegas/413159909)

84. Fully ACID Fully BASE
Redis
Photo by Marcus Vegas (http://www.flickr.com/photos/vegas/413159909)

85. Fully ACID Fully BASE
CouchDB Redis
Photo by Marcus Vegas (http://www.flickr.com/photos/vegas/413159909)

86. Fully ACID Fully BASE
CouchDB Redis Bigtable
Photo by Marcus Vegas (http://www.flickr.com/photos/vegas/413159909)

87. Questions to ask about data
• Where can I compromise aggregation?
• Where can I compromise where clauses?
• Where can I compromise consistency?
Where can I localize consistency?

88. CAP Theorem
Pick any two:

89. CAP Theorem
Pick any two:
• Consistency

90. CAP Theorem
Pick any two:
• Consistency
• Availability

91. CAP Theorem
Pick any two:
• Consistency
• Availability
• Partition tolerance

92. Summary

93. Summary
Every system at scale will have to
compromise consistency at some level.

94. Summary
Every system at scale will have to
compromise consistency at some level.

95. Summary
Every system at scale will have to
compromise consistency at some level.
Do it mindfully.

96. Questions?
Twitter: @sarahmei
Email: sarah@pivotallabs.com
Slides: http://bit.ly/9xS2PK
Please rate this talk! http://bit.ly/9MCtX9