Uploaded byC4Media
A Call for Sanity in NoSQL
The document discusses the complexities and challenges associated with NoSQL databases and schema-less designs, emphasizing the risks of mutability, denormalization, and eventual consistency. It proposes the use of a lambda architecture for better data management and accuracy while highlighting the importance of schema and structured approaches in preventing software bugs. The content emphasizes the necessity of balancing innovation with reliability in software development practices.
More Related Content
Similar to A Call for Sanity in NoSQL
A Call for Sanity in NoSQL
- 1. A call forsanity in NoSQL Nathan Marz @nathanmarz 1
- 2. InfoQ.com: News &Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations /nosql-cons http://www.infoq.com/presentati ons/nasa-big-data http://www.infoq.com/presentati ons/nasa-big-data
- 3. Presented at QConLondon www.qconlondon.com Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide
- 4.
- 5.
- 6.
- 7. The site isdown!
- 8.
- 9. Oops, I forgot theWHERE clause!
- 10.
- 11.
- 14.
- 16.
- 17.
- 18.
- 19.
- 20.
- 21.
- 22.
- 24.
- 25.
- 26.
- 27. ID Name LocationID 1 Sally 3 2 George 1 3 Bob 3 Location ID City State Population 1 NewYork NY 8.2M 2 San Diego CA 1.3M 3 Chicago IL 2.7M Normalized schema
- 28. Join is tooexpensive, so denormalize...
- 29. ID Name LocationID City State 1 Sally 3 Chicago IL 2 George 1 NewYork NY 3 Bob 3 Chicago IL Location ID City State Population 1 NewYork NY 8.2M 2 San Diego CA 1.3M 3 Chicago IL 2.7M Denormalized schema
- 30.
- 31. What is thesource of the insanity?
- 32. Code is deterministic. Iunderstand logic. Therefore, I can write correct code. Programming fallacy
- 33. Your code iswrong
- 34. Your code Dependency 1 Dependency2 Dependency 3
- 35.
- 36. Dependency 4 Dependency 6 Dependency9 Dependency 7 Dependency 8
- 37.
- 38.
- 39.
- 40.
- 41.
- 42. I think Ican safely say that nobody understands quantum mechanics. Richard Feynman
- 43. Your code iswrong
- 44.
- 45.
- 46. All the softwareyou’ve used has had bugs in it
- 47.
- 50. - Mutability - Schemalessdatabases - Eventual consistency / read-repair - Denormalization Insanity
- 51. Person Location Sally NewYork BobChicago Mutability
- 52. Person Location Sally London BobChicago Mutability
- 53. Person Location Time SallyNewYork 1318358351 Bob Chicago 1327928370 Immutability
- 54. Person Location Time SallyNewYork 1318358351 Bob Chicago 1327928370 Sally London 1338273801 Immutability
- 55. All data Precomputed batch view Query Precomputed realtime view Newdata stream Lambda Architecture
- 56.
- 57.
- 58.
- 59. What does adata system do?
- 60. Retrieve data thatyou previously stored? GetPut
- 61. Counterexamples Store location informationon people Where does Sally live? What are the most populous locations? How many people live in a particular location?
- 62. Counterexamples Store pageview information Howmany unique visitors over time? How many pageviews on September 2nd?
- 63. Counterexamples Store transaction historyfor bank account How much money do people spend on housing? How much money does George have?
- 64. What does adata system do? Query = Function(All data)
- 65. Example query Total numberof pageviews to a URL over a range of time
- 66.
- 67.
- 68.
- 69. Precomputed view Example query Alldata Pageview Pageview Pageview Pageview Pageview Query 2930
- 70.
- 71.
- 72.
- 73. Function that takesin all data as input
- 74.
- 75.
- 76. MapReduce is aframework for computing arbitrary functions on arbitrary data
- 77. Alldata Batch view #1 Batchview #2 MapReduce workflow MapReduce workflow MapReduce precomputation
- 78.
- 79.
- 80.
- 81. Not quite... • Abatch workflow is too slow • Views are out of date Absorbed into batch views Not absorbed Now Time Just a few hours of data!
- 82. Precomputation All data Precomputed batch view Query Precomputed realtime view Newdata stream Compute parallel realtime views
- 83. NoSQL databases New datastream Realtime view #1 Realtime view #2 Stream processor
- 84. Precomputation All data Precomputed batch view Query Precomputed realtime view Newdata stream Most complex part of system
- 85. Precomputation All data Precomputed batch view Query Precomputed realtime view Newdata stream But only represents few hours of data
- 86. Precomputation All data Precomputed batch view Query Precomputed realtime view Newdata stream So can be kept small
- 87. Precomputation All data Precomputed batch view Query Precomputed realtime view Newdata stream If anything goes wrong, auto-corrects
- 88. Precomputation All data Precomputed batch view Query Precomputed realtime view Newdata stream “Complexity isolation”
- 89. Precomputation All data Precomputed batch view Query Precomputed realtime view Newdata stream No random writes needed
- 90. All data Precomputed batch view Query Precomputed realtime view Newdata stream “Lambda Architecture”
- 91. Applying the LambdaArchitecture
- 92. Unique visitors overa range of hours A B C A B C D E C D E 1 2 3 4 Hours Pageviews
- 93. A B C A B C D E C D E 1 2 34 Hours Pageviews A = B B = D Equivs Unique visitors over a range of hours
- 94. What should viewlook like?
- 95. Attempt #1 1-1 3 Count 1-26 1-3 6 2-2 4 2-3 5 3-3 1 Hour rangeURL foo.com/blog 1-1 3 1-2 6 1-3 6 2-2 4 2-3 5 3-3 1 bar.com/foo
- 96. Attempt #2 1 3 Count 26 3 6 4 8 5 5 6 1 HourURL foo.com/blog 1 1 2 11 3 23 4 8 5 9 6 0 bar.com/foo
- 97. Attempt #3 1 A,B, C Visitors 2 A 3 D, E 4 A 5 B, C, D 6 F, G HourURL foo.com/blog 1 A 2 B, C 3 A, C 4 D 5 A, E 6 A bar.com/foo
- 98. Attempt #4 1 <HLL> HyperLogLogset 2 <HLL> 3 <HLL> 4 <HLL> 5 <HLL> 6 <HLL> HourURL foo.com/blog 1 <HLL> 2 <HLL> 3 <HLL> 4 <HLL> 5 <HLL> 6 <HLL> bar.com/foo
- 99. Final attempt 1 <HLL> HyperLogLogset 2 <HLL> 3 <HLL> 4 <HLL> 5 <HLL> 6 <HLL> HourURL foo.com/blog 1 <HLL> 2 <HLL> 3 <HLL> 4 <HLL> 5 <HLL> 6 <HLL> bar.com/foo 1 <HLL> HyperLogLog set 2 <HLL> 3 <HLL> 4 <HLL> 5 <HLL> 6 <HLL> MonthURL foo.com/blog 1 <HLL> 2 <HLL> 3 <HLL> 4 <HLL> 5 <HLL> 6 <HLL> bar.com/foo 1 <HLL> HyperLogLog set 2 <HLL> 3 <HLL> 4 <HLL> 5 <HLL> 6 <HLL> WeekURL foo.com/blog 1 <HLL> 2 <HLL> 3 <HLL> 4 <HLL> 5 <HLL> 6 <HLL> bar.com/foo
- 100. Batch workflow All data Normalize pageviewURLs Equiv connected- component labeling Normalize pageview userids Aggregate HyperLogLog sets per bucket Index into batch view
- 101.
- 102. Equiv graph 1 3 4 5 11 2 7 6 9 PersonA Person B
- 103. Equiv graph 1 ->A 2 -> A 3 -> A 4 -> A 5 -> A 11 -> A 6 -> B 7 -> B 9 -> B
- 104. Batch workflow All data Normalize pageviewURLs Equiv connected- component labeling Normalize pageview userids Aggregate HyperLogLog sets per bucket Index into batch view
- 105. Realtime workflow The equivproblem
- 106. The equiv problem 1<HLL> HyperLogLog set 2 <HLL> 3 <HLL> 4 <HLL> 5 <HLL> 6 <HLL> HourURL foo.com/blog 1 <HLL> 2 <HLL> 3 <HLL> 4 <HLL> 5 <HLL> 6 <HLL> bar.com/foo
- 107. The equiv problem 1A, B, C Visitors 2 A 3 D, E 4 A 5 B, C, D 6 F, G HourURL foo.com/blog 1 A 2 B, C 3 A, C 4 D 5 A, E 6 A bar.com/foo
- 108. Solving the equivproblem All data Equiv connected- component labeling Index into batch view of userid -> normalized userid Batch
- 109. Realtime workflow Batch All data Equivconnected- component labeling Index into batch view of userid -> normalized userid Pageviews stream Normalize userid Aggregate realtime HyperLogLog buckets Realtime view
- 110.
- 111.
- 112.
- 113.
- 114.
- 115.
- 116. Conclusions from example -Avoids every single insane complexity I talked about - Powerful to be able to extract more out of a piece of data the longer you have it - Eventual accuracy is a super useful technique - Schemas can be useful and non-painful - A Lambda Architecture is fundamentally easy to extend with new views/queries
- 117.
- 118. Watch the videowith slide synchronization on InfoQ.com! http://www.infoq.com/presentations/nosql- cons