NoSQL databases, the CAP theorem, and the theory of relativity


Published on

A presentation showing how the CAP theorem causes NoSQL databases to have BASE semantics. That is, they don't support ACID consistency. Then shows how CAP is related to Einstein's theory of relativity. And finally shows how Google Spanner and F1 provide ACID that scales.

Published in: Technology
  • Daren: To be honest I think theoretical physics is way, way beyond how database people approach their theory. It might be that some mathematical formalism may one day come out of databases and find an application in physics (that's happened with pure maths a number of times), but I kind of doubt it.
    Are you sure you want to  Yes  No
    Your message goes here
  • Interesting. I'm coming from physics, with only a minimal superficial understanding of databases. Databases, including NoSQL are becoming important, and I'm needing to learn this stuff to make a new career direction. I can't say I understood all of this presentation, but will certainly keep the URL handy for any fellow physicists interested in these things.

    I have seen how Vector Clocks and related ideas resemble some of the mathematical logic in relativity, so I especially enjoy that point of view in this slide show. Now I wonder if advances in database theory and NoSQL systems will feed fresh insights back to theoretical physics?
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

NoSQL databases, the CAP theorem, and the theory of relativity

  1. 1. NoSQL, CAP, and relativity 2013-09-18 Lars Marius Garshol,, 1
  2. 2. Agenda 2 • CAP theorem mostly • A bit about NoSQL databases • Are triple stores NoSQL? • Connection with Einstein’s theory of relativity • And, finally, a surprise
  3. 3. NoSQL databases 3
  4. 4. What makes a NoSQL database? 4 • Doesn’t use SQL as query language – usually more primitive query language – sometimes key/value only • BASE rather than ACID – that is, sacrifices consistency for availability – much more about this later • Schemaless – that is, data need not conform to a predefined schema
  5. 5. BASE vs ACID • ACID – Atomicity – Consistency – Isolation – Durability • BASE – Basically Available – Soft-state – Eventual consistency 5
  6. 6. Eventual consistency • A key property of non-ACID systems • Means – if no further changes made, – eventually all nodes will be consistent • In itself eventual consistency is a very weak guarantee – when is “eventually”? it doesn’t say – in practice it means the system can be inconsistent at any time • Stronger guarantees are sometimes made – with prediction and measuring, actual behaviour can be quantified – in practice, systems often appear strongly consistent 6
  7. 7. Implementing ev. consistency • Nodes must exchange information about writes – basically, after performing a write, node must inform all other replicas of the changed objects – signal OK to reader before or during replication – for example, by broadcast to all nodes • Must have some way to deal with conflicts – all nodes must agree on conflict resolution – common solution: embed clock value in write message, then let last writer win – clock need not be in sync for all nodes 7
  8. 8. What’s wrong with ACID? • Semantics are easier for developers – applications can lean back and just trust the db • However, doesn’t scale as well – that is, doesn’t scale with number of nodes – requires too much communication and agreement between nodes • Bigger web sites have therefore gone BASE – Facebook, Flickr,Twitter, ... 8
  9. 9. Other benefits of NoSQL • Schemaless – possible to write much more flexible code – schema evolution vastly easier • Avoid joins – document databases allow hierarchical non- normalized objects to be retrieved directly 9
  10. 10. Downsides to NoSQL • Everyone knows SQL – few people know your specific NoSQL database • Lack of validation – code will typically do anything the database lets it get away with (especially over time) • No standards – you can’t easily switch databases – (well, except with SPARQL) • Lack of maturity – lack of supporting tools, unpleasant surprises, ... • Weak query languages – means you have to do more in code – may hurt performance 10
  11. 11. Triple stores (SPARQL) • Non-SQL? Yes • BASE? No • Schemaless? Yes 11 Only two out of three, so whether triple stores are NoSQL databases is debatable. At the very least, they differ substantially from the core examples of NoSQL databases.
  12. 12. Can triple stores be BASE? • In theory, yes – nothing inherent in graph structure that prevents it • But, – how do you shard graph data? – no known way to do it that’s efficient • So, in practice this is hard 12
  13. 13. When should you use NoSQL? • If scalability is a concern – however, don’t forget that sites like Flickr and Wikipedia used RDBMSs for years – relational databases scale a long way • If schemalessness is important – sometimes it really is – seriously consider RDF for this use case • If fashion is a concern – for a surprising number of people, using something new and shiny is the main thing 13
  14. 14. The CAP Theorem 14
  15. 15. CAP 15 • Consistency – all nodes always give the same answer • Availability – nodes always answer queries and accept updates • Partition-tolerance – system continues working even if one or more nodes go quiet CAPTheorem:You can only have two of these. Partition tolerance: Without this, the cluster dies the moment one node goes silent. Can’t really drop this one.
  16. 16. C ≠ C 16 • C in ACID – means all data obeys constraints in schema • C in CAP – means all servers agree on the data • However, – ACID implementations also follow the C in CAP
  17. 17. History 17 • First formulated by Eric Brewer in 2000 – based on experience with Inktomi search engine – described the SQL/NoSQL divide very well – coined the BASE acronym • Formalized and proven in 2002 – by Seth Gilbert and Nancy Lynch • Today CAP is better understood – widely considered a key tradeoff in designing distributed systems – and particularly databases – in some ways gave rise to NoSQL databases
  18. 18. Consistent or Available? 18 request request request
  19. 19. Consistency 19 DB node 1 DB node 2Client 1 Client 2 read account X balance -> 100 set account X balance = 0 set account X balance = 0 set account X balance = 0 set account X balance = 0
  20. 20. Availability 20 DB node 1 DB node 2Client 1 Client 2 read account X balance -> 100 set account X balance = 0 set account X balance = 0 set account X balance = 0 read account X balance -> 100 set account X balance = 0 Happy customer walks away, richer by 200. Servers eventually agree balance is 0.
  21. 21. What exactly is the problem? 21 • The ordering of events affects the outcome • The different nodes do not necessarily observe the same order • However, it is possible to impose a consistent ordering of events • The trouble is, that involves communication with all nodes • Waiting for all of them takes time
  22. 22. Math digression 22
  23. 23. Time, Clocks and the Ordering of Events in a Distributed System 23 The origin of this paper was a note titled The Maintenance of Duplicate Databases by Paul Johnson and BobThomas. I believe their note introduced the idea of using message time- stamps in a distributed algorithm. I happen to have a solid, visceral understanding of special relativity (see [5]). This enabled me to grasp immediately the essence of what they were trying to do. ... I realized that the essence of Johnson andThomas's algorithm was the use of timestamps to provide a total ordering of events that was consistent with the causal order. ... It didn't take me long to realize that an algorithm for totally ordering events could be used to implement any distributed system.
  24. 24. Order theory 24 • An ordering relation is any relation ≤ such that – a ≤ a (reflexivity) – if a ≤ b and b ≤ a then a = b (antisymmetry) – if a ≤ b and b ≤ c then a ≤ c (transitivity) • A total order is an order such that – a ≤ b or b ≤ a (totality) • A partial order is any order which is not total – that is, for some pairs a and b, neither a ≤ b nor b ≤ a
  25. 25. Examples 25 • Total orders – normal ordering of numbers and letters • Partial orders – ordering sets by the subset relation (see figure) – “In general, the ·order-relation· on duration is a partial order since there is no determinate relationship between certain durations such as one month (P1M) and 30 days (P30D)” XML Schema, pt2
  26. 26. Relativity 26
  27. 27. History 27 • 1687 – Isaac Newton publishes Philosophiæ Naturalis Principia Mathematica – physics begins – no changes over next two centuries • 1905 – Albert Einstein publishes special relativity – abandons notions of fixed time and space • 1916 – Einstein’s general relativity – takes into account gravity – no changes since
  28. 28. 28 ... (we skip 270 slides)
  29. 29. The barn is too small 29 • Three people (M, F, and B) own a board (5m wide) and a barn (4m wide) • The board doesn’t fit inside the barn! • What to do? 4
  30. 30. 30
  31. 31. We have a solution! 31 As seen by F & B: board is 4m long (relativistic shortening), barn continues to be 4m wide. When the board is exactly inside the barn, F and B will close their doors simultaneously, and the problem will be solved. (As seen by M: board is 5m long (at rest relative to him), barn shortened to 3.2m wide. Pay no attention to this.)
  32. 32. What they observe F and B • When the board is just inside, both close their doors simultaneously • Right after, the board crashes through back door M • B shuts his door just as the front of the board reaches him • 0.6 seconds later, F closes his door • Board crashes through back door 32
  33. 33. The key point • They don’t agree on the order of events! – and this is not a paradox – it is in fact how the universe works • Change the story slightly and the three people could have three different orders of events • No total order of events exists on which all observers can agree – the ordering of events in the universe is a partial order • What then of causality? – ifA causes B, but some people think B happened beforeA, then what? 33
  34. 34. Resolution 34 • A, B, and C are events • The cone is the “light cone” from A – that is, the spread of light fromA • C is outside the cone – thereforeA cannot influence C – observers may disagree on order of A&C • B is inside the cone – thereforeA can influence B – observers may not disagree on order
  35. 35. 35 ... (we skip 532 slides)
  36. 36. Back to CAP 36
  37. 37. Relevance to the CAP theorem 37 • Distributed nodes will never agree on the order of events – unless a communication delay is introduced • Basically, only events inside the “light cone” can be totally ordered – communications delay can never be less than time taken by light to traverse physical distance – in practice, it will be quite a bit bigger – how big depends on hardware and design constraints
  38. 38. One solution: Paxos 38 • Protocol created by Leslie Lamport • Can be used to introduce logical clock – all nodes agree to always increase the number • Which again can order events • Allowing all nodes to agree on order of events
  39. 39. Another solution: ev. consistency 39 • Essentially, this solution says some % of errors is acceptable – Amazon may have to compensate purchasers with a gift card once every 100,000 transactions – business value of remaining available is higher than cost of errors • Even in banking this may apply – ATMs often continue working even if contact with bank is lost – allow withdrawals up to some limit – accept it if customers overcharge – (they’ll have to pay fees and interest, anyway)
  40. 40. CALM 40 • Client code may have to compensate for database inconsistencies – this can quickly become complex – complex means error-prone • However, there is a way around that – CALM = consistency as logical monotonicity – means facts used by clients to make decisions never change • A database that never deletes or overwrites is CALM – for example because it logs trades or other events – non-CALM databases may require ACID
  41. 41. A CALM example 41 • Client 1 reads A = 10 • Client 1 uses this information to write B = 5 • If Client 2 now reads B = 5, that client cannot read a value of A older than A = 10 • Doing so would violate what’s known as “causal consistency”
  42. 42. ACID 2.0 42 • A bit misleading, because it’s not really ACID at all • Basically requires update operations to have these properties – associativity a + (b + c) = (a + b) + c – commutativity a + b = b + a – idempotence f(x) = f(f(x)) – distributed • One approach is to use datatypes which guarantee these properties
  43. 43. SDShare updates 43 • These are actually – associative – commutative – idempotent • So SDShare is already ACID 2.0...
  44. 44. Another solution: datatypes 44 • CRDTs – commutative, replicated data types • Basically, – data types designed so that the order of operations doesn’t matter – the end result is always the same, regardless of the order of operations
  45. 45. An example 45 • A problem with our cash withdrawal example is the overwrite operation • What if the operation were “increment(account, -100)” instead? – this operation is associative and commutative – (not inherently idempotent, however) • Nodes can now apply incoming updates as they get them – the ordering of updates can be ignored – once all updates are applied, all nodes will agree (which is eventual consistency)
  46. 46. 46 Thus far, all is well, and everyone agrees
  47. 47. 47 “To go wildly faster, one must remove all four sources of the overhead discussed above.This is possible in either a SQL context or some other context.”
  48. 48. 48 What on earth is he talking about? Everyone knows SQL and ACID don’t scale!
  49. 49. Meanwhile, at Google... 49
  50. 50. AdWords database difficulties 50 This backend was originally based on a MySQL database that was manually sharded many ways.The uncompressed dataset is tens of terabytes, which is small compared to many NoSQL instances, but was large enough to cause difficulties with sharded MySQL.The MySQL sharding scheme assigned each customer and all related data to a fixed shard.This layout enabled the use of indexes and complex query processing on a per-customer basis, but required some knowledge of the sharding in application business logic. Resharding this revenue-critical database as it grew in the number of customers and their data was extremely costly. The last resharding took over two years of intense effort, and involved coordination and testing across dozens of teams to minimize risk.
  51. 51. More background 51 We store financial data and have hard requirements on data integrity and consistency.We also have a lot of experience with eventual consistency systems at Google. In all such systems, we find developers spend a significant fraction of their time building extremely complex and error-prone mechanisms to cope with eventual consistency and handle data that may be out of date. We think this is an unacceptable burden to place on developers and that consistency problems should be solved at the database level.
  52. 52. Yet more background 52 At least 300 applications within Google use Megastore (despite its relatively low per- formance) because its data model is simpler to manage than Bigtable’s, and because of its support for synchronous replication across datacenters. (Bigtable only supports eventually-consistent replication across data- centers.) Examples of well-known Google applications that use Megastore are Gmail, Picasa, Calendar, Android Market, and AppEngine.
  53. 53. Requirements • Scalability – scale simply by adding hardware – no manual sharding • Availability – no downtime, for any reason • Consistency – fullACID transactions • Usability – full SQL with indexes 53 Uh, didn’t we just learn that this is impossible?
  54. 54. Spanner 54 • Globally-distributed semi-relational db – SQL as query language – versioned data with non-locking read-only transactions • Externally consistent reads/writes • Atomic schema updates – even while transactions are running • High availability – experiment: killing 25 out of 125 servers has no effect (except on throughput)
  55. 55. Transaction model 55 • Fairly close to traditional MVCC – every row has a timestamp – reads have associated timestamp, see database as of that point in time • The key is a consistent order of timestamps across nodes
  56. 56. Spanner architecture 56
  57. 57. Spanner architecture #2 57
  58. 58. TrueTime 58 • Enables consistency in Spanner by giving transactions timestamps – that is, imposes a consistent ordering on transactions • Represents time with uncertainty interval – the bigger the uncertainty, the more careful nodes must be – bigger uncertainty leads to slower transactions • Uses two kinds of time servers to reduce uncertainty – GPS-based servers – atomic clocks
  59. 59. Use of Paxos is key • Combines Paxos withTrueTime to ensure timestamps are monotonically increasing • Paxos requires majority votes to agree – implies less than half of data centers can fail at any one time • AdWords therefore runs with 5 data centers – allows two simultaneous failures without effect – three on East Coast, two onWest Coast – (in Google East Coast +West Goast = globally) 59
  60. 60. Data model 60
  61. 61. F1 – the next layer up 61 • Builds on Spanner, adds – distributed SQL queries – including joins from external sources – transactionally consistent indexes – asynchronous schema changes – optimistic transactions – automatic change history
  62. 62. Why built-in change history? 62 “Many database users build mechanisms to log changes, either from application code or using database features like triggers. In the MySQL system that AdWords used before F1, our Java application libraries added change history records into all transactions.This was nice, but it was inefficient and never 100% reliable. Some classes of changes would not get history records, including changes written from Python scripts and manual SQL data changes.” Application code is not enough to enforce business rules, because many important changes are made behind the application code. For example, data conversion. Look at any database that’s a few years old, and you’ll find data disallowed by the application code, but allowed by the schema.
  63. 63. Distributed queries 63
  64. 64. Two interfaces • NoSQL interface – basically a simple key->row lookup – simpler in code for object lookup – faster because no SQL parsing • Full SQL interface – good for analytics and more complex interactions 64
  65. 65. Status • >100 terabyte of uncompressed data – distributed across 5 data centers – Five nines (99.999%) uptime • Serves up to hundreds of thousands of requests/second • SQL queries scan trillions of rows/day • No observable increase of latency compared to MySQL-based backend – but change tracking and sharding now invisible to application 65
  66. 66. Winding up 66
  67. 67. Conclusion 67 • NoSQL is mostly about BASE – to some degree also schemalessness • The CAPTheorem is key to understanding distributed systems – NoSQL is BASE because of CAP • The CAPTheorem is a consequence of the theory of relativity • New systems seem to indicate that ACID may scale, after all – basically, the speed of light is greater than we thought
  68. 68. Further reading 68 • NoSQL eMag, InfoQ, pilot issue May 2013 – • Brewer’s original presentation – 2004/PODC-keynote.pdf • Proof by Lynch & Gilbert – -SigAct.pdf • Why E=mc2?, Cox & Forshaw • Eventual ConsistencyToday: Limitations, Extensions, and Beyond, ACM Queue –
  69. 69. Further reading 69 • Spanner paper – • F1 papers – –