Convergent Replicated Data Types in Riak 2.0
•Download as PPTX, PDF•
2 likes•1,135 views
Talk by Gordon Guthrie, Senior Software Engineer at Basho Summary A review of the CAP Theorem and the difficulties of resolving conflicts in highly distributed systems. Covering the issues and various theories on how to resolve including the use CRDTs in Riak Details CRDTs are used to replicate data across multiple computers in a network, executing updates without the need for remote synchronisation. This leads to merge conflicts in systems using conventional eventual consistency technology, but CRDTs are designed such that conflicts are mathematically impossible. Under the constraints of the CAP theorem they provide the strongest consistency guarantees for available/partition-tolerant (AP) settings. The CRDT concept was first formally defined in 2007 by Marc Shapiro and Nuno Preguiça in terms of operation commutativity, and development was initially motivated by collaborative text editing. The concept of semilattice evolution of replicated states was first defined by Baquero and Moura in 1997, and development was initially motivated by mobile computing. The two concepts were later unified in 2011. Basho has worked with the EU and Marc Shapiro's team to push CRDTs into distributed systems. Riak v2.x is the first commercial product to include this functionality
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Convergent Replicated Data Types in Riak 2.0
Similar to Convergent Replicated Data Types in Riak 2.0 (20)
More from Big Data Spain
More from Big Data Spain (20)
Recently uploaded
Recently uploaded (20)
Convergent Replicated Data Types in Riak 2.0
- 1. Convergent Replicated Data Types in Riak 2.0 Gordon Guthrie Senior Software Engineer - Basho
- 2. OLAP OLTP Financial s 99.9999% 99.999% 99.99% 99.9% 99% Availability Pressure Consistency Pressure Year End Quarter End Month End Week End End Of Day End Of Hour End Of Minute Popping a Cap in the Enterprise’s Ass Customers Company
- 3. ‘Data Platform’ Riak Riak Available Replication Consistent At A Point In TimeCRDT’s – The Magic Customers Company
- 5. What is it for? • Riak is a highly-available, low-latency KV store • it is an open-source implementation of the Dynamo paper from Amazon • Dynamo underpins the Amazon shopping cart
- 6. Why does it matter? • availability matters • latency matters • Amazon found every 100ms of latency cost them 1% in sales • Google found an extra 0.5 seconds in search page generation time dropped traffic by 20%
- 8. The Ring
- 9. …it’s a hash function… If I have: • a ring of 16 slots • a hashed keyspace of 28 Then the slots are: • Slot 1: 0 – 15 • Slot 2: 16 – 31 • Slot 3: 32 – 47 • etc, etc {key: ‘Alice’, value: ‘hello’} hash(‘Alice’, 28) -> 17 Store it in Slot 2
- 10. Mapped To Machines Physical Machine 1 Physical Machine 2 Physical Machine 3 Physical Machine 4 Physical Machine 5
- 11. Machines Go Down Physical Machine 1 Physical Machine 2 Physical Machine 3 Physical Machine 4 Physical Machine 5
- 12. And Come Back Physical Machine 1 Physical Machine 2 Physical Machine 3 Physical Machine 4 Physical Machine 5
- 13. but that’s not enough That’s how the data survives a box dying
- 14. What happens if the network partitions
- 15. CAP Theorem
- 16. The Cap Theorem Pick 2 Consistency Availability Partition Tolerance Riak
- 17. • In a distributed computer system you trade: Consistency Availability Partition tolerance • Pick two CA (Relational) CP (Mongo, Hbase, Redis) AP (Riak, Cassandra, Couchbase, Dynamo DB) C A P CAP Theorem Relaxed consistency X
- 18. • Think of C, A and P as dials on a dashboard • In a good implementation (like Riak) they can be tuned to achieve the most appropriate balance of consistency, availability and partition tolerance for specific workloads 0 1 2 3 4 A 5 67 8 9 10 11 0 1 2 3 4 P 5 67 8 9 10 11 0 1 2 3 4 C 5 67 8 9 10 11 Tuning the CAP
- 19. Partitions Happen • because Riak is a highly-available data store it will continue to store your data • but your data will not be consistent
- 20. AP Physical Machine 1 Physical Machine 2 Physical Machine 3 Physical Machine 4 Physical Machine 5 Partition Val 1 Val 2
- 21. And Come Back Physical Machine 1 Physical Machine 2 Physical Machine 3 Physical Machine 4 Physical Machine 5
- 22. Now you have siblings Physical Machine 1 Physical Machine 2 Physical Machine 3 Physical Machine 4 Physical Machine 5 {Val 1, Val 2}
- 23. Siblings the developer’s problem that Riak 2.0 is trying to fix
- 24. Sibling Resolution • Developers hate siblings • Developers hate resolving siblings • Developers play Timestamp Roulette and go for Last Write Wins • but with intercontinental lag, NTP failure, clock skew and latency sometimes Timestamps kill your data
- 25. Google’s View • “Designing applications to cope with concurrency anomalies in their data is very error-prone, time- consuming, and ultimately not worth the performance gains.” • “We have a lot of experience with eventual consistency systems at Google.” • “We find developers spend a significant fraction of their time building extremely complex and error-prone mechanisms to cope with eventual consistency”
- 26. The Partition Cycle Consistent Available Available Partitioned Not Consistent Available (Eventually) Consistent Available partitioned unpartitioned Developer Hell Developer Heaven CRDTs
- 27. CRDTs Consistent Replicated Data Types
- 28. Understanding CRDTs • Not going to go into a lot of detail • Based on the idea of Lamport Vector Clocks • A Lamport Vector clock allows an ‘actor’ in a distributed data system to say 2 things: • this is when I changed this data • and when I changed it I knew all the things every other actor had done upto these times • Its a vector clock because it has a separate clock for every actor
- 29. Where Are The Actors? • Actors can be clients-side or server-side or both • Riak is implementing an architecture that allows users to use server-side CRDTs • to avoid the risk of ‘actor’ explosion
- 30. Client Side Actors • Alice’s Laptop • Alice’s phone • Alice’s iPad • Alice’s daughter’s iPad (her battery is flat) • Bob’s new laptop • Bob’s old laptop • etc, etc
- 32. How They Work • Lets take a simple example with two actions: • add an item to a shopping cart • empty the shopping cart • these actions don’t commute: • add item then clear gives an empty cart • empty cart then add item gives a cart with one item in it
- 33. How They Work Consistent Available Available Partitioned Not Consistent Available (Eventually) Consistent Available partitioned unpartitioned CRDTs ClearAdd Item [{a, 🕝}] [{b, 🕟}] {[{a, 🕝}], [{b, 🕟}]} The ‘actor’ clearing the shopping cart didn’t know there was anything in the cart Therefore there should be something in the cart after merging the two updates
- 34. How They Work Consistent Available Available Partitioned Not Consistent Available (Eventually) Consistent Available partitioned unpartitioned CRDTs ClearAdd Item [{a, 🕝}] [{a, 🕝},{b, 🕟}] {[{a, 🕝}], [{a, 🕝},{b, 🕟}]} The ‘actor’ clearing the shopping cart knew there was something in the cart Therefore there should be nothing in the cart after merging the two updates
- 35. Scary Maths Stuff • Join Semi-Lattices that have a Bottom and a Least Upper Bound • that have defined properties involving Associativity, Commutativity, Idempotency • with a Merge Function
- 36. References Your essential CRDT reading list http://christophermeiklejohn.com/crdt/2014/07/22/readings-in-crdts.html
- 37. CRDTs in Riak 2.0
- 38. Native Eventually Consistent Data Types • Flags • Registers • Counters • Sets • Maps • can contain Flags, Registers, Counter and Maps • the basis of composable data types
- 39. Flags • Can have one of two values: • enabled • disabled • Operations: • enable • disable • Examples - use like Booleans • has a tweet been sent? • has the customer signed up to a pricing plan?
- 40. Registers • Named Binaries that have contents which have a value • Operations: • store new value • Examples • store “My New Post” in the field “Blog Post Title”
- 41. Counters • Contain a number which can be incremented or decremented • Operations • increment • decrement • Examples • the number of “likes” in Facebook • the number of Twitter followers
- 42. Sets • Collections of unique binaries • Operations • add an element • remove an element • add a list of elements • remove a list of elements • Examples • shopping cart
- 43. Maps • Maps are like hash maps • Operations • add a named Flag, Register, Counter, Set or Map • remove a named Flag, Register, Counter, Set or Map • pass through an op for an element in the Map • Example • the Map is used to compose the data model
- 44. Questions? Fin