Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (15)
Similar to Data Consistency in Distributed Systems with Akka Distributed Data
Similar to Data Consistency in Distributed Systems with Akka Distributed Data (20)
Recently uploaded
Recently uploaded (20)
Data Consistency in Distributed Systems with Akka Distributed Data
- 1. Data Consistency in Distributed Systems with Akka DData by Dmitry Martyanov January, 2017
- 2. Preamble • Shared Mutable State
- 3. Preamble • Shared Mutable State • Locks
- 4. Preamble • Shared Mutable State • Locks • ACID
- 5. Preamble • Shared Mutable State • Locks • ACID • Scalability, Elasticity, Remote Deployment and so on
- 6. Preamble • Shared Mutable State • Locks • ACID • Scalability, Elasticity, Remote Deployment and so on • Eventual Consistency
- 7. Preamble • Shared Mutable State • Locks • ACID • Scalability, Elasticity, Remote Deployment and so on • Eventual Consistency • Affinity (sticky sessions) and Read-my-Writes
- 8. Preamble • Shared Mutable State • Locks • ACID • Scalability, Elasticity, Remote Deployment and so on • Eventual Consistency • Affinity (sticky sessions) and Read-my-Writes • Data Management Solutions
- 9. Preamble • Shared Mutable State • Locks • ACID • Scalability, Elasticity, Remote Deployment and so on • Eventual Consistency • Affinity (sticky sessions) and Read-my-Writes • Data Management Solutions • Event Sourcing
- 10. Problem Definition • cAP is not about consistency • Even with Immediate consistency Concurrent Access is essentially clients problem • In XDC fault tolerant environment assumptions are very expensive
- 11. CRDT
- 12. CRDT Conflict-free replicated data types • Every operation is a transformation from previous state to new one • Data type + rules
- 13. CRDT refresh math • Commutativity: a + b = b + a • Associativity: (a + b) + c = a + (b + c) • Idempotence: f(x) = f(f(x))
- 14. CmRDT commutative Conditions: + Commutativity + Associativity + ??? - Idempotence x1 x2 x3 f(x1) f(x2) f(x3) g(x2) g(x2) g(x2)
- 15. CmRDT commutative Conditions: + Commutativity + Associativity + Exactly once delivery - Idempotence x1 x2 x3 f(x1) f(x2) f(x3) g(x2) g(x2) g(x2)
- 16. CvRDT convergent Conditions: + Commutativity + Associativity + Idempotence - Exactly once delivery x1 x2 x3 f(x1) merge merge g(x2) merge merge
- 17. CRDT examples • G-Counter (m) • PN-Counter • LWW-Register • MV-Register • G-Set • 2P-Set • OR-Map • LWW-Set • OR-Set • …
- 19. Causality • Vector Clock (https://en.wikipedia.org/wiki/Vector_clock) • Version Vector (https://en.wikipedia.org/wiki/Version_vector) • Dotted Version Vector (https://github.com/ricardobcl/Dotted-Version- Vectors)
- 20. Causality DVV
- 21. Causality vs CRDT • CRDT to achieve convergence • Causality to achieve correct overriding
- 22. CRDT implementations Riak KV (http://docs.basho.com/riak/kv/2.2.0/developi ng/data-types/) Akka Distributed Data (http://doc.akka.io/docs/akka/2.4.16/scala/dis tributed-data.html)
- 24. Akka Cluster • Actors => • Akka Remote => • Akka Cluster • Gossip protocol • Failure detector • Singleton • Leader • Shard Coordinator 1 3 2 5 4 6 7
- 25. Akka Distributed Data • akka.cluster.ddata.Replicator • akka.cluster.ddata.DistributedData • akka.cluster.ddata.ReplicatedData
- 26. ddata.Replicator Subscribe[A <: ReplicatedData](key: Key[A], subscriber: ActorRef) Changed[A <: ReplicatedData](key: Key[A])(data: A) Get[A <: ReplicatedData](key: Key[A], consistency: ReadConsistency, request: Option[Any] = None) Update[A <: ReplicatedData](key: Key[A], writeConsistency: WriteConsistency, request: Option[Any])( val modify: Option[A] => A) Delete[A <: ReplicatedData](key: Key[A], consistency: WriteConsistency)
- 27. ddata.Replicator class ReplicatorDemo extends Actor with ActorLogging { val replicator = DistributedData(context.system).replicator implicit val node = Cluster(context.system) val DataKey = ORSetKey[String]("key") replicator ! Subscribe(DataKey, self) override def receive: Receive = { case AddString(str) => replicator ! Update(DataKey, ORSet.empty[String], WriteLocal)(_ + str) case RemoveString(str) => replicator ! Update(DataKey, ORSet.empty[String], WriteLocal)(_ - str) case c @ Changed(DataKey) => val data = c.get(DataKey) println(data.elements.mkString("; ")) } }
- 28. ddata.ReplicatedData trait ReplicatedData { type T <: ReplicatedData def merge(that: T): T }
- 29. ddata.ReplicatedData case class TwoPhaseSet( adds: GSet[String] = GSet.empty, removals: GSet[String] = GSet.empty) extends ReplicatedData { type T = TwoPhaseSet def add(element: String): TwoPhaseSet = copy(adds = adds.add(element)) def remove(element: String): TwoPhaseSet = copy(removals = removals.add(element)) def elements: Set[String] = adds.elements diff removals.elements override def merge(that: TwoPhaseSet): TwoPhaseSet = copy( adds = this.adds.merge(that.adds), removals = this.removals.merge(that.removals)) }
- 30. Summary
Editor's Notes
- Motivation. We all know that the root of most of problems in software is shared mutable state, it impacts on accidental complexity and as the result on scalability, correctness and cost of maintenance. Out of the Tar Pit is a very good paper about it. Recommended.
- As this problem appeared because of hardware architecture at the beginning of the era of programming, the most straightforward and durable solution was introduced. This solution is to use locks. There is variety of different locks, they are validated, they work pretty well, on every interview you would be asked about locks and so on.
- And the main philosophy in terms data persistence for system with locks is ACID. Atomicity Consistency Isolation Durability, which implies transactional data modification. It is what most of relational databases provide for you.
- But for last years there was a significant growth of internet traffic and new requirements have been appeared like scalability, elasticity. This gave birth to the need for distributed systems. And locks in distributed systems doesn’t work so well, they are slow, transactions fail often, overlaps happen - they limit your scalability.
- And usually at this moment someone in your company who is very smart and top talented like of course everyone in this room, suggests to use eventual consistency instead. And a lot of teams stop after this change, because it works fine. If your cluster replication happens faster than updates occur everything is fine, I like to call it good weather. But you need to keep in mind that your shared state become distributed state. And this fact come back when you start actually utilise your requirements. In highload environment we see that cluster works even worse than it was before, instead of transaction fails we have data losses and so on. We see some degree of divergence in our data, we are losing trust to our system. To solve it we begin to introduce some ad-hoc solutions which make your system even more complex
- Affinity is a tricky thing, in some case it helps you to increase predictivity, but you need to remember as much as it helps you as less your system really distributed
- Data management with coordinators has a lack of fault tolerance, limitation in elasticity
- Event sourcing is the 3rd solution to deal with distributed state, actually insert only strategy works fine in terms of shared state, but in my practise it works really bad in terms of business compatibility. The gap between you and business will grow significantly if you go with ES. By the way, has anybody already submitted any business application working in live based on event sourcing ? Are you proud of it ?
- So problem definition: Eventual Consistency is not about consistency under high load, when we talk about CAP theorem and choose AP we drop consistency at all. It doesn’t mean that we shift from immediate to eventually, we are dropping consistency at all. Even with Immediate consistency concurrency is clients problem In XDC fault tolerant environment assumptions are very expensive
- For persons who work in Finance, Banking or Insurance probably it is now new that there is like hype for last years about blockchain as technology. I am personally quite sceptical about it and I would like to share with you why. Blockchain is a technology that guarantees exactly once delivery in distributed environment without any assumption, which is very important. Because it is difficult to guarantee this property in some corner cases, like node initialisation, node showdown and so on, connection lost. But blockchain does it. So blockchain is an ingredient for this formula, if you have other 2 and you need exactly once delivery blockchain, it will help you.
- G-Counter - grow only counter like vector clock (counter per actor and merge all counters) PN-Counter - increment/decrement counter is comprised from 2 G-Counter LWW- Register - last write win register, based on timestamp MV-Register(LWWMAp) - multi value register G-Set - grow only set 2P-Set - once add once remove Set (2 G-Set) OR-Set - Observed-Removed Set - add/remove set with causality
- Dotted version allow to avoid false positives.
- Akka Distributed Data. So there is short disclaimer here, I am not an Akka Cluster advocate, please don’t ask me questions about it :)
- Akka Cluster. So for those who are new, akka is a framework that wraps an actor model, it implies the management of some lightweight logical units(actors) which can communicate to each other within message passing(Communicating Sequential Processes by Tony Hoar). All of this allows to achieve highly concurrent environment with minimum of side effects(Sounds like silver bullet - I know :) ). The next step was to create a capability to utilise more than 1 JVM for your actor system, it was introduction of akka remoting. And the current state is Akka Cluster which encapsulate all best practices related to issues existing with akka remote. Gossip protocol - to have all the nodes known about others Failure Detector - to detect whether the node is unreachable or it is just network interruption Singleton - for different single point management entities Leader - to include / exclude nodes from cluster Sharding Coordinator - to coordinate creation of stateful actors within each node of the cluster. This is exactly what we are going to talk about, because the idea is that it is very expensive and slow to communicate with remote node for CRUD operations, we would like to have a local copy with is somehow synchronized between all nodes. Here is a moment when data consistency comes in.
- So there is 3 main components for providing an access to DIstributed Data Replicator is an actor providing Access API to distributed KV store DistributedData is an Akka systen extension to create Replicators ReplicatedData is an abstract trait required to be mixed into your data structures to make the convergeable.
- Here I show you briefly what are the methods: Subscribe & Change - allows you to subscribe an actor of all the changes within the Key, subscribers receive Change message from replicator when the key was changed within replication. Get & Update & Delete - classical CRUD operations with the key, there is Consistency mode which allow you to leverage how many nodes will be reached. Replicator will reply to any of this message, this is exactly your consistency model.
- ReplicatedData is a basic trait for your distributed data types, it defines merge function, which extends your data to convergent CRDT.