This document summarizes key aspects of distributed databases. It discusses master-slave and multi-master replication approaches. It then covers the challenges of achieving availability, partition tolerance and consistency as defined by Brewer's CAP theorem. The rest of the document dives deeper into data distribution, replication, conflict resolution, membership protocols, and allowing the system to operate during network partitions or node failures. It provides examples of gossip protocols, vector clocks, hinted handoff, and anti-entropy processes used in distributed databases. Finally, it notes that building these systems requires clients to be aware of some internal workings and allows for extra credit in building your own distributed database.

1. todo lo que siempre quisiste
saber sobre:
Bases de datos
distribuídas de alta
disponibilidad
Javier Ramírez
@supercoco9
https://teowaki.com/services
MADRID · NOV 27-28 · 2015

2. IBM Data Center
in Japan during
and after
an earthquake

5. A squirrel did take out half of our
Santa Clara data centre two years back
Mike Christian, Yahoo Director of Engineering

7. Hayastan
Shakarian
a.k.a.
The Spade
Hacker

8. Cut-off
Armenia
from
the Internet
for almost
one day*
* By accident, while scavenging copper

9. I have no idea what
the internet is

10. Some data center outages reported in 2015:
* Amazon Web Services
* Apple iCloud
* Microsoft Azure
* IBM Softlayer
* Google Cloud Platform
* And of course every hosting with scheduled
maintenance operations (rackspace, digital
ocean, ovh...)

11. Complex systems can and will fail

12. You better distribute your data, or else...
Also, distributed databases can perform
better and run on cheaper hardware than
centralised ones

13. Most basic level:
Backup

14. And keep the copy
on a separate data centre*
* Vodafone once lost one year
of data on a fire because of this

15. Next
Level:
Replicas
(master-slave)

16. A main server sends a binary
log of changes to one or more
replicas
* Also known as Write Ahead Log or WAL

17. Master-slave is good but
* All the operations are replicated on all
slaves
* Good scalability on reads, but not on writes
* Cannot function during a network partition
* Single point of failure (SPOF)

18. Next Level:
Multi-Master Cluster
(master-master)

19. Every server can accept reads
or writes, and send its binary
log to all the other servers
* also referred as update-anywhere

20. Multi-master is great, but:
* All the operations are replicated on all masters.
* When synchronous, high latency (Consistency
achieved via locks, coordination and serializable
transactions)
* When asynchronous, typically poor conflict
resolution
*Hard to scale up or down automatically

21. What I want:
* A system that always can work, even with
network partitions
* That scales out both reads and writes
* On cheap commodity diverse hardware
* Running locally to your users (low latency)
* Can grow/shrink elastically and survive
server failures

22. Then you need to let go of
many convenient things you
take for granted in databases

23. Availability
Partition
Tolerance
Consistency
CA
AP
CP
CAP Theorem
Everything is a trade-off

24. Next Level:
Distributed Data
stores

26. Distributed DB design decisions
* data (keys) distribution
* data replication/durability
* conflict resolution
* membership
* status of the other peers
* operation under partitions and
during unavailability of peers
* incremental scalability

27. Data distribution
Consistent hashing based on the key
Usually implies operations work on single keys. Some
solutions, like Redis, allow the clients to group related
keys consistently. Some solutions, like BigTable, allow to
collocate data by group or family.
Queries are frequently limited to query by key or by
secondary indexes (say bye to the power of SQL)

28. Data distribution. The Ring

29. Data Replication
How many replicas of each? Typically at least 3, so in case of
conflicts there can be a quorum
Often, the distribution of keys is done taking into account the
physical location of nodes, so replicas live in different racks or
different datacentres

30. Replication: durability
If we want to have a durable system, we need at least to
make sure the data is replicated in at least 2 nodes before
confirming the transaction to the client.
This is called the write quorum, and in many cases it can be
configured individually.
Not all data are equally important, and not all systems have the
same R/W ratio.
Systems can be configured to be “always writable” or
“always readable”.

31. Conflict resolution
Can be done at Write time or at Read
time.
As long as R + W > N it's possible to
reach a quorum

32. Conflicts
I see a record that I thought was
deleted
I created a record but cannot see it
I have different values in two nodes
Something should be unique, but it's not

33. Conflict resolution strategies
Quorum-based systems: Paxos,
RAFT. Require coordination
of processes with continuous elections
of leaders and consensus.
Worse latency
Last Write Wins (LWW): Doesn't
require coordination. Good latency

34. But, what does “Last” mean?
* Google spanner uses atomic clocks
and servers with GPS clocks to
synchronize time
* Cassandra tries to sync clocks and
divides updates in small parts to
minimize conflict
* Dynamo-like use vector clocks

35. Vector clocks
* Don't need to sync time
* There are several
versions of a same item
* Need consolidation
to prune size
* Usually client needs to
fix the conflict and update

36. Alternatives to conflict resolution
* Conflict-Free-Replicated-Datatypes(CRDT).
Counters, Hashes, Maps
* Allowing for strong consistency on keys from the same
family
* The Uber solution with serialized tokens
* Some solutions are implementing immutability,
so no conflicts
* Peter David Bailis paper on Coordination Avoidance using
Read Atomic Multi-Partition transactions (Nov/15)

37. membership
gossip
infection-like
protocols

38. Gossip
A centralised server is a SPOF
Communicating state with each node is very time consuming
and doesn't support partitions
Gossip protocols communicate pairs of random nodes at
regular frequent intervals and exchange information.
Based on that information exchange, a new status is agreed

39. Gossip example

40. Incremental scalability
When a new node enters the system, the rest of nodes notice
via gossip.
The node claims a partition of the ring and asks
the replicas of the same partition to send data to it.
When the rest of nodes decide (after gossiping) that a node
has left the system and it's not a temporary failure, the data
assigned to the partitions of that node is copied to more
replicas to reach the N copies.
All the process is automatic and transparent.

41. Operation under partition:
Hinted Handoff
On a network partition, it can happen that we have less than
W nodes of the same segment in the current partition.
In this case, the data is replicated to W nodes, even if that
node wasn't responsible for the segment. The data is kept
with a “hint”, and stored in a special area.
Periodically, the server will try to contact the original
destination and will “hand off” the data to it.

42. Operation under partition:
Hinted Handoff

43. Anti Entropy
A system with handoffs can be chaotic and not very
effective
Anti Entropy is implemented to make sure hints are
handed off or synchronized to other nodes
Anti entropy is usually achieved by using Merkle Trees, a
hash of hashes structure very efficient to compare
differences between nodes

44. All this features mean your clients need to
be aware of some internals of the system

45. Clients must
* Know which close nodes are responsible for each
segment of the ring, and hash locally**
* Be aware of when nodes become available or
unavailable**
* Decide on durability
* Handle conflict resolution, unless under LWW
** some solutions offer a load balancer proxy to abstract the client
from that complexity, but trading off latency

46. now you know how it works
* A system that always can work, even with
network partitions
* That scales out both reads and writes
* On cheap commodity diverse hardware
* Running locally to your users (low latency)
* Can grow/shrink elastically and survive
server failures

47. Extra level: Build your
own distributed database
Netflix dynomite, built in Java
Uber ringpop, built in JavaScript

48. Not
Scared
Of You
Anymore

49. aprendoaprogramar.com
… y si tienes hijas o hijos en edad escolar

50. Find related links at
https://teowaki.com/teams/javier-community/link-categories/distributed-systems
Gracias!
Javier Ramírez
@supercoco9
need help with distributed or big data?
https://teowaki.com/services