In this talk we'll discuss technical foundations behind Conflict-free Replicated Data Types (CRDT), which let us create collaborative client applications - systems where no reliance on central servers and offline-first capabilities are one of the founding principles. We'll cover some of the challenges bound to this approach and how to address them. Finally we'll present Yrs - Rust library, that allows us to build rich collaborative applications on desktop and browser.

1. RICH COLLABORATIVE DATA
STRUCTURES FOR EVERYONE

2. INTRODUCTION
Bartosz Sypytkowski
▪ @Horusiath
▪ b.sypytkowski@gmail.com
▪ bartoszsypytkowski.com

3.  Motivation & challenges
 Timestamps and concurrency tracking
 Deterministic conflict resolution
 Optimizations
AGENDA

4. PROBLEM STATEMENT
COLLABORATIVE TEXT EDITOR

5. COLLABORATIVE
TEXT EDITOR
Alice
a
Carol
a
Bob
a

6. COLLABORATIVE
TEXT EDITOR
Alice
a
Carol
a
Bob
a
insert(1, ‘b’) insert(1, ‘c’)

7. COLLABORATIVE
TEXT EDITOR
Alice
a
Carol
ac
Bob
ab

8. COLLABORATIVE
TEXT EDITOR
Alice
a
Carol
ac
Bob
ab
insert(1, ‘b’)
insert(1, ‘c’)

9. COLLABORATIVE
TEXT EDITOR
Alice
abc
Carol
abc
Bob
acb

10. MAYBE JUST SYNCHRONIZE VIA CENTRAL SERVER?

11. SERVER-DRIVEN
TEXT EDITOR
Alice
a
Carol
a
Bob
a
server
a

12. SERVER-DRIVEN
TEXT EDITOR
Alice
a
Carol
a
Bob
a
server
a
insert(1, ‘b’) insert(1, ‘c’)

13. SERVER-DRIVEN
TEXT EDITOR
Alice
a
Carol
a
Bob
a
server
a
insert(1, ‘b’) insert(1, ‘c’)
E1 E2

14. SERVER-DRIVEN
TEXT EDITOR
Alice
a
Carol
a
Bob
a
server
ab
E2
insert(1,
‘b’)

15. SERVER-DRIVEN
TEXT EDITOR
Alice
ab
Carol
ab
Bob
ab
server
acb
insert(1,
‘c’)

16. SERVER-DRIVEN
TEXT EDITOR
Alice
acb
Carol
acb
Bob
acb
server
acb

17. COMPAR
ING
VECTOR
CLOCKS
ISSUES
1. Latency
2. Online-only / Network issues
3. Characters interleaving
4. Server: bottleneck & point of failure

18. FIELDS SHARING
SIMILAR
PROBLEMS
1. Collaborative text editors
2. Cross-continental data replication
3. Vehicle apps
4. Remote areas

19. CONFLICT AVOIDANCE
“Let the majority decide on
the correct order.”
Decisions are made by
quorum

20. CONFLICT AVOIDANCE CONFLICT RESOLUTION
“Let the majority decide on
the correct order.”
“Given enough context
everyone should come to the
same conclusion.”
Decisions are made by
quorum
Decisions are made
individually

21. CONFLICT RESOLUTION
IT’S ALL ABOUT PRESERVING THE ORIGINAL INTENT

22. DON’T USE INDEXES FOR POSITIONING
THEIR MEANING CHANGES OVER TIME

23. LAMPORT CLOCK
A:1
Unique peer ID
Sequence
Number

24. CONTEXT-AWARE
INSERTION
insert_between(A:1, B:1, (B:2, ‘b’))

25. CONTEXT-AWARE
INSERTION
insert_between(A:1, B:1, (B:2, ‘b’))
inserted block ID
inserted block
content

26. CONTEXT-AWARE
INSERTION
insert_between(A:1, B:1, (B:2, ‘b’))
left neighbor
right neighbor
inserted block ID
inserted block
content

27. CONFLICT RESOLUTION

28. A:1
ID
NULL
LEFT
NULL
RIGHT
CONTENT
“a”
STRING
B:1
ID
A:1
LEFT
NULL
RIGHT
CONTENT
“d”
STRING
Document state
B:2
ID
A:1
LEFT
B:1
RIGHT
CONTENT
“c”
STRING
A:2
ID
A:1
LEFT
B:1
RIGHT
CONTENT
“b”
STRING
CONFLICT
RESOLUTION

29. A:1
ID
NULL
LEFT
NULL
RIGHT
CONTENT
“a”
STRING
B:1
ID
A:1
LEFT
NULL
RIGHT
CONTENT
“d”
STRING
Document state
B:2
ID
A:1
LEFT
B:1
RIGHT
CONTENT
“c”
STRING
A:2
ID
A:1
LEFT
B:1
RIGHT
CONTENT
“b”
STRING
CONFLICT
RESOLUTION

30. A:1
ID
NULL
LEFT
NULL
RIGHT
CONTENT
“a”
STRING
B:1
ID
A:1
LEFT
NULL
RIGHT
CONTENT
“d”
STRING
Document state
B:2
ID
A:1
LEFT
B:1
RIGHT
CONTENT
“c”
STRING
A:2
ID
A:1
LEFT
B:1
RIGHT
CONTENT
“b”
STRING
CONFLICT
RESOLUTION
?

31. A:1
ID
NULL
LEFT
NULL
RIGHT
CONTENT
“a”
STRING
B:1
ID
A:1
LEFT
NULL
RIGHT
CONTENT
“d”
STRING
Document state
B:2
ID
A:1
LEFT
B:1
RIGHT
CONTENT
“c”
STRING
A:2
ID
A:1
LEFT
B:1
RIGHT
CONTENT
“b”
STRING
CONFLICT
RESOLUTION
Use block IDs to skip over blocks
with lower precedence

32. A:1
ID
NULL
LEFT
NULL
RIGHT
CONTENT
“a”
STRING
B:1
ID
A:1
LEFT
NULL
RIGHT
CONTENT
“d”
STRING
Document state
B:2
ID
A:1
LEFT
B:1
RIGHT
CONTENT
“c”
STRING
A:2
ID
A:1
LEFT
B:1
RIGHT
CONTENT
“b”
STRING
CONFLICT
RESOLUTION

33. OPTIMIZATIONS
CAUSE METADATA CAN BE HEAVY

34. OPTIMIZATIONS
BLOCK MERGING
A:1
ID
NULL
LEFT
NULL
RIGHT
CONTENT
“a”
STRING
A:2
ID
A:1
LEFT
NULL
RIGHT
CONTENT
“b”
STRING
Document state

35. OPTIMIZATIONS
BLOCK MERGING
A:1
ID
NULL
LEFT
NULL
RIGHT
CONTENT
“a”
STRING
A:2
ID
A:1
LEFT
NULL
RIGHT
CONTENT
“b”
STRING
Document state
Both blocks have sequential IDs

36. OPTIMIZATIONS
BLOCK MERGING
A:1
ID
NULL
LEFT
NULL
RIGHT
CONTENT
“a”
STRING
A:2
ID
A:1
LEFT
NULL
RIGHT
CONTENT
“b”
STRING
Document state
Block was intended to be placed
sequentially

37. OPTIMIZATIONS
BLOCK MERGING
A:1
ID
NULL
LEFT
NULL
RIGHT
CONTENT
“ab”
STRING
Document state
Block A:1 is responsible for holding 2
elements now (range from A:1 to A:2)

38. OPTIMIZATIONS
BLOCK MERGING
A:1
ID
NULL
LEFT
NULL
RIGHT
CONTENT
“ab”
STRING
A:3
ID
A:2
LEFT
NULL
RIGHT
CONTENT
“c”
STRING
Document state
Next block ID = last block ID + last
block length
insert_between(A:2, NULL, (A:3, ‘c’))

39. OPTIMIZATIONS
BLOCK MERGING
A:1
ID
NULL
LEFT
NULL
RIGHT
CONTENT
“ab”
STRING
A:3
ID
A:2
LEFT
NULL
RIGHT
CONTENT
“c”
STRING
Document state
Next block ID = last block ID + last
block length
insert_between(A:2, NULL, (A:3, ‘c’))
Left neighbor point to last ID
not block ID

40. WHAT IF WE WANT TO INSERT ELEMENT WITHIN BLOCK?

41. A:1
ID
NULL
LEFT
NULL
RIGHT
CONTENT
“helo”
STRING
A:5
ID
A:3
LEFT
A:4
RIGHT
CONTENT
“l”
STRING
Document state
insert_between(A:3, A:4, (A:5, ‘l’))
OPTIMIZATIONS
BLOCK SPLITTING

42. A:4
ID
A:3
LEFT
NULL
RIGHT
CONTENT
“o”
STRING
A:5
ID
A:3
LEFT
A:4
RIGHT
CONTENT
“l”
STRING
Document state
insert_between(A:3, A:4, (A:5, ‘l’))
OPTIMIZATIONS
BLOCK SPLITTING
A:1
ID
NULL
LEFT
A:4
RIGHT
CONTENT
“hel”
STRING
Split blocks to create space

43. A:4
ID
A:3
LEFT
NULL
RIGHT
CONTENT
“o”
STRING
A:5
ID
A:3
LEFT
A:4
RIGHT
CONTENT
“l”
STRING
Document state
OPTIMIZATIONS
BLOCK SPLITTING
A:1
ID
NULL
LEFT
A:4
RIGHT
CONTENT
“hel”
STRING

44. COMPAR
ING
VECTOR
CLOCKS
OTHER STUFF
1. Tombstones
2. Other data types eg. Maps, Counters
3. Undo/Redo
4. Transactions

45. SUMMARY

46.  Yjs/Yrs: https://crates.io/crates/yrs
 Automerge: https://automerge.org/
 Ditto.live: https://www.ditto.live/
 RiakDB: https://riak.com/
 Amazon DynamoDB: https://aws.amazon.com/dynamodb/
 AntidoteDB: https://www.antidotedb.eu/
CRDT PROJECTS

47.  CRDTs deep dive: https://bartoszsypytkowski.com/tag/crdt/
 List of aggregated CRDT articles: https://crdt.tech
 Making CRDTs faster: https://josephg.com/blog/crdts-go-brrr/
REFERENCES

48. THANK YOU