Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Grokking #9
Building an offline & real-time
editing service with Couchbase
Vu Nguyen
vu.nguyen@will.vn
In this talk
You’ll walk away with
• Our approach for an unknown architecture
• How do we design architecture for real-tim...
Talk structure
1. Problems of a real-time & offline editing service
2. Evaluate real-time frameworks and databases
3. Firs...
A real-time and offline editing service
What is it?
A real-time and offline editing services
1. Users can edit at the same time
2. Can work without the internet
3. Synchroniz...
Evernote does not resolve conflict!
Hackpad disable editing while offline!
And Asana!
First problem:
How to handle conflict?
A real-time and offline editing service
Google Docs
Google Docs
Google Docs
Operational transformation
0 , 0
1 , 0 0 , 1
1 , 1
2 , 1 1 , 2
2 , 2
2 , 0 0 , 2
client
Operational transformation
0 , 0
1 , 0 0 , 1
1 , 1
2 , 1 1 , 2
2 , 2
2 , 0 0 , 2
client server
Operational transformation
0 , 0
1 , 0 0 , 1
1 , 1
2 , 1 1 , 2
2 , 2
2 , 0 0 , 2
client server
Operational transformation
0 , 0
1 , 0 0 , 1
1 , 1
2 , 1 1 , 2
2 , 2
2 , 0 0 , 2
client server
Operational transformation
0 , 0
1 , 0 0 , 1
1 , 1
2 , 1 1 , 2
2 , 2
2 , 0 0 , 2
client server
Conflict!
Operational transformation
0 , 0
1 , 0 0 , 1
1 , 1
2 , 1 1 , 2
2 , 2
2 , 0 0 , 2
client server
Conflict!
Example for operational transformation
client server
“ABCDE” “ABCDE”
delete 2 delete 4
“ACDE” “ABCE”
Example for operational transformation
client server
“ABCDE” “ABCDE”
delete 2 delete 4
“ACDE” “ABCE”
“ACD” “ACE”
Example for operational transformation
client server
“ABCDE” “ABCDE”
delete 2 delete 4
“ACDE” “ABCE”
“ACE” “ACE”
delete 3
Example for operational transformation
client server
“ABCDE” “ABCDE”
delete 2 delete 4
“ACDE” “ABCE”
“ACE” “ACE”
delete 3
Google Docs
1. Apply Operation Transformation
2. Client and server exchange “change action”
3. Same code runs on both clie...
A real-time and offline editing service
Git
Commit & Revision
1
2
3
3A 3B
4
client server
Conflict!
Git
1. Change unit is “line”
2. Commit and revision
3. Possible actions
• Create new file / line
• Edit a file / line
• De...
Second problem:
Which real-time & offline solution?
Real-time frameworks and databases
1. Firebase
2. Deepstream.io
3. RethinkDB
4. CouchDB
5. PostgreSQL, PinelineDB, … (?)
Real-time frameworks and databases
1. Firebase
2. Deepstream.io
3. RethinkDB
4. CouchDB
5. PostgreSQL, PinelineDB, … (?)
Other problems
1. Work on web and mobile (Android, iOS)
2. Keep client in sync with server
• Can resume from a specific po...
First version
Firebase
Why Firebase?
1. Real-time database (as a service)
2. Work on web and mobile (Android, iOS)
3. Store value as a big JSON
4...
Sample code
var ref = new Firebase("https://<MY-FIREBASE-APP>.firebaseio.com");
ref.set({ name: "Alex Wolfe" });
ref.on("v...
Data as a tree
How do we structure data in Firebase
{
nodes: {
“u12345”: {
“a3f0”: {
parent_id: “b02c”,
order: 18
},
“b02c”: {
parent_id:...
How do we structure data in Firebase
{
users: {
“u12345”: { … },
},
nodes: {
“u12345”: {
“a3f0”: … ,
“b41c”: …
}
},
shares...
When we want to share a sub-tree
{
users: {
“u12345”: { … },
},
nodes: {
“u12345”: {
“a3f0”: … ,
“b41c”: …
}
},
shares: {
...
When we want to share a sub-tree
{
users: {
“u12345”: { … },
},
nodes: {
“u12345”: {
“a3f0”: … ,
“b41c”: …
}
},
shares: {
...
Why not Firebase?
1. Firebase does not support “move” action
2. Permission is defined on sub-tree
• When sharing some node...
Second version
Couchbase Mobile
Couchbase Mobile?
1. Couchbase Server
2. Couchbase Sync Gateway
3. Couchbase Lite on Device
- Use CouchDB protocol
- Pouch...
Sample code in PouchBD
var db = new PouchDB('dbname');
db.put({
_id: 'dave@gmail.com',
name: 'David',
age: 69
});
db.chang...
Why Couchbase Mobile?
1. Real-time database (self-deployment)
2. Work on web and mobile (Android, iOS)
3. Store value as a...
Why Couchbase Mobile?
1. Real-time database (self-deployment)
2. Work on web and mobile (Android, iOS)
3. Store value as a...
How Couchbase handle conflict?
1
2A
2_
2B
db.put({
_id: 'dave@gmail.com',
_rev: '1-af2345c1',
name: 'David',
age: 69,
});
...
How Couchbase handle conflict?
1
2A
2_
2B
1. Each change must include _rev
2. When two changes conflict
1. Choose arbitrar...
Couchbase Mobile Permission
Master DB
User1 DB User2 DB User3 DB UserN DB
User1 DB User2 DB User3 DB UserN DB
Couchbase DB...
How Couchbase handle changes?
GET /<db>/_changes?since=900
{
{"results":[
{"seq":909,"id":"_user/1NEzeQ","changes":[]},
{"...
How Couchbase handle changes?
{
"_deleted": true,
"_sync": {
"rev": "1-91fe6048079942fb279c6c51cfd039f0",
"sequence": 1039...
How Couchbase handle changes?
1. Each update include a sequence number
2. Client can request changes by ?since=seq
• Only ...
Why Couchbase Mobile?
1. Real-time database (self-deployment)
2. Work on web and mobile (Android, iOS)
3. Store value as a...
How do we structure data in Couchbase Mobile
{
_id: “a3f0”,
_rev: “…”
parent_id: “b02c”,
order: 18
shares: {
“u231b”: …
}
...
How do we structure data in Couchbase Mobile
{
_id: “a3f0”,
_rev: “…”
parent_id: “b02c”,
order: 18
shares: {
“u231b”: …
}
...
Why not Couchbase Mobile?
1. Permission based on each document
• It does not understand parent-child relationship
2. We ca...
Third version
Build our own solution with
Couchbase
What did we learn?
1. We need our own data structure and API
• Pre-built solutions are nice. But they do not fit to our ap...
What do we have to do?
1. Our own data structure and API
2. Sync service
3. SDK for JavaScript and mobiles (Android and iO...
Why still Couchbase?
1. Both database and memcached
2. Data Change Protocol for replication (DCP)
• Can resume at a specif...
Sample DCP receiver
func (r *Receiver) OnError(err error)
func (r *Receiver) DataUpdate(vbucketId uint16, key []byte, seq ...
How do we structure data in Couchbase
// Tree
{
_id: “…”,
_rev: “…”,
shares: { … },
items: []
}
// Node
{
_id: “…”,
_rev: ...
How do we handle changes?
GET /change?since=524
{
trees: [
{ _id: “…”, _rev: “…” }
],
nodes: [
{ _id: “…”, _rev: “…” }
],
...
How do we share?
<tree1>
<tree2>
<tree1>/<node1>
<tree2>/<node2>
1. Only define permission on trees
2. Split tree when nee...
Remaining problems
1. Change set & history
2. Moving & tree conflict
3. Optimize loading: load important data first
4. Imp...
THANK YOU
Vu Nguyen
vu.nguyen@will.vn
Grokking #9
Grokking #9
Building an offline & real-time
editing service with Couchbase
Vu Nguyen
vu.nguyen@will.vn
Grokking TechTalk 9 - Building a realtime & offline editing service from scratch with Couchbase
Grokking TechTalk 9 - Building a realtime & offline editing service from scratch with Couchbase
Upcoming SlideShare
Loading in …5
×

Grokking TechTalk 9 - Building a realtime & offline editing service from scratch with Couchbase

414 views

Published on

https://www.youtube.com/watch?v=_Wqy1B8PXD4&feature=youtu.be

Talk presented by Vu Nguyen, CTO @ Liti Book (Vietnamese)

Brief intro: In this talk, I would like to share how we build a system for LitiBook that can handle (1) real-time editing, (2) offline editing, (3) synchronizing between devices and (4) conflict between different editing sessions. There are not many applications out there can do all these above things. (Evernote does not resolve conflict. Hackpad, Trello and Asana do not support offline). So the challenge is really interesting.

About speaker: Vu Nguyen is a young and passionate engineer who founded Liti Book with his friend. Liti Book aimed to develp the next generation of productivity tool which is support more collaboratoin, more real-time editing, ...

www.grokking.org

Published in: Engineering
  • Be the first to comment

  • Be the first to like this

Grokking TechTalk 9 - Building a realtime & offline editing service from scratch with Couchbase

  1. 1. Grokking #9 Building an offline & real-time editing service with Couchbase Vu Nguyen vu.nguyen@will.vn
  2. 2. In this talk You’ll walk away with • Our approach for an unknown architecture • How do we design architecture for real-time & offline • Different between Couchbase and other real-time databases
  3. 3. Talk structure 1. Problems of a real-time & offline editing service 2. Evaluate real-time frameworks and databases 3. First prototype with Firebase 4. Second version with Couchbase Mobile 5. Time to make our own one! 6. Why Couchbase?
  4. 4. A real-time and offline editing service What is it?
  5. 5. A real-time and offline editing services 1. Users can edit at the same time 2. Can work without the internet 3. Synchronize as soon as online 4. Resolve conflict between editing sessions
  6. 6. Evernote does not resolve conflict!
  7. 7. Hackpad disable editing while offline!
  8. 8. And Asana!
  9. 9. First problem: How to handle conflict?
  10. 10. A real-time and offline editing service Google Docs
  11. 11. Google Docs
  12. 12. Google Docs
  13. 13. Operational transformation 0 , 0 1 , 0 0 , 1 1 , 1 2 , 1 1 , 2 2 , 2 2 , 0 0 , 2 client
  14. 14. Operational transformation 0 , 0 1 , 0 0 , 1 1 , 1 2 , 1 1 , 2 2 , 2 2 , 0 0 , 2 client server
  15. 15. Operational transformation 0 , 0 1 , 0 0 , 1 1 , 1 2 , 1 1 , 2 2 , 2 2 , 0 0 , 2 client server
  16. 16. Operational transformation 0 , 0 1 , 0 0 , 1 1 , 1 2 , 1 1 , 2 2 , 2 2 , 0 0 , 2 client server
  17. 17. Operational transformation 0 , 0 1 , 0 0 , 1 1 , 1 2 , 1 1 , 2 2 , 2 2 , 0 0 , 2 client server Conflict!
  18. 18. Operational transformation 0 , 0 1 , 0 0 , 1 1 , 1 2 , 1 1 , 2 2 , 2 2 , 0 0 , 2 client server Conflict!
  19. 19. Example for operational transformation client server “ABCDE” “ABCDE” delete 2 delete 4 “ACDE” “ABCE”
  20. 20. Example for operational transformation client server “ABCDE” “ABCDE” delete 2 delete 4 “ACDE” “ABCE” “ACD” “ACE”
  21. 21. Example for operational transformation client server “ABCDE” “ABCDE” delete 2 delete 4 “ACDE” “ABCE” “ACE” “ACE” delete 3
  22. 22. Example for operational transformation client server “ABCDE” “ABCDE” delete 2 delete 4 “ACDE” “ABCE” “ACE” “ACE” delete 3
  23. 23. Google Docs 1. Apply Operation Transformation 2. Client and server exchange “change action” 3. Same code runs on both client and server • Google Web Toolkit
  24. 24. A real-time and offline editing service Git
  25. 25. Commit & Revision 1 2 3 3A 3B 4 client server Conflict!
  26. 26. Git 1. Change unit is “line” 2. Commit and revision 3. Possible actions • Create new file / line • Edit a file / line • Delete a file / line • Move a file 4. Manual conflict resolving
  27. 27. Second problem: Which real-time & offline solution?
  28. 28. Real-time frameworks and databases 1. Firebase 2. Deepstream.io 3. RethinkDB 4. CouchDB 5. PostgreSQL, PinelineDB, … (?)
  29. 29. Real-time frameworks and databases 1. Firebase 2. Deepstream.io 3. RethinkDB 4. CouchDB 5. PostgreSQL, PinelineDB, … (?)
  30. 30. Other problems 1. Work on web and mobile (Android, iOS) 2. Keep client in sync with server • Can resume from a specific point in time 3. Edit history • Can rollback to a specific version
  31. 31. First version Firebase
  32. 32. Why Firebase? 1. Real-time database (as a service) 2. Work on web and mobile (Android, iOS) 3. Store value as a big JSON 4. Permission on sub-tree 5. Quickly make a prototype
  33. 33. Sample code var ref = new Firebase("https://<MY-FIREBASE-APP>.firebaseio.com"); ref.set({ name: "Alex Wolfe" }); ref.on("value", function(data) { var name = data.val().name; alert("My name is " + name); });
  34. 34. Data as a tree
  35. 35. How do we structure data in Firebase { nodes: { “u12345”: { “a3f0”: { parent_id: “b02c”, order: 18 }, “b02c”: { parent_id: “”, order: 1 } } } nodes/<user>/<id1> nodes/<user>/<id2> nodes/<user>/<id3>
  36. 36. How do we structure data in Firebase { users: { “u12345”: { … }, }, nodes: { “u12345”: { “a3f0”: … , “b41c”: … } }, shares: { “s98765”: { “2abf”: … } } } nodes/<user>/<id1> nodes/<user>/<id2> nodes/<user>/<id3>
  37. 37. When we want to share a sub-tree { users: { “u12345”: { … }, }, nodes: { “u12345”: { “a3f0”: … , “b41c”: … } }, shares: { “s98765”: { “2abf”: … } } } nodes/<user>/<id1> shares/<sid>/<id2> nodes/<user>/<id3>
  38. 38. When we want to share a sub-tree { users: { “u12345”: { … }, }, nodes: { “u12345”: { “a3f0”: … , “b41c”: … } }, shares: { “s98765”: { “2abf”: … } } } nodes/<user>/<id1> shares/<sid>/<id2> nodes/<user>/<id3> - Firebase does not support “move” action - Permission is defined on sub-tree - We have to copy data from “nodes” to “shares”
  39. 39. Why not Firebase? 1. Firebase does not support “move” action 2. Permission is defined on sub-tree • When sharing some nodes, we have to copy them from “nodes” to “shares” • This defeats synchronization! 3. Does not resolve conflict • Latest update win. • Workaround solutions like hack!
  40. 40. Second version Couchbase Mobile
  41. 41. Couchbase Mobile? 1. Couchbase Server 2. Couchbase Sync Gateway 3. Couchbase Lite on Device - Use CouchDB protocol - PouchDB for offline JS database
  42. 42. Sample code in PouchBD var db = new PouchDB('dbname'); db.put({ _id: 'dave@gmail.com', name: 'David', age: 69 }); db.changes().on('change', function() { console.log('Ch-Ch-Changes'); }); db.replicate.to('http://example.com/mydb');
  43. 43. Why Couchbase Mobile? 1. Real-time database (self-deployment) 2. Work on web and mobile (Android, iOS) 3. Store value as a JSON documents 4. Permission using channels 5. It does handle conflict!
  44. 44. Why Couchbase Mobile? 1. Real-time database (self-deployment) 2. Work on web and mobile (Android, iOS) 3. Store value as a JSON documents 4. Permission using channels 5. It does handle conflict!
  45. 45. How Couchbase handle conflict? 1 2A 2_ 2B db.put({ _id: 'dave@gmail.com', _rev: '1-af2345c1', name: 'David', age: 69, }); db.replicate.to(“http://...”)
  46. 46. How Couchbase handle conflict? 1 2A 2_ 2B 1. Each change must include _rev 2. When two changes conflict 1. Choose arbitrary winner based on a “deterministic algorithm” 2. The document has two _rev numbers 3. Client know that a document is in conflicting status and can decide to resolve. 4. Deleted documents are kept in db { _id: “”, _rev: “”, _deleted: true }
  47. 47. Couchbase Mobile Permission Master DB User1 DB User2 DB User3 DB UserN DB User1 DB User2 DB User3 DB UserN DB Couchbase DB Sync Gateway … … Couchbase Lite / Pouch DB Replication
  48. 48. How Couchbase handle changes? GET /<db>/_changes?since=900 { {"results":[ {"seq":909,"id":"_user/1NEzeQ","changes":[]}, {"seq":1317,"id":"mIM0Uuscg0MC","deleted":true, "changes":[{"rev":"1-96307afa"}]}, {"seq":1318,"id":"mQQ0NwKUM0Dy","changes":[{"rev":"1-5fdcaba"}]}, {"seq":1319,"id":"sLX0SKTpM41X","changes":[{"rev":"1-d15ee59"}]} } }
  49. 49. How Couchbase handle changes? { "_deleted": true, "_sync": { "rev": "1-91fe6048079942fb279c6c51cfd039f0", "sequence": 10392, "history": { "revs": [ "1-91fe6048079942fb279c6c51cfd039f0" ], } }
  50. 50. How Couchbase handle changes? 1. Each update include a sequence number 2. Client can request changes by ?since=seq • Only latest document revisions are returned 3. Client can request by long-polling or websocket
  51. 51. Why Couchbase Mobile? 1. Real-time database (self-deployment) 2. Work on web and mobile (Android, iOS) 3. Store value as a JSON documents 4. Permission using channels 5. It does handle conflict!
  52. 52. How do we structure data in Couchbase Mobile { _id: “a3f0”, _rev: “…” parent_id: “b02c”, order: 18 shares: { “u231b”: … } } <Id1> <id2> <id3>
  53. 53. How do we structure data in Couchbase Mobile { _id: “a3f0”, _rev: “…” parent_id: “b02c”, order: 18 shares: { “u231b”: … } } <id1> <id3> - Couchbase Mobile does not understand parent-child relationship - We have to update “shares” property on all child nodes => All child nodes will be updated and synchronized to all clients <id2>
  54. 54. Why not Couchbase Mobile? 1. Permission based on each document • It does not understand parent-child relationship 2. We can not easily share a sub-tree • Have to update all children nodes • This defeats synchronization purpose!
  55. 55. Third version Build our own solution with Couchbase
  56. 56. What did we learn? 1. We need our own data structure and API • Pre-built solutions are nice. But they do not fit to our app. 2. We learned how Couchbase Sync Gateway implement synchronization and conflict resolving 3. We inherit code from Couchbase Sync Gateway • It is written in Go!
  57. 57. What do we have to do? 1. Our own data structure and API 2. Sync service 3. SDK for JavaScript and mobiles (Android and iOS) 4. Synchronization and conflict solution 5. Sharing solution
  58. 58. Why still Couchbase? 1. Both database and memcached 2. Data Change Protocol for replication (DCP) • Can resume at a specific time • Can be used to replicate whole database content
  59. 59. Sample DCP receiver func (r *Receiver) OnError(err error) func (r *Receiver) DataUpdate(vbucketId uint16, key []byte, seq uint64, req *gomemcached.MCRequest) error func (r *Receiver) DataDelete(vbucketId uint16, key []byte, seq uint64, req *gomemcached.MCRequest) error func (r *Receiver) SetMetaData(vbucketId uint16, value []byte) error func (r *Receiver) GetMetaData(vbucketId uint16) func (r *Receiver) Rollback(vbucketId uint16, rollbackSeq uint64) error
  60. 60. How do we structure data in Couchbase // Tree { _id: “…”, _rev: “…”, shares: { … }, items: [] } // Node { _id: “…”, _rev: “…”, content: [] } <tree1> <tree2> <tree1>/<node1> <tree2>/<node2>
  61. 61. How do we handle changes? GET /change?since=524 { trees: [ { _id: “…”, _rev: “…” } ], nodes: [ { _id: “…”, _rev: “…” } ], current_seq: “…” } <tree1> <tree2> <tree1>/<node1> <tree2>/<node2>
  62. 62. How do we share? <tree1> <tree2> <tree1>/<node1> <tree2>/<node2> 1. Only define permission on trees 2. Split tree when needed 3. Tree has it own revisions 4. Nodes are used to store data (without explicit permission)
  63. 63. Remaining problems 1. Change set & history 2. Moving & tree conflict 3. Optimize loading: load important data first 4. Improve caching 5. Stress test 6. …
  64. 64. THANK YOU Vu Nguyen vu.nguyen@will.vn Grokking #9
  65. 65. Grokking #9 Building an offline & real-time editing service with Couchbase Vu Nguyen vu.nguyen@will.vn

×