Today mobile apps are everywhere. These apps cannot count on a reliable and constant internet connection: working in offline mode is becoming a common pattern. This is quite easy for read-only apps but it becomes rapidly tricky for apps that create data in offline mode. This talk is a case study about a possible architecture for enabling data synchronization in these situations. Some of the topics touched will be: - id generation - hierarchical data - managing differente data types - sync algorithm

1. Implementing data
synchronization API for
mobile apps

2. Michele Orselli
CTO@Ideato
micheleorselli / ideatosrl
_orso_
mo@ideato.it

3. Agenda
scenario design choices
implementation alternative approaches

6. Dealing with conflicts
A1
A2
?

7. Brownfield project
several mobile apps for tracking user generated
data (calendar, notes, bio data)
iOS & Android
~10 K users steadily growing at 1.2 K/month
Scenario

8. MongoDB
Legacy App based on Codeigniter
Existing RPC-wannabe-REST API for data sync
Scenario

9. For every resource
get updates:
POST /m/:app/get/:user_id/:res/:updated_from
create/send updates:
POST /m/:app/update/:user_id/:res_id/:dev_id/:res
Scenario

10. api

11. ~6 different resources, ~12 calls per sync
apps sync by polling every 30 sec
every call sync little data
Scenario

12. Rebuild sync API for old apps + 2 incoming
Enable image synchronization
More efficient than previous API
Challenge

14. Existing Solutions
Tstamps,
Vector clocks,
CRDTs
syncML,
syncano
Azure Data
sync
Algorithms Protocols/API
Platform
couchDB,
riak
Storage

15. Not Invented Here?
Don't Reinvent The Wheel,
UnlessYou Plan on Learning More About Wheels
J. Atwood

16. 2 different mobile platforms
Several teams with different skill level
Changing storage wasn’t an option
Forcing a particular technology client side wasn’t
an option
Architecture

17. Architecture
c1
server
c2
c3
sync logic
conflicts resolution
thin clients

18. In the sync domain all resources are managed in
the same way
Implementation

19. For every app:
one endpoint for getting new data
one endpoint for pushing changes
one endpoint for uploading images
Implementation

20. GET /apps/:app/users/:user_id/changes[?from=:from]
POST /apps/:app/users/:user_id/merge
POST /upload/:res_id/images
The new APIs

21. Silex Implementation

22. Silex Implementation
Col 1
Col 2
Col 3

23. Silex Implementation
Col 1
Col 2
Col 3
Sync Service

24. Silex Implementation
Col 1
Col 2
Col 3
Sync Service

25. Silex Implementation
Col 1
Col 2
Col 3
Sync Service

26. Silex Implementation
Col 1
Col 2
Col 3
Sync Service

27. Silex Implementation
Col 1
Col 2
Col 3
Sync Service

28. Silex Implementation
$app->get(“/apps/{mApp}/users/{userId}/merge”,
function ($mApp, $userId, $app, $request)
{
$lastSync = $request->get('from', null);
$data = $request->get(‘data’, false);
$syncService = $app[‘syncService’];
$syncService->merge($data, $lastSync, $userId);
$response = new JsonResponse(
$syncService->getResult()
);
return $response;
}

29. Silex Implementation
$app->get(“/apps/{mApp}/users/{userId}/merge”,
function ($mApp, $userId, $app, $request)
{
$lastSync = $request->get('from', null);
$data = $request->get(‘data’, false);
$syncService = $app[‘syncService’];
$syncService->merge($data, $lastSync, $userId);
$response = new JsonResponse(
$syncService->getResult()
);
return $response;
}

30. Silex Implementation
$app->get(“/apps/{mApp}/users/{userId}/merge”,
function ($mApp, $userId, $app, $request)
{
$lastSync = $request->get('from', null);
$data = $request->get(‘data’, false);
$syncService = $app[‘syncService’];
$syncService->merge($data, $lastSync, $userId);
$response = new JsonResponse(
$syncService->getResult()
);
return $response;
}

31. Silex Implementation
$app->get(“/apps/{mApp}/users/{userId}/merge”,
function ($mApp, $userId, $app, $request)
{
$lastSync = $request->get('from', null);
$data = $request->get(‘data’, false);
$syncService = $app[‘syncService’];
$syncService->merge($data, $lastSync, $userId);
$response = new JsonResponse(
$syncService->getResult()
);
return $response;
}

32. Silex Implementation
$app->get(“/apps/{mApp}/users/{userId}/merge”,
function ($mApp, $userId, $app, $request)
{
$lastSync = $request->get('from', null);
$data = $request->get(‘data’, false);
$syncService = $app[‘syncService’];
$syncService->merge($data, $lastSync, $userId);
$response = new JsonResponse(
$syncService->getResult()
);
return $response;
}

33. Silex Implementation
$app['mongodb'] = new MongoDb(…);
$app[‘changesRepo’] = new ChangesRepository(
$app[‘mongodb’]
);
$app[‘syncService’] ? new SyncService(
$app[‘changesRepo’]
);

34. GET /apps/:app/users/:user_id/changes?from=:from
Get changes
timestamp?

35. timestamp are inaccurate
server suggests the “from” parameter to be used
in the next request
Server suggest the sync time

36. Server suggest the sync time
c1 server
GET /changes
{ ‘next’ : 12345,
‘data’: […] }

37. Server suggest the sync time
c1 server
GET /changes
{ ‘next’ : 12345,
‘data’: […] }
GET /changes?from=12345
{ ‘next’ : 45678,
‘data’: […] }

38. data format
{id:‘1’, ’type’:‘measure’,‘_deleted’: true}
{id: 2’,‘type’:‘note’}
{id:‘3’,‘type’:‘note’}
ps: soft delete all the things!
what to transfer

39. How do we generate an unique id in a distributed
system?
unique identifiers

40. How do we generate an unique id in a distributed
system?
UUID (RFC 4122): several implementations in PHP
(https://github.com/ramsey/uuid)
unique identifiers

41. How do we generate an unique id in a distributed
system?
Local/Global Id: only the server generates GUIDs
clients use local ids to manage their records
unique identifiers

42. unique identifiers
c1 server
POST /merge
{ ‘data’: [
{’lid’:‘1’, …},
{‘lid’:‘2’, …}
] }
{ ‘data’: [
{‘guid’:‘58f0bdd7-1400’, ’lid’:‘1’, …},
{‘guid’:‘6f9f3ec9-1400’,‘lid’:‘2’, …}
] }

43. mobile generated data are “temporary” until sync
to server
server handles conflicts resolution
conflict resolution algorithm (plain data)

44. conflict resolution:
domain indipendent: e.g. last-write wins
domain dipendent: use domain knowledge to
resolve
conflict resolution algorithm (plain data)

45. function sync($data) {
foreach ($data as $newRecord) {
$s = findByGuid($newRecord->getGuid());
if (!$s) {
add($newRecord);
send($newRecord);
continue;
}
if ($newRecord->updated > $s->updated) {
update($s, $newRecord);
send($newRecord);
continue;
}
updateRemote($newRecord, $s);
}
conflict resolution algorithm (plain data)

46. function sync($data) {
foreach ($data as $newRecord) {
$s = findByGuid($newRecord->getGuid());
if (!$s) {
add($newRecord);
send($newRecord);
continue;
}
if ($newRecord->updated > $s->updated) {
update($s, $newRecord);
send($newRecord);
continue;
}
updateRemote($newRecord, $s);
}
conflict resolution algorithm (plain data)

47. function sync($data) {
foreach ($data as $newRecord) {
$s = findByGuid($newRecord->getGuid());
if (!$s) {
add($newRecord);
send($newRecord);
continue;
}
if ($newRecord->updated > $s->updated) {
update($s, $newRecord);
send($newRecord);
continue;
}
updateRemote($newRecord, $s);
}
conflict resolution algorithm (plain data)
no conflict

48. function sync($data) {
foreach ($data as $newRecord) {
$s = findByGuid($newRecord->getGuid());
if (!$s) {
add($newRecord);
send($newRecord);
continue;
}
if ($newRecord->updated > $s->updated) {
update($s, $newRecord);
send($newRecord);
continue;
}
updateRemote($newRecord, $s);
}
conflict resolution algorithm (plain data)
remote wins

49. function sync($data) {
foreach ($data as $newRecord) {
$s = findByGuid($newRecord->getGuid());
if (!$s) {
add($newRecord);
send($newRecord);
continue;
}
if ($newRecord->updated > $s->updated) {
update($s, $newRecord);
send($newRecord);
continue;
}
updateRemote($newRecord, $s);
}
conflict resolution algorithm (plain data)
server wins

50. conflict resolution algorithm (plain data)
c1
{ ‘lid’:‘1’,
‘guid’:‘af54d’,
‘data’ :‘AAA’
‘updated’: ’100’ }
{ ‘lid’:‘2’,
‘data’ :‘hello!’,
‘updated’: ’15’ }
server
{ ’guid’:‘af54d’,
‘data’:‘BBB’,
‘updated’ : ’20’ }

51. conflict resolution algorithm (plain data)
c1 server
{ ‘lid’:‘1’,
‘guid’:‘af54d’,
‘data’ :‘AAA’
‘updated’: ’100’ }
{ ‘lid’:‘2’,
‘data’ :‘hello!’,
‘updated’: ’15’ } POST /merge
{ ’guid’:‘af54d’,
‘data’:‘BBB’,
‘updated’ : ’20’ }

52. conflict resolution algorithm (plain data)
c1 server
{ ‘lid’:‘1’,
‘guid’:‘af54d’,
‘data’ :‘AAA’
‘updated’: ’100’ }
{ ‘lid’:‘2’,
‘data’ :‘hello!’,
‘updated’: ’15’ } POST /merge
{ ‘guid’:‘e324f’,
‘data’ :‘hello!’,
‘updated’: ’15’ }
{ ’guid’:‘af54d’,
‘data’:‘BBB’,
‘updated’ : ’20’ }

53. conflict resolution algorithm (plain data)
c1 server
{ ‘lid’:‘1’,
‘guid’:‘af54d’,
‘data’ :‘AAA’
‘updated’: ’100’ }
{ ‘lid’:‘2’,
‘data’ :‘hello!’,
‘updated’: ’15’ } POST /merge
{ ‘guid’:‘e324f’,
‘data’ :‘hello!’,
‘updated’: ’15’ }
{ ’guid’:‘af54d’,
‘data’:‘BBB’,
‘updated’ : ’20’ }

54. conflict resolution algorithm (plain data)
c1 server
{ ‘lid’:‘1’,
‘guid’:‘af54d’,
‘data’ :‘AAA’
‘updated’: ’100’ }
{ ‘lid’:‘2’,
‘data’ :‘hello!’,
‘updated’: ’15’ } POST /merge
{ ‘guid’:‘e324f’,
‘data’ :‘hello!’,
‘updated’: ’15’ }
{ ’guid’:‘af54d’,
‘data’:‘AAA’,
‘updated’ : ’100’ }

55. conflict resolution algorithm (plain data)
c1 server
{ ‘lid’:‘1’,
‘guid’:‘af54d’,
‘data’ :‘AAA’
‘updated’: ’100’ }
{ ’guid’:‘af54d’,
‘data’:‘AAA’,
‘updated’ : ’100’ }
{ ‘lid’:‘2’,
‘data’ :‘hello!’,
‘updated’: ’15’ } POST /merge
{ ‘guid’:‘e324f’,
‘data’ :‘hello!’,
‘updated’: ’15’ }
{‘ok’ : { ’guid’:‘af54d’ }}
{‘update’ : { lid:‘2’, ’guid’:‘e324f’ }}

56. conflict resolution algorithm (hierarchical data)
How to manage hierarchical data?
{
‘lid’ : ‘123456’,
‘type’ : ‘baby’,
…
}
{
‘lid’ : ‘123456’,
‘type’ : ‘temperature’,
‘baby_id : ‘123456’
}

57. conflict resolution algorithm (hierarchical data)
How to manage hierarchical data?
1) sync root record
2) update ids
3) sync child records
{
‘lid’ : ‘123456’,
‘type’ : ‘baby’,
…
}
{
‘lid’ : ‘123456’,
‘type’ : ‘temperature’,
‘baby_id : ‘123456’
}

58. function syncHierarchical($data) {
sortByHierarchy($data);
foreach ($data as $newRootRecord) {
$s = findByGuid($newRootRecord->getGuid());
if($newRecord->isRoot()) {
if (!$s) {
add($newRootRecord);
updateRecordIds($newRootRecord, $data);
send($newRootRecord);
continue;
}
…
conflict resolution algorithm (hierarchical data)

59. function syncHierarchical($data) {
sortByHierarchy($data);
foreach ($data as $newRootRecord) {
$s = findByGuid($newRootRecord->getGuid());
if($newRecord->isRoot()) {
if (!$s) {
add($newRootRecord);
updateRecordIds($newRootRecord, $data);
send($newRootRecord);
continue;
}
…
conflict resolution algorithm (hierarchical data)
parent records first

60. function syncHierarchical($data) {
sortByHierarchy($data);
foreach ($data as $newRootRecord) {
$s = findByGuid($newRootRecord->getGuid());
if($newRecord->isRoot()) {
if (!$s) {
add($newRootRecord);
updateRecordIds($newRootRecord, $data);
send($newRootRecord);
continue;
}
…
conflict resolution algorithm (hierarchical data)

61. function syncHierarchical($data) {
sortByHierarchy($data);
foreach ($data as $newRootRecord) {
$s = findByGuid($newRootRecord->getGuid());
if($newRecord->isRoot()) {
if (!$s) {
add($newRootRecord);
updateRecordIds($newRootRecord, $data);
send($newRootRecord);
continue;
}
…
conflict resolution algorithm (hierarchical data)
no conflict

62. …
if ($newRootRecord->updated > $s->updated) {
update($s, $newRecord);
updateRecordIds($newRootRecord, $data);
send($newRootRecord);
continue;
} else {
updateRecordIds($s, $data);
updateRemote($newRecord, $s);
}
} else {
sync($data);
}
}
conflict resolution algorithm (hierarchical data)
remote wins

63. …
if ($newRootRecord->updated > $s->updated) {
update($s, $newRecord);
updateRecordIds($newRootRecord, $data);
send($newRootRecord);
continue;
} else {
updateRecordIds($s, $data);
updateRemote($newRecord, $s);
}
} else {
sync($data);
}
}
conflict resolution algorithm (hierarchical data)
server wins

64. conflict resolution algorithm (hierarchical data)
{ ‘lid’:‘1’,
‘data’ :‘AAA’
‘updated’: ’100’ }
{ ‘lid’:‘2’,
‘parent’:‘1’,
‘data’ :‘hello!’,
‘updated’: ’15’ }
c1 server
POST /merge

65. conflict resolution algorithm (hierarchical data)
c1
{ ‘lid’:‘1’,
‘data’ :‘AAA’
‘updated’: ’100’ }
{ ‘lid’:‘2’,
‘parent’:‘1’,
‘data’ :‘hello!’,
‘updated’: ’15’ }
server
POST /merge
{ ‘lid’:‘1’,
‘guid’ :‘32ead’,
‘data’ :‘AAA’
‘updated’: ’100’ }

66. conflict resolution algorithm (hierarchical data)
c1
{ ‘lid’:‘1’,
‘data’ :‘AAA’
‘updated’: ’100’ }
{ ‘lid’:‘2’,
‘parent’:‘32ead’,
‘data’ :‘hello!’,
‘updated’: ’15’ }
server
POST /merge
{ ‘lid’:‘1’,
‘guid’ :‘32ead’,
‘data’ :‘AAA’
‘updated’: ’100’ }

67. conflict resolution algorithm (hierarchical data)
c1
{ ‘lid’:‘1’,
‘data’ :‘AAA’
‘updated’: ’100’ }
{ ‘lid’:‘2’,
‘parent’:‘32ead’,
‘data’ :‘hello!’,
‘updated’: ’15’ }
server
POST /merge
{ ‘lid’:‘1’,
‘guid’ :‘32ead’,
‘data’ :‘AAA’
‘updated’: ’100’ }
{ ‘lid’:‘2’,
‘parent’:‘32ead’,
‘data’ :‘hello!’,
‘updated’: ’15’ }
{‘update’ : { ‘lid’:‘1’, ’guid’:‘af54d’ }}
{‘update’ : { lid:‘2’, ’guid’:‘e324f’ }}

68. e.g.“only one temperature can be registered in a
given day”
how to we enforce domain constraints on data?
enforcing domain constraints

69. e.g.“only one temperature can be registered in a
given day”
how to we enforce domain constraints on data?
1) relax constraints
enforcing domain constraints

70. e.g.“only one temperature can be registered in a
given day”
how to we enforce domain constraints on data?
1) relax constraints
2) integrate constraints in sync algorithm
enforcing domain constraints

71. from findByGuid to findSimilar
first lookup by GUID then by domain rules
“two measures are similar if are referred to the
same date”
enforcing domain constraints

72. enforcing domain constraints
c1 server

73. enforcing domain constraints
c1 server
{ ’guid’:‘af54d’,
‘when’:‘20141005’ }

74. enforcing domain constraints
c1 server
{ ‘lid’:‘1’,
‘when’:‘20141005’ }
{ ’guid’:‘af54d’,
‘when’:‘20141005’ }

75. enforcing domain constraints
c1 server
{ ‘lid’:‘1’,
‘when’:‘20141005’ }
{ ’guid’:‘af54d’,
‘when’:‘20141005’ }
POST /merge

76. enforcing domain constraints
c1 server
{ ‘lid’:‘1’,
‘when’:‘20141005’ }
{ ’guid’:‘af54d’,
‘when’:‘20141005’ }
POST /merge

77. enforcing domain constraints
c1 server
{ ‘lid’:‘1’,
‘when’:‘20141005’ }
{ ’guid’:‘af54d’,
‘when’:‘20141005’ }
POST /merge
{ ’guid’:‘af54d’,
‘when’:‘20141005’ }

78. Binary data uploaded via custom endpoint
Sync data remains small
Uploads can be resumed
dealing with binary data

79. Two steps*
1) data are synchronized
2) related images are uploaded
* this means record without file for a given time
dealing with binary data

80. dealing with binary data
c1 server
POST /merge
POST /upload/ac435-f8345/image
{ ‘lid’ : 1,
‘type’ :‘baby’,
‘image’ :‘myimage.jpg’ }
{ ‘lid’ : 1,
‘guid’ :‘ac435-f8345’ }

81. Implementing this stuff is tricky
Explore existing solution if you can
Understanding the domain is important
What we learned

82. vector clocks

83. Conflict-free Replicated Data Types (CRDTs)
Constraining the types of operations in order to:
- ensure convergence of changes to shared data by
uncoordinated, concurrent actors
- eliminate network failure modes as a source of
error
CRDT

84. Gateways handles sync
Data flows through channels
- partition data set
- authorization
- limit the data
Use revision trees
Couchbase Mobile

85. Distributed DB
Eventually/Strong Consistency
Data Types
Configurable conflict resolution
- db level for built-in data types
- application level for custom
data
Riak

86. See you inVerona!
jsDay 13th-14th of May
http://2015.jsday.it/
phpDay 15th-16th of May
http://2015.phpday.it/
Questions?

87. http://www.objc.io/issue-10/sync-case-study.html
http://www.objc.io/issue-10/data-synchronization.html
https://dev.evernote.com/media/pdf/edam-sync.pdf
http://blog.helftone.com/clear-in-the-icloud/
http://strongloop.com/strongblog/node-js-replication-mobile-offline-sync-loopback/
http://blog.denivip.ru/index.php/2014/04/data-syncing-in-core-data-based-ios-apps/?lang=en
http://inessential.com/2014/02/15/vesper_sync_diary_8_the_problem_of_un
http://culturedcode.com/things/blog/2010/12/state-of-sync-part-1.html
http://programmers.stackexchange.com/questions/206310/data-synchronization-in-mobile-apps-
multiple-devices-multiple-users
http://bricklin.com/offline.htm
http://blog.couchbase.com/why-mobile-sync
Links

88. Vector Clocks
http://basho.com/why-vector-clocks-are-easy/
http://www.datastax.com/dev/blog/why-cassandra-doesnt-need-vector-clocks
http://basho.com/why-vector-clocks-are-hard/
http://blog.8thlight.com/rylan-dirksen/2013/10/04/synchronization-in-a-distributed-system.html
CRDTs
http://christophermeiklejohn.com/distributed/systems/2013/07/12/readings-in-distributed-systems.html
http://www.infoq.com/presentations/problems-distributed-systems
https://www.youtube.com/watch?v=qyVNG7fnubQ
Riak
http://docs.basho.com/riak/latest/dev/using/conflict-resolution/
Couchbase Sync Gateway
http://docs.couchbase.com/sync-gateway/
http://www.infoq.com/presentations/sync-mobile-data
API
http://developers.amiando.com/index.php/REST_API_DataSync
https://login.syncano.com/docs/rest/index.html
Links

89. phones https://www.flickr.com/photos/15216811@N06/14504964841
wat http://uturncrossfit.com/wp-content/uploads/2014/04/wait-what.jpg
darth http://www.listal.com/viewimage/3825918h
blueprint: http://upload.wikimedia.org/wikipedia/commons/5/5e/Joy_Oil_gas_station_blueprints.jpg
building: http://s0.geograph.org.uk/geophotos/02/42/74/2427436_96c4cd84.jpg
brownfield: http://s0.geograph.org.uk/geophotos/02/04/54/2045448_03a2fb36.jpg
no connection: https://www.flickr.com/photos/77018488@N03/9004800239
no internet con https://www.flickr.com/photos/roland/9681237793
vector clocks: http://en.wikipedia.org/wiki/Vector_clock
crdts: http://www.infoq.com/presentations/problems-distributed-systems
Credits