Sync	
  Gateway	
  In-­‐Depth
Chris	
  Anderson	
  
June	
  24,	
  2014
V1.0 - 3/10/14
Architecture
What’s	
  It	
  For?
1.	
  Syncing	
  With	
  Couchbase	
  Lite
• MulH-­‐master	
  sync,	
  scalable	
  to	
  any	
  number	
  of	
  clients	
 ...
2.	
  AuthenHcaHon
• Sync	
  Gateway	
  is	
  publicly	
  reachable	
  outside	
  the	
  firewall	
  
• Security	
  barrier...
3.	
  AuthorizaHon
• Channel	
  tags	
  grant	
  read	
  access	
  to	
  documents	
  
Each	
  user	
  has	
  access	
  to...
4.	
  App	
  Workflow
• Channels	
  route	
  documents	
  between	
  users	
  
• Sync	
  funcHon	
  drives	
  the	
  enHre	...
Architecture	
  Diagram
Architecture
Server	
  3Server	
  1 Server	
  2
Couchbase	
  Server
Channel
Sync	
  Gateway
Channel Channel
Sync	
  Gatewa...
Sync	
  Gateway	
  Components
Sync	
  
REST	
  API
Couchbase	
  
Smart	
  Client
Revision/
Conflict	
  
Management
AuthenHc...
Sync	
  Gateway:	
  Incoming	
  Changes
Sync	
  
REST	
  API AuthenHcaHon External	
  Auth	
  
Services
!
• Pushes	
  chan...
Sync	
  Gateway:	
  App	
  Logic	
  &	
  Storage
Couchbase	
  
Smart	
  Client
Revision/
Conflict	
  
Management
App’s	
  S...
Sync	
  Gateway	
  Components
Sync	
  
REST	
  API
Couchbase	
  
Smart	
  Client
Revision/
Conflict	
  
Management
AuthenHc...
MulH-­‐Master	
  ReplicaHon
Features
• Any	
  number	
  of	
  clients	
  
• Arbitrary	
  topologies	
  (from	
  centralized	
  to	
  P2P)	
  
• Occasi...
ReplicaHon	
  Types
• One-­‐direcHonal	
  
“Push”	
  to	
  server	
  
“Pull”	
  from	
  server	
  
• One-­‐shot	
  or	
  c...
“Push”	
  ReplicaHon
• Consult	
  local	
  db	
  for	
  revisions	
  added	
  since	
  last	
  checkpoint	
  
• POST	
  li...
“Pull”	
  replicaHon
• Read	
  server’s	
  “_changes”	
  feed	
  
StarEng	
  from	
  just	
  past	
  last	
  checkpoint	
 ...
Changes	
  Feed
• Most	
  difficult	
  part	
  of	
  enHre	
  project	
  
Linear	
  history	
  of	
  all	
  document	
  chan...
Demo	
  ApplicaHon
Sync	
  FuncHon
JSON	
  Document	
  Schema
Sync	
  FuncHon
Sync	
  FuncHon
• Task	
  documents	
  belong	
  to	
  lists
Sync	
  FuncHon
• List	
  documents	
  specify	
  owners	
  and	
  members	
  
!
!
!
!
!
• This	
  app	
  shares	
  the	
 ...
Sync	
  FuncHon
• Profile	
  documents	
  are	
  distributed	
  to	
  all	
  users	
  
• Owned	
  by	
  the	
  user	
  they...
Admin	
  API
Port	
  :4985
• Bind	
  only	
  to	
  localhost!!!!	
  
• Superset	
  of	
  the	
  REST	
  API	
  on	
  :4984
Read	
  all	
  changes
• Bypasses	
  authenHcaHon	
  /	
  authorizaHon	
  
• Great	
  for	
  admin	
  tasks
Admin	
  UI
• Browse	
  channels	
  
• Simulate	
  results	
  of	
  changing	
  the	
  sync	
  funcHon
hOp://localhost:498...
Edit	
  User	
  Accounts
• Add	
  admin_channels,	
  change	
  password,	
  etc
Document	
  Model
Differences	
  from	
  Couchbase	
  Server
• Metadata	
  is	
  inside	
  the	
  JSON	
  
“_”-­‐prefixed	
  fields	
  (“_id”,	...
Revision	
  Trees
• Documents	
  store	
  revision	
  trees	
  (“hash	
  histories”)	
  
• Tree	
  stores	
  metadata	
  
...
Fiong	
  This	
  Into	
  Couchbase	
  Server
{
"_sync": {
"channels": {
"short": null,
"word": null
},
"history": {
"bodie...
Document	
  Types
• ApplicaHon	
  data	
  documents	
  
• “Local”	
  documents	
  
Used	
  by	
  client	
  replicators	
  ...
Coexistence	
  With	
  Couchbase	
  Apps
Sharing	
  Isn’t	
  Easy
• App	
  servers	
  reading	
  from	
  Gateway’s	
  bucket	
  
“What’s	
  this	
  ‘_sync’	
  crap...
Bucket	
  Shadowing
• Give	
  app	
  and	
  Gateway	
  their	
  own	
  buckets	
  
• Shadower	
  task	
  watches	
  both	
...
Read-­‐only	
  direct	
  access
• Reads,	
  writes	
  and	
  channel	
  
subscripHons	
  via	
  Gateway	
  
• Map	
  reduc...
Webinar- Couchbase Mobile Sync Gateway Configuration and Management Webinar mobile   advanced couchbase sync gateway(1)
Webinar- Couchbase Mobile Sync Gateway Configuration and Management Webinar mobile   advanced couchbase sync gateway(1)
Upcoming SlideShare
Loading in...5
×

Webinar- Couchbase Mobile Sync Gateway Configuration and Management Webinar mobile advanced couchbase sync gateway(1)

1,678

Published on

Get an in-depth look into Couchbase Sync Gateway, and learn more about how it works and what is the best practice for how it should be used.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,678
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
44
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Webinar- Couchbase Mobile Sync Gateway Configuration and Management Webinar mobile advanced couchbase sync gateway(1)

  1. 1. Sync  Gateway  In-­‐Depth Chris  Anderson   June  24,  2014 V1.0 - 3/10/14
  2. 2. Architecture
  3. 3. What’s  It  For?
  4. 4. 1.  Syncing  With  Couchbase  Lite • MulH-­‐master  sync,  scalable  to  any  number  of  clients   • Based  on  CouchDB  replicaHon  protocol
  5. 5. 2.  AuthenHcaHon • Sync  Gateway  is  publicly  reachable  outside  the  firewall   • Security  barrier  between  clients  and  Couchbase  Server   • User  Accounts   Stored  as  docs  in  bucket   Passwords  stored  as  digests,  using  bcrypt   • Logins   HTTP  Basic  auth   Cookie-­‐based  sessions   App  server  can  generate  sessions  for  custom  auth
  6. 6. 3.  AuthorizaHon • Channel  tags  grant  read  access  to  documents   Each  user  has  access  to  a  specific  set  of  channels   Channel  access  can  be  configured  via  REST  API   Sync  funcEon  can  grant  access  on  the  fly   • Sync  funcHon  performs  data  validaHon   JavaScript  funcEon  defined  by  app  developer,  provided  in  config   Called  on  every  document  update   Throws  excepEon  to  reject  a  change   Must  programmaEcally  enforce  write  access  to  channels
  7. 7. 4.  App  Workflow • Channels  route  documents  between  users   • Sync  funcHon  drives  the  enHre  workflow   Tagging  documents  with  channels   GranEng  channel  access  to  users   ValidaEng  data  
  8. 8. Architecture  Diagram
  9. 9. Architecture Server  3Server  1 Server  2 Couchbase  Server Channel Sync  Gateway Channel Channel Sync  Gateway Channel Couchbase  Lite  for  iOS  and  Android On Premise In the cloud
  10. 10. Sync  Gateway  Components Sync   REST  API Couchbase   Smart  Client Revision/ Conflict   Management AuthenHcaHon App’s  Sync   FuncHon Channel   Change   Tracking External  Auth   Services to  client to  Couchbase  Server
  11. 11. Sync  Gateway:  Incoming  Changes Sync   REST  API AuthenHcaHon External  Auth   Services ! • Pushes  changes  from  client:   • POST  /db/_revs_diff   • POST  /db/_bulk_docs ! • HTTP  Basic  (over  SSL),  or   • Session  cookie   ! • Facebook   • Persona  (email-­‐based)   • Custom   • LDAP,  etc.  *
  12. 12. Sync  Gateway:  App  Logic  &  Storage Couchbase   Smart  Client Revision/ Conflict   Management App’s  Sync   FuncHon to  Couchbase  Server !function(doc,  oldDoc)  {   …   requireUser(oldDoc.owner);   …   channel(doc.channel);   …   access(doc.members,  doc.roomID);   } validaHon rouHngaccess  ctrl rev 1 rev 2 rev 3a rev 3b AuthenHcaHon
  13. 13. Sync  Gateway  Components Sync   REST  API Couchbase   Smart  Client Revision/ Conflict   Management AuthenHcaHon App’s  Sync   FuncHon Channel   Change   Tracking TAP  feed  from  server
  14. 14. MulH-­‐Master  ReplicaHon
  15. 15. Features • Any  number  of  clients   • Arbitrary  topologies  (from  centralized  to  P2P)   • Occasionally-­‐connected  clients   • Conflict  resoluHon   No  data  loss   Usually  client-­‐driven   Asynchronous   • Some  delta  opHmizaHons   Unchanged  aOachments  aren’t  sent   Lots  of  room  to  opEmize  here  (delta  encoding)
  16. 16. ReplicaHon  Types • One-­‐direcHonal   “Push”  to  server   “Pull”  from  server   • One-­‐shot  or  conHnuous   ConEnuous  offers  low-­‐latency  changes   but  locks  up  a  server  socket   Polling  is  a  compromise   • Always  client-­‐iniHated   Sync  Gateway  is  passive
  17. 17. “Push”  ReplicaHon • Consult  local  db  for  revisions  added  since  last  checkpoint   • POST  list  of  {doc  ID,  rev  ID}  tuples  to  _revs_diff   Response  contains  subset  that  are  new  to  the  server   • plus  latest  rev  IDs  known  to  server   • PUT  each  new  revision  to  server   Including  revision  history  to  incorporate  into  tree   and  aOachments  added  since  server’s  last  known  revision   • Save  checkpoint  with  latest  sequence  processed
  18. 18. “Pull”  replicaHon • Read  server’s  “_changes”  feed   StarEng  from  just  past  last  checkpoint  sequence   List  of  {sequence,  doc  ID,  leaf  rev  ID(s)}  tuples   • Consult  local  db  to  find  unknown  revisions   • GET  each  revision   Tell  server  latest  rev  ID(s)  I  have,  to  prune  unchanged  aOachments   Server  includes  rev  history  to  incorporate  into  tree   Response  usually  MIME  mulEpart/related   • Save  checkpoint  with  latest  sequence  processed
  19. 19. Changes  Feed • Most  difficult  part  of  enHre  project   Linear  history  of  all  document  changes,  per  channel   Must  be  efficient,  scalable,  reliable   • Source  of  truth:  a  view   But  view  queries  are  expensive   • Source  of  speed:  Tap  feed   Parse  incoming  document  changes   Queue  in  sequence  order   Cache  by  channel   Consult  view  for  older  changes for  (channel  in  doc.channels)  {      emit([channel,  doc.sequence],                doc.revID,  doc.deleted);   }
  20. 20. Demo  ApplicaHon
  21. 21. Sync  FuncHon
  22. 22. JSON  Document  Schema
  23. 23. Sync  FuncHon
  24. 24. Sync  FuncHon • Task  documents  belong  to  lists
  25. 25. Sync  FuncHon • List  documents  specify  owners  and  members   ! ! ! ! ! • This  app  shares  the  list  info  with  all  members  (easy  to   change)
  26. 26. Sync  FuncHon • Profile  documents  are  distributed  to  all  users   • Owned  by  the  user  they  describe
  27. 27. Admin  API
  28. 28. Port  :4985 • Bind  only  to  localhost!!!!   • Superset  of  the  REST  API  on  :4984
  29. 29. Read  all  changes • Bypasses  authenHcaHon  /  authorizaHon   • Great  for  admin  tasks
  30. 30. Admin  UI • Browse  channels   • Simulate  results  of  changing  the  sync  funcHon hOp://localhost:4985/_admin/
  31. 31. Edit  User  Accounts • Add  admin_channels,  change  password,  etc
  32. 32. Document  Model
  33. 33. Differences  from  Couchbase  Server • Metadata  is  inside  the  JSON   “_”-­‐prefixed  fields  (“_id”,  “_rev”,  etc.)   • More-­‐robust  MVCC   Digest-­‐based  “_rev”  property,  not  uint64  CAS   “_rev”  idenEfies  a  revision  globally  across  all  replicas   Ties  into  revision  tree  (q.v.)   • Anachments   Arbitrary-­‐size  binary  blobs   Tagged  with  name  and  MIME  type   Metadata  visible  as  “_aOachments”  property
  34. 34. Revision  Trees • Documents  store  revision  trees  (“hash  histories”)   • Tree  stores  metadata   Revision  ID  (based  on  SHA-­‐1  digest  of  contents)   DeleEon  status  (“tombstone”)   JSON  contents  of  old  revs  deleted  during  compacEon   • “Pruning”  eventually  deletes  oldest  tree  items   • Tree  structure  supports  conflicts   Conflicts  are  not  errors!   ResoluEon  can  be  deferred  unEl  convenient,  or  never   There  is  always  a  single  “default”  or  “winning”  revision
  35. 35. Fiong  This  Into  Couchbase  Server { "_sync": { "channels": { "short": null, "word": null }, "history": { "bodies": [ ""], "channels": [ ["short","word"] ], "parents": [ -1 ], "revs": [ "1-86effb929acbf953905dd0e3974f6051" ] }, "rev": "1-86effb929acbf953905dd0e3974f6051", "sequence": 1, "time_saved": "0001-01-01T00:00:00Z" }, "word": "cat" }
  36. 36. Document  Types • ApplicaHon  data  documents   • “Local”  documents   Used  by  client  replicators  to  store  checkpoints   • User  accounts   • Roles   • Binary  anachments   • Obsolete  revisions   Removed  when  database  is  compacted   • A  single  sequence  counter
  37. 37. Coexistence  With  Couchbase  Apps
  38. 38. Sharing  Isn’t  Easy • App  servers  reading  from  Gateway’s  bucket   “What’s  this  ‘_sync’  crap  in  my  data?”   “What  are  all  these  extra  docs  like  ‘_sync:user:snej’”?   • App  servers  updaHng  docs  in  the  bucket  is  worse   Gateway:  “Who  moved  my  cheese?!”   App  removing  “_sync”  property  is  disastrous   App  preserving  “sync”  property  is  sEll  bad:   • Rev  tree  isn’t  updated   • Sequence  number  isn’t  bumped   • But  Gateway  can’t  tell  anything’s  wrong
  39. 39. Bucket  Shadowing • Give  app  and  Gateway  their  own  buckets   • Shadower  task  watches  both  buckets’  Tap  feeds   Adds  changes  from  app  bucket  to  Gateway  docs  as  new  revisions   Copies  current  rev  of  Gateway  doc  to  app  bucket   • Asynchronous  replicaHon   Shadowing  is  best  for  adding  sync  to  exisEng  high-­‐volume  web  apps
  40. 40. Read-­‐only  direct  access • Reads,  writes  and  channel   subscripHons  via  Gateway   • Map  reduce  queries  directly   to  Couchbase  Server
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×