Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Creating a data pipeline with Couchbase Mobile – Couchbase Connect 2016

631 views

Published on

In this session, Fujio will cover Couchbase Mobile from an operational perspective. He will dive into replication topologies, deployment strategies, and how Couchbase Mobile is being used full-stack database solution in the enterprise. Specifically, he will cover:

– How Sync Gateway filtered replication works

– How to simplify data flow in an organization

– Common tactics we see in enterprise applications including push vs. pull

Published in: Software
  • Be the first to comment

Creating a data pipeline with Couchbase Mobile – Couchbase Connect 2016

  1. 1. Couchbase Mobile Enterprise Solutions NOT just Sync
  2. 2. ©2015 Couchbase Inc. 2 Couchbase Mobile Solution Couchbase Lite Sync Gateway Couchbase Server Full featured embedded NoSQL database Secure data synchronization over the web Our flagship, highly scalable NoSQL database in the cloud
  3. 3. ©2015 Couchbase Inc. 3 Replication Topologies STAR TIERED MESH Full multi-master replication Continuous or ad-hoc in either direction
  4. 4. ©2015 Couchbase Inc. 4 Topics How ChannelsWorks Past, Present & Future Simplify Data Flow in an Organization Push vs Pull Scaling
  5. 5. ©2015 Couchbase Inc. 5 Syncing a single user’s database is EASY
  6. 6. ©2015 Couchbase Inc. 6 What about syncing x,000,000?
  7. 7. ©2015 Couchbase Inc. 7 Read-side filtering doc.owner == 'bob'
  8. 8. ©2015 Couchbase Inc. 8 Read-side filtering is too slow
  9. 9. ©2015 Couchbase Inc. 9 Bob Jim channel(doc.owner) Partition data during writes instead
  10. 10. ©2015 Couchbase Inc. 10 Now syncing X,000,000 is easy
  11. 11. ©2015 Couchbase Inc. 11 Sync Gateway & Channels – Pull Feed { “_id”:”12”, “_rev”:”3-hno..”, “channels”:[“yellow”] } { “_id”:”78”, “_rev”:”5-u9b..”, “channels”:[“yellow”] } 78 31 20 17
  12. 12. ©2015 Couchbase Inc. 12 Sync Gateway & Channels • Private • Public • Share Private
  13. 13. ©2015 Couchbase Inc. 13 Sync Gateway & Channels – Multi-channel Feed { “_id”:”12”, “_rev”:”3-hno..”, “channels”:[“yellow”] } { “_id”:”xzq”, “_rev”:”1-7tpb..”, “channels”:[“blue”] } { “_id”:”81x”, “_rev”:”2-jba..”, “channels”:[“red”] } 78 172031 1963
  14. 14. ©2015 Couchbase Inc. 14 Sync Gateway & Channels – Filtered Feed by Channel(s) 78 172031 63 19 Continuous Every 2 min Every 15 min & Only Wi-Fi
  15. 15. ©2015 Couchbase Inc. 15 Couchbase Mobile to My Current Systems Your App for: Your Business Logic 24 2625
  16. 16. ©2015 Couchbase Inc. 16 Couchbase Mobile to My Current Systems 24 2625 Get me channel feed: since=25 26 27 28 “Failure”
  17. 17. ©2015 Couchbase Inc. 17 Traditional - Data Flow 3rd Party Public Internal Mobile Mobile Internal Mobile 3rd Party in 2010
  18. 18. ©2015 Couchbase Inc. 18 Mobile the HardWay - 1st Request / Response Data RequestApp Auth
  19. 19. ©2015 Couchbase Inc. 19 Mobile the HardWay – 2nd Online/Offline Data Sync Query/Filter by User(s)/Group(s) Auth Data RequestApp Auth Buffer / Que Data ProcessingApp
  20. 20. ©2015 Couchbase Inc. 20 Mobile the HardWay - 3rd , 4th , 5th … etc. • Conflict Resolution ? • 2x users = 2x Reads & 2xWrites ? • New Release = New Features = New Schema = Schema Migration
  21. 21. ©2015 Couchbase Inc. 21 Simple Mobile - Data Flow 3rd Party Public Internal Mobile Mobile Internal Mobile 3rd Party
  22. 22. ©2015 Couchbase Inc. 22 Who has the “Truth” ? Big Applications Mini Applications
  23. 23. ©2015 Couchbase Inc. 23 Sync Gateway – the “Truth” “What is my value add ?”
  24. 24. ©2015 Couchbase Inc. 24 Sync Gateways –Value Add & Data Sourcing { “_id”:”flight::100::sfo”, “_rev”:”2-aoru”, “seats”:[ {“a1”:”open”} ………. ], “status”:”onTime”, “channel”:”sfo::100” } { “_id”:”weather::sfo”, “_rev”:”5-8oum”, “windSpeed”:”75knots”, ”windDirection”:”sw”, “channel”:”sfo” } { “_id”:”price::100::sfo::a1”, “_rev”:”3-d90j’”, “src”:{“_rev”:”2-aoru”}, “price”:450.00 “cost”: { “weather”:-55.00, ”src”:{ “_rev”:”5-8oum” }, }, “channel”:”price::sfo::100” } ?
  25. 25. ©2015 Couchbase Inc. 25 Market Place of Micro Services Source 2 Source 1 Source 4 Source 3 Different: Data , Locations and/or Sync Functions Public Data Private Data
  26. 26. ©2015 Couchbase Inc. 26 Replication SG-Replicate XDCR • Replicate via REST/HTTP • Master/Master or Master/Slave • Fine Grain Data Replication vi channel(s) • Replication offline for years is OK. • Replicate via Memory to Memory • Master/Slave • Full Data Copy • ReplicationOffline for few days is OK. or
  27. 27. ©2015 Couchbase Inc. 27 “Chaining” Sync Gateways – Data Locality Channels[“A”,”B”] Channels[“A”,”T”,”Z”] DC Failed? No Problem go to the main source DC1 WEST DC2 EAST via SG-Replicate (built-into Sync Gateway v1.3)
  28. 28. ©2015 Couchbase Inc. 28 “Chaining” Sync Gateways - Options Unidirectional Bidirectional& & Future Changes
  29. 29. ©2015 Couchbase Inc. 29 Push PUSH {“data”} Q: Who is responsible for getting the data to the BLUE app? Q: How dose the BLUE app know if the publisher is available? A: “it depends” or “its complicated” A: BLUE app polls the publisher or publisher send empty packets
  30. 30. ©2015 Couchbase Inc. 30 Pull – Streaming or Polling PULL {“data”} A: The BLUE app A: Connection timeout to publisher Q: Who is responsible for getting the data to the BLUE app? Q: How dose the BLUE app know if the publisher is available? Q: How dose the BLUE app know what data it needs? A: Query publisher or last sequence number since=25
  31. 31. ©2015 Couchbase Inc. 31 Couchbase Mobile - Before Load Balancer Couchbase Mobile v1.1.1 & lower 20,000 Max Concurrent CRUD , Security & Seq. Number HA Data Storage & Channel Index
  32. 32. ©2015 Couchbase Inc. 32 Problems – Single Global Sequence Number Updating a single key at scale causes too many problems
  33. 33. ©2015 Couchbase Inc. 33 Solution - DCP (Database Change Protocol) Cluster data replication via sequential streaming feed from each vBucket (shard) on each node called “DCP”
  34. 34. ©2015 Couchbase Inc. 34 Sync Gateway Accelerator (Enterprise Only) Couchbase Mobile v1.2+ Accelerator listens to DCP to create k/v channel indexes & uses DCP sequence numbers Same vLAN
  35. 35. ©2015 Couchbase Inc. 35 Horizontally Scale AllThree Parts in v1.2+ LB x,000,000+ Concurrent CRUD & Security HA Data Storage Channel Index & Seq. Number
  36. 36. ©2015 Couchbase Inc. 36 Consolidation Key/Value & Document Query & Index Mobile Search Analytics

×