• Like

CouchConf London Syncpoint

  • 624 views
Uploaded on

 

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
624
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
6
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • Everyone is familiar with this from grade school science classPrecipitation is brings water from the clouds to the groundAggregated through rivers and streams into lakes, oceansThen returns to the cloud through evaporation
  • Authoritative source of data in the cloudDevices used to browse, search and present informationNew/Updated information capturedCollected, Aggregated and Reconciled with the data in the cloud
  • More and more mobile devices at both ends of the cycleTablets have become hugely popular way of consuming informationSmart phones in our pockets have a variety of sensors2 cameras, GPS, compass, accelerometers, gyroscope
  • A LOT of devices2012 more mobile connected devices than humans on earthJOKE about all my mobile devices
  • Ultimately data is about getting work doneIn mobile space many non-obvious examplesWe’re not JUST talking about phones
  • Sensors in the field, measure soil moisture, pH, temperatureLow powered radios, only powerful enough to transmit to other farm equipmentRolled up Analyzed, identify problems, improve productivity of farm
  • Soldier as sensorInformation needs to flow from people on the ground to key decision makersAbsolutely needs to be distributed, if satellite is taken out, must not affect local opsSecurity is critical, 2 soldiers meeting on battle field need to pair devices,BUT not the enemy
  • Fleet management, not just Fedex and UPS any more, any company with fleet of vehicles has satellite tracking of vehiclesFrequently, whether its delivery or repair, staff have mobile devices to access things like inventory, instructions, etcWarehouse , hand-held barcode scannersBoth of these cases, MUST be able to continue work in disconnected state
  • In Hospital environment, doctors want iPads, often no radio transmissions in some locations, sync up at nurses stationPoint of Sale, we have distributed retail locations with sales and inventory dataEven have restaurants using mobile devices to take orders and process paymentsMobile truly is everywhere
  • Characterized as 3 different types, important to understand there are any number of hybrids Type 1 is syncing MY stuff, useful to get your data on all your devices, not so good at sharing data with othersType 2 has large master database, users need access to a subset of that on a device, lots of variety in how these subsets are definedType 3 involves rolling up machine generated data into the cloud for analytics
  • Type 1 supported well by iCloud and their competitorsType 3 SmartMeter Network, Fame data aggregationWe’ll be talking a lot more about supporting type 2
  • Supporting type 2 in a general way has all kinds of challenges
  • The system of record is in the cloud, but now we have replica copies on potentially millions of devices…
  • Scalability of Master - Use Couchbase.Provision new users and devices.Selection, scalability: temporal, geospatial, user specificTransport, bandwidth, conserving battery and memoryConflict detection and resolution
  • ElasticSearch for defining ChannelsChannels represent the subsets of the master database available for syncingExcellent clustering supportLeverage existing lucene indexing knowledgeSyncpoint databases provide sync API between device and cloudAlso manage conflicts between device and couchbase server
  • Document mutation comes into the Couchbase ClusterWe use Couchbase facility known as a TAP stream to ensure all data gets to ESThen we use percolation feature of ES to determine which documents belong to which channel
  • Some examples of the higher-level API that we plan to offerEstablishing session removes a lot of authentication/provisioning detailsSubscribing to channels simplifies the logic of filtering the data
  • TouchDB is the foundation for storage on the mobile devicesBased on everything we learned from initially porting the Erlang version of CouchDB to iOS and AndroidUses SQLite for persistent storage and has CouchDB compatible APISyncpoint API sits on top of this and adds the session and channel capabilities
  • TouchDB betas for iOS and Android available on githubDemo TimeGrocery sync shows replication between TouchDBiOS, Apache CouchDB, and TouchDBAndroid

Transcript

  • 1. Connecting Millions ofMobile Devices to the Cloud Marty Schoch @mschoch 1
  • 2. THE WATER CYCLE 2
  • 3. The water cycle Fun fact: In 20 minutes, one thunderstorm can drop 125,000,000 gallons of water. 3
  • 4. The water data cycle 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 4
  • 5. Increasingly, it starts and ends with a mobile device Collect 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 Capture Consume 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 1110101101011010001010101110101010101110101001010100101010101000101010101 5
  • 6. Actually, it starts and ends with a LOT of mobile devices Mobile-broadband The number of mobile- subscriptions have grown connected devices will 45% annually over the last exceed the world’s four years and today there population in 2012; and are twice as many mobile- monthly global mobile data broadband as fixed- traffic will surpass 10 broadband subscriptions. exabytes in 2016. Fun Fact: By the end of 1999, the sum of human-produced information (including all audio, video recordings and text/books) was about 12 exabytes of data, according to Berkeley study. 6
  • 7. PUTTING DATA TO WORK 7
  • 8. Make farming data work 8
  • 9. Make intelligence data work 9
  • 10. Make industrial data work 10
  • 11. Make business data work 11
  • 12. Connecting millions of mobile devices to the cloud analysis other farms Type 1 Type 2 Type 3 backup and sync sync relevant portion collect machine- my stuff of a database generated data 12
  • 13. We’ll focus on Type 2 Type 2 sync relevant portion of a database Most applications combine patterns. 13
  • 14. CHALLENGES … SOLUTIONS 14
  • 15. Type 2 Mobile Sync The ultimate home of the data for a business application is not on the device, but is a central data store in the cloud. 15
  • 16. Challenges Schema evolution Security Scalability of consolidated store Selection performance Bandwidth conservationReferential integrity on interruption Battery and memory conservation Deletion propagation New user provisioning Which data – temporal, spatial, user? Conflict detection and resolution Hard at “enterprise scale.” Excruciating at “web scale.” 16
  • 17. A multi-tier, scalable type 2 sync architecture App Server DOCUMENTS Elastic Search Couchbase M DOCUMENTS (TO INDEX) CHANNEL MEMBERSHIP Syncpoint F-channel C-channel Load balance DOCUMENTS SYNC METADATA 17
  • 18. DetailsSubject to Change 18
  • 19. ElasticSearch Queries and Syncpoint ChannelRegister ElasticSearch query named strongBeerfor percolation { "range": { "abv": { "from": 7, "to": 10 } } }Then define a Syncpoint channel with strongBeeras a member query 19
  • 20. The architecture, in more detail Document Mutation Couchbase Cluster TAP Stream Percolation Results ElasticSearch Cluster Syncpoint Cluster • Q1 • Q2 • C1 (Q1, Q2) • Q3 • C2 (Q2) • C3 (Q1, Q2, Q3) 20
  • 21. Syncpoint Client APIMobile Client Syncpoint Cluster Establish Session • Pluggable Authentication • Pairs user with device • Establishes control database Subscribe to Channel(s) • Creates a local installation of this channel • Bi-directional sync of channel documents 21
  • 22. TouchDB – Foundation for Syncpoint Native Library 22
  • 23. TouchDB Betas – Works with CouchDB Today Not a complete product but a functional open source projects with production deployments. Open source, and available from Couchbase labs. TouchDB for TouchDB for iOS Android 23
  • 24. JOIN THE COMMUNITYhttp://groups.google.com/group/mobile-couchbase 24
  • 25. THANKSMARTY@COUCHBASE.COM 25