Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Couchbase Server architecture: an overview – Connect Silicon Valley 2017

301 views

Published on

Speaker: David Haikney, Dave Rigby

Blazing performance, rich indexing, query and search capabilities plus high availability ingrained to the core. In this session we will demonstrate Couchbase Server 5.0’s industry-leading features and explore behind the scenes to uncover the architecture that underpins it. This is the grand tour of Couchbase Server, so this is the session for you if you are a master architect, developer, or administrator of platforms.

Published in: Technology
  • Be the first to comment

Couchbase Server architecture: an overview – Connect Silicon Valley 2017

  1. 1. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. ARCHITECTURE TO START A REVOLUTION Dave Rigby, Principal Engineer David Haikney, Director of Engineering
  2. 2. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. AGENDA 01/ 02/ 03/ For You, For Your Customers Architecture Fundamentals Demo 04/ Overview of Services
  3. 3. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 1 FOR YOU, FOR YOUR CUSTOMERS
  4. 4. Anywhere Anytime Customers now expect exceptional digital experiences Personalized Feature-rich Quick
  5. 5. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 5 Agility to adapt to changes in demand Time to Market What businesses need to succeed in this new world Multi-channel, multi-device engagement 5Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. •Meet expectations of a large and growing customer base
  6. 6. Every aspect of our architecture is designed to meet these needs Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 6
  7. 7. Powerful Capabilities + Ease of Management Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 7 COUCHBASE SERVER NODE Analytics Full Text Search Data Query Global Index Cluster Manager • Rich Set of Services • Pay For What You Use • Single Install • Single UI • Industry Leading Performance • Dynamically Scalable 5.0 5.0
  8. 8. Built For Developers Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 8 • Flexible Schema • Access the data the way YOU want • KV / Query / Search / REST interfaces • Rich set of supported SDKs • Focus on what you do best • Automatic sharding & replication • Mobile synchronisation built-in App Server SDK Application
  9. 9. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 2 ARCHITECTURE HIGHLIGHTS
  10. 10. Choose the features you want, the way you want them Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 10
  11. 11. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 11 Multidimensional Scaling Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 11 NODE 1 NODE 12 Cluster Manager Data Global Index Search AnalyticsQuery
  12. 12. Multidimensional Scaling: Examples Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 12 NODE 1 SearchAnalytics Global Index Query Data Cluster Manager NODE 4 Example Setup: Q/A NODE 1 Analytics Search Data Query Global Index Cluster Manager NODE 2 Example Setup: Development
  13. 13. Automatic Sharding Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 13
  14. 14. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 14 Automatic Sharding Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 14 CLUSTER MANAGER DATA SERVICE App Server SDK Application doc_connect_2017 HASH(“doc_connect_2017”) ==> 27 VB Node 0 A 1 B 2 C … 27 C 1023 D { … } NODE A NODE B NODE C NODE D
  15. 15. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 15 Automatic Sharding Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 15 CLUSTER MANAGER DATA SERVICE App Server SDK Application doc_connect_2017 VB Node 0 A 1 B 2 C … 27 C 1023 D { … } NODE A NODE B NODE C NODE D
  16. 16. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 16 Automatic Sharding Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 16 CLUSTER MANAGER DATA SERVICE App Server SDK Application doc_connect_2017 HASH(“doc_connect_2017”) ==> 27 VB Node 0 A 1 B 2 C … 27 C 1023 D { … } NODE A NODE B NODE C NODE D VB Node 0 A 1 B 2 C … 27 E 1023 D NODE E
  17. 17. Efficient Data Propagation Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 17
  18. 18. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 18 The Database Change Protocol EXTERNAL SERVICES • Backup • Mobile Sync Gateway • Hadoop • Kafka • High Performance / In-Memory • De-Duplication • Ordered, predictable and consistent • Restartable Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 18 Data Service DCP
  19. 19. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 3 DEMO
  20. 20. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 4 OVERVIEW OF SERVICES
  21. 21. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 21 The Cluster Manager COUCHBASE SERVER NODE Analytics Full Text Search Data Query Global Index Cluster Manager
  22. 22. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 22 The Cluster Manager (ns_server) The control plane of the server: • Cluster membership • Status & health monitoring • Service layout • Data placement • Rebalance • 5-Second Failover • Authentication • Admin APIs per-node-&-bucket services Generic distributed facilities Generic local facilities Logging and Other Services Distributed node discovery Master Services • Cluster-wide operations • Data placement • Rebalance • Auto-failover Admin Console & REST API Global config (gossip replication) Local config Store Per-node Services • Heartbeats • Babysitter Bucket services • DCP management • Stats collection & monitoring Authentication
  23. 23. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 23 The Data Service COUCHBASE SERVER NODE Analytics Full Text Search Query Global Index Cluster Manager Data
  24. 24. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 24 Data Service Architecture … Database Engine (kv_engine) Listener vBucket Manager Item Pager Expiry Pager Checkpoint Manager Cache ..Partition Hash Tables AuthNetwork IO Flusher Scheduler Reader IO Writer IO Non IO Batch Reader The (low-level) data plane of the server: • Key/Value access • Full Document and Sub-Document • Persistent and Ephemeral Buckets • Active Memory Defragmentation • Map/Reduce • Database Change Protocol Data Service
  25. 25. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 25 Query COUCHBASE SERVER NODE Analytics Full Text Search Data Global Index Cluster Manager Query
  26. 26. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 26 The Query Service ©2017 Couchbase. All rights reserved. 26 Global Index CLIENT Key/Value API FetchScanParse Plan Join Filter Pre- Aggregate Offset Limit Project Data-parallel — Query is N data streams over N cores* Memory-based Pluggable architecture — datastore, index… REQUEST RESPONSE SortAggregate
  27. 27. 11:15am N1QL: what's new in Couchbase 5.0 Keshav Murthy, Senior Director, Couchbase R&D, Couchbase Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved.
  28. 28. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 28 Global Secondary Indexing COUCHBASE SERVER NODE Analytics Full Text Search Data Query Global Index Cluster Manager
  29. 29. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 29 Data Service Projector & Router Indexing Service • The Power behind efficient queries • New Plasma Storage Engine Query ServiceIndex Service Supervisor Index maintenance & Scan coordinator Index#2Index#1 DCP Index#4Index#3 Plasma Storage Engine Bucket 1 Bucket 2
  30. 30. 2:30pm Indexes: Batman of the data world Venkat Subramanian, Product Manager, Couchbase Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved.
  31. 31. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 31 Full Text Search COUCHBASE SERVER NODE Analytics Full Text Search Data Query Global Index Cluster Manager
  32. 32. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 32 Full Text Search “Googling for your JSON documents” • NOW GA! • Index Fields or Documents • Lexical Analysis and Stemming • Flexible Query Capabilities • Available and Scalable Config { Node Definitions: … Index Definitions: … Plan Indexes: … } Planner Janitor PIndex bleve REST Interface Manager PIndex bleve DCP
  33. 33. 3:45pm Enabling Full Text Search for Couchbase documents Marty Schoch, Senior Software Engineer for Search, Couchbase Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved.
  34. 34. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 34 The Analytics Service COUCHBASE SERVER NODE Analytics Full Text Search Data Query Global Index Cluster Manager
  35. 35. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 35 Analytics Service • Currently Developer Preview • Gain more complex insight into your data • Fed via DCP • No impact on application’s key / value ops • Shadow is a Query-specific distribution and organization of data, aggregation, index, and query processing • Longer term data can be persisted even when operational data is expired/deleted ©2017 Couchbase. All rights reserved. 35 Analytics Service Shadow DCPDCP SHARD 1 SHARD 3 SHARD SHARD SHARD 6 SHARD Data Service 1 SHARD 4 SHARD 7 SHARD SHARD SHARD 8 SHARD Data Service 2 SHARD 5 SHARD 2 SHARD SHARD SHARD 9 SHARD Data Service 3 DCP Node 1 Node 2 Node 3 ShadowShadow
  36. 36. 4:30pm Couchbase Analytics: an overview Till Westmann, Senior Director, Server Development, Couchbase Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved.
  37. 37. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. THANK YOU

×