Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Overview of Couchbase Server architecture – Couchbase Connect 2016

1,102 views

Published on

In this session we will look at the Couchbase Server architecture in detail and provide attendees with an understanding of how the cluster manager, cache engine, and storage engine plug together with the data, query, and index services to give you a best-of-breed NoSQL engine for big data processing. This is the grand tour of Couchbase Server, so this is the session for you if you are a master architect, developer, or administrator of platforms.

Published in: Software
  • Be the first to comment

  • Be the first to like this

Overview of Couchbase Server architecture – Couchbase Connect 2016

  1. 1. ©2016 Couchbase Inc. Couchbase Server Architecture 1
  2. 2. ©2016 Couchbase Inc.©2016 Couchbase Inc. LookWho’sTalking David Haikney Director of Engineering, EMEA 2 Dave Rigby Technical Lead, Data Service
  3. 3. ©2016 Couchbase Inc.©2016 Couchbase Inc. WhatTo Expect •Whistle-StopTour with demos •Key Architectural Concepts •Exploration of each Service •Where to hear more @ Connect! 3
  4. 4. ©2016 Couchbase Inc. 4 Key Concepts
  5. 5. ©2016 Couchbase Inc.©2016 Couchbase Inc. Concept 1 –Your Choice of Services STORAGE Couchbase Server 1 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster Manager Cluster Manager Data Service Index Service Query Service Full Text Search
  6. 6. ©2016 Couchbase Inc.©2016 Couchbase Inc. Concept 1 –Your Choice of Services STORAGE Couchbase Server 1 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster Manager Cluster Manager Data Service Index Service Query Service STORAGE Couchbase Server 2 Managed Cache Cluster Manager Cluster Manager STORAGE Couchbase Server 3 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster Manager Cluster Manager STORAGE Couchbase Server 4 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster Manager Cluster Manager STORAGE Couchbase Server 5 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster Manager Cluster Manager STORAGE Couchbase Server 6 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster Manager Cluster Manager Full Text Search
  7. 7. ©2016 Couchbase Inc.©2016 Couchbase Inc. Concept 1 –Your Choice of Services STORAGE Couchbase Server 1 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster Manager Cluster Manager Data Service STORAGE Couchbase Server 2 Managed Cache Cluster Manager Data Service STORAGE Couchbase Server 3 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster Manager Data Service STORAGE Couchbase Server 4 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster Manager Index Service STORAGE Couchbase Server 5 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster Manager Index Service STORAGE Couchbase Server 6 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster Manager Query Service FTS Service
  8. 8. ©2016 Couchbase Inc.©2016 Couchbase Inc. Find Out More! Preparing for success: the top deployment 10 do's and don'ts • Today 11:00 am - 11:50 am • OperationsTrack • Great America 1 8 Karthik Sekar Solutions Architect, Couchbase Ian McCloy Technical Support Manager, EMEA, Couchbase
  9. 9. ©2016 Couchbase Inc.©2016 Couchbase Inc. Find Out More! Picking the right API for the right job • Wednesday 2:00 pm - 2:50 pm • DevelopmentTrack • Great America K 9 Michael Nitschinger JVM Engineer, Couchbase
  10. 10. ©2016 Couchbase Inc.©2016 Couchbase Inc. Application Concept 2 – Automatic Sharding Cluster 10 Key = connect2016 HASH(“connect2016”) Server A Server CServer B VB Server 0 A 1 B 2 C … 27 B 1023 C { … }
  11. 11. ©2016 Couchbase Inc.©2016 Couchbase Inc. Application Concept 2 – Automatic Sharding Cluster 11 Key = connect2016 HASH(“connect2016”) Server A Server CServer B VB Server 0 A 1 B 2 C … 27 D 1023 C { … } Server D
  12. 12. ©2016 Couchbase Inc.©2016 Couchbase Inc. Concept 3 – Exceptional Data Streaming • DatabaseChange Protocol (DCP) • High Performance / In-Memory • De-Duplication • Ordered, predictable and consistent • Restartable 12 Data Service Index Service Map/ Reduce DCP Cross Data Center Cluster Hadoop Client/App Notifications (future) Backup/ Export XDCR Full Text Search Data Replication
  13. 13. ©2016 Couchbase Inc. 13 STORAGE SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster Manager Cluster Manager Data Service Index Service Query Service Full Text Search
  14. 14. ©2016 Couchbase Inc. 14 Cluster Manager
  15. 15. ©2016 Couchbase Inc.©2016 Couchbase Inc. The Cluster Manager STORAGE SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster Manager Cluster Manager Data Service Index Service Query Service Full Text Search The control plane of the server: • Cluster membership • Status & health monitoring • Service layout • Data placement • Rebalance • Failover • Authentication • Admin APIs Implemented in Erlang
  16. 16. ©2016 Couchbase Inc.©2016 Couchbase Inc. The Cluster Manager The control plane of the server: • Cluster membership • Status & health monitoring • Service layout • Data placement • Rebalance • Failover • Authentication • Admin APIs Implemented in Erlang Generic distributed facilities • Global configuration (gossip replication) • Distributed node discovery Generic local facilities Master services • Cluster-wide operations • Data placement • Rebalance • Auto-failover Admin Console & REST API Per-node services • Heartbeats • Babysitter Bucket services • DCP management • Stats collection & monitoring Authentication • Local configuration store • Logging and Other Services • Vclock, uuid • Work queue • Misc
  17. 17. ©2016 Couchbase Inc. 17 Data Service
  18. 18. ©2016 Couchbase Inc.©2016 Couchbase Inc. The Data Service The (low-level) data plane of the server: • Key/Value access • Map/ReduceViews [*] KV-Engine: • Evolution of memcached; adding persistence, replication, enhanced data access APIs • Asynchronous networking supports ~10K clients. • Memory-centric architecture. • Disk IO performed via background threads. C++11 18 STORAGE SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster Manager Cluster Manager Data Service Index Service Query Service Full Text Search
  19. 19. ©2016 Couchbase Inc.©2016 Couchbase Inc. Data Service Architecture Database Engine (ep-engine) Dispatcher Request / Response Scheduler Reader IO Aux IONon IO … KV-Bucket … KV-Engine Item Pager Expiry Pager Checkpoint Manager Managed Cache Partition Hash Tables (Active and Replica) Flusher Batch Reader Partition Hash Tables (Active and Replica) Writer IO DCP Streaming Authentication
  20. 20. ©2016 Couchbase Inc. 20 Journey of a K/V operation
  21. 21. ©2016 Couchbase Inc.©2016 Couchbase Inc. Journey of a K/V operation 21 STORAGE Couchbase Server 1 Managed Cache Cluster Manager Data Service (1x replica) STORAGE Couchbase Server 0 Managed Cache Cluster Manager Data Service (1x replica) SDK Application DCP Replication 'tweets::Dhaikney' -> SD.upsert('hashtags', 'Connect2016')
  22. 22. ©2016 Couchbase Inc. 22 Indexing Service
  23. 23. ©2016 Couchbase Inc.©2016 Couchbase Inc. Index Service Three “indexing” services: • Incremental Map / ReduceViews • Javascript map() & reduce() functions applied to all mutations • Supports geo-spatial views • Co-located with Data service • Global Secondary Indexes (GSI) • Full-Text Search – more later GSI: efficient indexes for secondary lookups and ad-hoc query processing 23 STORAGE SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster Manager Cluster Manager Data Service Index Service Query Service Full Text Search
  24. 24. ©2016 Couchbase Inc.©2016 Couchbase Inc. Data Service Projector & Router Indexing Service Bucket 1 DCP Stream Projector and Router: 1 Projector and Router per node 1 stream of changes per buckets per supervisor Bucket 2
  25. 25. ©2016 Couchbase Inc.©2016 Couchbase Inc. Data Service Projector & Router Indexing Service Query ServiceIndex Service Supervisor Index maintenance & Scan coordinator Index#2Index#1 Query Processor cbq-engine Bucket 1 DCP Stream Index#4Index#3 B u c k e t # 2 B u c k e t # 1 Projector and Router: 1 Projector and Router per node 1 stream of changes per buckets per supervisor ForestDB Storage Engine Supervisor 1 Supervisor per node Many indexes per Supervisor Bucket 2 Memory- Optimized Indexes
  26. 26. ©2016 Couchbase Inc.©2016 Couchbase Inc. Find Out More! Memory-optimized indexes: how they work Today, 4:00 pm - 4:50 pm • Great America 2 26 Sarath Lakshman Senior Software Engineer, Couchbase
  27. 27. ©2016 Couchbase Inc. 27 Query Service
  28. 28. ©2016 Couchbase Inc.©2016 Couchbase Inc. Query Service Client FetchParse Plan Join Filter Pre-Aggregate Offset Limit ProjectSortAggregateScan Query Service Index Service Data Service
  29. 29. ©2016 Couchbase Inc.©2016 Couchbase Inc. Find Out More! How to understand and use the query optimizer Today, 11:00 am - 11:50 am • Great America 2 29 Keshav Murthy Director, Query Development, Couchbase
  30. 30. ©2016 Couchbase Inc.©2016 Couchbase Inc. Find Out More! Going beyond SELECT, top 10 ways to use N1QL Wednesday 10:00 am - 10:50 am • Great America K 30 PrasadVarakur Product Manager,Couchbase
  31. 31. ©2016 Couchbase Inc. 31 FullText Search
  32. 32. ©2016 Couchbase Inc.©2016 Couchbase Inc. FullText Search “Googling for your JSON documents” • Index Fields or Documents • LexicalAnalysis and Stemming • Flexible Query Capabilities • Available and Scalable 32 STORAGE SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster Manager Cluster Manager Data Service Index Service Query Service Full Text Search
  33. 33. ©2016 Couchbase Inc.©2016 Couchbase Inc. FullText Search “Googling for your JSON documents” • Index Fields or Documents • LexicalAnalysis and Stemming • Flexible Query Capabilities • Available and Scalable 33 Config { Node Definitions: … Index Definitions: … Plan Indexes: … } Planner Janitor PIndex bleve DCP Feed REST Interface Manager DCP Feed PIndex bleve
  34. 34. ©2016 Couchbase Inc.©2016 Couchbase Inc. Find Out More! Full-text search: how it works and what it can do Wednesday, 1:00 pm - 1:50 pm Great America 2 34 SteveYen Co-Founder, Couchbase Marty Schoch Senior Software Engineer, Couchbase
  35. 35. ©2016 Couchbase Inc. 35 New! Analytics Service
  36. 36. ©2016 Couchbase Inc.©2016 Couchbase Inc. New: Analytics Service! STORAGE SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster Manager Cluster Manager Data Service Index Service Query Service Full Text Search Analytics Service Sneak Peak: Couchbase Analytics Wednesday, 2pm – 2:50pm • Great America 2 TillWestman, Senior Director, Server Development, Couchbase
  37. 37. ©2016 Couchbase Inc.©2016 Couchbase Inc. Summary • The architecture of a next generation database • The addition of new features whilst retaining our principles: • Performance • Scalability • Availability • Ease of use • More technical meat available atConnect • Ask questions and get involved! 37
  38. 38. ©2016 Couchbase Inc. ThankYou! 38

×