Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Multi-Dimensional Scaling: A New Architecture for Scaling Big Data Applications: Couchbase Connect 2015

1,323 views

Published on

Couchbase Server 4.0 not only supports homogenous scaling in one dimension but it now supports heterogeneous scaling in multiple dimensions. Data, querying, and indexing services can now be deployed into "zones" within the cluster. The new Multi-Dimensional Scaling capability enables enterprises to optimize their database infrastructure with unparalleled precision. The enterprise benefits of Multi-Dimensional Scaling include: increased application performance and stability, increased operational efficiency and agility, and reduced hardware costs. In this talk, we will also review the new services available in Couchbase Server 4.0 that allows independent scaling of query, indexing and core data workloads, reducing interference and maximizing throughput.

Published in: Technology
  • Be the first to comment

Multi-Dimensional Scaling: A New Architecture for Scaling Big Data Applications: Couchbase Connect 2015

  1. 1. MULTI-DIMENSIONAL SCALING – A NEW ARCHITECTURE FOR SCALING BIG DATA APPLICATION Anil Kumar, Senior Product Manager Couchbase
  2. 2. ©2015 Couchbase Inc. 2 About Me Anil Kumar Sr. Product Manager, Couchbase anil@couchbase.com @anilkumar1129
  3. 3. ©2015 Couchbase Inc. 3 Next 40 minutes …  Brief history of scaling in database  Scaling up  Scaling out  NoSQL workloads and scalability model  Core data operations, indexing and querying  Homogenous HW scaling  Introducing multi-dimensional scaling  Isolation of workloads through a services architecture  Independent HW scaling  Demo  Q & A
  4. 4. Brief History of Scaling in Database
  5. 5. ©2015 Couchbase Inc. 5 Scaling up  Scale-up architecture  Cluster processors – hyper-threading to cores  Locally partition workload among processors  Communicate over memory
  6. 6. ©2015 Couchbase Inc. 6 Scaling Up – Pros & Cons Pros  Can result in major performance improvement  Machines can now support having many cores and terabytes of RAM Cons  Expensive  Requires downtime  Performance bounded – at some point the database engine itself becomes the bottleneck  Limited in scalability and elasticity
  7. 7. ©2015 Couchbase Inc. 7 Scaling out  Scale-out architecture  Cluster of commodity HW  Horizontal partitioning of data on cluster nodes  Communicate over network
  8. 8. ©2015 Couchbase Inc. 8 Scaling Out – Pros & Cons Pros  Simple and easily scalable  Data evenly split across cluster of nodes  Scales linearly with throughput  Highly available  No single point of failure Cons  Not great for all workloads – data, index and query
  9. 9. ©2015 Couchbase Inc. 9 So Which Model is the Right Model ? Scale up or scale out?
  10. 10. ©2015 Couchbase Inc. 10 Scaling Up vs. Scale Out Link to whitepaper - http://www.msr- waypoint.com/pubs/204499/ a20-appuswamy.pdf
  11. 11. NoSQLWorkloads & Scalability Model
  12. 12. ©2015 Couchbase Inc. 12 NoSQLWorkloads  One database, many workloads  Core data processing:GETs & SETs for a given key  Indexing: Index maintenance and lookups  Querying: Combine index and data with complex just-in-time data re-shaping, ordering, grouping, aggregations, and more Varying resource requirements - CPU, RAM, I/O, Network Varying methods to optimize latency & throughput for each
  13. 13. ©2015 Couchbase Inc. 13 Scalability ModelToday Homogenous Scaling  Each node get a slice of the workload  Simple to do… But...  Workloads compete and interfere with each other  Can’t fine tune each workload  Core Data operation are partition-able so great with wider fan-out  Indexing and queries aren’t always partitionable, so worse with wider fan-out IndexWorkload Couchbase Cluster QueryWorkload DataWorkload node1 node8
  14. 14. Introducing Multi-Dimensional Scaling
  15. 15. ©2015 Couchbase Inc. 15 Modern Architecture What is Multi-Dimensional Scalability? MDS is the architecture that enables independent scaling of data, query and indexing workloads. Index Service Couchbase Cluster Query Service Data Service node1 node8
  16. 16. ©2015 Couchbase Inc. 16 Couchbase Cluster node1 node8 Modern Architecture  Isolated Service for minimized interference  Independent “zones” for Query, Index and Data Services Minimize indexing and query overhead on core key-value operations. Index ServiceQuery Service Data Service
  17. 17. ©2015 Couchbase Inc. 17 Modern Architecture  Independent Scalability for Best Computational Capacity per Service Heavier indexing (index more fields) : scale up index service nodes More RAM for query processing: scale up query service nodes Couchbase Cluster node1 node8 node9 Data Service Index ServiceQuery Service
  18. 18. Services Architecture Data, Index, & Query
  19. 19. ©2015 Couchbase Inc. 19 Full Cluster Architecture STORAGE Couchbase Server 1 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster Manager Cluster Manager Managed Cache Storage Data Service Index Service Query Service STORAGE Couchbase Server 2 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster Manager Cluster Manager Managed Cache Storage Data Service Index Service Query Service STORAGE Couchbase Server 3 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster Manager Cluster Manager Managed Cache Storage Data Service Index Service Query Service STORAGE Couchbase Server 4 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster Manager Cluster Manager Managed Cache Storage Data Service Index Service Query Service STORAGE Couchbase Server 5 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster Manager Cluster Manager Managed Cache Storage Data Service Index Service Query Service STORAGE Couchbase Server 6 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster Manager Cluster Manager Managed Cache Storage Data Service Index Service Query Service
  20. 20. ©2015 Couchbase Inc. 20 Full Cluster Architecture STORAGE Couchbase Server 1 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster Manager Cluster Manager Managed Cache Storage Data Service Index Service Query Service STORAGE Couchbase Server 2 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster Manager Cluster Manager Managed Cache Storage Data Service Index Service Query Service STORAGE Couchbase Server 3 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster Manager Cluster Manager Managed Cache Storage Data Service Index Service Query Service STORAGE Couchbase Server 4 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster Manager Cluster Manager Managed Cache Storage Data Service Index Service Query Service STORAGE Couchbase Server 5 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster Manager Cluster Manager Managed Cache Storage Data Service Index Service Query Service STORAGE Couchbase Server 6 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster Manager Cluster Manager Managed Cache Storage Data Service Index Service Query Service
  21. 21. ©2015 Couchbase Inc. 21 Storage Engine Cluster Manager Data Service Projector & Router New Services in Couchbase Server 4.0 Query ServiceIndex Service Supervisor Index maintenance & Scan coordinator Index#2Index#1 Query Processor cbq-engine Bucket#1 Bucket#2 DCP Stream Index#4Index#3 ... B u c k e t # 2 B u c k e t # 1 1809311211 18901 ManagedCache
  22. 22. Demo !!!
  23. 23. ©2015 Couchbase Inc. 23 Recap  MDS enables unprecedented control of scalability with Couchbase Server  Separate out competing workloads to independent services  Independently scale each service “zone” within the cluster  Couchbase Server with MDS maximizes scalability and performance  Improves scale and performance to degrees not possible with other NoSQL or big-data engines on premise or in the cloud  Improved price/performance and squeezes more performance and throughput for mission-critical systems
  24. 24. Thank you.
  25. 25. Get Started withCouchbase Server 4.0: www.couchbase.com/beta GetTrained on Couchbase: training.couchbase.com

×