Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Preparing for success: the top deployment do’s and don’ts – Couchbase Connect 2016


Published on

This session will provide an overview of Couchbase operations, ensuring properly sized clusters, best practices and tips in indexing, querying, high availability, and disaster recovery. This session will also discuss some specific architecture and deployment considerations as well as the effects of using different Couchbase features.

Published in: Software
  • Be the first to comment

Preparing for success: the top deployment do’s and don’ts – Couchbase Connect 2016

  1. 1. ©2016 Couchbase Inc. Best Practices: Preparing for Success-Top Deployment Do’s & Don’ts - Ian McCloy & Karthik Sekar
  2. 2. ©2016 Couchbase Inc. 2 Ian McCloy CouchbaseTechnical Support Manager – EMEA Ian McCloy is the Technical Support Manager for Couchbase in EMEA. Ian is based in Couchbase’s Manchester U.K. engineering lab. Previously, Ian was a senior inventor and team lead with IBM’s System and Technology Group’s Storage Systems division. Ian specializes in OS platforms, virtualization, distributed systems, storage, and networking. Ian is a VMware certified engineer and has authored many patents in a diverse range of technologies from server hardware, virtualization, software testing, security, and language translation.
  3. 3. ©2016 Couchbase Inc. 3 Karthik Sekar Solutions Architect- WW Field Operations Karthik is a Solutions Architect, part of Professional Services - Worldwide Technical Field Operations at Couchbase with expertise in BigData, NoSQL technologies, cloud, and distributed computing platforms and he specializes in setting up, configuring, securing, tuning, architecting and managing mission critical systems. His current major responsibilities are providing architecture reviews, technical product assistance & sharing best practices with the customers.
  4. 4. ©2016 Couchbase Inc.©2016 Couchbase Inc. Agenda • OSTuning • PlatformTuning • Bucket Recommendations • Sizing-Why Sizing is Crucial for Success • Best Practices –Views & Indexing • The Breaking Point
  5. 5. ©2016 Couchbase Inc.©2016 Couchbase Inc. OSTuning (Linux) Linux DisableTHP (Transparent Huge Pages) Use XFS File System Create Swap Space Set Swappiness to 0
  6. 6. ©2016 Couchbase Inc.©2016 Couchbase Inc. OSTuning (Windows) Windows IncreaseTCP ephemeral ports TuneVirus Scanners Tune Backup Utilities
  7. 7. ©2016 Couchbase Inc.©2016 Couchbase Inc. PlatformTuning (Virtualization) Virtual Machine Avoid a Single Point of Failure Do not over-commit your resource Avoid Live Migration Increase Auto-FailoverThreshold
  8. 8. ©2016 Couchbase Inc.©2016 Couchbase Inc. System Clocks Ensure NTP (NetworkTime Protocol) is configured correctly before adding a node to the cluster TTL (TimeTo Live) Document Expiry is per Couchbase Node
  9. 9. ©2016 Couchbase Inc.©2016 Couchbase Inc. Server Quota Keep the per node Server Quota to best practice of 80% maximum total system RAM Provision enough capacity before the demand exceeds the availability of the cluster, failure to do this can prevent normal recovery procedures File-System Cache needs RAM Proactive monitoring of workload and cluster capacity
  10. 10. ©2016 Couchbase Inc.©2016 Couchbase Inc. BucketTuning Use as few buckets as possible. Disable “Flush” on production buckets to prevent accidental data loss.
  11. 11. ©2016 Couchbase Inc.©2016 Couchbase Inc. Replica Recommendations As a rule of thumb, we recommend the following: One replica for up to five data service nodes. One or two replicas for five to ten nodes. One, two, or three replicas for over ten nodes.
  12. 12. ©2016 Couchbase Inc. Sizing
  13. 13. ©2016 Couchbase Inc.©2016 Couchbase Inc. Why Sizing Matters one of the most common support issues is due to under-provisioned clusters • Data Explosion : Explosive growth of applications for Digital Economy - the web, mobile and IoT • Performance Demands – Low latency access • Capacity Planning - Sufficient capacity to the handle peak loads • High Availability – For today’s mission critical apps, high availability is no longer a ‘nice to have’ but is essential. 13
  14. 14. ©2016 Couchbase Inc.©2016 Couchbase Inc. Hardware Minimums RAM: At least ~8GB (highly dependent on data set) Disk: Fastest “local” storage available -SSD is better -RAID 0,10 -Avoid SAN, if Possible CPU (minimums): 8 cores + 1-per bucket + 1-per design document (if Views are used) + 1-per 10 K Mutations/sec + 1-per XDCR stream Hardware requirements/recommendations are the intersection of what’s needed versus what’s available.
  15. 15. ©2016 Couchbase Inc.©2016 Couchbase Inc. Sizing Couchbase Server • Multi-Dimensional Scalability (MDS) – Optionally Scale each service independently: • Data • Index • Query MDS is the architecture that enables independent scaling of data, query, and indexing workloads while being managed as one cluster.
  16. 16. ©2016 Couchbase Inc.©2016 Couchbase Inc. Sizing Couchbase Server - Data • Data Service: • Same as previous Couchbase Server 3.x version • Enough RAM to cache reads • Enough Disk to eventually persist writes • CPU primarily forViews and XDCR • At least 3 nodes – Replication at the bucket level • Minimum requirements: 4GB RAM, 8 Cores CPU
  17. 17. ©2016 Couchbase Inc.©2016 Couchbase Inc. Sizing Couchbase Server - Index • Index service: • Primarily RAM and Disk IO bound • IndexTypes • Standard Global Secondary Indexes (GSI) • Memory Optimized Indexes (MOI) • At least 2 nodes for HA, each index replicated individually • Minimum Requirements: 32 GB RAM, 16 core CPU, “fast disk”, “as much RAM as you need for MOI”
  18. 18. ©2016 Couchbase Inc.©2016 Couchbase Inc. Sizing Couchbase Server - Query • Query Service : • Primarily CPU bound • Optimized for multi-core systems • Very low RAM and disk requirements • At least 2 nodes for HA – Queries automatically load balanced • Minimum Requirements: 8 GB RAM, 8+ Core CPU
  19. 19. ©2016 Couchbase Inc. 5 Factors of Sizing
  20. 20. ©2016 Couchbase Inc.©2016 Couchbase Inc. How Many Nodes? 20 5 Key Factors determine number of nodes needed: 1) RAM 2) Disk 3) CPU 4) Network 5) Data Distribution/Safety (per-bucket, multiple buckets aggregate)
  21. 21. ©2016 Couchbase Inc.©2016 Couchbase Inc. 21 RAM sizing 1) Total RAM § Managed document cache: § Working set § Metadata § Active+Replicas § Index caching (I/O buffer) Keep working set in RAM for best read performance Server Give me document A Here is document A A A A Reading Data Application Server
  22. 22. ©2016 Couchbase Inc.©2016 Couchbase Inc. Working set depends on your application • Resident item ratio shows the total number of active documents that reside in memory.Typically you want your working set (actively accessed documents) to be in memory for low latencies and an awesome user experience. • Recommended best practices for the working set ( doc resident ratio) is always over 20 % 22 Metadata Overhead Indicates that a bucket is now using more than 50% of the allocated RAM for storing metadata and keys, reducing the amount of RAM available for data values.This is a helpful indicator that you may need to add nodes to your cluster.
  23. 23. ©2016 Couchbase Inc.©2016 Couchbase Inc. 23 Disk Sizing: Space and I/O 2) Disk § Sustained write rate § Rebalance capacity § Backups § XDCR § Views/Indexes § Compaction § Total dataset: § (active + replicas + indexes) § Append-only I/O Space Please store document A OK, I stored document A A Server A A Writing Data Application Server
  24. 24. ©2016 Couchbase Inc.©2016 Couchbase Inc. Disk Sizing: Space and I/O • DiskWrites are Buffered • Bursts of data expand the disk write queue • Sustained writes need corresponding throughput • Disk throughput affected by disk speed • SSD > 10K RPM > EBS • SSDs give a huge boost to write throughput and startup/warmup times • RAID can provide redundancy and increase throughput • Throughput = read/write+compaction+indexing+XDCR • Best to configure different paths for data and indexes • Plan on about 3x space (append-only, compaction, backups, etc)
  25. 25. ©2016 Couchbase Inc.©2016 Couchbase Inc. CPU Sizing 25 3) CPU § Disk writing § Views/compaction/XDCR § RAM r/w performance not impacted § Min. production requirement: 8 cores +1 per bucket +1 core per Design Doc (If Views are used) +1 core per XDCR stream + 1 core for every 10K Mutations/sec
  26. 26. ©2016 Couchbase Inc.©2016 Couchbase Inc. Network Sizing 26 4) Network § Client traffic § Replication (writes) § Rebalancing § XDCR Reads+Writes Replication (multiply writes) and Rebalancing network networknetwork Couchbase Server Couchbase Server Couchbase Server Application Server Application ServerApplication Server
  27. 27. ©2016 Couchbase Inc.©2016 Couchbase Inc. Network Considerations • Low latency, high throughput (LAN) - within cluster • Eliminate router hops: • Within Cluster nodes • Between clients and cluster • Check who else is sharing the network • Increase bandwidth by: • Add more nodes (will scale linearly) • Upgrade routers/switches/NIC’s/etc
  28. 28. ©2016 Couchbase Inc. Best Practices withViews
  29. 29. ©2016 Couchbase Inc. ©2015 Couchbase Inc. 29 Couchbase Data Access • Everything is built on top of Key Value • A Document store is a special case of Key-Value • Views provide aggregation and real-time analytics through incremental map-reduce • Global Secondary Indexes provide low latency/high throughput indexes • N1QL is a language that provides a powerful and expressive way of accessing documents
  30. 30. ©2016 Couchbase Inc.©2016 Couchbase Inc. Best Practices - Selection, Projection, Aggregation • Try avoid computing too many things in aView • Check for attribute existence • Pre-Filter data to avoid unnecessary entries in theView • Use document types to makeViews more selective • Project (map) only necessary data by emitting it as part of the value • Do not emit the full document • Back-reference via the original document id • Use the built-in reduce functions if possible
  31. 31. ©2016 Couchbase Inc.©2016 Couchbase Inc. Best Practices - Selection, Projection, Aggregation
  32. 32. ©2016 Couchbase Inc. Database Design Considerations forViews
  33. 33. ©2016 Couchbase Inc.©2016 Couchbase Inc. Number of Design Documents per Bucket • Indexers are allocated per Design Document • Bad cases • One Design Document contains allViews ØAllViews A lot to do for the Indexer are updated the same time • OneView per Design Document ØResource intensive because one Indexer perView • Good balance!
  34. 34. ©2016 Couchbase Inc. The Breaking Point
  35. 35. ©2016 Couchbase Inc.©2016 Couchbase Inc. Data Distribution 35 5) Data Distribution / Safety (assuming one replica): § 1 node = Single point of failure § 2 nodes = +Replication § 3+ nodes = Best to start with, for Test/production § Auto failover § Upgrade-ability § Further scale-ability § 7+ nodes = 3 Data nodes + 2 Index nodes + 2 Query nodes – Minimum for MDS ( for Production) § Single node deployment is not for Test/Production, however for development, it is fine to start with Servers fail, be prepared. The more nodes, the less impact a failure will have.
  36. 36. ©2016 Couchbase Inc.©2016 Couchbase Inc. As your Dataset Grows… 36 Effects on scale/sizing: • Your RAM needs will grow: • Metadata needs increase with item count • Is your working set increasing? • Your disk space will likely grow (duh?) Indications: • Dropping resident ratio • Rising ejections/cache miss ratio What to do: • Revise sizing calculations, add more nodes • Remove un-needed data This is the most common need for scaling and will most likely result in needing more nodes
  37. 37. ©2016 Couchbase Inc.©2016 Couchbase Inc. To Ensure Success… • Test, Deploy, Monitor…rinse and repeat • Drills for FailoverTesting • Simulating Disk Failures, Network Failures, Process Hung, Cluster Failures • Proactive monitoring, efficient cluster, and process management • Define the SLA, RPO, RTO • Stress testing over Nx times than the normal operations • Isolate systems interfering with the performance
  38. 38. ©2016 Couchbase Inc.©2016 Couchbase Inc. Sizing is tricky business… • Work with the CouchbaseTeam • Validate your “on-paper” numbers with testing • Constantly monitor production • Gather your workload and dataset requirements: § Item counts and sizes, read/write/delete ratios • Review our documentation and formulas
  39. 39. ©2016 Couchbase Inc. ThankYou