Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Containerizing Couchbase with Swarm & Kubernetes – Couchbase Connect 2016

318 views

Published on

Couchbase is supported on a wide variety of orchestration frameworks for Docker containers, such as Kubernetes and Docker Swarm. You might have wondered what the best practices are for containerizing Couchbase in these orchestration frameworks. This session covers how the Couchbase engineering team uses Swarm and Kubernetes, and shares the best practices for doing so.

Published in: Software
  • Be the first to comment

  • Be the first to like this

Containerizing Couchbase with Swarm & Kubernetes – Couchbase Connect 2016

  1. 1. ©2016 Couchbase Inc. Containerizing Couchbase with Swarm and Kubernetes 1
  2. 2. ©2016 Couchbase Inc. 2 Eric Cooper Manager, Test Automation Eric.cooper@couchbase.com IMAGE GOES HERE
  3. 3. ©2016 Couchbase Inc. 3 Tommie McAfee Senior Engineer, Test Automation tommie@couchbase.com IMAGE GOES HERE
  4. 4. ©2016 Couchbase Inc.©2016 Couchbase Inc. Agenda • Containers and their popularity • Orchestration Frameworks • Functional Testing at Couchbase with Docker • Interfacing with Docker • System testing with Sequoia Framework and Swarm •Demo • Orchestrating Couchbase with Kubernetes • System testing with Swarm • Best practices 4
  5. 5. ©2016 Couchbase Inc.©2016 Couchbase Inc. Containers and their popularity •Customizable images • e.g. the same Couchbase Server build and OS, mix and match • Include all necessary libraries in the image •One solution from Dev -> production • Deploy the same image in dev, preproduction and production across all platforms • Packageable • can share a container for troubleshooting • Lightweight • time to bring up a container with Couchbase • “clean slate” • Popular • 500K+ Couchbase pulls from Dockerhub 5
  6. 6. ©2016 Couchbase Inc.©2016 Couchbase Inc. Orchestration and their frameworks • Aggregation • Deploy apps implemented with multiple containers in pods • Scaling • Increased the replication count across multiple servers • Computing resources per container •Robustness • The ability to restart failing containers and hosts • Services • Service discovery load balancing 6
  7. 7. ©2016 Couchbase Inc. 7 Orchestration and Couchbase Test Environments
  8. 8. ©2016 Couchbase Inc.©2016 Couchbase Inc. Functional Testing at Couchbase with Docker • Couchbase Server Functional Testing Overview • Multiple test suites per component (e.g. Views, XDCR, N1QL etc.) • Each test suite: • Contains 5 to 50 test cases • May use between 1 and 8 Couchbase Servers • Runs from 30 minutes to 10 hours • Total of 150 test suites across 15 components • All regression suites runs in 12 hours 8
  9. 9. ©2016 Couchbase Inc.©2016 Couchbase Inc. Test Infrastructure - historical Test client - takes as input a test suite descriptor and cluster config (Python) Couchbase Server 1 Couchbase Server n REST/CLI/memcached VM VM Jenkins Job
  10. 10. ©2016 Couchbase Inc.©2016 Couchbase Inc. Challenges • A multitude of Jenkins jobs (one per test suite) • Ordering is difficult • Test suite and cluster configuration implicitly stored in Jenkins jobs • Dedicated VMs • Hard to maintain - if a VM is replaced with a new IP many Jenkins jobs must be edited • Fixed OS - testing with a different OS is cumbersome
  11. 11. ©2016 Couchbase Inc.©2016 Couchbase Inc. Docker solution 11 Test Suite Descriptors (Couchbase) Test Suite Dispatcher Test Suite Executor Jenkins Job (multiple instances dependent on hardware capacity) Couchbase Server 1 on Docker Test Client Test Suite N Couchbase Server n on Docker Couchbase Server 1 on Docker Test Client Test Suite 1 Couchbase Server n on Docker
  12. 12. ©2016 Couchbase Inc.©2016 Couchbase Inc. Docker based Jenkins Test Executor
  13. 13. ©2016 Couchbase Inc.©2016 Couchbase Inc. Couchbase Functional Test Suites and Kubernetes (Work in progress) • Test suites are heterogeneous - differing number of Couchbase servers depending on the suite • Test suite runs are not persistent - when they are done the containers are not reused • Test client and Couchbase servers pod configuration: • in 1 pod, or • spread across pods? • Pods per common cluster configs? • Kubernetes Jobs may be a solution: • Job per test suites? Auto generated • Control parallelism?
  14. 14. ©2016 Couchbase Inc. 14 Interfacing With Docker Container Based Testing
  15. 15. ©2016 Couchbase Inc.©2016 Couchbase Inc. Requirements 15 Topology Changes Data Loading Query • Building a framework for system testing • JSON and binary documents with CRUD operations • Map/Reduce, N1QL, Full Text Search, with index operations • Add and remove servers, rebalance and failover cluster
  16. 16. ©2016 Couchbase Inc.©2016 Couchbase Inc. The Good News 16 Topology Changes Data Loading Query • No problem...we already have tools to perform these operations • pillowfight, cbdocloader, spring, gideon • cbq, rest_client, vegetta • Couchbase_cli, rest_client
  17. 17. ©2016 Couchbase Inc.©2016 Couchbase Inc. The Good News 17 Topology Changes Data Loading Query • No problem...we already have tools to perform these operations Package Together and we’re done!
  18. 18. ©2016 Couchbase Inc.©2016 Couchbase Inc. The Bad News 18 Topology Changes Data Loading Query • Problem...tools are written if different languages GO C Python Java Bash
  19. 19. ©2016 Couchbase Inc.©2016 Couchbase Inc. The Docker News 19 Topology Changes Data Loading Query • That’s fine...free containers for everyone! GO C Python Java Bash
  20. 20. ©2016 Couchbase Inc.©2016 Couchbase Inc. The Docker News 20 Topology Changes Data Loading Query • From Implementing to Interfacing GO C Python Java Bash I N T E R F A C E
  21. 21. ©2016 Couchbase Inc.©2016 Couchbase Inc. The Docker News 21 • For Example Topology Changes Data Loading Query T E S T • run pillowfight -U 172.23.100.10 • run cbq -script=’create primary index on default’ • run couchbase-cli rebalance -c 172.23.100.10
  22. 22. ©2016 Couchbase Inc.©2016 Couchbase Inc. More Requirements 22 • What if I want to test a different cluster? Topology Changes Data Loading Query T E S T • run pillowfight -U 172.23.100.10 • run cbq -script=’create primary index on default’ • run couchbase-cli rebalance -c 172.23.100.10
  23. 23. ©2016 Couchbase Inc.©2016 Couchbase Inc. More Requirements 23 • Just change IP’s? Topology Changes Data Loading Query T E S T • run pillowfight -U 10.1.2.10 • run cbq -script=’create primary index on default’ • run couchbase-cli rebalance -c 10.1.2.11
  24. 24. ©2016 Couchbase Inc.©2016 Couchbase Inc. More Requirements 24 • Scope Abstraction S C O P E Topology Changes Data Loading Query T E S T • run cbq -script=’create primary index on default’ • run couchbase-cli rebalance -c {{.Orchestrator}} • run pillowfight -U {{.Orchestrator}} Orchestrator: 10.1.2.10
  25. 25. ©2016 Couchbase Inc.©2016 Couchbase Inc. More Requirements 25 • What if I wanted to test against a cluster with 100 vms? S C O P E Topology Changes Data Loading Query T E S T • run cbq -script=’create primary index on default’ • run couchbase-cli rebalance -c {{.Orchestrator}} • run pillowfight -U {{.NthDataNode 70}} …DataNode[0]: 10.1.2.10 QueryNode[0]: 10.1.2.11 DataNode[98]: 10.1.2.110
  26. 26. ©2016 Couchbase Inc.©2016 Couchbase Inc. Houston 26 • How about 100 containers? S C O P E Topology Changes Data Loading Query T E S T • run cbq -script=’create primary index on default’ • run couchbase-cli rebalance -c {{.Orchestrator}} • run pillowfight -U {{.NthDataNode 70}} …DataNode[0]: ??? QueryNode[0]: ??? DataNode[98]: ???
  27. 27. ©2016 Couchbase Inc.©2016 Couchbase Inc. Solution 27 • Provider Abstraction S C O P E Topology Changes Data Loading Query T E S T • run cbq -script=’create primary index on default’ • run couchbase-cli rebalance -c {{.Orchestrator}} • run pillowfight -U {{.NthDataNode 70}} PROVIDER
  28. 28. ©2016 Couchbase Inc.©2016 Couchbase Inc. TADA! 28 • Sequoia Framework Topology Changes Data Loading Query • Container Pluggability = More Tests + Less Coding • Scope Portability = Define Once + Run Anywhere • Resource Scalability = Your Laptop || The Lab
  29. 29. ©2016 Couchbase Inc. 29 Testing With Docker Using Sequoia Framework
  30. 30. ©2016 Couchbase Inc.©2016 Couchbase Inc. Test expression • Tests are a series of docker commands 30 • docker run cbq -script=’create primary index on default’ • docker run couchbase-cli rebalance -c 172.10.10.2 • docker run pillowfight -U 172.10.10.2
  31. 31. ©2016 Couchbase Inc.©2016 Couchbase Inc. Test expression • With sequoia we express this using YAML files 31
  32. 32. ©2016 Couchbase Inc.©2016 Couchbase Inc. Altogether • Scale it up! 32
  33. 33. ©2016 Couchbase Inc.©2016 Couchbase Inc. Altogether •With concurrency 33
  34. 34. ©2016 Couchbase Inc. 34 Scaling With Swarm Deploying Containers As a Service With Swarm
  35. 35. ©2016 Couchbase Inc.©2016 Couchbase Inc. • New Swarm Provider •Containers are encapsulated in services (networking, multi-host) Scaling With Swarm 35 DOCKER SWARM containers services
  36. 36. ©2016 Couchbase Inc.©2016 Couchbase Inc. •Spread Strategy • Default behavior is to evenly distribute containers across couchbase servers Scaling With Swarm 36 Manager 1 Manager 2 Manager 3 CB CLCB Manager 4 CB CLCB CB CLCB CB CLCB
  37. 37. ©2016 Couchbase Inc.©2016 Couchbase Inc. •Spread Strategy • Default behavior is to evenly distribute containers across couchbase servers Scaling With Swarm 37 Manager 1 Manager 2 Manager 3 CB CLCB Manager 4 CB CLCB CB CLCB CB CLCB
  38. 38. ©2016 Couchbase Inc.©2016 Couchbase Inc. •Adding Constraints • Apply Labels for workload isolation Scaling With Swarm 38 Manager 1 Manager 2 Manager 3 CB CB Manager 4 CB CB CB CB CLCL CL CL CB CB CB Server Server Server Client • ‘docker node update --label-add zone=client manager-4’
  39. 39. ©2016 Couchbase Inc.©2016 Couchbase Inc. •Adding Constraints • Apply Labels for workload isolation Scaling With Swarm 39 Manager 1 Manager 2 Manager 3 D D Manager 4 D D D D CLCL CL CL I I Q Server Server Server Client
  40. 40. ©2016 Couchbase Inc.©2016 Couchbase Inc. •Adding Constraints • Apply Labels for workload isolation Scaling With Swarm 40 Manager 1 Manager 2 Manager 3 D D Manager 4 D D D D CLCL CL CL I I Q Server Server Server Client
  41. 41. ©2016 Couchbase Inc.©2016 Couchbase Inc. •Service Level Constraints • We can further isolate nodes by service Scaling With Swarm 41 Manager 1 Manager 2 Manager 3 D D Manager 4 I I CLCLQ CL CL D Data Index Query Client DDD
  42. 42. ©2016 Couchbase Inc.©2016 Couchbase Inc. Big Wins •Multi-Host Load-Balanced Scalability with Isolation • Container automatically distributed evenly across Swarm nodes • Labeling allows service load balancing • Easy scale out • For FREE! We don’t have to rewrite a single test! 42
  43. 43. ©2016 Couchbase Inc. 43 Demo • Kubernetes and Couchbase service • System test with Swarm
  44. 44. ©2016 Couchbase Inc. 44 Best Practices
  45. 45. ©2016 Couchbase Inc.©2016 Couchbase Inc. Best Practices • Isolate Isolate! • Consider CPU Pinning • Set Quotas • Share the memory 45
  46. 46. ©2016 Couchbase Inc.©2016 Couchbase Inc. Best Practices (continued) • Be aware of when system calls reflect the host perspective e.g. available RAM, set time • Google and stackoverflow are your friends 46
  47. 47. ©2016 Couchbase Inc. Thank You! 47

×