Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Overcoming 5 Common Docker Challenges: How We Do It at RightScale

1,847 views

Published on

We highlight solutions to common Docker challenges that you may encounter as you move from initial experiments toward full-fledged Docker adoption. At RightScale, we’ve been sharing our lessons learned as we move toward a fully containerized environment leveraging a “sea of containers.” We’re now in the middle stages of that journey and will share some of the challenges we’ve encountered and how we’ve overcome them.

Published in: Technology
  • Be the first to comment

Overcoming 5 Common Docker Challenges: How We Do It at RightScale

  1. 1. OVERCOMING 5 COMMON DOCKER CHALLENGES: HOW WE DID IT AT RIGHTSCALE
  2. 2. Panelists • Ryan O’Leary: Moderator • Senior Director, Product • Tony Spataro • Senior Systems Architect • Mark Dotson • Principal System Administrator
  3. 3. • The State of Docker • RightScale’s Docker Journey • Development Challenges • Managing Docker images • Multi-host Docker development • Ops Challenges • Orchestrating Docker in production: • service discovery, configuration, and auditability • Dynamic monitoring and alerting • Increasing container density per host Agenda 2
  4. 4. RightScale Vision: Manage Any Resource Pool 3 Public Clouds Private Clouds Virtual Environments Bare Metal Servers Containers Containers Containers Containers Orchestration, Management, and Governance for Any Environment
  5. 5. 2% 3% 3% 4% 6% 7% 9% 20% 27% 32% 32% 8% 10% 12% 11% 16% 18% 12% 14% 35% 18% 19% Rancher Rocket Docker Tutum Mesosphere Docker Swarm Kubernetes Salt Ansible Docker Puppet Chef Respondents Using DevOps Tools Use today Plan to use 2016 DevOps Tools – All Respondents Source: RightScale 2016 State of the Cloud Report
  6. 6. 3% 6% 10% 13% 28% 24% 3% 9% 20% 27% 32% 32% Rocket Salt Ansible Docker Chef Puppet Respondents Using DevOps Tools 2016 2015 DevOps Tools YoY – All Respondents Source: RightScale 2016 State of the Cloud Report
  7. 7. RIGHTSCALE’S DOCKER JOURNEY
  8. 8. • Architectural role of VMs didn’t change • Services other then app lived outside container • Deployed (generally) 1 app container onto each VM, using nginx on VM to route to app container. • Static Mapping Step 1: Containerize Code 7 syslog smtp my-awesome-app Application Server 1..n Container No container nginx
  9. 9. • Deploy N containers in a non-cloud VM • Integration test in real time • No persistence, HA, etc Step 2: Composing in dev-and-test 8 Developer Laptop nginx my-awesome-app smtp syslog
  10. 10. • Deploy N ‘good neighbor’ containers onto a VM • Host-local balancer • Service discovery • Supports microservices • Supports “traditional” services Step 3: Bay of Containers 9 Application Logical Grouping 1..n balancer A smtp syslog srv. dsc. B C D Sidecar B balancer E smtp syslog srv. dsc. F G H Z
  11. 11. DEVELOPMENT AND DOCKER
  12. 12. CHALLENGE 1 MAKING (USEFUL) IMAGES
  13. 13. Images: Build Args # In my Dockerfile ARG gitref=unknown LABEL git.ref=${gitref} # During my build commit=`git rev-parse --verify HEAD` docker build --build-arg gitref=${commit} 1 2 # In my built image‘s metadata Id: "sha256:11bb...", RepoTags: [ "rightscale/right_api:latest” ], Labels: { git.ref: "f131ac4047...” } 3 Metadata-rich images Conditional content (e.g. debug support) Your idea here! (Careful…)
  14. 14. Fetch dependencies before the build; install them during the build. # Add fetched dependencies to # the image COPY vendor/cache vendor/cache/ # Build them inside the container # for matched shared libraries &c RUN bundle install # No key material in my image! Images: The Golden Rule # Continuous integration script # Use an SSH private key to # fetch private dependencies bundle package docker build Secure - no embedded private keys Efficient layer-cached image builds Repeatable dependency resolution
  15. 15. • Developers want to integration test without using production- grade orchestration tools • Docker images that know how to dance • Two (or more) ways to dance • Fail fast • Eventual consistency Images: Choreography
  16. 16. # I fail at startup if my database # is not reachable, or if it isn’t # in the ideal state for me. CMD bin/webapp --database=$DB_HOST Brittle containers Resilient compositions # docker-compose will relaunch me # every time I fail. # Not suitable for real deployment! webapp: image: rightscale/webapp:latest restart: always env: DB_HOST: postgres links: - postgres Choreography: Fail Fast + Implicit failure is less likely - Causes log spam at startup
  17. 17. # Wrapper script runs my actual # service CMD bin/entrypoint.sh # Wrapper converges on an ideal state # before passing control to my app while [ “$?” != “0” ]; do psql –qc "select now()" $DB_HOST sleep 1 done Resilient containers Simple compositions # Failures are an error, not an # expected outcome webapp: image: rightscale/webapp:latest restart: never env: DB_HOST: postgres links: - postgres Choreography: Eventual consistency + Allows powerful and sophisticated setup logic - Developer tooling built into images; may ship to production!
  18. 18. CHALLENGE 2 MULTI-HOST DOCKER DEVELOPMENT
  19. 19. Mixed containers/processes Pure-container Which Runtime Model?
  20. 20. Split Connected Which Networking Model?
  21. 21. Mixed runtime; split networks
  22. 22. Pure-container runtime; connected network
  23. 23. • Many options • Experiment & play • As simple as possible • …but no simpler! Multi-Host Docker
  24. 24. OPERATIONS AND DOCKER
  25. 25. CHALLENGE 3 ORCHESTRATION
  26. 26. • Existing, proven architecture • Orchestration tools already in place • SSAE-16 certified key management & auditing practices • Managing a mixture of resources • VMs • Containers-on-VMs • Databases, load balancers, networks, other non-compute resources • Need for cross-cutting solutions • Service discovery that spans VMs and containers • Key/value store that works for with VM-hosted services Orchestration: No Clean Slate
  27. 27. Orchestration 26 • Haproxy • Host-local load balancer. • Consul • Service Discovery • Key-Value Store • Consul-Template haproxy A smtp syslog consul B C D Sidecar B
  28. 28. Orchestration: HAProxy 27 • Acts as host-level balancer between boundry balancers and containers. • Allows for enforcement of an container-level maximum connections allowed to a given service • Dynamically creates or destroys app listener pools with upstream hosts (containers) using service discovery/consul + consul-template.
  29. 29. Orchestration: Consul 28 • Service Discovery • Registers service offering along with randomly assigned local host port map • Key-Value Store • Universal port mapping • Application configuration (“inputs”) information • Audit trail • Consul-template • Consul + HAProxy = Win
  30. 30. CHALLENGE 4 MONITORING & ALERTING
  31. 31. Monitoring & alerting 30 • Application specific monitoring for services in containers • Deployment information in consul key-value • Collectd + docker exec • Application specific alerts for services in containers • Alert definations stored in consul key-value • Creation and destruction calls to add to vm at container runtime # alert stored as json hash and into kv { “deployment”: { “alerts”: { "container - High Activity Delay": { "name": "container - High Activity Delay", "description": "This graph represents the delay before an activity completing a request", "file": "cwf_stats/gauge-median_activity_delay", "variable": "value", "condition": ">=", "threshold": "60", "duration": 10, "escalation_name": "critical" }, …
  32. 32. CHALLENGE 5 INCREASED HOST DENSITY
  33. 33. Increased host density 32 • Old model was 1 properly-sized vm to 1 container. • At the end of the day, bringing up a new service still meant… launching a new instance. • New model is 1 properly-sized vm to many “good neighbor” containers. • Have some excess capacity due to your usage patterns on a host/set of hosts? Add that new service right on in! haproxy A smtp syslog consul B C D Sidecar B
  34. 34. WHAT COMES NEXT?
  35. 35. Sea of Containers 34 VM VM VM A A A A A A C C B B B B VM VM VM A A A C C A A A C C B B B B B Container Management B • N(×M) containers • 0..N VMs • Elastic mesh network • Declarative everything • Resource scheduling A A
  36. 36. ECS Kubernetes Rancher Swarm Docker CLI X X X Orchestration API, compose API compose compose Constraints replication placement placement Shared Networking layer-3 overlay (IPSec) overlay (VXLAN) Shared Storage X X D.I.Y. Load Balancer X X X D.I.Y. Self-Healing replace replace restart restart Portability low medium medium high Overview of Solutions
  37. 37. • Using Docker and RightScale • www.rightscale.com/docker Q&A 36

×