Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

DockerCon SF 2015: Enabling Microservices @Orbitz

5,498 views

Published on

The slides from Steve Hoffman and Rick Fast's presentation at DockerCon SF 2015 -

Talk Description:
In this talk we will discuss how we enabled decomposition of one of our 250+ system components into a continously deployed microservice cluster.

This includes building a standardized Docker server composed of various local companion services along side the Docker daemon including: dynamic service discovery via Consul, a log relay to a centralized Elasticsearch cluster, and forwarding/batching of Dropwizard metrics to Graphite.

Building on this we'll cover our Jenkins driven automated pipeline for building Docker images and rolling deployments via Ansible using static placement on existing infrastructure while prototyping dynamic placement using Docker + Apache Mesos.

Published in: Technology
  • Be the first to comment

DockerCon SF 2015: Enabling Microservices @Orbitz

  1. 1. Enabling Microservices @Orbitz Steve Hoffman @bacoboy Rick Fast @tortiepoint Senior Principal Engineers
  2. 2. Agenda • Brief Architecture Overview/History • From Monolithic to Services to Microservices with Docker • Automated Pipelines • Questions
  3. 3. #dockercon 2000
  4. 4. #dockercon 2000 Yes, we are STILL hiring! @OrbitzTalent
  5. 5. #dockercon 2000 Web Layer Business Layer
  6. 6. #dockercon 2000
  7. 7. #dockercon 2000
  8. 8. #dockercon 2000 Business Layer
  9. 9. #dockercon Business Layer 20032000
  10. 10. #dockercon Business Layer 2003
  11. 11. #dockercon 2003 Business Layer
  12. 12. #dockercon 20042003
  13. 13. #dockercon 2004
  14. 14. #dockercon 20042012
  15. 15. #dockercon 2012 • Multiple Brands • Websites • Webservices • Multiple Backends • 500+ apps / thousands of instances • Deployments Daily (sometimes more) 2015
  16. 16. #dockercon Process Overkill
  17. 17. #dockercon Different Provisioning Tools DEV OPS Application Platform!=
  18. 18. #dockercon A New Experiment • Microservices? • Decompose single “service” into the actual 40+ sub-services - Any change to sub-service was a deployment of many • Could it be a simple Spring Boot App in Docker (12 factor?) • Backward compatible with existing service infrastructure. • Code to Production w/o help from other Humans
  19. 19. #dockercon The Docker Slave aka “The Rickbot” App App App
  20. 20. #dockercon 10.10.10.10 10.10.10.10:31002 App App App Register & Lookup
  21. 21. #dockercon App App App
  22. 22. #dockercon App App App
  23. 23. #dockercon
  24. 24. #dockercon
  25. 25. #dockercon http://consul.io
  26. 26. #dockercon 10.10.10.10 10.10.10.10:51515 App App App
  27. 27. #dockercon 10.10.10.10 10.10.10.10:1337 App App App
  28. 28. #dockercon Almost done… App App App
  29. 29. #dockercon App App App
  30. 30. #dockercon App App App VIP https://github.com/QubitProducts/bamboo
  31. 31. #dockercon App App App
  32. 32. #dockercon App App App
  33. 33. #dockercon App App App
  34. 34. A Little More Background
  35. 35. #dockercon
  36. 36. #dockercon Module Module Module Module
  37. 37. #dockercon Editorial Module • A Continuously Deployed Microservice • www.orbitz.com - (scroll down)
  38. 38. #dockercon Orbitz Content Orchestration Service Content (Solr) Editorial Module Search Module Hotel Module
  39. 39. #dockercon Orbitz Content Orchestration Service Content (Solr) Editorial Module Search Module Hotel Module Hotel Team Search Team Content Team
  40. 40. #dockercon Orbitz Content Orchestration Service Content (Solr) Editorial Module Search Module Hotel Module
  41. 41. #dockercon Orbitz Content Orchestration Service Content (Solr) Editorial Module Search Module Hotel Module Hotel Team Search Team Content Team
  42. 42. #dockercon Pre-Continuous Delivery 1.2-BETA-20150401-113002 Test Discard Test Discard 1.2-BETA-20150402-093002 Build Build And so on…
  43. 43. #dockercon Pre-Continuous Delivery 1.2-BETA-20150401-113002 Test Discard Test Discard 1.2-BETA-20150402-093002 Build Build Test Deploy 1.2 Build
  44. 44. Continuous Delivery with Jenkins, Docker, and Ansible (And then Marathon)
  45. 45. #dockercon Yo rfast-mbp:git rfast$ yo microservice _-----_ | | .--------------------------. |--(o)--| | Welcome to the kickass | `---------´ | Microservice | ( _´U`_ ) | generator! | /___A___ '--------------------------' | ~ | __'.___.'__ ´ ` |° ´ Y ` ? Enter the name of your service. (E.G. "my-service")
  46. 46. #dockercon Committer Pull Request
  47. 47. #dockercon Committer Reviewer Pull Request
  48. 48. #dockercon Committer Reviewer Pull Request
  49. 49. #dockercon Committer Reviewer Pull Request
  50. 50. Jenkins Pipeline
  51. 51. #dockercon if(…) { x ++; } Merge
  52. 52. #dockercon Pipeline (Simplified) build unit test publish deploy dev acceptance tests deploy qa deploy staging open RFC deploy prod close RFC
  53. 53. #dockercon Build if(…) { x ++; } 1.2 Merge
  54. 54. #dockercon Build if(…) { x ++; } 1.2 Merge 1.2. editorial-module.jar ./gradlew build {BUILD_NUMBER}
  55. 55. #dockercon Build 1.2 Merge editorial-module.jar if(…) { x ++; } 171.2.
  56. 56. #dockercon Build 1.2 Merge editorial-module.jar FROM orbitz/java-8 ADD build/editorial-module.jar /opt/orbitz CWD /opt/orbitz CMD java -jar editorial-module.jar if(…) { x ++; } 171.2.
  57. 57. #dockercon Build 1.2 Merge editorial-module.jar orbitz/editorial-module:1.2.17 if(…) { x ++; } 171.2.
  58. 58. #dockercon Build orbitz/editorial-module:1.2.17
  59. 59. #dockercon Deploy - hosts: dev - tasks: - name: find previous images docker_facts: image=orbitz/{{application}} register: previous - name: deploy new image docker: image=orbitz/{{application}}:{{version}} … - name: wait for service wait_for: port={{port}} … - name: check health endpoint uri: url="http://{{fqdn}}:{{port}}/health" … - name: kill old image docker: {{previous}} git pull playbook.yml (abridged)
  60. 60. #dockercon host-001 host-002 host-003 host-004 host-005 - hosts: dev
  61. 61. #dockercon Deploy host-001 1.2.16
  62. 62. #dockercon Deploy 1.2.16 1.2.17 host-001
  63. 63. #dockercon 1.2.16 1.2.17 host-001 Deploy DATABASE_URL=jdbc://whatever/db/stuff CONSUL_HOST={{ansible_fqdn}} LOGSTASH_HOST={{ansible_fqdn}} GRAPHITE_HOST={{ansible_fqdn}}
  64. 64. #dockercon Deploy 1.2.16 1.2.17 /health host-001
  65. 65. #dockercon Deploy 1.2.16 1.2.17 /health 200 OK host-001
  66. 66. #dockercon Deploy 1.2.16 1.2.17 host-001
  67. 67. #dockercon Deploy 1.2.17 host-001
  68. 68. #dockercon Deploy host-002 1.2.16 And so on… host-001 1.2.17
  69. 69. #dockercon What’s next?
  70. 70. #dockercon How to handle failure?
  71. 71. #dockercon What happens when a VM is moved?
  72. 72. #dockercon What if I need to add capacity?
  73. 73. #dockercon Deploy (Second Attempt) - hosts: dev - tasks: - name: find previous images docker_facts: image=orbitz/{{application}} register: previous - name: deploy new image docker: image=orbitz/{{application}}:{{version}} … - name: wait for service wait_for: port={{port}} … - name: check health endpoint uri: url="http://{{fqdn}}:{{port}}/health" … - name: kill old image docker: {{previous}} git pull playbook.yml (abridged)
  74. 74. #dockercon Deploy (Second Attempt) - hosts: dev - tasks: - name: find previous images docker_facts: image=orbitz/{{application}} register: previous - name: deploy new image docker: image=orbitz/{{application}}:{{version}} … - name: wait for service wait_for: port={{port}} … - name: check health endpoint uri: url="http://{{fqdn}}:{{port}}/health" … - name: kill old image git pull playbook.yml (abridged) - hosts: localhost - tasks: - name: marathon deploy marathon: image=orbitz/{{application}}:{{version}} instances=3
  75. 75. #dockercon Deploy (Second Attempt) New host setup
  76. 76. #dockercon Deploy (Second Attempt) Mesos Agent New host setup 1.2.16
  77. 77. #dockercon Deploy (Second Attempt) - tasks: marathon: … playbook.yml 1.2.16 1.2.16 1.2.16
  78. 78. #dockercon Deploy (Second Attempt) - tasks: marathon: … 1.2.16 1.2.16 1.2.16
  79. 79. #dockercon Deploy (Second Attempt) - tasks: marathon: … PUT /apps/editorial-module { “image”: “orbitz/editorial-module:1.2.17” … } 1.2.16 1.2.16 1.2.16
  80. 80. #dockercon Deploy (Second Attempt) PUT /apps/editorial-module { “image”: “orbitz/editorial-module:1.2.17” … } - tasks: marathon: … 1.2.16 1.2.16 1.2.16 app = GET /v2/apps/editorial-module if not app then deploy_id = POST /v2/apps { “image”: “orbitz/editorial-module:1.2.17”, “id”: “editorial-module” } else deploy_id = PUT /v2/apps/editorial-module { “image”: “orbitz/editorial-module:1.2.17” } end if while GET /v2/deployments contains deploy_id // still deploying end // deploy complete
  81. 81. #dockercon Deploy (Second Attempt) - tasks: marathon: … PUT /apps/editorial-module { “image”: “orbitz/editorial-module:1.2.17” … } 1.2.16 1.2.16 1.2.16
  82. 82. #dockercon Deploy (Second Attempt) - tasks: marathon: … 1.2.16 1.2.16 1.2.16
  83. 83. #dockercon Deploy (Second Attempt) - tasks: marathon: … 1.2.16 1.2.16 1.2.16 1.2.17 1.2.17 1.2.17
  84. 84. #dockercon Deploy (Second Attempt) - tasks: marathon: … /health 1.2.16 1.2.16 1.2.16 1.2.17 1.2.17 1.2.17
  85. 85. #dockercon Deploy (Second Attempt) - tasks: marathon: … /health 200 OK 200 OK 200 OK 1.2.16 1.2.16 1.2.16 1.2.17 1.2.17 1.2.17
  86. 86. #dockercon - tasks: marathon: … 1.2.17 1.2.17 1.2.17
  87. 87. #dockercon And off to the next environment… 1.2.17 1.2.17 1.2.17
  88. 88. #dockercon 1.2.17 1.2.17 1.2.17 What if?
  89. 89. #dockercon 1.2.17 1.2.17
  90. 90. #dockercon 1.2.17 1.2.17
  91. 91. #dockercon 1.2.17 1.2.17 1.2.17
  92. 92. #dockercon /health 200 OK 1.2.17 1.2.17 1.2.17
  93. 93. #dockercon 1.2.17 1.2.17 1.2.17
  94. 94. #dockercon Smoke/Acceptance Testing 1.2.17 1.2.17 1.2.17
  95. 95. #dockercon Paper Trail create tickets commit(s)
  96. 96. #dockercon Paper Trail fail! close/fail
  97. 97. #dockercon Paper Trail ok close
  98. 98. Build Unit Test Deploy Dev Deploy Prod Deploy Staging Acceptance Test Code Review & Push Production Pre-Production Open RFC Close RFC
  99. 99. #dockercon What’s inside the containers? • Spring Boot • Dropwizard Metrics (formerly Coda Hale, Yammer) • Consul Registration/Discovery • Logstash + Logback • Swagger • Hystrix • Retrofit + Consul OrbitzWorldwide/consul-client
  100. 100. #dockercon –You “Why didn’t you use {{.x}}?”
  101. 101. #dockercon Keeping an eye on… Kubernetes Amazon Elastic Container Service Docker Swarm
  102. 102. #dockercon Summary • Create a shared platform for docker deployments using shared and app-specific “localhost” helpers — this was ours, yours SHOULD look different — adapt to change, don’t fight it. • Take people out of the release process - Docker - repeatable applications - Chef - repeatable infrastructure — Environment aware - Jenkins - repeatable releases • Delineate configuration concerns: - Known at Compile time — Bake into Docker image - Known at Boot time — Bake into Playbook/Launcher - parameter to Docker run - Changes Anytime — Externalize (consul K/V, etcd, zookeeper)
  103. 103. Thanks! Steve Hoffman @bacoboy Rick Fast @tortiepoint

×