Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Enabling Microservice @ Orbitz - GOTO Chicago 2016

297 views

Published on

In this talk we will discuss how we enabled decomposition of one of our 500+ system components into a continuously deployed microservice cluster. Our platform is comprised of Apache Mesos/Marathon, Docker, and a number of local services including Consul for service discovery, Logstash for diskless logging, and a custom metrics forwarder to Graphite. Building on this, we'll detail our CI pipeline using Jenkins workflows to build and publish microservices as Docker images, test and deploy via Marathon/Mesos, and automated change tickets. Finally, we'll discuss lessons learned from building our own enterprise PaaS and scaling it out to a large organization.

Published in: Technology
  • Be the first to comment

Enabling Microservice @ Orbitz - GOTO Chicago 2016

  1. 1. Enabling Microservices
 @ Orbitz Steve Hoffman Senior Principal Engineer @bacoboy Rick Fast Principal Engineer @tortiepoint
  2. 2. #GOTOChgo 2016
  3. 3. #GOTOChgo 2016 Agenda • Brief Architecture Overview/History • From Monolithic to Services to Microservices with Docker • Automated Pipelines • Questions
  4. 4. #GOTOChgo 2016
  5. 5. #GOTOChgo 2016 2000
  6. 6. #GOTOChgo 2016 2000 Web Layer Business Layer
  7. 7. #GOTOChgo 2016 2000
  8. 8. #GOTOChgo 2016 2000
  9. 9. #GOTOChgo 2016 2000 Business Layer
  10. 10. #GOTOChgo 2016 2000 Business Layer
  11. 11. #GOTOChgo 2016 Business Layer 2000
  12. 12. #GOTOChgo 2016 Business Layer 2003
  13. 13. #GOTOChgo 2016 Business Layer 2003
  14. 14. #GOTOChgo 2016 Business Layer 2003
  15. 15. #GOTOChgo 2016 2003
  16. 16. #GOTOChgo 2016 2004
  17. 17. #GOTOChgo 2016 2004
  18. 18. #GOTOChgo 2016 2004
  19. 19. #GOTOChgo 2016 2012
  20. 20. #GOTOChgo 2016 2012
  21. 21. #GOTOChgo 2016 2012 • Multiple Brands • Websites • Webservices • Multiple Backends • 500+ apps / thousands of instances • Deployments Daily (sometimes more)
  22. 22. #GOTOChgo 2016 2015 • Multiple Brands • Websites • Webservices • Multiple Backends • 500+ apps / thousands of instances • Deployments Daily (sometimes more)
  23. 23. #GOTOChgo 2016 • Multiple Brands • Websites • Webservices • Multiple Backends • 500+ apps / thousands of instances • Deployments Daily (sometimes more) 2015
  24. 24. #GOTOChgo 2016 2016 • Multiple Brands • Websites • Webservices • Multiple Backends • 500+ apps / thousands of instances • Deployments Daily (sometimes more)
  25. 25. #GOTOChgo 2016 Process Overkill
  26. 26. #GOTOChgo 2016 Different Provisioning Tools DEV OPS
  27. 27. #GOTOChgo 2016 Different Provisioning Tools DEV OPS Application Platform!=
  28. 28. #GOTOChgo 2016 A New Experiment • Microservices? • Decompose single “service” into the actual 40+ sub-services • Any change to sub-service was a deployment of many • Could it be a simple Spring Boot App in Docker (12 factor?) • Backward compatible with existing service infrastructure. • Code to Production w/o help from other Humans
  29. 29. #GOTOChgo 2016 The Docker Slave aka “The Rickbot”
  30. 30. #GOTOChgo 2016 The Docker Slave aka “The Rickbot” App App App
  31. 31. #GOTOChgo 2016 App App App
  32. 32. #GOTOChgo 2016 App App App
  33. 33. #GOTOChgo 2016 App App App Register & Lookup
  34. 34. #GOTOChgo 2016 10.10.10.10 10.10.10.10:31002 App App App Register & Lookup
  35. 35. #GOTOChgo 2016 App App App
  36. 36. #GOTOChgo 2016 App App App
  37. 37. #GOTOChgo 2016 10.10.10.10 App App App
  38. 38. #GOTOChgo 2016 10.10.10.10 10.10.10.10:51515 App App App
  39. 39. #GOTOChgo 2016 App App App
  40. 40. #GOTOChgo 2016 App App App
  41. 41. #GOTOChgo 2016 App App App
  42. 42. #GOTOChgo 2016 10.10.10.10 App App App
  43. 43. #GOTOChgo 2016 10.10.10.10 10.10.10.10:1337 App App App
  44. 44. #GOTOChgo 2016 Almost done… App App App
  45. 45. #GOTOChgo 2016 Mesos Scheduler http://mesos.apache.org/
  46. 46. #GOTOChgo 2016 Mesos Scheduler Marathon Scheduler Slave xNMaster x3 ZK x3
  47. 47. #GOTOChgo 2016 App App App
  48. 48. #GOTOChgo 2016 App App App
  49. 49. #GOTOChgo 2016 App App App
  50. 50. #GOTOChgo 2016 App App App
  51. 51. #GOTOChgo 2016 App App App VIP
  52. 52. #GOTOChgo 2016 App App App VIP
  53. 53. #GOTOChgo 2016 App App App VIP
  54. 54. #GOTOChgo 2016 App App App
  55. 55. #GOTOChgo 2016 App App App
  56. 56. #GOTOChgo 2016 App App App
  57. 57. #GOTOChgo 2016 App App App
  58. 58. #GOTOChgo 2016 A Little More Background…
  59. 59. #GOTOChgo 2016
  60. 60. #GOTOChgo 2016 Module Module Module Module
  61. 61. #GOTOChgo 2016 Editorial Module A Continuously Deployed Microservice www.orbitz.com (scroll down)
  62. 62. #GOTOChgo 2016 Orbitz Content Orchestration Service Content (Solr) Editorial Module Search Module Hotel Module
  63. 63. #GOTOChgo 2016 Orbitz Content Orchestration Service Content (Solr) Editorial Module Search Module Hotel Module Hotel Team Search Team Content Team
  64. 64. #GOTOChgo 2016 Orbitz Content Orchestration Service Content (Solr) Editorial Module Search Module Hotel Module
  65. 65. #GOTOChgo 2016 Orbitz Content Orchestration Service Content (Solr) Editorial Module Search Module Hotel Module Hotel Team Search Team Content Team
  66. 66. #GOTOChgo 2016 Pre-Continuous Delivery 1.2-BETA-20150401-113002 Test Discard Test Discard 1.2-BETA-20150402-093002 Build Build And so on…
  67. 67. #GOTOChgo 2016 Pre-Continuous Delivery 1.2-BETA-20150401-113002 Test Discard Test Discard 1.2-BETA-20150402-093002 Build Build Test Deploy 1.2 Build
  68. 68. #GOTOChgo 2016 Continuous Delivery with Jenkins, Docker, Ansible and Marathon
  69. 69. #GOTOChgo 2016 Yo rfast-mbp:git rfast$ yo microservice _-----_ | | .--------------------------. |--(o)--| | Welcome to the kickass | `---------´ | Microservice | ( _´U`_ ) | generator! | /___A___ '--------------------------' | ~ | __'.___.'__ ´ ` |° ´ Y ` ? Enter the name of your service. (E.G. "my-service")
  70. 70. #GOTOChgo 2016 Committer Pull Request
  71. 71. #GOTOChgo 2016 Committer Reviewer Pull Request
  72. 72. #GOTOChgo 2016 Committer Reviewer Pull Request
  73. 73. #GOTOChgo 2016 Committer Reviewer Pull Request
  74. 74. #GOTOChgo 2016 Jenkins Pipeline
  75. 75. #GOTOChgo 2016 if(…) { x ++; } Merge
  76. 76. #GOTOChgo 2016 Pipeline (Simplified) build unit test publish deploy dev acceptance tests deploy qa deploy staging open RFC deploy prod close RFC
  77. 77. #GOTOChgo 2016 Build if(…) { x ++; } 1.2 Merge
  78. 78. #GOTOChgo 2016 Build if(…) { x ++; } 1.2 Merge 1.2. editorial-module.jar ./gradlew build {BUILD_NUMBER}
  79. 79. #GOTOChgo 2016 Build 1.2 Merge editorial-module.jar if(…) { x ++; } 171.2.
  80. 80. #GOTOChgo 2016 Build 1.2 Merge editorial-module.jar FROM orbitz/java-8 COPY build/editorial-module.jar /opt/orbitz WORKDIR /opt/orbitz ENTRYPOINT [“java”,”-jar”,”editorial-module.jar”] if(…) { x ++; } 171.2.
  81. 81. #GOTOChgo 2016 Build 1.2 Merge editorial-module.jar orbitz/editorial-module:1.2.17 if(…) { x ++; } 171.2.
  82. 82. #GOTOChgo 2016 Build orbitz/editorial-module:1.2.17
  83. 83. #GOTOChgo 2016 Deploy - hosts: dev - tasks: - name: find previous images docker_facts: image=orbitz/{{application}} register: previous - name: deploy new image docker: image=orbitz/{{application}}:{{version}} … - name: wait for service wait_for: port={{port}} … - name: check health endpoint uri: url="http://{{fqdn}}:{{port}}/health" … - name: kill old image docker: {{previous}} git pull playbook.yml (abridged)
  84. 84. #GOTOChgo 2016 - hosts: dev host-001 host-002 host-003 host-004
  85. 85. #GOTOChgo 2016 - hosts: dev host-001 host-002 host-003 host-004
  86. 86. #GOTOChgo 2016 - hosts: dev host-001 host-002 host-003 host-004
  87. 87. #GOTOChgo 2016 Downsides • How to handle failure with YAML files? • What happens when a VM is moved (destroyed/created)? • What if I need to add capacity?
  88. 88. #GOTOChgo 2016 Deploy (Second Attempt) - hosts: dev - tasks: - name: find previous images docker_facts: image=orbitz/{{application}} register: previous - name: deploy new image docker: image=orbitz/{{application}}:{{version}} … - name: wait for service wait_for: port={{port}} … - name: check health endpoint uri: url="http://{{fqdn}}:{{port}}/health" … - name: kill old image docker: {{previous}} git pull playbook.yml (abridged)
  89. 89. #GOTOChgo 2016 Deploy (Second Attempt) - hosts: dev - tasks: - name: find previous images docker_facts: image=orbitz/{{application}} register: previous - name: deploy new image docker: image=orbitz/{{application}}:{{version}} … - name: wait for service wait_for: port={{port}} … - name: check health endpoint uri: url="http://{{fqdn}}:{{port}}/health" … - name: kill old image git pull playbook.yml (abridged) - hosts: localhost - tasks: - name: marathon deploy marathon: image=orbitz/{{application}}:{{version}} instances=3
  90. 90. #GOTOChgo 2016 Deploy (Second Attempt) git pull playbook.yml (abridged) - hosts: localhost - tasks: - name: marathon deploy marathon: image=orbitz/{{application}}:{{version}} instances=3
  91. 91. #GOTOChgo 2016 Deploy (Second Attempt) New host setup
  92. 92. #GOTOChgo 2016 Deploy (Second Attempt) Mesos Agent New host setup 1.2.16
  93. 93. #GOTOChgo 2016 - tasks: marathon: … playbook.yml 1.2.16 1.2.16 1.2.16
  94. 94. #GOTOChgo 2016 - tasks: marathon: … 1.2.16 1.2.16 1.2.16
  95. 95. #GOTOChgo 2016 - tasks: marathon: … 1.2.16 1.2.16 1.2.16
  96. 96. #GOTOChgo 2016 - tasks: marathon: … 1.2.16 1.2.16 1.2.16 PUT /apps/editorial-module { “image”: “orbitz/editorial-module:1.2.17” … }
  97. 97. #GOTOChgo 2016 PUT /apps/editorial-module { “image”: “orbitz/editorial-module:1.2.17” … } - tasks: marathon: … 1.2.16 1.2.16 1.2.16 app = GET /v2/apps/editorial-module if not app then deploy_id = POST /v2/apps { “image”: “orbitz/editorial-module:1.2.17”, “id”: “editorial-module” } else deploy_id = PUT /v2/apps/editorial-module { “image”: “orbitz/editorial-module:1.2.17” } end if while GET /v2/deployments contains deploy_id // still deploying end // deploy complete
  98. 98. #GOTOChgo 2016 - tasks: marathon: … 1.2.16 1.2.16 1.2.16 “env”: { “SOME_DATABASE_URL”: “jdbc://whatever”, “SOME_DATABASE_USER”: “rickfast”, “SOME_OTHER_VALUE”: “whatever” } PUT /apps/editorial-module { “image”: “orbitz/editorial-module:1.2.17” … }
  99. 99. #GOTOChgo 2016 - tasks: marathon: … PUT /apps/editorial-module { “image”: “orbitz/editorial-module:1.2.17” … } 1.2.16 1.2.16 1.2.16
  100. 100. #GOTOChgo 2016 - tasks: marathon: … 1.2.16 1.2.16 1.2.16
  101. 101. #GOTOChgo 2016 - tasks: marathon: … 1.2.16 1.2.16 1.2.16 1.2.17 1.2.17 1.2.17
  102. 102. #GOTOChgo 2016 - tasks: marathon: … /health 1.2.16 1.2.16 1.2.16 1.2.17 1.2.17 1.2.17
  103. 103. #GOTOChgo 2016 - tasks: marathon: … /health 200 OK 200 OK 200 OK 1.2.16 1.2.16 1.2.16 1.2.17 1.2.17 1.2.17
  104. 104. #GOTOChgo 2016 - tasks: marathon: … 1.2.17 1.2.17 1.2.17
  105. 105. #GOTOChgo 2016 And off to the next environment… 1.2.17 1.2.17 1.2.17
  106. 106. #GOTOChgo 2016 1.2.17 1.2.17 1.2.17 What if?
  107. 107. #GOTOChgo 2016 1.2.17 1.2.17
  108. 108. #GOTOChgo 2016 1.2.17 1.2.17
  109. 109. #GOTOChgo 2016 1.2.17 1.2.17 1.2.17
  110. 110. #GOTOChgo 2016 /health 200 OK 1.2.17 1.2.17 1.2.17
  111. 111. #GOTOChgo 2016 1.2.17 1.2.17 1.2.17
  112. 112. #GOTOChgo 2016 Smoke/Acceptance Testing 1.2.17 1.2.17 1.2.17
  113. 113. #GOTOChgo 2016 Paper Trail create tickets commit(s)
  114. 114. #GOTOChgo 2016 Paper Trail fail! close/fail
  115. 115. #GOTOChgo 2016 Paper Trail ok close
  116. 116. #GOTOChgo 2016 Build Unit Test Deploy Dev Deploy Prod Deploy Staging Acceptance Test Code Review & Push Production Pre-Production Open RFC Close RFC
  117. 117. #GOTOChgo 2016 Pipeline as code! node { git ‘https://stash.orbitz.com/editorial-module.git‘ sh ‘./gradlew clean build’ def image = docker.build “editorial-module:${env.BUILD_TAG}" image() }
  118. 118. #GOTOChgo 2016 Pipeline as code! node { git ‘http://stash.orbitz.com/deploy-playbook.git' sh ‘./ansible-playbook … editorial-module:${env.BUILD_TAG} test’ } node { git ‘https://stash.orbitz.com/editorial-module.git‘ sh ‘./gradlew clean build’ def image = docker.build “editorial-module:${env.BUILD_TAG}" image.push() }
  119. 119. #GOTOChgo 2016 Pipeline as code! pipeline.json { “steps”: [ { “type”: “gradle-build” }, { “type”: “marathon-deploy”, “environment”: “test” } ] }
  120. 120. #GOTOChgo 2016 Pipeline as code! def pipeline = new JsonSluper().parse(‘pipeline.json’) pipeline.steps.each { step -> if (step.type == ‘gradle-build’) { node(‘java’) { git ‘https://stash.orbitz.com/${service}.git’ sh ‘./gradlew clean build’ def image = docker.build “${service}:${env.BUILD_TAG}” image.push() } } … }
  121. 121. #GOTOChgo 2016 Jenkins CI/CD - on Mesos? • Existing Solution • Dedicated Jenkins Slaves • Hand created • Snapshotted & Rolled Back to “Clean” state after each Job • Hard to Manage Build Environment for 300+ apps across many OSes, Java versions, Ruby versions, perl versions, python versions, protocol buffer compiler versions, etc…
  122. 122. #GOTOChgo 2016 Master Slave Commit/Push or Pull Request/Merge Before
  123. 123. #GOTOChgo 2016 Master Slave Poll or Push Trigger Commit/Push or Pull Request/Merge Clone & Build Push Artifacts Before
  124. 124. #GOTOChgo 2016 Master Slave Before
  125. 125. #GOTOChgo 2016 Jenkins mesos-plugin
  126. 126. #GOTOChgo 2016 Master Master Commit/Push or Pull Request/Merge After Slave env:jenkins
  127. 127. #GOTOChgo 2016 Master Master Slave Poll or Push Trigger Commit/Push or Pull Request/Merge Clone & Build Push Artifacts After Slave env:jenkins
  128. 128. #GOTOChgo 2016 Master Master After Slave env:jenkins
  129. 129. #GOTOChgo 2016 Master Master Slave Slave env:jenkins env:ci
  130. 130. #GOTOChgo 2016 Master Master Slave Slave Deploy App Clone & Run Slave env:jenkins env:ci docker
 pull
  131. 131. #GOTOChgo 2016 Master Master Slave App Slave env:jenkins env:ci
  132. 132. #GOTOChgo 2016 Prod Network Pre-Prod Network env:ci env:qa env:staging env:jenkins Master env:prod
  133. 133. #GOTOChgo 2016 DC: East DC: West env:ci
 dc:west env:qa
 dc:west env:staging
 dc:west env:jenkins
 dc:west Master env:prod
 dc:east env:prod
 dc:west
  134. 134. #GOTOChgo 2016 Learnings • Create a shared platform for docker deployments using shared and app-specific “localhost” helpers — this was ours, yours SHOULD look different — adapt to change, don’t fight it. • Take people out of the release process • Docker & Packer AMIs - repeatable applications/servers • Chef - repeatable infrastructure & configuration — Environment aware • Jenkins - repeatable releases • Delineate configuration concerns: • Known at Compile time — Bake into Docker image • Known at Boot time — Pass from Playbook/Launcher/Jenkins - parameter to Docker run/AMI • Changes Anytime — Externalize (consul K/V, etcd, zookeeper)
  135. 135. #GOTOChgo 2016

×