Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Enabling Microservices @Orbitz - Velocity Conf 2015

2,237 views

Published on

Slides from Orbitz's use case for microservices on docker & mesos at Velocity Santa Clara 2015 conference.

Details: http://velocityconf.com/web-mobile-business-conf-2015/public/schedule/detail/40700

Published in: Technology

Enabling Microservices @Orbitz - Velocity Conf 2015

  1. 1. Enabling Microservices @Orbitz Steve Hoffman Senior Principal Engineer @bacoboy Rick Fast Senior Principal Engineer @tortiepoint
  2. 2. #velocityconf 2015
  3. 3. #velocityconf 2015
  4. 4. #velocityconf 2015
  5. 5. #velocityconf 2015 Outline • Brief Architecture Overview/History • From Monolithic to Services to Microservices/Docker • Automated Pipelines • Questions
  6. 6. #velocityconf 2015
  7. 7. #velocityconf 2015 2000
  8. 8. #velocityconf 2015 2000 Yes, we are STILL hiring!
  9. 9. #velocityconf 2015 2000 Yes, we are STILL hiring! @OrbitzTalent
  10. 10. #velocityconf 2015 2000 Web Layer Business Layer
  11. 11. #velocityconf 2015 2000
  12. 12. #velocityconf 2015 2000
  13. 13. #velocityconf 2015 2000 Business Layer
  14. 14. #velocityconf 2015 2000 Business Layer
  15. 15. #velocityconf 2015 Business Layer 2000
  16. 16. #velocityconf 2015 Business Layer 2003
  17. 17. #velocityconf 2015 Business Layer 2003
  18. 18. #velocityconf 2015 Business Layer 2003
  19. 19. #velocityconf 2015 2003
  20. 20. #velocityconf 2015 2004
  21. 21. #velocityconf 2015 2004
  22. 22. #velocityconf 2015 2004
  23. 23. #velocityconf 2015 2012
  24. 24. #velocityconf 2015 2012
  25. 25. #velocityconf 2015 2012 • Multiple Brands • Websites • Webservices • Multiple Backends • 500+ apps / thousands of instances • Deployments Daily (sometimes more)
  26. 26. #velocityconf 2015 2015 • Multiple Brands • Websites • Webservices • Multiple Backends • 500+ apps / thousands of instances • Deployments Daily (sometimes more)
  27. 27. #velocityconf 2015 Process Overkill
  28. 28. #velocityconf 2015 Different Provisioning Tools DEV OPS
  29. 29. #velocityconf 2015 Different Provisioning Tools DEV OPS Application Platform!=
  30. 30. #velocityconf 2015 A New Experiment • Microservices? • Decompose single “service” into the actual 40+ sub-services • Any change to sub-service was a deployment of many • Could it be a simple Spring Boot App in Docker (12 factor?) • Backward compatible with existing service infrastructure. • Code to Production w/o help from other Humans - Minutes not Days
  31. 31. #velocityconf 2015 The Docker Slave aka “The Rickbot”
  32. 32. #velocityconf 2015 The Docker Slave aka “The Rickbot” App App App
  33. 33. #velocityconf 2015 App App App
  34. 34. #velocityconf 2015 App App App
  35. 35. #velocityconf 2015 App App App Register & Lookup
  36. 36. #velocityconf 2015 10.10.10.10 10.10.10.10:31002 App App App Register & Lookup
  37. 37. #velocityconf 2015 App App App
  38. 38. #velocityconf 2015 App App App
  39. 39. #velocityconf 2015
  40. 40. #velocityconf 2015
  41. 41. #velocityconf 2015
  42. 42. #velocityconf 2015
  43. 43. #velocityconf 2015
  44. 44. #velocityconf 2015 http://consul.io
  45. 45. #velocityconf 2015 App App App
  46. 46. #velocityconf 2015 App App App
  47. 47. #velocityconf 2015 10.10.10.10 App App App
  48. 48. #velocityconf 2015 10.10.10.10 10.10.10.10:51515 App App App
  49. 49. #velocityconf 2015 App App App
  50. 50. #velocityconf 2015 App App App
  51. 51. #velocityconf 2015 App App App
  52. 52. #velocityconf 2015 10.10.10.10 App App App
  53. 53. #velocityconf 2015 10.10.10.10 10.10.10.10:1337 App App App
  54. 54. #velocityconf 2015 Almost done… App App App
  55. 55. #velocityconf 2015 App App App
  56. 56. #velocityconf 2015 App App App
  57. 57. #velocityconf 2015 App App App
  58. 58. #velocityconf 2015 App App App VIP
  59. 59. #velocityconf 2015 App App App VIP
  60. 60. #velocityconf 2015 App App App VIP
  61. 61. #velocityconf 2015 App App App VIP https://github.com/QubitProducts/bamboo
  62. 62. #velocityconf 2015 App App App
  63. 63. #velocityconf 2015 App App App
  64. 64. #velocityconf 2015 App App App
  65. 65. #velocityconf 2015 App App App
  66. 66. #velocityconf 2015 A Little More Background
  67. 67. #velocityconf 2015
  68. 68. #velocityconf 2015 Module Module Module Module
  69. 69. #velocityconf 2015 Editorial Module A Continuously Deployed Microservice www.orbitz.com (scroll down)
  70. 70. #velocityconf 2015 Orbitz Content Orchestration Service Content (Solr) Editorial Module Search Module Hotel Module
  71. 71. #velocityconf 2015 Orbitz Content Orchestration Service Content (Solr) Editorial Module Search Module Hotel Module Hotel Team Search Team Content Team
  72. 72. #velocityconf 2015 Orbitz Content Orchestration Service Content (Solr) Editorial Module Search Module Hotel Module
  73. 73. #velocityconf 2015 Orbitz Content Orchestration Service Content (Solr) Editorial Module Search Module Hotel Module Hotel Team Search Team Content Team
  74. 74. #velocityconf 2015 Pre-Continuous Delivery 1.2-­‐BETA-­‐20150401-­‐113002 Test Discard Test Discard 1.2-­‐BETA-­‐20150402-­‐093002 Build Build And so on…
  75. 75. #velocityconf 2015 Pre-Continuous Delivery 1.2-­‐BETA-­‐20150401-­‐113002 Test Discard Test Discard 1.2-­‐BETA-­‐20150402-­‐093002 Build Build Test Deploy 1.2 Build
  76. 76. #velocityconf 2015 Continuous Delivery with Jenkins, Docker, and Ansible (And then Marathon)
  77. 77. #velocityconf 2015 rfast-mbp:git rfast$ yo microservice _-----_ | | .--------------------------. |--(o)--| | Welcome to the kickass | `---------´ | Microservice | ( _´U`_ ) | generator! | /___A___ '--------------------------' | ~ | __'.___.'__ ´ ` |° ´ Y ` ? Enter the name of your service. (E.G. "my-service") Yo
  78. 78. #velocityconf 2015 Committer Pull Request
  79. 79. #velocityconf 2015 Committer Reviewer Pull Request
  80. 80. #velocityconf 2015 Committer Reviewer Pull Request
  81. 81. #velocityconf 2015 Committer Reviewer Pull Request
  82. 82. #velocityconf 2015 Jenkins Pipeline
  83. 83. #velocityconf 2015 if(…) { x ++; } Merge
  84. 84. #velocityconf 2015 build unit test publish deploy dev acceptance tests deploy qa deploy staging open RFC deploy prod close RFC Pipeline (Simplified)
  85. 85. #velocityconf 2015 if(…) { x ++; } 1.2 Merge Build
  86. 86. #velocityconf 2015 if(…) { x ++; } 1.2 Merge 1.2. editorial-module.jar ./gradlew build {BUILD_NUMBER} Build
  87. 87. #velocityconf 2015 1.2 Merge editorial-module.jar if(…) { x ++; } 171.2. Build
  88. 88. #velocityconf 2015 1.2 Merge editorial-module.jar FROM orbitz/java-8 ADD build/editorial-module.jar /opt/orbitz CWD /opt/orbitz CMD java -jar editorial-module.jar if(…) { x ++; } 171.2. Build
  89. 89. #velocityconf 2015 1.2 Merge editorial-module.jar orbitz/editorial-module:1.2.17 if(…) { x ++; } 171.2. Build
  90. 90. #velocityconf 2015 orbitz/editorial-module:1.2.17 Build
  91. 91. #velocityconf 2015 - hosts: dev - tasks: - name: find previous images docker_facts: image=orbitz/{{application}} register: previous - name: deploy new image docker: image=orbitz/{{application}}:{{version}} … - name: wait for service wait_for: port={{port}} … - name: check health endpoint uri: url="http://{{fqdn}}:{{port}}/health" … - name: kill old image docker: {{previous}} git pull playbook.yml (abridged) Deploy
  92. 92. #velocityconf 2015 host-001 host-002 host-003 host-004 host-005 - hosts: dev Deploy
  93. 93. #velocityconf 2015 host-001 1.2.16 Deploy
  94. 94. #velocityconf 2015 1.2.16 1.2.17 host-001 Deploy
  95. 95. #velocityconf 2015 1.2.16 1.2.17 host-001 Deploy DATABASE_URL=jdbc://whatever/db/stuff CONSUL_HOST={{ansible_fqdn}} LOGSTASH_HOST={{ansible_fqdn}} GRAPHITE_HOST={{ansible_fqdn}}
  96. 96. #velocityconf 2015 1.2.16 1.2.17 /health host-001 Deploy
  97. 97. #velocityconf 2015 1.2.16 1.2.17 /health 200 OK host-001 Deploy
  98. 98. #velocityconf 2015 1.2.16 1.2.17 host-001 Deploy
  99. 99. #velocityconf 2015 1.2.17 host-001 Deploy
  100. 100. #velocityconf 2015 host-002 1.2.16 And so on… host-001 1.2.17 Deploy
  101. 101. #velocityconf 2015 What’s next?
  102. 102. #velocityconf 2015 How to handle failure?
  103. 103. #velocityconf 2015 What happens when a VM is moved?
  104. 104. #velocityconf 2015 What if I need to add capacity?
  105. 105. #velocityconf 2015 - hosts: dev - tasks: - name: find previous images docker_facts: image=orbitz/{{application}} register: previous - name: deploy new image docker: image=orbitz/{{application}}:{{version}} … - name: wait for service wait_for: port={{port}} … - name: check health endpoint uri: url="http://{{fqdn}}:{{port}}/health" … - name: kill old image docker: {{previous}} git pull playbook.yml (abridged) Deploy
  106. 106. #velocityconf 2015 - hosts: dev - tasks: - name: find previous images docker_facts: image=orbitz/{{application}} register: previous - name: deploy new image docker: image=orbitz/{{application}}:{{version}} … - name: wait for service wait_for: port={{port}} … - name: check health endpoint uri: url="http://{{fqdn}}:{{port}}/health" … - name: kill old image git pull playbook.yml (abridged) - hosts: localhost - tasks: - name: marathon deploy marathon: image=orbitz/{{application}}:{{version}} instances=3 Deploy
  107. 107. #velocityconf 2015 git pull playbook.yml (abridged) - hosts: localhost - tasks: - name: marathon deploy marathon: image=orbitz/{{application}}:{{version}} instances=3 Deploy
  108. 108. #velocityconf 2015 New host setup Deploy
  109. 109. #velocityconf 2015 Mesos Agent New host setup 1.2.16 Deploy
  110. 110. #velocityconf 2015 - tasks: marathon: … playbook.yml 1.2.16 1.2.16 1.2.16 Deploy
  111. 111. #velocityconf 2015 - tasks: marathon: … 1.2.16 1.2.16 1.2.16 Deploy
  112. 112. #velocityconf 2015 - tasks: marathon: … 1.2.16 1.2.16 1.2.16 Deploy
  113. 113. #velocityconf 2015 - tasks: marathon: … PUT /apps/editorial-module { “image”: “orbitz/editorial-module:1.2.17” … } 1.2.16 1.2.16 1.2.16 Deploy
  114. 114. #velocityconf 2015 Deploy PUT /apps/editorial-module { “image”: “orbitz/editorial-module:1.2.17” … } - tasks: marathon: … 1.2.16 1.2.16 1.2.16 app = GET /v2/apps/editorial-module if not app then deploy_id = POST /v2/apps { “image”: “orbitz/editorial-module:1.2.17”, “id”: “editorial-module” } else deploy_id = PUT /v2/apps/editorial-module { “image”: “orbitz/editorial-module:1.2.17” } end if while GET /v2/deployments contains deploy_id // still deploying end // deploy complete
  115. 115. #velocityconf 2015 - tasks: marathon: … PUT /apps/editorial-module { “image”: “orbitz/editorial-module:1.2.17” … } 1.2.16 1.2.16 1.2.16 Deploy
  116. 116. #velocityconf 2015 - tasks: marathon: … 1.2.16 1.2.16 1.2.16 Deploy
  117. 117. #velocityconf 2015 - tasks: marathon: … 1.2.16 1.2.16 1.2.16 1.2.17 1.2.17 1.2.17 Deploy
  118. 118. #velocityconf 2015 - tasks: marathon: … /health 1.2.16 1.2.16 1.2.16 1.2.17 1.2.17 1.2.17 Deploy
  119. 119. #velocityconf 2015 - tasks: marathon: … /health 200 OK 200 OK 200 OK 1.2.16 1.2.16 1.2.16 1.2.17 1.2.17 1.2.17 Deploy
  120. 120. #velocityconf 2015 - tasks: marathon: … 1.2.17 1.2.17 1.2.17 Deploy
  121. 121. #velocityconf 2015 And off to the next environment… 1.2.17 1.2.17 1.2.17 Deploy
  122. 122. #velocityconf 2015 1.2.17 1.2.17 1.2.17 What if?
  123. 123. #velocityconf 2015 1.2.17 1.2.17
  124. 124. #velocityconf 2015 1.2.17 1.2.17
  125. 125. #velocityconf 2015 1.2.17 1.2.17 1.2.17
  126. 126. #velocityconf 2015 /health 200 OK 1.2.17 1.2.17 1.2.17
  127. 127. #velocityconf 2015 1.2.17 1.2.17 1.2.17
  128. 128. #velocityconf 2015 Smoke/Acceptance Testing 1.2.17 1.2.17 1.2.17
  129. 129. #velocityconf 2015 Paper Trail create tickets commit(s)
  130. 130. #velocityconf 2015 Paper Trail fail! close/fail
  131. 131. #velocityconf 2015 Paper Trail ok close
  132. 132. #velocityconf 2015 Build Unit Test Deploy Dev Deploy Prod Deploy Staging Acceptance Test Code Review & Push Production Pre-Production Open RFC Close RFC
  133. 133. #velocityconf 2015 What’s inside the containers? • Spring Boot • Dropwizard Metrics (formerly Coda Hale, Yammer) • Consul Registration/Discovery OrbitzWorldwide/consul-client • Logstash + Logback • Swagger • Hystrix • Retrofit + Consul
  134. 134. #velocityconf 2015 “Why didn’t you use {{.x}} ?”
  135. 135. #velocityconf 2015 Keeping an eye on… Kubernetes Amazon Elastic Container Service Docker Swarm
  136. 136. #velocityconf May 2015 Summary • Create a platform for docker deployments using shared and app-specific “localhost” helpers — this was ours, yours SHOULD look different — adapt to change, don’t fight it. • People don’t scale, use automation • Docker - repeatable applications (environment portable apps) • Chef - repeatable infrastructure (environment aware services) • Jenkins - repeatable releases (pipelines) • Delineate configuration concerns: • Known at Compile time — Bake into Docker image • Known at Boot time — Bake into Playbook/Launcher - parameter to Docker run • Changes Anytime — Externalize (consul K/V, etcd, zookeeper)
  137. 137. #velocityconf 2015 Questions?

×