Enabling
Microservices
@Orbitz
Steve Hoffman
Senior Principal Engineer
@bacoboy
Rick Fast
Senior Principal Engineer
@tortiepoint
#velocityconf
2015
#velocityconf
2015
#velocityconf
2015
#velocityconf
2015
Outline
• Brief Architecture Overview/History
• From Monolithic to Services to Microservices/Docker
• Automated Pipelines
• Questions
#velocityconf
2015
#velocityconf
2015
2000
#velocityconf
2015
2000
Yes, we are STILL hiring!
#velocityconf
2015
2000
Yes, we are STILL hiring!
@OrbitzTalent
#velocityconf
2015
2000
Web Layer
Business Layer
#velocityconf
2015
2000
#velocityconf
2015
2000
#velocityconf
2015
2000
Business Layer
#velocityconf
2015
2000
Business Layer
#velocityconf
2015
Business Layer
2000
#velocityconf
2015
Business Layer
2003
#velocityconf
2015
Business Layer
2003
#velocityconf
2015
Business Layer
2003
#velocityconf
2015
2003
#velocityconf
2015
2004
#velocityconf
2015
2004
#velocityconf
2015
2004
#velocityconf
2015
2012
#velocityconf
2015
2012
#velocityconf
2015
2012
• Multiple Brands
• Websites
• Webservices
• Multiple Backends
• 500+ apps / thousands of
instances
• Deployments Daily (sometimes
more)
#velocityconf
2015
2015
• Multiple Brands
• Websites
• Webservices
• Multiple Backends
• 500+ apps / thousands of
instances
• Deployments Daily (sometimes
more)
#velocityconf
2015
Process Overkill
#velocityconf
2015
Different Provisioning Tools
DEV OPS
#velocityconf
2015
Different Provisioning Tools
DEV OPS
Application Platform!=
#velocityconf
2015
A New Experiment
• Microservices?
• Decompose single “service” into the actual 40+ sub-services
• Any change to sub-service was a deployment of many
• Could it be a simple Spring Boot App in Docker (12 factor?)
• Backward compatible with existing service infrastructure.
• Code to Production w/o help from other Humans - Minutes not Days
#velocityconf
2015
The Docker Slave
aka “The Rickbot”
#velocityconf
2015
The Docker Slave
aka “The Rickbot”
App
App
App
#velocityconf
2015
App
App
App
#velocityconf
2015
App
App
App
#velocityconf
2015
App
App
App Register & Lookup
#velocityconf
2015
10.10.10.10
10.10.10.10:31002
App
App
App Register & Lookup
#velocityconf
2015
App
App
App
#velocityconf
2015
App
App
App
#velocityconf
2015
#velocityconf
2015
#velocityconf
2015
#velocityconf
2015
#velocityconf
2015
#velocityconf
2015
http://consul.io
#velocityconf
2015
App
App
App
#velocityconf
2015
App
App
App
#velocityconf
2015
10.10.10.10
App
App
App
#velocityconf
2015
10.10.10.10
10.10.10.10:51515
App
App
App
#velocityconf
2015
App
App
App
#velocityconf
2015
App
App
App
#velocityconf
2015
App
App
App
#velocityconf
2015
10.10.10.10
App
App
App
#velocityconf
2015
10.10.10.10
10.10.10.10:1337
App
App
App
#velocityconf
2015
Almost done…
App
App
App
#velocityconf
2015
App
App
App
#velocityconf
2015
App
App
App
#velocityconf
2015
App
App
App
#velocityconf
2015
App
App
App
VIP
#velocityconf
2015
App
App
App
VIP
#velocityconf
2015
App
App
App
VIP
#velocityconf
2015
App
App
App
VIP
https://github.com/QubitProducts/bamboo
#velocityconf
2015
App
App
App
#velocityconf
2015
App
App
App
#velocityconf
2015
App
App
App
#velocityconf
2015
App
App
App
#velocityconf
2015
A Little More
Background
#velocityconf
2015
#velocityconf
2015
Module
Module
Module Module
#velocityconf
2015
Editorial Module
A Continuously Deployed Microservice
www.orbitz.com (scroll down)
#velocityconf
2015
Orbitz
Content
Orchestration
Service
Content
(Solr)
Editorial
Module
Search
Module
Hotel
Module
#velocityconf
2015
Orbitz
Content
Orchestration
Service
Content
(Solr)
Editorial
Module
Search
Module
Hotel
Module
Hotel Team Search Team Content Team
#velocityconf
2015
Orbitz
Content
Orchestration
Service
Content
(Solr)
Editorial
Module
Search
Module
Hotel
Module
#velocityconf
2015
Orbitz
Content
Orchestration
Service
Content
(Solr)
Editorial
Module
Search
Module
Hotel
Module
Hotel Team
Search Team
Content Team
#velocityconf
2015
Pre-Continuous Delivery
1.2-­‐BETA-­‐20150401-­‐113002
Test Discard
Test Discard
1.2-­‐BETA-­‐20150402-­‐093002
Build
Build
And so on…
#velocityconf
2015
Pre-Continuous Delivery
1.2-­‐BETA-­‐20150401-­‐113002
Test Discard
Test Discard
1.2-­‐BETA-­‐20150402-­‐093002
Build
Build
Test Deploy
1.2
Build
#velocityconf
2015
Continuous Delivery with
Jenkins, Docker, and Ansible
(And then Marathon)
#velocityconf
2015
rfast-mbp:git rfast$ yo microservice
_-----_
| | .--------------------------.
|--(o)--| | Welcome to the kickass |
`---------´ | Microservice |
( _´U`_ ) | generator! |
/___A___ '--------------------------'
| ~ |
__'.___.'__
´ ` |° ´ Y `
? Enter the name of your service. (E.G. "my-service")
Yo
#velocityconf
2015
Committer
Pull Request
#velocityconf
2015
Committer
Reviewer
Pull Request
#velocityconf
2015
Committer
Reviewer
Pull Request
#velocityconf
2015
Committer
Reviewer
Pull Request
#velocityconf
2015
Jenkins Pipeline
#velocityconf
2015
if(…) {
x ++;
}
Merge
#velocityconf
2015
build
unit test
publish
deploy
dev
acceptance
tests
deploy
qa
deploy
staging
open RFC
deploy
prod
close RFC
Pipeline (Simplified)
#velocityconf
2015
if(…) {
x ++;
}
1.2
Merge
Build
#velocityconf
2015
if(…) {
x ++;
}
1.2
Merge
1.2.
editorial-module.jar
./gradlew build
{BUILD_NUMBER}
Build
#velocityconf
2015
1.2
Merge
editorial-module.jar
if(…) {
x ++;
}
171.2.
Build
#velocityconf
2015
1.2
Merge
editorial-module.jar
FROM orbitz/java-8
ADD build/editorial-module.jar /opt/orbitz
CWD /opt/orbitz
CMD java -jar editorial-module.jar
if(…) {
x ++;
}
171.2.
Build
#velocityconf
2015
1.2
Merge
editorial-module.jar orbitz/editorial-module:1.2.17
if(…) {
x ++;
}
171.2.
Build
#velocityconf
2015
orbitz/editorial-module:1.2.17
Build
#velocityconf
2015
- hosts: dev
- tasks:
- name: find previous images
docker_facts: image=orbitz/{{application}}
register: previous
- name: deploy new image
docker: image=orbitz/{{application}}:{{version}} …
- name: wait for service
wait_for: port={{port}} …
- name: check health endpoint
uri: url="http://{{fqdn}}:{{port}}/health" …
- name: kill old image
docker: {{previous}}
git pull
playbook.yml (abridged)
Deploy
#velocityconf
2015
host-001 host-002 host-003 host-004 host-005
- hosts: dev
Deploy
#velocityconf
2015
host-001
1.2.16
Deploy
#velocityconf
2015
1.2.16 1.2.17
host-001
Deploy
#velocityconf
2015
1.2.16 1.2.17
host-001
Deploy
DATABASE_URL=jdbc://whatever/db/stuff
CONSUL_HOST={{ansible_fqdn}}
LOGSTASH_HOST={{ansible_fqdn}}
GRAPHITE_HOST={{ansible_fqdn}}
#velocityconf
2015
1.2.16 1.2.17
/health
host-001
Deploy
#velocityconf
2015
1.2.16 1.2.17
/health
200 OK
host-001
Deploy
#velocityconf
2015
1.2.16 1.2.17
host-001
Deploy
#velocityconf
2015
1.2.17
host-001
Deploy
#velocityconf
2015
host-002
1.2.16
And so on…
host-001
1.2.17
Deploy
#velocityconf
2015
What’s next?
#velocityconf
2015
How to handle failure?
#velocityconf
2015
What happens when a
VM is moved?
#velocityconf
2015
What if I need to add
capacity?
#velocityconf
2015
- hosts: dev
- tasks:
- name: find previous images
docker_facts: image=orbitz/{{application}}
register: previous
- name: deploy new image
docker: image=orbitz/{{application}}:{{version}} …
- name: wait for service
wait_for: port={{port}} …
- name: check health endpoint
uri: url="http://{{fqdn}}:{{port}}/health" …
- name: kill old image
docker: {{previous}}
git pull
playbook.yml (abridged)
Deploy
#velocityconf
2015
- hosts: dev
- tasks:
- name: find previous images
docker_facts: image=orbitz/{{application}}
register: previous
- name: deploy new image
docker: image=orbitz/{{application}}:{{version}} …
- name: wait for service
wait_for: port={{port}} …
- name: check health endpoint
uri: url="http://{{fqdn}}:{{port}}/health" …
- name: kill old image
git pull
playbook.yml (abridged)
- hosts: localhost
- tasks:
- name: marathon deploy
marathon: image=orbitz/{{application}}:{{version}}
instances=3
Deploy
#velocityconf
2015
git pull
playbook.yml (abridged)
- hosts: localhost
- tasks:
- name: marathon deploy
marathon: image=orbitz/{{application}}:{{version}}
instances=3
Deploy
#velocityconf
2015
New host setup
Deploy
#velocityconf
2015
Mesos Agent
New host setup
1.2.16
Deploy
#velocityconf
2015
- tasks:
marathon: …
playbook.yml
1.2.16
1.2.16
1.2.16
Deploy
#velocityconf
2015
- tasks:
marathon: …
1.2.16
1.2.16
1.2.16
Deploy
#velocityconf
2015
- tasks:
marathon: …
1.2.16
1.2.16
1.2.16
Deploy
#velocityconf
2015
- tasks:
marathon: …
PUT /apps/editorial-module
{
“image”: “orbitz/editorial-module:1.2.17” …
}
1.2.16
1.2.16
1.2.16
Deploy
#velocityconf
2015
Deploy
PUT /apps/editorial-module
{
“image”: “orbitz/editorial-module:1.2.17” …
}
- tasks:
marathon: …
1.2.16
1.2.16
1.2.16
app = GET /v2/apps/editorial-module
if not app then
deploy_id =
POST /v2/apps
{
“image”: “orbitz/editorial-module:1.2.17”,
“id”: “editorial-module”
}
else
deploy_id =
PUT /v2/apps/editorial-module
{ “image”: “orbitz/editorial-module:1.2.17” }
end if
while GET /v2/deployments contains deploy_id
// still deploying
end
// deploy complete
#velocityconf
2015
- tasks:
marathon: …
PUT /apps/editorial-module
{
“image”: “orbitz/editorial-module:1.2.17” …
}
1.2.16
1.2.16
1.2.16
Deploy
#velocityconf
2015
- tasks:
marathon: …
1.2.16
1.2.16
1.2.16
Deploy
#velocityconf
2015
- tasks:
marathon: …
1.2.16
1.2.16
1.2.16
1.2.17
1.2.17
1.2.17
Deploy
#velocityconf
2015
- tasks:
marathon: …
/health
1.2.16
1.2.16
1.2.16
1.2.17
1.2.17
1.2.17
Deploy
#velocityconf
2015
- tasks:
marathon: …
/health
200 OK
200 OK
200 OK
1.2.16
1.2.16
1.2.16
1.2.17
1.2.17
1.2.17
Deploy
#velocityconf
2015
- tasks:
marathon: …
1.2.17
1.2.17
1.2.17
Deploy
#velocityconf
2015
And off to the next environment…
1.2.17
1.2.17
1.2.17
Deploy
#velocityconf
2015
1.2.17
1.2.17
1.2.17
What if?
#velocityconf
2015
1.2.17
1.2.17
#velocityconf
2015
1.2.17
1.2.17
#velocityconf
2015
1.2.17 1.2.17
1.2.17
#velocityconf
2015
/health
200
OK
1.2.17 1.2.17
1.2.17
#velocityconf
2015
1.2.17 1.2.17
1.2.17
#velocityconf
2015
Smoke/Acceptance Testing
1.2.17
1.2.17
1.2.17
#velocityconf
2015
Paper Trail
create
tickets
commit(s)
#velocityconf
2015
Paper Trail
fail!
close/fail
#velocityconf
2015
Paper Trail
ok
close
#velocityconf
2015
Build
Unit Test
Deploy
Dev
Deploy
Prod
Deploy
Staging
Acceptance
Test
Code
Review &
Push
Production
Pre-Production
Open
RFC
Close
RFC
#velocityconf
2015
What’s inside the containers?
• Spring Boot
• Dropwizard Metrics (formerly Coda Hale, Yammer)
• Consul Registration/Discovery OrbitzWorldwide/consul-client
• Logstash + Logback
• Swagger
• Hystrix
• Retrofit + Consul
#velocityconf
2015
“Why didn’t you use {{.x}} ?”
#velocityconf
2015
Keeping an eye on…
Kubernetes Amazon
Elastic Container
Service
Docker
Swarm
#velocityconf
May 2015
Summary
• Create a platform for docker deployments using shared and app-specific “localhost” helpers
— this was ours, yours SHOULD look different — adapt to change, don’t fight it.
• People don’t scale, use automation
• Docker - repeatable applications (environment portable apps)
• Chef - repeatable infrastructure (environment aware services)
• Jenkins - repeatable releases (pipelines)
• Delineate configuration concerns:
• Known at Compile time — Bake into Docker image
• Known at Boot time — Bake into Playbook/Launcher - parameter to Docker run
• Changes Anytime — Externalize (consul K/V, etcd, zookeeper)
#velocityconf
2015
Questions?

Enabling Microservices @Orbitz - Velocity Conf 2015