Building a smarter application Stack by Tomas Doran from Yelp

D
Building a smarter application stack
Tomas	
  Doran	
  
@bobtfish	
  
2014-­‐06-­‐10
Docker is the future
• Preaching to the converted here ;)

• Game changing technology

• No silver bullets (ever)
• Introduces it’s own set of problems and issues
• Dependency discovery / wiring
• Scheduling
2
Smartstack
• One possible solution to discovery
problems

• This talk:
• Application architecture
• Problem(s) we’re solving
• Why this solution works well for us
• Alternate solutions
3
Microservices - also the future!
• The same as SOA
• But one API per service.
• Own data store!

• Lots of services (dozens, maybe 100s)

• All individually very simple
• Easy to reason about.
• Easy to replace
4
Don’t break the site - ever!!!
• Microservices are individually deployable!
• When we say “Don’t break the site”
• We mean
• Don’t break all of the site!
5
Don’t break the site - ever!!!
• If you have graceful degradation…
• You can ignore MTBF in the backend services!
• You only care about MTTR.
6
“I’ll just break this out into it’s own
application, as it’ll be easier to
maintain in 10 years time”
- Pre seed funding
nobody, ever!
7
Monolith - the reality
• Everyone has one of these :)
• If you’re far enough down the path, you call
this ‘The presentation layer’.

• Still poses a challenge
• need async requests
• need graceful degradation
8
Monolith - the reality
• Most popular service

• Most dependencies
• Call into 10s or 100s of other services in a
request!

• Needs circuit breakers + dynamic
configuration
9
No silver bullet = No one solution
• You should always have 2.
• Nagios / Sensu
• RRDs + Ganglia / Graphite + Diamond
• YAML files / Zookeeper
10
No silver bullet = No one solution
• ‘Top down’ architecture sucks.
• Instead, broad goals + ’Bottom up’ architecture
• Internal competition!
• Replacing the incumbent solution happens
organically
• If your thing works better, people will want to
move! 

• Not perfect! Better than top-down!
11
“Humans are bad at predicting the performance of
complex systems […]. Our ability to create large
and complex systems fools us into believing that
we’re also entitled to understand them”

- Carlos Bueno “Mature optimization handbook”

12
Distributed complexity
• Distributed systems introduce their own set of complexity
• Reasoning about the whole system challenging
• Timing/profiling/performance analysis non-trivial
• Resource scheduling also non-trivial
• 2nd order effects
!
• Can’t reason about emergent behavior
13
14
15
?
16
?
What the heck happened at 16:46?
17
?
And why did it stop at 17:00?
Dynamic architecture
• Cattle not pets
• AWS and VMs in ‘the cloud’ started this
• Docker takes it a step further

• Explicitly manage persistent state
• Explicit regular recycling
• All updates are redeploys

18
Dependency nightmares
• Almost everything has some dependencies
• Simple example, web app with a mysql DB
• App config in a YAML file

• Mysql container address changes when you restart mysql!
• Oops, app can’t find mysql!

• Do I need to restart every application using mysql?
• Sucks!
• Do I need to rebuild application containers using mysql?
• To edit the config YAML file!
• Super super sucks! 19
Runtime wiring
• mysql failovers - the simple case!

• Presentation layer talking to service REST layers
• Different deployment schedules
• No downtime

• Only possible solution: wiring dependencies at runtime
• A challenge
• Also an opportunity

• DNS is workable in some cases
20
Dynamic discovery
• Discovery becomes a core problem

• DNS re-resolving not generally trustworthy
• You need to test everything for this

• DNS balancing (internally) is awful
• Failed node + multiple connections/requests
• DNS round robin
• Everything sees failure
• Slow to shift traffic
• Round robin is crappy for load balancing

21
Externalized wiring
• Remove a lot of complexity from the application domain

• Run a load balancer (haproxy) on each machine

• Applications always connect to load balancer on fixed
host/port
• localhost on traditional metal/VMs
• supplied by —link or environment variables in Docker

• Applications become wiring agnostic!
22
Externalized wiring
23
‘Client side load balancing’
• Lots of projects use this approach:
• Project Atomic
• Marathon + Mesos-Docker
• vulcand (https://github.com/mailgun/vulcand)
• Frontrunner (https://github.com/Wizcorp/
frontrunner)
• Consul
!
• Smartstack
24
Legacy infrastructure
• Physical machines
• Application images in AMIs
• kvm

• Can’t just use container links or a Docker only
solution

• Want to use the same (uniform) solution
everywhere.
25
Entropy reduction
• You can’t change everything at once!

• Everything will tend towards chaos
• ‘Old infrastructure’
• ‘New infrastructure’
• ‘New new infrastructure’
!
• Solution specifically chosen so it could be generic.
26
SmartStack
• 2 parts
• Synapse
• Nerve
!
• Conceptually simple
• Very flexible
• Easy to hack on
• Plays well on traditional machines
• Plays well in docker
27
Synapse
• Does discovery against a pluggable backend
• Writes out a haproxy configuration

• Assign a well known port to all services
• Application connects to that port
• haproxy forwards to an available backend

• Your application doesn’t need to know about discovery!

• Technology agnostic - works the same on metal/VMs/Docker
28
29
Why synapse?
• haproxy is a well known solution
• ruby - easy to modify
• simple (has one job)
• Pluggable - discovery with multiple methods:
• JSON config (static)
• zookeeper
• etcd
• docker API
• ec2 tags
• Flexible
• Deploy one per instance
• Or pairs as dedicated lbs 30
Nerve
• Health checks services
• Health checks are pluggable.
• HTTP (flexible) + mysql come out the box
• Registers service information to backend
• zookeeper
• etcd (beta)
31
32
Connector agnostic containers
• On ‘real servers’ or VMs, running a synapse
instance per box is fine.

• In docker, we want to abstract more than that
• Make containers connector agnostic!
• They don’t need to know or care
• Upgrade independently.
33
Synapse <3 ambassador
containers
• ‘Ambassador pattern’
• Run a synapse ‘ambassador’ container on each host
for each service
• Link each application to the ambassador for each of
it’s dependencies
• Environment variables to each service’s haproxy
• Separates synapse management (i.e. changing the
wiring) from application management (i.e.
upgrading the app version).
34
35
Container links
• Ambassador for service A presents:
• port 8000 for HTTP REST service
• port 8443 for HTTPS REST service
!
• Container linking to ambassador sees:
• SRVA_PORT_8000_TCP=tcp://172.17.0.8:6379
• SRVA_PORT_8000_TCP_PROTO=tcp
• SRVA_PORT_8000_TCP_ADDR=172.17.0.8
• SRVA_PORT_8000_TCP_PORT=6379
• SRVA_PORT_8443_TCP=tcp://172.17.0.8:6380
• SRVA_PORT_8443_TCP_PROTO=tcp
• SRVA_PORT_8443_TCP_ADDR=172.17.0.8
• SRVA_PORT_8443_TCP_PORT=6380
36
Nerve registration container
• Each app container gets a Nerve instance
• Nerve registers its 1 app
• Nerve instance can be generic
• Make services all have a standard /health
endpoint
• Healthchecks standard
• Only need one nerve container image!
37
Alternate options
• Just register the contents of the docker API into etcd
• http://coreos.com/blog/docker-dynamic-ambassador-
powered-by-etcd/
• No health checks
• Docker only
• confd
• Consul
• frontrunner - discovery from Marathon
• Uses haproxy too
• Less health checking options
38
Vulcand
39
Issues
• If you have lots of machines + services, you have a lot
of Synapses
• haproxy health checks can become expensive on
end user apps
• Nerve helps with this

• Lots of small load balancers is harder to reason about
than a few big ones
40
Live demo?
41
Thanks
• Slides will be online

http://slideshare.net/bobtfish

• Official Smartstack site:

http://nerds.airbnb.com/smartstack-service-discovery-cloud/

• Pre-built containers to play with + blog post

http://engineeringblog.yelp.com/

https://index.docker.io/u/bobtfish/synapse-etcd-amb/

https://index.docker.io/u/bobtfish/nerve-etcd/

• Questions?
42
1 of 42

Recommended

DockerCon Keynote Ben Golub by
DockerCon Keynote Ben GolubDockerCon Keynote Ben Golub
DockerCon Keynote Ben GolubdotCloud
21.4K views43 slides
Docker at Spotify - Dockercon14 by
Docker at Spotify - Dockercon14Docker at Spotify - Dockercon14
Docker at Spotify - Dockercon14dotCloud
56.7K views32 slides
Deploying containers and managing them on multiple Docker hosts, Docker Meetu... by
Deploying containers and managing them on multiple Docker hosts, Docker Meetu...Deploying containers and managing them on multiple Docker hosts, Docker Meetu...
Deploying containers and managing them on multiple Docker hosts, Docker Meetu...dotCloud
17.3K views35 slides
Docker in pratice -chenyifei by
Docker in pratice -chenyifeiDocker in pratice -chenyifei
Docker in pratice -chenyifeidotCloud
19.6K views31 slides
Are VM Passé? by
Are VM Passé? Are VM Passé?
Are VM Passé? dotCloud
7.3K views35 slides
Docker worshop @Twitter - How to use your own private registry by
Docker worshop @Twitter - How to use your own private registryDocker worshop @Twitter - How to use your own private registry
Docker worshop @Twitter - How to use your own private registrydotCloud
7.4K views19 slides

More Related Content

What's hot

Immutable infrastructure with Docker and EC2 by
Immutable infrastructure with Docker and EC2Immutable infrastructure with Docker and EC2
Immutable infrastructure with Docker and EC2dotCloud
57.8K views29 slides
Docker and Containers overview - Docker Workshop by
Docker and Containers overview - Docker WorkshopDocker and Containers overview - Docker Workshop
Docker and Containers overview - Docker WorkshopJonas Rosland
2K views31 slides
Basic docker for developer by
Basic docker for developerBasic docker for developer
Basic docker for developerWeerayut Hongsa
3.6K views261 slides
Docker HK Meetup - 201707 by
Docker HK Meetup - 201707Docker HK Meetup - 201707
Docker HK Meetup - 201707Clarence Ho
995 views109 slides
DockerCon EU 2015: Speed Up Deployment: Building a Distributed Docker Registr... by
DockerCon EU 2015: Speed Up Deployment: Building a Distributed Docker Registr...DockerCon EU 2015: Speed Up Deployment: Building a Distributed Docker Registr...
DockerCon EU 2015: Speed Up Deployment: Building a Distributed Docker Registr...Docker, Inc.
1.5K views12 slides
Docker and stuff by
Docker and stuffDocker and stuff
Docker and stuffVarun Sharma
112 views16 slides

What's hot(20)

Immutable infrastructure with Docker and EC2 by dotCloud
Immutable infrastructure with Docker and EC2Immutable infrastructure with Docker and EC2
Immutable infrastructure with Docker and EC2
dotCloud57.8K views
Docker and Containers overview - Docker Workshop by Jonas Rosland
Docker and Containers overview - Docker WorkshopDocker and Containers overview - Docker Workshop
Docker and Containers overview - Docker Workshop
Jonas Rosland2K views
Docker HK Meetup - 201707 by Clarence Ho
Docker HK Meetup - 201707Docker HK Meetup - 201707
Docker HK Meetup - 201707
Clarence Ho995 views
DockerCon EU 2015: Speed Up Deployment: Building a Distributed Docker Registr... by Docker, Inc.
DockerCon EU 2015: Speed Up Deployment: Building a Distributed Docker Registr...DockerCon EU 2015: Speed Up Deployment: Building a Distributed Docker Registr...
DockerCon EU 2015: Speed Up Deployment: Building a Distributed Docker Registr...
Docker, Inc.1.5K views
Docker Introduction by Hao Fan
Docker IntroductionDocker Introduction
Docker Introduction
Hao Fan281 views
Write Once and REALLY Run Anywhere | OpenStack Summit HK 2013 by dotCloud
Write Once and REALLY Run Anywhere | OpenStack Summit HK 2013Write Once and REALLY Run Anywhere | OpenStack Summit HK 2013
Write Once and REALLY Run Anywhere | OpenStack Summit HK 2013
dotCloud6.7K views
DockerCon SF 2015: Keynote Day 1 by Docker, Inc.
DockerCon SF 2015: Keynote Day 1DockerCon SF 2015: Keynote Day 1
DockerCon SF 2015: Keynote Day 1
Docker, Inc.19K views
Docker Introduction by Peng Xiao
Docker IntroductionDocker Introduction
Docker Introduction
Peng Xiao1.5K views
John Engates Keynote at Dockercon 14 by dotCloud
John Engates Keynote at Dockercon 14John Engates Keynote at Dockercon 14
John Engates Keynote at Dockercon 14
dotCloud16.4K views
DCSF19 CMD and Conquer: Containerizing the Monolith by Docker, Inc.
DCSF19 CMD and Conquer: Containerizing the Monolith  DCSF19 CMD and Conquer: Containerizing the Monolith
DCSF19 CMD and Conquer: Containerizing the Monolith
Docker, Inc.245 views
How we dockerized a startup? #meetup #docker by Jonathan Martin
How we dockerized a startup? #meetup #docker How we dockerized a startup? #meetup #docker
How we dockerized a startup? #meetup #docker
Jonathan Martin6.6K views
OpenStack - Docker - Rackspace HQ by dotCloud
OpenStack - Docker - Rackspace HQOpenStack - Docker - Rackspace HQ
OpenStack - Docker - Rackspace HQ
dotCloud10.3K views
Docker on Google App Engine by Docker, Inc.
Docker on Google App EngineDocker on Google App Engine
Docker on Google App Engine
Docker, Inc.8K views
Microservices using relocatable Docker containers by Mauricio Garavaglia
Microservices using relocatable Docker containersMicroservices using relocatable Docker containers
Microservices using relocatable Docker containers
Mauricio Garavaglia3.4K views
Docker and containers - For Boston Docker Meetup Workshop in March 2015 by Jonas Rosland
Docker and containers - For Boston Docker Meetup Workshop in March 2015Docker and containers - For Boston Docker Meetup Workshop in March 2015
Docker and containers - For Boston Docker Meetup Workshop in March 2015
Jonas Rosland1.6K views
Docker Container As A Service - March 2016 by Patrick Chanezon
Docker Container As A Service - March 2016Docker Container As A Service - March 2016
Docker Container As A Service - March 2016
Patrick Chanezon3.5K views

Viewers also liked

Docker introduction by
Docker introductionDocker introduction
Docker introductiondotCloud
455.8K views29 slides
Introduction to dockerfile, SF Peninsula Software Development Meetup @Guidewire by
Introduction to dockerfile, SF Peninsula Software Development Meetup @Guidewire Introduction to dockerfile, SF Peninsula Software Development Meetup @Guidewire
Introduction to dockerfile, SF Peninsula Software Development Meetup @Guidewire dotCloud
2.8K views15 slides
Wot2013云计算架构师峰会 -陈轶飞2 by
Wot2013云计算架构师峰会 -陈轶飞2Wot2013云计算架构师峰会 -陈轶飞2
Wot2013云计算架构师峰会 -陈轶飞2dotCloud
6.6K views33 slides
Dockerizing your applications - Docker workshop @Twitter by
Dockerizing your applications - Docker workshop @TwitterDockerizing your applications - Docker workshop @Twitter
Dockerizing your applications - Docker workshop @TwitterdotCloud
10K views12 slides
Introduction to Docker - Docker workshop @Twitter by
Introduction to Docker - Docker workshop @TwitterIntroduction to Docker - Docker workshop @Twitter
Introduction to Docker - Docker workshop @TwitterdotCloud
3K views8 slides
MySQL At Yelp by
MySQL At YelpMySQL At Yelp
MySQL At YelpYelp Engineering
3.6K views30 slides

Viewers also liked(13)

Docker introduction by dotCloud
Docker introductionDocker introduction
Docker introduction
dotCloud455.8K views
Introduction to dockerfile, SF Peninsula Software Development Meetup @Guidewire by dotCloud
Introduction to dockerfile, SF Peninsula Software Development Meetup @Guidewire Introduction to dockerfile, SF Peninsula Software Development Meetup @Guidewire
Introduction to dockerfile, SF Peninsula Software Development Meetup @Guidewire
dotCloud2.8K views
Wot2013云计算架构师峰会 -陈轶飞2 by dotCloud
Wot2013云计算架构师峰会 -陈轶飞2Wot2013云计算架构师峰会 -陈轶飞2
Wot2013云计算架构师峰会 -陈轶飞2
dotCloud6.6K views
Dockerizing your applications - Docker workshop @Twitter by dotCloud
Dockerizing your applications - Docker workshop @TwitterDockerizing your applications - Docker workshop @Twitter
Dockerizing your applications - Docker workshop @Twitter
dotCloud10K views
Introduction to Docker - Docker workshop @Twitter by dotCloud
Introduction to Docker - Docker workshop @TwitterIntroduction to Docker - Docker workshop @Twitter
Introduction to Docker - Docker workshop @Twitter
dotCloud3K views
Scaling Traffic from 0 to 139 Million Unique Visitors by Yelp Engineering
Scaling Traffic from 0 to 139 Million Unique VisitorsScaling Traffic from 0 to 139 Million Unique Visitors
Scaling Traffic from 0 to 139 Million Unique Visitors
Yelp Engineering3K views
"Using ElasticSearch to Scale Near Real-Time Search" by John Billings (Presen... by Yelp Engineering
"Using ElasticSearch to Scale Near Real-Time Search" by John Billings (Presen..."Using ElasticSearch to Scale Near Real-Time Search" by John Billings (Presen...
"Using ElasticSearch to Scale Near Real-Time Search" by John Billings (Presen...
Yelp Engineering17.9K views
VMCS14 Pool Your REsources: Building Cross Sector Collaborations for High Impact by VolunteerMatch
VMCS14 Pool Your REsources: Building Cross Sector Collaborations for High ImpactVMCS14 Pool Your REsources: Building Cross Sector Collaborations for High Impact
VMCS14 Pool Your REsources: Building Cross Sector Collaborations for High Impact
VolunteerMatch1.7K views
Portfolio-Hany Tadross by Hany Tadross
Portfolio-Hany TadrossPortfolio-Hany Tadross
Portfolio-Hany Tadross
Hany Tadross199 views
Energy Management in Water Supply Systems - Pradeep Kumar , Alliance to Save ... by www.theurbanvision.com
Energy Management in Water Supply Systems - Pradeep Kumar , Alliance to Save ...Energy Management in Water Supply Systems - Pradeep Kumar , Alliance to Save ...
Energy Management in Water Supply Systems - Pradeep Kumar , Alliance to Save ...
ICAWC 2014 - Workshop - Shelter Building, Design For Daily Routines - Matt Ta... by Dogs Trust
ICAWC 2014 - Workshop - Shelter Building, Design For Daily Routines - Matt Ta...ICAWC 2014 - Workshop - Shelter Building, Design For Daily Routines - Matt Ta...
ICAWC 2014 - Workshop - Shelter Building, Design For Daily Routines - Matt Ta...
Dogs Trust860 views

Similar to Building a smarter application Stack by Tomas Doran from Yelp

Microservices pros and cons dark by
Microservices pros and cons darkMicroservices pros and cons dark
Microservices pros and cons darkAndrew Siemer
985 views20 slides
Microservices pros and cons - houston tech fest by
Microservices pros and cons - houston tech festMicroservices pros and cons - houston tech fest
Microservices pros and cons - houston tech festAndrew Siemer
273 views22 slides
Jelastic - Containers Live Migration Behind the Scene by
Jelastic - Containers Live Migration Behind the SceneJelastic - Containers Live Migration Behind the Scene
Jelastic - Containers Live Migration Behind the SceneJelastic Multi-Cloud PaaS
2.1K views18 slides
The impact of cloud NSBCon NY by Yves Goeleven by
The impact of cloud NSBCon NY by Yves GoelevenThe impact of cloud NSBCon NY by Yves Goeleven
The impact of cloud NSBCon NY by Yves GoelevenParticular Software
439 views47 slides
Allyourbase by
AllyourbaseAllyourbase
AllyourbaseAlex Scotti
3.1K views43 slides
PromCon EU 2022 - Centralized vs Decentralized Prometheus Scraping Architectu... by
PromCon EU 2022 - Centralized vs Decentralized Prometheus Scraping Architectu...PromCon EU 2022 - Centralized vs Decentralized Prometheus Scraping Architectu...
PromCon EU 2022 - Centralized vs Decentralized Prometheus Scraping Architectu...Eric D. Schabell
62 views17 slides

Similar to Building a smarter application Stack by Tomas Doran from Yelp(20)

Microservices pros and cons dark by Andrew Siemer
Microservices pros and cons darkMicroservices pros and cons dark
Microservices pros and cons dark
Andrew Siemer985 views
Microservices pros and cons - houston tech fest by Andrew Siemer
Microservices pros and cons - houston tech festMicroservices pros and cons - houston tech fest
Microservices pros and cons - houston tech fest
Andrew Siemer273 views
PromCon EU 2022 - Centralized vs Decentralized Prometheus Scraping Architectu... by Eric D. Schabell
PromCon EU 2022 - Centralized vs Decentralized Prometheus Scraping Architectu...PromCon EU 2022 - Centralized vs Decentralized Prometheus Scraping Architectu...
PromCon EU 2022 - Centralized vs Decentralized Prometheus Scraping Architectu...
Eric D. Schabell62 views
The Need of Cloud-Native Application by Emiliano Pecis
The Need of Cloud-Native ApplicationThe Need of Cloud-Native Application
The Need of Cloud-Native Application
Emiliano Pecis2.4K views
John adams talk cloudy by John Adams
John adams   talk cloudyJohn adams   talk cloudy
John adams talk cloudy
John Adams3.3K views
Platform Clouds, Containers, Immutable Infrastructure Oh My! by Stuart Charlton
Platform Clouds, Containers, Immutable Infrastructure Oh My!Platform Clouds, Containers, Immutable Infrastructure Oh My!
Platform Clouds, Containers, Immutable Infrastructure Oh My!
Stuart Charlton919 views
Latest (storage IO) patterns for cloud-native applications by OpenEBS
Latest (storage IO) patterns for cloud-native applications Latest (storage IO) patterns for cloud-native applications
Latest (storage IO) patterns for cloud-native applications
OpenEBS166 views
Rami Sayar - Node microservices with Docker by Web à Québec
Rami Sayar - Node microservices with DockerRami Sayar - Node microservices with Docker
Rami Sayar - Node microservices with Docker
Web à Québec810 views
Scaling a MeteorJS SaaS app on AWS by Brett McLain
Scaling a MeteorJS SaaS app on AWSScaling a MeteorJS SaaS app on AWS
Scaling a MeteorJS SaaS app on AWS
Brett McLain1.8K views
LASUG Online: Introduction to Docker and Docker Tools by Vasiliy Fomichev
LASUG Online: Introduction to Docker and Docker ToolsLASUG Online: Introduction to Docker and Docker Tools
LASUG Online: Introduction to Docker and Docker Tools
Vasiliy Fomichev991 views
To Build My Own Cloud with Blackjack… by Sergey Dzyuban
To Build My Own Cloud with Blackjack…To Build My Own Cloud with Blackjack…
To Build My Own Cloud with Blackjack…
Sergey Dzyuban110 views
Digital Transformation with Kubernetes, Containers, and Microservices by Lightbend
Digital Transformation with Kubernetes, Containers, and MicroservicesDigital Transformation with Kubernetes, Containers, and Microservices
Digital Transformation with Kubernetes, Containers, and Microservices
Lightbend1.3K views
FreeSWITCH as a Microservice by Evan McGee
FreeSWITCH as a MicroserviceFreeSWITCH as a Microservice
FreeSWITCH as a Microservice
Evan McGee3.4K views
Migrating Hundreds of Legacy Applications to Kubernetes - The Good, the Bad, ... by QAware GmbH
Migrating Hundreds of Legacy Applications to Kubernetes - The Good, the Bad, ...Migrating Hundreds of Legacy Applications to Kubernetes - The Good, the Bad, ...
Migrating Hundreds of Legacy Applications to Kubernetes - The Good, the Bad, ...
QAware GmbH2.3K views
The Good, the Bad and the Ugly of Migrating Hundreds of Legacy Applications ... by Josef Adersberger
 The Good, the Bad and the Ugly of Migrating Hundreds of Legacy Applications ... The Good, the Bad and the Ugly of Migrating Hundreds of Legacy Applications ...
The Good, the Bad and the Ugly of Migrating Hundreds of Legacy Applications ...
Josef Adersberger672 views

More from dotCloud

Introduction to Docker and all things containers, Docker Meetup at RelateIQ by
Introduction to Docker and all things containers, Docker Meetup at RelateIQIntroduction to Docker and all things containers, Docker Meetup at RelateIQ
Introduction to Docker and all things containers, Docker Meetup at RelateIQdotCloud
2.2K views52 slides
Introduction to Docker at SF Peninsula Software Development Meetup @Guidewire by
Introduction to Docker at SF Peninsula Software Development Meetup @GuidewireIntroduction to Docker at SF Peninsula Software Development Meetup @Guidewire
Introduction to Docker at SF Peninsula Software Development Meetup @GuidewiredotCloud
3.1K views44 slides
Dockerizing stashboard - Docker meetup at Twilio by
Dockerizing stashboard - Docker meetup at TwilioDockerizing stashboard - Docker meetup at Twilio
Dockerizing stashboard - Docker meetup at TwiliodotCloud
7.8K views9 slides
Docker links | Docker workshop #2 at Twitter by
Docker links | Docker workshop #2 at TwitterDocker links | Docker workshop #2 at Twitter
Docker links | Docker workshop #2 at TwitterdotCloud
2.6K views7 slides
Dockerfile Basics | Docker workshop #2 at twitter, 2013-11-05 by
Dockerfile Basics | Docker workshop #2 at twitter, 2013-11-05Dockerfile Basics | Docker workshop #2 at twitter, 2013-11-05
Dockerfile Basics | Docker workshop #2 at twitter, 2013-11-05dotCloud
3.3K views15 slides
Intro Docker october 2013 by
Intro Docker october 2013Intro Docker october 2013
Intro Docker october 2013dotCloud
6.7K views27 slides

More from dotCloud(15)

Introduction to Docker and all things containers, Docker Meetup at RelateIQ by dotCloud
Introduction to Docker and all things containers, Docker Meetup at RelateIQIntroduction to Docker and all things containers, Docker Meetup at RelateIQ
Introduction to Docker and all things containers, Docker Meetup at RelateIQ
dotCloud2.2K views
Introduction to Docker at SF Peninsula Software Development Meetup @Guidewire by dotCloud
Introduction to Docker at SF Peninsula Software Development Meetup @GuidewireIntroduction to Docker at SF Peninsula Software Development Meetup @Guidewire
Introduction to Docker at SF Peninsula Software Development Meetup @Guidewire
dotCloud3.1K views
Dockerizing stashboard - Docker meetup at Twilio by dotCloud
Dockerizing stashboard - Docker meetup at TwilioDockerizing stashboard - Docker meetup at Twilio
Dockerizing stashboard - Docker meetup at Twilio
dotCloud7.8K views
Docker links | Docker workshop #2 at Twitter by dotCloud
Docker links | Docker workshop #2 at TwitterDocker links | Docker workshop #2 at Twitter
Docker links | Docker workshop #2 at Twitter
dotCloud2.6K views
Dockerfile Basics | Docker workshop #2 at twitter, 2013-11-05 by dotCloud
Dockerfile Basics | Docker workshop #2 at twitter, 2013-11-05Dockerfile Basics | Docker workshop #2 at twitter, 2013-11-05
Dockerfile Basics | Docker workshop #2 at twitter, 2013-11-05
dotCloud3.3K views
Intro Docker october 2013 by dotCloud
Intro Docker october 2013Intro Docker october 2013
Intro Docker october 2013
dotCloud6.7K views
[Open stack] heat + docker by dotCloud
[Open stack] heat + docker[Open stack] heat + docker
[Open stack] heat + docker
dotCloud3.8K views
Dockerizing WordPress by dotCloud
Dockerizing WordPressDockerizing WordPress
Dockerizing WordPress
dotCloud18.2K views
Building images from dockerfiles by dotCloud
Building images from dockerfilesBuilding images from dockerfiles
Building images from dockerfiles
dotCloud9.7K views
Docker at DevTable by dotCloud
Docker at DevTableDocker at DevTable
Docker at DevTable
dotCloud2.7K views
Docker presentation | Paris Docker Meetup by dotCloud
Docker presentation | Paris Docker MeetupDocker presentation | Paris Docker Meetup
Docker presentation | Paris Docker Meetup
dotCloud11.3K views
Lightweight Virtualization with Linux Containers and Docker | YaC 2013 by dotCloud
Lightweight Virtualization with Linux Containers and Docker | YaC 2013Lightweight Virtualization with Linux Containers and Docker | YaC 2013
Lightweight Virtualization with Linux Containers and Docker | YaC 2013
dotCloud14.2K views
Installing and running Postfix within a docker container from the command line by dotCloud
Installing and running Postfix within a docker container from the command lineInstalling and running Postfix within a docker container from the command line
Installing and running Postfix within a docker container from the command line
dotCloud34.9K views
LXC, Docker, and the future of software delivery | LinuxCon 2013 by dotCloud
LXC, Docker, and the future of software delivery | LinuxCon 2013LXC, Docker, and the future of software delivery | LinuxCon 2013
LXC, Docker, and the future of software delivery | LinuxCon 2013
dotCloud12.8K views
Docker Presentation at the OpenStack Austin Meetup | 2013-09-12 by dotCloud
Docker Presentation at the OpenStack Austin Meetup | 2013-09-12Docker Presentation at the OpenStack Austin Meetup | 2013-09-12
Docker Presentation at the OpenStack Austin Meetup | 2013-09-12
dotCloud4.9K views

Recently uploaded

Serverless computing with Google Cloud (2023-24) by
Serverless computing with Google Cloud (2023-24)Serverless computing with Google Cloud (2023-24)
Serverless computing with Google Cloud (2023-24)wesley chun
11 views33 slides
【USB韌體設計課程】精選講義節錄-USB的列舉過程_艾鍗學院 by
【USB韌體設計課程】精選講義節錄-USB的列舉過程_艾鍗學院【USB韌體設計課程】精選講義節錄-USB的列舉過程_艾鍗學院
【USB韌體設計課程】精選講義節錄-USB的列舉過程_艾鍗學院IttrainingIttraining
58 views8 slides
MVP and prioritization.pdf by
MVP and prioritization.pdfMVP and prioritization.pdf
MVP and prioritization.pdfrahuldharwal141
31 views8 slides
Unit 1_Lecture 2_Physical Design of IoT.pdf by
Unit 1_Lecture 2_Physical Design of IoT.pdfUnit 1_Lecture 2_Physical Design of IoT.pdf
Unit 1_Lecture 2_Physical Design of IoT.pdfStephenTec
12 views36 slides
Five Things You SHOULD Know About Postman by
Five Things You SHOULD Know About PostmanFive Things You SHOULD Know About Postman
Five Things You SHOULD Know About PostmanPostman
36 views43 slides
Microsoft Power Platform.pptx by
Microsoft Power Platform.pptxMicrosoft Power Platform.pptx
Microsoft Power Platform.pptxUni Systems S.M.S.A.
53 views38 slides

Recently uploaded(20)

Serverless computing with Google Cloud (2023-24) by wesley chun
Serverless computing with Google Cloud (2023-24)Serverless computing with Google Cloud (2023-24)
Serverless computing with Google Cloud (2023-24)
wesley chun11 views
【USB韌體設計課程】精選講義節錄-USB的列舉過程_艾鍗學院 by IttrainingIttraining
【USB韌體設計課程】精選講義節錄-USB的列舉過程_艾鍗學院【USB韌體設計課程】精選講義節錄-USB的列舉過程_艾鍗學院
【USB韌體設計課程】精選講義節錄-USB的列舉過程_艾鍗學院
Unit 1_Lecture 2_Physical Design of IoT.pdf by StephenTec
Unit 1_Lecture 2_Physical Design of IoT.pdfUnit 1_Lecture 2_Physical Design of IoT.pdf
Unit 1_Lecture 2_Physical Design of IoT.pdf
StephenTec12 views
Five Things You SHOULD Know About Postman by Postman
Five Things You SHOULD Know About PostmanFive Things You SHOULD Know About Postman
Five Things You SHOULD Know About Postman
Postman36 views
Special_edition_innovator_2023.pdf by WillDavies22
Special_edition_innovator_2023.pdfSpecial_edition_innovator_2023.pdf
Special_edition_innovator_2023.pdf
WillDavies2218 views
PharoJS - Zürich Smalltalk Group Meetup November 2023 by Noury Bouraqadi
PharoJS - Zürich Smalltalk Group Meetup November 2023PharoJS - Zürich Smalltalk Group Meetup November 2023
PharoJS - Zürich Smalltalk Group Meetup November 2023
Noury Bouraqadi132 views
Voice Logger - Telephony Integration Solution at Aegis by Nirmal Sharma
Voice Logger - Telephony Integration Solution at AegisVoice Logger - Telephony Integration Solution at Aegis
Voice Logger - Telephony Integration Solution at Aegis
Nirmal Sharma39 views
Igniting Next Level Productivity with AI-Infused Data Integration Workflows by Safe Software
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Safe Software280 views
Business Analyst Series 2023 - Week 3 Session 5 by DianaGray10
Business Analyst Series 2023 -  Week 3 Session 5Business Analyst Series 2023 -  Week 3 Session 5
Business Analyst Series 2023 - Week 3 Session 5
DianaGray10300 views
Automating a World-Class Technology Conference; Behind the Scenes of CiscoLive by Network Automation Forum
Automating a World-Class Technology Conference; Behind the Scenes of CiscoLiveAutomating a World-Class Technology Conference; Behind the Scenes of CiscoLive
Automating a World-Class Technology Conference; Behind the Scenes of CiscoLive

Building a smarter application Stack by Tomas Doran from Yelp

  • 1. Building a smarter application stack Tomas  Doran   @bobtfish   2014-­‐06-­‐10
  • 2. Docker is the future • Preaching to the converted here ;)
 • Game changing technology
 • No silver bullets (ever) • Introduces it’s own set of problems and issues • Dependency discovery / wiring • Scheduling 2
  • 3. Smartstack • One possible solution to discovery problems
 • This talk: • Application architecture • Problem(s) we’re solving • Why this solution works well for us • Alternate solutions 3
  • 4. Microservices - also the future! • The same as SOA • But one API per service. • Own data store!
 • Lots of services (dozens, maybe 100s)
 • All individually very simple • Easy to reason about. • Easy to replace 4
  • 5. Don’t break the site - ever!!! • Microservices are individually deployable! • When we say “Don’t break the site” • We mean • Don’t break all of the site! 5
  • 6. Don’t break the site - ever!!! • If you have graceful degradation… • You can ignore MTBF in the backend services! • You only care about MTTR. 6
  • 7. “I’ll just break this out into it’s own application, as it’ll be easier to maintain in 10 years time” - Pre seed funding nobody, ever! 7
  • 8. Monolith - the reality • Everyone has one of these :) • If you’re far enough down the path, you call this ‘The presentation layer’.
 • Still poses a challenge • need async requests • need graceful degradation 8
  • 9. Monolith - the reality • Most popular service
 • Most dependencies • Call into 10s or 100s of other services in a request!
 • Needs circuit breakers + dynamic configuration 9
  • 10. No silver bullet = No one solution • You should always have 2. • Nagios / Sensu • RRDs + Ganglia / Graphite + Diamond • YAML files / Zookeeper 10
  • 11. No silver bullet = No one solution • ‘Top down’ architecture sucks. • Instead, broad goals + ’Bottom up’ architecture • Internal competition! • Replacing the incumbent solution happens organically • If your thing works better, people will want to move! 
 • Not perfect! Better than top-down! 11
  • 12. “Humans are bad at predicting the performance of complex systems […]. Our ability to create large and complex systems fools us into believing that we’re also entitled to understand them”
 - Carlos Bueno “Mature optimization handbook”
 12
  • 13. Distributed complexity • Distributed systems introduce their own set of complexity • Reasoning about the whole system challenging • Timing/profiling/performance analysis non-trivial • Resource scheduling also non-trivial • 2nd order effects ! • Can’t reason about emergent behavior 13
  • 14. 14
  • 15. 15 ?
  • 16. 16 ? What the heck happened at 16:46?
  • 17. 17 ? And why did it stop at 17:00?
  • 18. Dynamic architecture • Cattle not pets • AWS and VMs in ‘the cloud’ started this • Docker takes it a step further
 • Explicitly manage persistent state • Explicit regular recycling • All updates are redeploys
 18
  • 19. Dependency nightmares • Almost everything has some dependencies • Simple example, web app with a mysql DB • App config in a YAML file
 • Mysql container address changes when you restart mysql! • Oops, app can’t find mysql!
 • Do I need to restart every application using mysql? • Sucks! • Do I need to rebuild application containers using mysql? • To edit the config YAML file! • Super super sucks! 19
  • 20. Runtime wiring • mysql failovers - the simple case!
 • Presentation layer talking to service REST layers • Different deployment schedules • No downtime
 • Only possible solution: wiring dependencies at runtime • A challenge • Also an opportunity
 • DNS is workable in some cases 20
  • 21. Dynamic discovery • Discovery becomes a core problem
 • DNS re-resolving not generally trustworthy • You need to test everything for this
 • DNS balancing (internally) is awful • Failed node + multiple connections/requests • DNS round robin • Everything sees failure • Slow to shift traffic • Round robin is crappy for load balancing
 21
  • 22. Externalized wiring • Remove a lot of complexity from the application domain
 • Run a load balancer (haproxy) on each machine
 • Applications always connect to load balancer on fixed host/port • localhost on traditional metal/VMs • supplied by —link or environment variables in Docker
 • Applications become wiring agnostic! 22
  • 24. ‘Client side load balancing’ • Lots of projects use this approach: • Project Atomic • Marathon + Mesos-Docker • vulcand (https://github.com/mailgun/vulcand) • Frontrunner (https://github.com/Wizcorp/ frontrunner) • Consul ! • Smartstack 24
  • 25. Legacy infrastructure • Physical machines • Application images in AMIs • kvm
 • Can’t just use container links or a Docker only solution
 • Want to use the same (uniform) solution everywhere. 25
  • 26. Entropy reduction • You can’t change everything at once!
 • Everything will tend towards chaos • ‘Old infrastructure’ • ‘New infrastructure’ • ‘New new infrastructure’ ! • Solution specifically chosen so it could be generic. 26
  • 27. SmartStack • 2 parts • Synapse • Nerve ! • Conceptually simple • Very flexible • Easy to hack on • Plays well on traditional machines • Plays well in docker 27
  • 28. Synapse • Does discovery against a pluggable backend • Writes out a haproxy configuration
 • Assign a well known port to all services • Application connects to that port • haproxy forwards to an available backend
 • Your application doesn’t need to know about discovery!
 • Technology agnostic - works the same on metal/VMs/Docker 28
  • 29. 29
  • 30. Why synapse? • haproxy is a well known solution • ruby - easy to modify • simple (has one job) • Pluggable - discovery with multiple methods: • JSON config (static) • zookeeper • etcd • docker API • ec2 tags • Flexible • Deploy one per instance • Or pairs as dedicated lbs 30
  • 31. Nerve • Health checks services • Health checks are pluggable. • HTTP (flexible) + mysql come out the box • Registers service information to backend • zookeeper • etcd (beta) 31
  • 32. 32
  • 33. Connector agnostic containers • On ‘real servers’ or VMs, running a synapse instance per box is fine.
 • In docker, we want to abstract more than that • Make containers connector agnostic! • They don’t need to know or care • Upgrade independently. 33
  • 34. Synapse <3 ambassador containers • ‘Ambassador pattern’ • Run a synapse ‘ambassador’ container on each host for each service • Link each application to the ambassador for each of it’s dependencies • Environment variables to each service’s haproxy • Separates synapse management (i.e. changing the wiring) from application management (i.e. upgrading the app version). 34
  • 35. 35
  • 36. Container links • Ambassador for service A presents: • port 8000 for HTTP REST service • port 8443 for HTTPS REST service ! • Container linking to ambassador sees: • SRVA_PORT_8000_TCP=tcp://172.17.0.8:6379 • SRVA_PORT_8000_TCP_PROTO=tcp • SRVA_PORT_8000_TCP_ADDR=172.17.0.8 • SRVA_PORT_8000_TCP_PORT=6379 • SRVA_PORT_8443_TCP=tcp://172.17.0.8:6380 • SRVA_PORT_8443_TCP_PROTO=tcp • SRVA_PORT_8443_TCP_ADDR=172.17.0.8 • SRVA_PORT_8443_TCP_PORT=6380 36
  • 37. Nerve registration container • Each app container gets a Nerve instance • Nerve registers its 1 app • Nerve instance can be generic • Make services all have a standard /health endpoint • Healthchecks standard • Only need one nerve container image! 37
  • 38. Alternate options • Just register the contents of the docker API into etcd • http://coreos.com/blog/docker-dynamic-ambassador- powered-by-etcd/ • No health checks • Docker only • confd • Consul • frontrunner - discovery from Marathon • Uses haproxy too • Less health checking options 38
  • 40. Issues • If you have lots of machines + services, you have a lot of Synapses • haproxy health checks can become expensive on end user apps • Nerve helps with this
 • Lots of small load balancers is harder to reason about than a few big ones 40
  • 42. Thanks • Slides will be online
 http://slideshare.net/bobtfish
 • Official Smartstack site:
 http://nerds.airbnb.com/smartstack-service-discovery-cloud/
 • Pre-built containers to play with + blog post
 http://engineeringblog.yelp.com/
 https://index.docker.io/u/bobtfish/synapse-etcd-amb/
 https://index.docker.io/u/bobtfish/nerve-etcd/
 • Questions? 42