Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Eric Holmes & Michael Barrett, Remind
October 20...
What to Expect from the Session
• A brief introduction about why we decided to build an
internal platform at Remind, and t...
About Us
• Eric Holmes & Michael Barrett
• Infrastructure engineers at Remind
• We build things for developers
• You can f...
Remind
• A messaging platform for teachers.
• Chat/announcements/files
• Over 30 million users
• Used actively in ~50% of ...
A Little History
A Little History
• Started as a “monorail”
• Scaling challenges during BTS
• Migrated to an SOA/micro-service architecture
Heroku was great, but…
• Every app on Heroku is publicly accessible
• Databases need to be exposed to Internet traffic
• L...
What we want from a PaaS
• AWS
• Flexibility
• Shared patterns for deployment
• Easy service operation
• Containers/Docker
Why Containers?
• Fast build + deploy iteration cycles
• Isolate dependencies
• Better dev/prod parity
• Immutable images
...
Building an Empire
Design Goals
• Easy to operate
• Open source
• Support 12-factor stateless apps (12factor.net)
• Swappable scheduling back...
Components of a PaaS
Scheduler Router Control Plane
Scheduler :: Cluster Management
Join Leave
Scheduler :: Task Placement
Find Host Run Job
CPU/
Memory
Container
Cluster
Scheduler :: Task Placement
type App []Process
type Scheduler interface {
Run(App)
Remove(App)
Scale(Process)
Tasks(App) [...
Empire :: V1
Scheduler Router Control Plane
+ Fleet
etcd + registrator + confd
Heroku Platform API
Spec + hk CLI
Amazon EC2 Container Service
• Managed cluster manager and scheduler
• Supports Docker
• Built-in service scheduler
• Inte...
Amazon EC2 Container Service :: Resources
• Clusters
• Task definitions
• Tasks
• Services
Scheduler Interface
Run(App)
Remove(App)
Scale(Process)
Tasks(App) []Task
Stop(Task)
Amazon ECS Scheduler Implementation
A...
Empire :: V2
Scheduler Router Control Plane
ECS ELB
Heroku Platform API
Spec + emp CLI
Empire :: V2
An open-source, self-hosted PaaS for running
twelve-factor Docker apps backed by AWS
services
Twelve-Factor
Twelve-Factor Tenants
I. Codebase
II. Dependencies
III. Config
IV. Backing Services
V. Build, release, run
VI. Processes
V...
12factor :: Dependencies
“Explicitly declare and isolate dependencies”
FROM ruby
RUN apt-get install imagemagick
RUN bundl...
12factor :: Build, release, run
“Strictly separate build and run stages”
Empire
12factor :: Build
$ git push
12factor :: Release, Run
Config
{}
Release
Amazon ECS
12factor :: Release, Run
$ cat Procfile
web: ./bin/web
worker: ./bin/worker
$ aws ecs list-services
arn:aws:ecs:us-east-1:...
Service Discovery
$ aws ecs describe-services --service api--web
"loadBalancers": [{
"containerName": "web”,
"containerPor...
12factor :: Concurrency
“Scale out via the process model”
$ emp scale web=10
$ aws ecs describe-service --service api--web...
12factor :: Dev/prod parity
“Keep development, staging, and production as similar as
possible”
$ docker run --env-file <(e...
12factor :: Logs
“Treat logs as event streams”
$ emp log
“GET / HTTP/1.1” 200
STDOUT Amazon Kinesis
12factor :: Admin processes
“Run admin/management tasks as one-off processes”
$ emp run rake db:migrate
Migrated
Demo
Pain Points & Lessons Learned
Container Instance Rollout
1. Update AMI in AWS CloudFormation stack.
2. Kill 1 host
3. Wait for new Amazon ECS services t...
Logging
Logspout
Logging
SOURCE=<app>.<process>.<version>
SYSLOG_STRUCTURED_DATA=app={{ .Container.Config.Env "SOURCE" }}
Docker Monolith
= Bad Times
Docker Performance
• Early versions of Docker had abysmal push/pull
performance
• Use Docker >= 1.8.1
• Make your Dockerfi...
This space moves fast!
• Containers have been around, but Docker made them
accessible
• New tools coming out ever day
• AW...
Remember to complete
your evaluations!
Thank you!
Upcoming SlideShare
Loading in …5
×

(DVO308) Docker & ECS in Production: How We Migrated Our Infrastructure from Heroku to AWS

4,660 views

Published on

This session will introduce you to Empire, a new self-hosted PaaS built on top of Amazon’s EC2 Container Service (ECS). Empire is a recently open-sourced project that provides a mostly Heroku-compatible API. It allows engineering teams to deploy and manage applications in a method similar to Heroku, but with the added flexibility and control of running your own ECS container instances. We'll talk about why Remind decided to move its infrastructure from Heroku to AWS, introduce you to ECS and the open source platform we built on top of it to make migration easier, and then we'll demo Empire to show you how you can try it today.

Published in: Technology
  • Be the first to comment

(DVO308) Docker & ECS in Production: How We Migrated Our Infrastructure from Heroku to AWS

  1. 1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Eric Holmes & Michael Barrett, Remind October 2015 DVO308 Docker & ECS in Production How We Migrated Our Infrastructure from Heroku to AWS
  2. 2. What to Expect from the Session • A brief introduction about why we decided to build an internal platform at Remind, and the lessons we learned along the way • An introduction to the open source PaaS we built called Empire, and how we’re leveraging Amazon ECS • Demo • Q&A
  3. 3. About Us • Eric Holmes & Michael Barrett • Infrastructure engineers at Remind • We build things for developers • You can find our open source stuff at: • https://github.com/ejholmes • https://github.com/phobologic
  4. 4. Remind • A messaging platform for teachers. • Chat/announcements/files • Over 30 million users • Used actively in ~50% of U.S. public schools • Over 2 billion messages delivered • ~50 employees. ~30 engineers.
  5. 5. A Little History
  6. 6. A Little History • Started as a “monorail” • Scaling challenges during BTS • Migrated to an SOA/micro-service architecture
  7. 7. Heroku was great, but… • Every app on Heroku is publicly accessible • Databases need to be exposed to Internet traffic • Limited visibility and control
  8. 8. What we want from a PaaS • AWS • Flexibility • Shared patterns for deployment • Easy service operation • Containers/Docker
  9. 9. Why Containers? • Fast build + deploy iteration cycles • Isolate dependencies • Better dev/prod parity • Immutable images • Better resource utilization
  10. 10. Building an Empire
  11. 11. Design Goals • Easy to operate • Open source • Support 12-factor stateless apps (12factor.net) • Swappable scheduling back-ends • Stability! • Docker images as a unit of deployment
  12. 12. Components of a PaaS Scheduler Router Control Plane
  13. 13. Scheduler :: Cluster Management Join Leave
  14. 14. Scheduler :: Task Placement Find Host Run Job CPU/ Memory Container Cluster
  15. 15. Scheduler :: Task Placement type App []Process type Scheduler interface { Run(App) Remove(App) Scale(Process) Tasks(App) []Task Stop(Task) }
  16. 16. Empire :: V1 Scheduler Router Control Plane + Fleet etcd + registrator + confd Heroku Platform API Spec + hk CLI
  17. 17. Amazon EC2 Container Service • Managed cluster manager and scheduler • Supports Docker • Built-in service scheduler • Integrates with Elastic Load Balancing
  18. 18. Amazon EC2 Container Service :: Resources • Clusters • Task definitions • Tasks • Services
  19. 19. Scheduler Interface Run(App) Remove(App) Scale(Process) Tasks(App) []Task Stop(Task) Amazon ECS Scheduler Implementation Amazon ECS API RegisterTaskDefinition -> CreateService/UpdateService DeleteService UpdateService ListTasks StopTask
  20. 20. Empire :: V2 Scheduler Router Control Plane ECS ELB Heroku Platform API Spec + emp CLI
  21. 21. Empire :: V2 An open-source, self-hosted PaaS for running twelve-factor Docker apps backed by AWS services
  22. 22. Twelve-Factor
  23. 23. Twelve-Factor Tenants I. Codebase II. Dependencies III. Config IV. Backing Services V. Build, release, run VI. Processes VII. Port binding VIII.Concurrency IX. Disposability X. Dev/prod parity XI. Logs XII. Admin processes
  24. 24. 12factor :: Dependencies “Explicitly declare and isolate dependencies” FROM ruby RUN apt-get install imagemagick RUN bundle install
  25. 25. 12factor :: Build, release, run “Strictly separate build and run stages” Empire
  26. 26. 12factor :: Build $ git push
  27. 27. 12factor :: Release, Run Config {} Release Amazon ECS
  28. 28. 12factor :: Release, Run $ cat Procfile web: ./bin/web worker: ./bin/worker $ aws ecs list-services arn:aws:ecs:us-east-1:***:service/api--web arn:aws:ecs:us-east-1:***:service/api--worker $ emp deploy org/api:latest Status: Created v1 release.
  29. 29. Service Discovery $ aws ecs describe-services --service api--web "loadBalancers": [{ "containerName": "web”, "containerPort": 9001, "loadBalancerName”: "2888...a31d4c” }] $ curl http://api Ok
  30. 30. 12factor :: Concurrency “Scale out via the process model” $ emp scale web=10 $ aws ecs describe-service --service api--web “desired-count”: 10
  31. 31. 12factor :: Dev/prod parity “Keep development, staging, and production as similar as possible” $ docker run --env-file <(emp env -a api) org/api
  32. 32. 12factor :: Logs “Treat logs as event streams” $ emp log “GET / HTTP/1.1” 200 STDOUT Amazon Kinesis
  33. 33. 12factor :: Admin processes “Run admin/management tasks as one-off processes” $ emp run rake db:migrate Migrated
  34. 34. Demo
  35. 35. Pain Points & Lessons Learned
  36. 36. Container Instance Rollout 1. Update AMI in AWS CloudFormation stack. 2. Kill 1 host 3. Wait for new Amazon ECS services to start running on new host 4. Rinse and repeat
  37. 37. Logging Logspout
  38. 38. Logging SOURCE=<app>.<process>.<version> SYSLOG_STRUCTURED_DATA=app={{ .Container.Config.Env "SOURCE" }}
  39. 39. Docker Monolith = Bad Times
  40. 40. Docker Performance • Early versions of Docker had abysmal push/pull performance • Use Docker >= 1.8.1 • Make your Dockerfile’s use the layer cache efficiently • https://github.com/remind101/conveyor
  41. 41. This space moves fast! • Containers have been around, but Docker made them accessible • New tools coming out ever day • AWS’s offerings have been incredibly stable and feature rich
  42. 42. Remember to complete your evaluations!
  43. 43. Thank you!

×