Successfully reported this slideshow.
From Vagrant to production
Mark Eijsermans
HootSuite - Software Engineer
@markeijsermans
http://code.hootsuite.com
HootSuite
• 
• 
• 
• 
• 
• 
• 

Social media dashboard
8M users
40 people committing code
4 ops
AWS (2/3) & private cloud ...
•  Monolithic web app (PHP)
•  Transitioning towards service oriented
architecture (Scala)
In the beginning

dev server
(LAMP)

smb

dev 1

svn

smb

dev 2

production
(LAMP)
In the beginning
Release anytime
•  Small team
•  Is intimately aware of prod
•  Low overhead to release

DevOps?
..a while later
web

web

web

web

web

web

web

web

svn

dev server

web

web

web

web

gearman
smb

gearman

0mq wor...
..a while later

1 release / 2-4 weeks
• 
• 
• 
• 
• 

Medium sized team
Branching, pre-release code freezes
Only some dev...
…and a few years more (now)
Vagrant
dev

dev

dev

dev

dev

dev

dev

dev

dev

dev

dev

dev

dev

dev

dev

dev

dev

d...
…and a few years more (now)
6 - 10 release / day
• 
• 
• 
• 
• 
• 

Larger team
No branching
Anyone deploys
Automated proc...
Perfection is the enemy of good
•  You’re never going to get a perfect system
•  How far along are we?
•  How many things ...
The move to artifacts
•  svn
•  svn mirror
•  PHP
•  phar
•  tar + zsync

•  Scala
•  debian packages
Dev

Build

Deploy

Jenkins
project code

unit test

create
artifact

Artifact Repo
build #124
build #123

smoke test

sta...
Broke the build?

“I’m on it”
Vagrant
•  Working on a shared web server over
samba was simply painful
•  extremely volatile environment

•  Vagrant simp...
Dev

Build

Deploy

Jenkins
project code
(vagrant)

unit test

create
artifact

Artifact Repo
build #124
build #123

smoke...
Dev

Build

Deploy

Jenkins
project code
(vagrant)

Ansible

unit test

create
artifact

smoke test

build vagrant
box

Ar...
Ansible (and why we love it)
…yes, use any CM, but why we chose Ansible:
•  New team members up to speed in a few days:
• ...
Digital archeology
Auditing our infrastructure
There are gems in that dirt
•  logs
•  you mean logs aren’t saved to /dev/null ?
•  app was ro...
Cache for gold
•  With configuration management gold images
become a caching layer to the build process
•  Private cloud - ...
Simplify for sanity
•  200+ server types is hard to manage
•  Amortize the stack
•  Custom bespoke vs. generic appliances
Devs on call
•  Learning happens in production
•  Production is still running at 3am
“People who are really
serious about software should
make their own hardware”
Alan Kay
Devs on call
•  Is your monitoring and tooling truly effective?
•  Make sure they can ssh into the damn box! (if they

have...
Is this working?
Working out loud
•  Yammer (& HipChat for on call)
•  Tried IRC, was too silo’ed

•  5 whys post mortems
•  Accountability...
Learning happens in production
Thank you!
@markeijsermans
http://code.hootsuite.com
From vagrant to production - Mark Eijsermans
From vagrant to production - Mark Eijsermans
From vagrant to production - Mark Eijsermans
From vagrant to production - Mark Eijsermans
From vagrant to production - Mark Eijsermans
From vagrant to production - Mark Eijsermans
From vagrant to production - Mark Eijsermans
From vagrant to production - Mark Eijsermans
From vagrant to production - Mark Eijsermans
From vagrant to production - Mark Eijsermans
From vagrant to production - Mark Eijsermans
Upcoming SlideShare
Loading in …5
×

From vagrant to production - Mark Eijsermans

1,378 views

Published on

Published in: Technology
  • Be the first to comment

From vagrant to production - Mark Eijsermans

  1. 1. From Vagrant to production Mark Eijsermans HootSuite - Software Engineer @markeijsermans http://code.hootsuite.com
  2. 2. HootSuite •  •  •  •  •  •  •  Social media dashboard 8M users 40 people committing code 4 ops AWS (2/3) & private cloud (1/3) 100M requests/day (hootsuite.com) 70M requests/day (ow.ly)
  3. 3. •  Monolithic web app (PHP) •  Transitioning towards service oriented architecture (Scala)
  4. 4. In the beginning dev server (LAMP) smb dev 1 svn smb dev 2 production (LAMP)
  5. 5. In the beginning Release anytime •  Small team •  Is intimately aware of prod •  Low overhead to release DevOps?
  6. 6. ..a while later web web web web web web web web svn dev server web web web web gearman smb gearman 0mq worker dev dev dev dev dev dev dev dev dev dev dev dev dev dev dev dev dev dev dev dev dev cron dev dev cron dev dev cron 0mq worker ops ops cron
  7. 7. ..a while later 1 release / 2-4 weeks •  •  •  •  •  Medium sized team Branching, pre-release code freezes Only some devs knowledgeable of prod Ops mostly handles deploy Complicated process Devs vs. Ops?
  8. 8. …and a few years more (now) Vagrant dev dev dev dev dev dev dev dev dev dev dev dev dev dev dev dev dev dev dev monolithic web app dev dev dev dev dev dev dev dev ops ops ops automate build test QA Service A Service B Service C Service D
  9. 9. …and a few years more (now) 6 - 10 release / day •  •  •  •  •  •  Larger team No branching Anyone deploys Automated process Commit to production in 15min Much higher % of devs understand prod
  10. 10. Perfection is the enemy of good •  You’re never going to get a perfect system •  How far along are we? •  How many things are actually: •  Automated •  Reproducible •  In a state we want them to be •  When can we start feeling good about it?
  11. 11. The move to artifacts •  svn •  svn mirror •  PHP •  phar •  tar + zsync •  Scala •  debian packages
  12. 12. Dev Build Deploy Jenkins project code unit test create artifact Artifact Repo build #124 build #123 smoke test staging dash4deploy production production service A/B/C
  13. 13. Broke the build? “I’m on it”
  14. 14. Vagrant •  Working on a shared web server over samba was simply painful •  extremely volatile environment •  Vagrant simply kicks ass •  Dev / prod parity •  Encourages experimentation •  go ahead and try out that new X
  15. 15. Dev Build Deploy Jenkins project code (vagrant) unit test create artifact Artifact Repo build #124 build #123 smoke test staging dash4deploy production production service A/B/C
  16. 16. Dev Build Deploy Jenkins project code (vagrant) Ansible unit test create artifact smoke test build vagrant box Artifact Repo vagrant.box build #124 build #123 dash4deploy production production service A/B/C staging
  17. 17. Ansible (and why we love it) …yes, use any CM, but why we chose Ansible: •  New team members up to speed in a few days: •  Gentle learning curve – YAML •  Push over ssh - easy to understand •  The more declarative it is, the more it documents •  It's all about disseminating information •  Agentless model suits immutable ephemeral instances
  18. 18. Digital archeology
  19. 19. Auditing our infrastructure There are gems in that dirt •  logs •  you mean logs aren’t saved to /dev/null ? •  app was rotating logs, vs. using logrotate •  memcached, mongodb servers not all the same version •  what the hell is server X for ???
  20. 20. Cache for gold •  With configuration management gold images become a caching layer to the build process •  Private cloud - idle instances are cheap •  AMI - need auto spinning to save $$$ & time
  21. 21. Simplify for sanity •  200+ server types is hard to manage •  Amortize the stack •  Custom bespoke vs. generic appliances
  22. 22. Devs on call •  Learning happens in production •  Production is still running at 3am
  23. 23. “People who are really serious about software should make their own hardware” Alan Kay
  24. 24. Devs on call •  Is your monitoring and tooling truly effective? •  Make sure they can ssh into the damn box! (if they have to) •  Throwing some poor dev onto a broken system they know nothing about doesn't work •  Choose your own adventure cookbooks •  Create registry of specialists
  25. 25. Is this working?
  26. 26. Working out loud •  Yammer (& HipChat for on call) •  Tried IRC, was too silo’ed •  5 whys post mortems •  Accountability •  Build fails - “I'm on it”
  27. 27. Learning happens in production
  28. 28. Thank you! @markeijsermans http://code.hootsuite.com

×