1. From Vagrant to production
Mark Eijsermans
HootSuite - Software Engineer
@markeijsermans
http://code.hootsuite.com

2. HootSuite
•
•
•
•
•
•
•
Social media dashboard
8M users
40 people committing code
4 ops
AWS (2/3) & private cloud (1/3)
100M requests/day (hootsuite.com)
70M requests/day (ow.ly)

3. • Monolithic web app (PHP)
• Transitioning towards service oriented
architecture (Scala)

4. In the beginning
dev server
(LAMP)
smb
dev 1
svn
smb
dev 2
production
(LAMP)

5. In the beginning
Release anytime
• Small team
• Is intimately aware of prod
• Low overhead to release
DevOps?

6. ..a while later
web
web
web
web
web
web
web
web
svn
dev server
web
web
web
web
gearman
smb
gearman
0mq worker
dev
dev
dev
dev
dev
dev
dev
dev
dev
dev
dev
dev
dev
dev
dev
dev
dev
dev
dev
dev
dev
cron
dev
dev
cron
dev
dev
cron
0mq worker
ops
ops
cron

7. ..a while later
1 release / 2-4 weeks
•
•
•
•
•
Medium sized team
Branching, pre-release code freezes
Only some devs knowledgeable of prod
Ops mostly handles deploy
Complicated process
Devs vs. Ops?

8. …and a few years more (now)
Vagrant
dev
dev
dev
dev
dev
dev
dev
dev
dev
dev
dev
dev
dev
dev
dev
dev
dev
dev
dev
monolithic
web app
dev
dev
dev
dev
dev
dev
dev
dev
ops
ops
ops
automate
build
test
QA
Service A
Service B
Service C
Service D

9. …and a few years more (now)
6 - 10 release / day
•
•
•
•
•
•
Larger team
No branching
Anyone deploys
Automated process
Commit to production in 15min
Much higher % of devs understand prod

11. Perfection is the enemy of good
• You’re never going to get a perfect system
• How far along are we?
• How many things are actually:
• Automated
• Reproducible
• In a state we want them to be
• When can we start feeling good about it?

12. The move to artifacts
• svn
• svn mirror
• PHP
• phar
• tar + zsync
• Scala
• debian packages

13. Dev
Build
Deploy
Jenkins
project code
unit test
create
artifact
Artifact Repo
build #124
build #123
smoke test
staging
dash4deploy
production
production
service A/B/C

21. Broke the build?
“I’m on it”

23. Vagrant
• Working on a shared web server over
samba was simply painful
• extremely volatile environment
• Vagrant simply kicks ass
• Dev / prod parity
• Encourages experimentation
• go ahead and try out that new X

24. Dev
Build
Deploy
Jenkins
project code
(vagrant)
unit test
create
artifact
Artifact Repo
build #124
build #123
smoke test
staging
dash4deploy
production
production
service A/B/C

25. Dev
Build
Deploy
Jenkins
project code
(vagrant)
Ansible
unit test
create
artifact
smoke test
build vagrant
box
Artifact Repo
vagrant.box
build #124
build #123
dash4deploy
production
production
service A/B/C
staging

28. Ansible (and why we love it)
…yes, use any CM, but why we chose Ansible:
• New team members up to speed in a few days:
• Gentle learning curve – YAML
• Push over ssh - easy to understand
• The more declarative it is, the more it documents
• It's all about disseminating information
• Agentless model suits immutable ephemeral instances

29. Digital archeology

30. Auditing our infrastructure
There are gems in that dirt
• logs
• you mean logs aren’t saved to /dev/null ?
• app was rotating logs, vs. using logrotate
• memcached, mongodb servers not all the same version
• what the hell is server X for ???

31. Cache for gold
• With configuration management gold images
become a caching layer to the build process
• Private cloud - idle instances are cheap
• AMI - need auto spinning to save $$$ & time

32. Simplify for sanity
• 200+ server types is hard to manage
• Amortize the stack
• Custom bespoke vs. generic appliances

33. Devs on call
• Learning happens in production
• Production is still running at 3am

34. “People who are really
serious about software should
make their own hardware”
Alan Kay

35. Devs on call
• Is your monitoring and tooling truly effective?
• Make sure they can ssh into the damn box! (if they
have to)
• Throwing some poor dev onto a broken system
they know nothing about doesn't work
• Choose your own adventure cookbooks
• Create registry of specialists

36. Is this working?

37. Working out loud
• Yammer (& HipChat for on call)
• Tried IRC, was too silo’ed
• 5 whys post mortems
• Accountability
• Build fails - “I'm on it”

38. Learning happens in production

39. Thank you!
@markeijsermans
http://code.hootsuite.com