How serv.sg deliver features using AWS infrastructure: programatically building and provisioning a stack before destroying it. This talk will explain what they do, why they do it and the tools they use (packer.io, terraform.io, ansible etc.). Includes a live demo of the whole cycle. Video of the talk: http://engineers.sg/v/474

1. How we build features at
Serv.sg
in a Pandan Leaf...
January 06th 2016
Nicolas Mas
Nicolas@serv.sg
Nicolas.Mas@gmail.com
https://www.linkedin.com/in/nicolasmas
https://github.com/nicolasmas

2. Acknowledgment
“Alone we can do so little; together we can
do so much”
Sandeep Isac Marco Yang Alex Hayden
Alexander Bulat Andrei Ping Yang

3. <ME>
Nicolas Mas
Nicolas@serv.sg
Nicolas.Mas@gmail.com
https://www.linkedin.com/in/nicolasmas
https://github.com/nicolasmas
</ME>

4. What this talk won’t do
⬗ Entertain you with funny
pictures (sorry). This will be
the only one.
⬗ Describe the tools being
used, unless there is a
specific request (let’s keep
the focus).
⬗ Guarantee that Murphy’s law
won’t impact the demo.

5. So what is it going to be?
⬗ What do we do at Serv.sg?
⬗ Our (current) technology stack
⬗ Problem we solve
⬗ Feature build (+ DEMO)
⬗ Back to our initial problem...
⬗ Limits
⬗ Enhancements
⬗ B Tracks: if we really have time
Disclaimer: I am a DevOps padawan, what is described tonight is probably
not the best solution ever. Any suggestion is welcome~

6. What we do at Serv.sg
✓ Real time Mobile workforce location tracking
✓ Automated scheduling of service appointments
✓ Automated email / SMS reminders of upcoming appointments and contracts.
✓ Manage customer data and all history of service appointments
✓ Manage Services offered, including recurring contracts and packages, etc.
✓ Advanced reports on employee utilization as well as business

7. Technologies (Stack… for now)
Key
backend
technology
Data
storage
Key
frontend
technology

8. Technologies (DevOps)
For now… (bis)

9. The Problem We Hope to solve...
Feature A
Feature B
… Feature N
PRODUCT
BACKLOG
QA / UAT /
PRODUCTION
Build - All or Nothing
Feature A
Feature B
Monolithic App Development

10. Overview
JIRA-456
JIRA-456.serv.sg
JIRA-456.db.serv.sg
/serv-branch-build
Branch JIRA-456
JOB JIRA-456
On “New Branch”
JOB Monitor Branches

11. Overview
On “Branch Deleted”
JIRA-456
JIRA-456.serv.sg
JIRA-456.db.serv.sg
/serv-branch-build
Branch JIRA-456
JOB JIRA-456
JOB Monitor Branches

12. It all starts in Jira
⬗ The developer takes ownership of, let’s say the item “JIRA-456” during a
sprint.
⬘ Actually, he does not do it in JIRA, but from github by branching out the
codebase.
⬘ The Branch HAS TO BE named after the jira item, e.g. JIRA-456

13. It continues on Slack
⬗ The moment the branch is created, we can actually build the
infrastructure:
⬘ In slack: “/serv-branch-build”
⬗ Slack is asking a Jenkins job to monitor the github repository for
new/deleted branches

14. What happens in the jenkins monitoring job
⬗ The Jenkins job will do a few things:
⬘ 1. It will connect to github and get all the branches
⬘ 2. It will detect our new branch
⬘ 3. A custom jenkins job will be created for that branch
⬘ Terraform will be invocated
⬗ It will also keep you updated of the progress in Slack

15. The Infra is built...
...But not yet provisioned !!
Note: we use a pre-baked OS image for
the instances

16. Building the branch
⬗ In Jenkins

17. Provisioning the branch
⬗ Ansible playbook
⬘ Site.yml dynamically set
⬗ Roles
⬘ Hostname set
⬘ Supervisord
⬘ Yum additional packages
⬘ Web conf
⬘ DB conf
⬘ DB Load
⬘ (...)

18. End Result

19. Destroying the build
⬗ The features has been validated or ditched, the branch is either merged
or destroyed on github
Note: If we need a few back and forth between the product and the dev for the
feature, it’s just a matter of running the jenkins branch job to deploy again

20. Back in Slack
⬗ “/serv-branch-build”
⬗ AWS quick check.. the instances are terminated

21. Looking back at our initial problem...
⬗ Deploying by feature is not new. Immutable infra seems to be less
common. The two combined? Anyone doing it?
⬗ From a business/product perspective
⬘ More responsive to bug fixing, features deliveries
⬘ No confusion on what has been deployed
⬘ The branch can be hotfixed => one step closer to CD
⬗ From a technical perspective
⬘ Leveraging the cloud infrastructure, programmatically
⬘ No environment recycling or reconfiguring with the related uncertainty
⬘ As close to production as we can
⬘ A good way to craft baked OS with the latest update and test it

22. Limits/Challenges
⬗ Hard to keep track of the full cycle, from the OS baking to the feature
deployed:
⬘ Packer: need to manually describe the target provisioned OS. Hard to test,
need to manage the produced artifacts
⬘ Ansible: need to manually write the roles, hard to test and a pain to
maintain
⬘ Terraform: need to manually describe the infra, hard to test, painful to
maintain and evolve. Some bugs with route 53
⬘ Dependency on Jenkins
⬘ A fair amount of templating (jinja2 style) with a risk induced by the dynamic
aspect of the build (hostnames, EIP...)
⬗ Many pieces in movement, dependency
⬗ Need to be in sync with the product team to check and act upon the
feature as soon as it is available. We don’t want a stack running for ages..

23. Enhancement
⬗ Use a tool to orchestrate the branch + infra build aspect (Spinnaker.io?
StackStorm.io?) or build a simple micro service to do it
⬗ Choose the branch to build in slack instead of all the branches with the
pattern “JIRA-XYZ”
⬗ Improve the provisioning and better manage the ansible playbook
(Tower.io?)
⬗ How is this compatible with our next architecture steps? (micro services)

24. Bonus Track - Did you know ?
“The mother of all demos” http://en.wikipedia.
org/wiki/The_Mother_of_All_Demos
⬗ Douglas Engelbart's December 9, 1968
⬗ 90-minute live public demonstration of the online system, NLS, they had
been working on since 1962
⬗ 1000 attendees and a demo of almost all features we use today
⬘ Mouse
⬘ hypertext, object addressing and dynamic file linking
⬘ shared-screen collaboration involving two persons at different sites
communicating over a network with audio and video interface.
http://web.stanford.
edu/dept/SUL/library/extra4/sloan/mousesite/1968Demo.html

25. https://www.youtube.com/playlist?list=PL76DBC8D6718B8FD3
Watch it !!

26. Bye Bye !