Startup Track

2. * as Code
Árni Birgisson
Infrastructure Engineer @arnibirgisson

3. What is QuizUp?

4. What is QuizUp?
• The biggest trivia game in the world!
• Record setting 1M in first week
• 1000+ curated topics in 6 languages
• MyQuizUp topics
• Millions of questions

5. QuizUp
Infrastructure

6. • QuizUp-Admin
• Codebase python
• Created AMI’s along
with ASG, ELB AMI
[Python App]
AMI
[Python App]
AMI
[Python App]
AMI
[Python App]
AMI
[Python App]
{ASG
ELB
Datastores

7. • Multiple services
• AMI + ASG per service
• ELB for service-to-
service
AMI
[Python App]
AMI
[Python App]
AMI
[Python App]
AMI
[Python App]
AMI
[Python App]
{ASG
ELB
AMI #2
[Java App]
ELB
AMI #2
[Java App]
AMI #2
[Java App]
Datastores

8. • Increasing number of services
• SOA / Micro Service architecture
• Routers (nginx)
• ZooKeeper
• ZooRunner AMI
[Python App]
AMI
[Python App]
AMI
[Python App]
AMI
[Python App]
AMI#1
[Python App]
ASG
Routers
AMI #2
[Java App]
AMI #2
[Java App]
AMI #2
[Java App]
Datastores
RoutersRoutersRouters
AMI #2
[Java App]
{ AMI #2
[Java App]
AMI #3
[Java App]

9. • Docker
• Python, Node, Go, Java, Scala,
C++
• More generic AMIs
AMI
[Python App]
AMI
[Python App]
AMI
[Python App]
AMI
[Python App]
AMI
[Docker App]
ASG
Routers
AMI #2
[Java App]
AMI #2
[Java App]
AMI #2
[Docker App]
Datastores
RoutersRoutersRouters
AMI #2
[Java App]
{ AMI #2
[Java App]
AMI #3
[Docker App]

10. • ~50 different services
• Mostly JVM
• All services run in docker
• Single AMI
• Datastores too* AMI
[Python App]
AMI
[Python App]
AMI
[Python App]
AMI
[Python App]
Single AMI
[App]
[Dependancies]
ASG
Routers
AMI #2
[Java App]
AMI #2
[Java App]
Single AMI
[App]
[Dependancies]
Datastores
RoutersRoutersRouters
AMI #2
[Java App]
{
AMI #2
[Java App]
Single AMI
[App]
[Dependancies]
… x 50

11. Why IaC?

12. Why IaC?
• EC2 Classic - VPC
• QuizUp-Admin outdated
• Increasing complexity
• Iteration speed
• Clean setup, not a VPC version of the old
stuff.

13. Experiments

14. Experiments
• CloudFormation, Troposphere, Terraform
• VPC with one service

15. Experiments
• VPC
• Subnets
• Routing Table
• Auto-Scale Group
• Launch Configuration
• Security Group (+ rules)
• Support for variables

16. Experiments
• Decided to use Terraform
• Plan / Apply (preview changes)
• Why is that important?

17. Infrastructure as Code

18. What’s Included
• Repository
• Terraform
• CI
• Module for a Service
• Modules for DynamoDB, ELBs, etc.

20. deploy
staging
iterate
Test
deploy
production
PR/Revie
w
merge
CI JOB checks syntax and
outputs plan
Branch with
proposed change
Iteration

21. Single AMI
• Can become any service
• Packer + Puppet
• QuizUp Docker Manager
• Logging & Metrics

22. Single AMI
Quizup Docker Manager
statsd log shipper
Instance metadata ZooKeeper
application dependancy dependancy
Docker registry

23. Configuration as Code

24. Configuration
• Servers fetch config from ZooKeeper
• ZooKeeper manually updated
• Can we use a similar flow?

25. Yes!
• In house git repository
• git/filesystem to ZooKeeper sync tool
• Jenkins CI
• Configuration through same workflow
Branches, PR

26. What have we gained?

27. What have we gained?
• Production infrastructure changes worry-
free non-events
• Code = Documentation
• Peer reviews, PRs, Comments
• Disaster recovery
• History (+ reverting!)

28. What have we gained?
• Anyone can propose infrastructure changes
• Faster and more consistent changes
• Staging is a small copy of production

29. What have we gained?
Infrastructure as Code
helps maintain a devops culture
by using the same workflow
to modify applications, infrastructure and
configuration

30. What’s not great?

31. State
• State file must be kept safe!
• Hard to “inject” resources into state
• (Currently) no way to create Terraform
description of current resources

32. Speed
• Must choose between:
• Apply everything always (slower)
Infrastructure = Code
• Apply in parts (speedy)
Requires plumbing. Can diverge.

33. No sidelining
• No manual changes
• They are reset during next apply
• Pro or Con ?

34. Thank You