This document summarizes the Skypilot project at Demonware to build a containerized continuous delivery pipeline for game services. Key points: - Demonware provides online services for popular games but faced challenges with their legacy monolithic architecture - Skypilot aimed to deliver game services for Skylanders Imaginators through containers and automation to improve agility - It introduced tools like Jenkins, Mesos/Marathon, CoreOS, Docker, and defined workflows for building, configuring and deploying containers - Skylanders launched successfully using this new pipeline, reducing deployment times from hours to minutes and recovery times.

1. Skypilot
Thomas Shaw
Build Engineer, Demonware
(Activision)

2. Delivering Amazing
Game Experiences
through Containerized
Pipelines
Skypilot

3. 1. Who are Demonware?
2. What is Skypilot?
3. Skypilot Principles
4. Workflows
5. Takeaways
Agenda

4. Who are Demonware?
We provide online services and
infrastructure for some of the world’s
most popular video game franchises.

6. A brief history of Demonware time
2009(MW2)
Company size
Concurrent users
Size of our monolith
Number of services
Operational overhead
Gastric ulcer size
2013 (Ghosts) 2015 (BO3)

7. What was Skypilot ?
Goal : deliver and run game services for a
production title, Skylanders Imaginators.
How : through a containerised continuous
delivery pipeline.
Length : 9 months.

8. Deliver an amazing build, test and deploy pipeline that
aims to be so reliable, effective and easy to use that our
product and title departments will end up writing high
value gaming services without giving a second thought
to how they may reliably deliver these in record time.
Unlocking engineering agility allows us to deliver
“amazing value in record time”.
Mission Statement

9. Building, configuring, testing, deploying [and running]
services
• requires specialist knowledge
• is not safe (incomplete and/or unreliable automation)
• is not efficient
• is not consistent across environments
• does not allow us to work in small batch sizes
The problem

10. “Evolution of our
existing tools was
doomed to fail. What
we needed was a
revolution”
- Morgan Brickley, Dev Lead

11. This wasn’t a greenfield project. We had some legacy issues
and tech debt that we needed to address
• Services had always run on bare metal
• Monolithic codebase
• Internal tooling didn’t support container deployment
• Software choices made a decade ago no longer made sense
• Processes were inconsistent across teams
• Processes were immature
Some other challenges

12. We want to work in small batch sizes in order to
• reduce iteration time
• increase quality
• fulfil our mandate: unlock engineering agility
We also wanted to
• codify our deployments
• become lean
• reduce the entry-level to service deployment
The solution

13. From hero to zero
1 - Initial
2 - Repeatable
3 - Defined
4 - Managed
5 - Optimizing
Process
M
aturity

14. “What’s the smallest number of
steps, with the smallest number of
people and the smallest amount
of ceremony required to get new
code running on your servers?”
- Erik Kastner

15. Reliable
Simple
Fast
Skypilot Principles

16. Keeping it familiar and simple
● git
● docker
● docker-registry
● docker-compose

17. Introducing something new
● Jenkins
● Mesos/Marathon
● CoreOS

18. Databases
● MySQL in containers
● Data stored on GP2 EBS volumes
● Simple master/slave topology (no master
promotion)
● Slaves attempt sync on startup

19. Workflows
● Building service containers
● Adding configuration
● Deploying service containers

20. Developer
Developer
Developer
Centralized
Artifact Repo
Building Service Containers

21. Adding Configuration
Configuration
Shared Build Infrastructure

22. Deploying Service Containers
Configuration
Shared Build Infrastructure Production Infrastructure
Restricted
Production
Artifact Repo

23. So how did it work out?
Skylanders was successfully launched on
October 13th, 2016 without issue.

24. Efficiencies
Time to build a shared cluster days → 15 - 20 minutes
Time to deploy a title environment hours → 10 - 15 minutes
Time to recover from a full outage hours → 8 - 15 minutes
Time to recover from a database failure hours → 40 - 60 seconds

25. What next ?
● Provide a better UX for deployments
● Reduce new deployment time by
parallelizing the CD test stage
● Prepare for larger scale deployments
● Educate

26. “I was able to update a
service from scratch within
45 mins. This included setup
time, learning git and 30
mins of user error
debugging”
- Lisa Reilly, Project Manager

27. “With zero Ops
experience, I was able
to create a Production
cluster, on my own, in
minutes”
- Anar Rahimli, Skypilot customer

28. Thank You!
@docker #dockercon @demonware