2. "Technologies that are going to affect our lives in
the next decade are being tested and developed in
the video game sphere"
Keith Stuart, The Guardian
3. Thomas Shaw :
❏ Build Engineer, Demonware (Activision)
❏ Docker Captain
❏ Co-founder and organizer of ShipItCon 2017
Lesser known facts :
Originally from a small town called Carrickfergus.
Once worked as Santa.
4. Who are Demonware?
We provide online services and
infrastructure for some of the world’s most
popular video game franchises.
5. Demonware in numbers
• 170 employees
• 3 offices; Dublin, Vancouver and Shanghai
• 469 million registered gamers
• 4000 servers
• 132 billion API calls per month
• 15.1 million requests per second at peak
• 1.2+ million metrics a minute
• 25 million database queries per second
6. A brief history of Demonware time
Size of our monolith
Number of services
Gastric ulcer size
2013 (Ghosts) 2015 (BO3)
13. What was Skypilot ?
Goal : deliver and run game services for a
production title, Skylanders Imaginators.
How : through a containerised continuous
Length : 9 months.
15. Current Challenges
Building, configuring, testing, deploying [and running] services
• requires specialist knowledge
• is not safe (incomplete and/or unreliable automation)
• is not efficient
• is not consistent across environments
• does not allow us to work in small batch sizes
16. Legacy Challenges
We had a few other issues that we needed to address
• Services had always run on bare metal
• Monolithic codebase
• Internal tooling didn’t support container deployment
• Processes were inconsistent across teams
• Processes were immature
17. Industry Challenges
• Shorter development cycles
• Increasingly complex services
• Supporting more platforms/customers/markets
• Exceeding customers expectations
• More innovation to meet customer diversity
18. The Solution
We want to work in small batch sizes in order to
• reduce iteration time
• increase quality
• fulfil our mandate: unlock engineering agility
We also wanted to
• codify our deployments
• become lean
• reduce the entry-level to service deployment
26. So how did it work out?
Skylanders was successfully launched on
October 13th, 2016 without issue.
Time to build a production cluster days → 15 - 20 minutes
Time to deploy a title environment hours → 10 - 15 minutes
Time to recover from a full outage hours → 8 - 15 minutes
Time to recover from database failure hours → 40 - 60 seconds
28. What’s next ?
● Provide a better UX for deployments
● Reduce new deployment time by
parallelizing the CD test stage
● Prepare for larger scale deployments
● Leverage container orchestration
29. “With zero Ops experience, I was able to
create a Production cluster, on my own, in
Anar Rahimli, Skypilot customer
30. “I was able to update a service from
scratch within 45 mins. This included
setup time, learning git and 30 mins of
user error debugging”
Lisa Reilly, Project Manager
32. Tips for container adoption
● Build a coalition of users
● Encourage the coalition to standardize
● Image Registry is a must (Check out Harbor by VMware)
● Demonstrate tooling in the Docker Ecosystem
● Organize Meetups, Bootcamps, Hackathons
33. "The most important thing about
containers is the process of using them,
not the things themselves"
Justin Warren, PivotNine