"Technologies that are going to affect our lives in
the next decade are being tested and developed in
the video game sphere"
Keith Stuart, The Guardian
Thomas Shaw :
❏ Build Engineer, Demonware (Activision)
❏ Docker Captain
❏ Co-founder and organizer of ShipItCon 2017
Lesser known facts :
Originally from a small town called Carrickfergus.
Once worked as Santa.
Who are Demonware?
We provide online services and
infrastructure for some of the world’s most
popular video game franchises.
Demonware in numbers
• 170 employees
• 3 offices; Dublin, Vancouver and Shanghai
• 469 million registered gamers
• 4000 servers
• 132 billion API calls per month
• 15.1 million requests per second at peak
• 1.2+ million metrics a minute
• 25 million database queries per second
A brief history of Demonware time
Size of our monolith
Number of services
Gastric ulcer size
2013 (Ghosts) 2015 (BO3)
What was Skypilot ?
Goal : deliver and run game services for a
production title, Skylanders Imaginators.
How : through a containerised continuous
Length : 9 months.
Building, configuring, testing, deploying [and running] services
• requires specialist knowledge
• is not safe (incomplete and/or unreliable automation)
• is not efficient
• is not consistent across environments
• does not allow us to work in small batch sizes
We had a few other issues that we needed to address
• Services had always run on bare metal
• Monolithic codebase
• Internal tooling didn’t support container deployment
• Processes were inconsistent across teams
• Processes were immature
• Shorter development cycles
• Increasingly complex services
• Supporting more platforms/customers/markets
• Exceeding customers expectations
• More innovation to meet customer diversity
We want to work in small batch sizes in order to
• reduce iteration time
• increase quality
• fulfil our mandate: unlock engineering agility
We also wanted to
• codify our deployments
• become lean
• reduce the entry-level to service deployment
So how did it work out?
Skylanders was successfully launched on
October 13th, 2016 without issue.
Time to build a production cluster days → 15 - 20 minutes
Time to deploy a title environment hours → 10 - 15 minutes
Time to recover from a full outage hours → 8 - 15 minutes
Time to recover from database failure hours → 40 - 60 seconds
What’s next ?
● Provide a better UX for deployments
● Reduce new deployment time by
parallelizing the CD test stage
● Prepare for larger scale deployments
● Leverage container orchestration
“With zero Ops experience, I was able to
create a Production cluster, on my own, in
Anar Rahimli, Skypilot customer
“I was able to update a service from
scratch within 45 mins. This included
setup time, learning git and 30 mins of
user error debugging”
Lisa Reilly, Project Manager
Tips for container adoption
● Build a coalition of users
● Encourage the coalition to standardize
● Image Registry is a must (Check out Harbor by VMware)
● Demonstrate tooling in the Docker Ecosystem
● Organize Meetups, Bootcamps, Hackathons
"The most important thing about
containers is the process of using them,
not the things themselves"
Justin Warren, PivotNine