This document summarizes the Skypilot project at Demonware to deliver game services for Skylanders Imaginators through a containerized continuous delivery pipeline over 9 months. It introduced Docker, Jenkins Pipelines, and CoreOS to build, configure, test, and deploy services automatically in a simple, fast, and reliable manner. This reduced deployment times from hours to minutes and improved recovery times. It was successfully used to launch Skylanders without issues.

1. Skypilot
Delivering Amazing
Game Experiences
through
Containerized
Pipelines

2. "Technologies that are going to affect our lives in
the next decade are being tested and developed in
the video game sphere"
Keith Stuart, The Guardian

3. Thomas Shaw :
❏ Build Engineer, Demonware (Activision)
❏ Docker Captain
❏ Co-founder and organizer of ShipItCon 2017
❏ Husband/Father
Lesser known facts :
Originally from a small town called Carrickfergus.
Once worked as Santa.
Enjoys coffee.

4. Who are Demonware?
We provide online services and
infrastructure for some of the world’s most
popular video game franchises.

5. Demonware in numbers
• 170 employees
• 3 offices; Dublin, Vancouver and Shanghai
• 469 million registered gamers
• 4000 servers
• 132 billion API calls per month
• 15.1 million requests per second at peak
• 1.2+ million metrics a minute
• 25 million database queries per second

6. A brief history of Demonware time
2009(MW2)
Company size
Concurrent users
Size of our monolith
Number of services
Operational overhead
Gastric ulcer size
2013 (Ghosts) 2015 (BO3)

8. Demonware Stack

9. Popular Services
• Matchmaking
• Leaderboards
• Marketplace
• Storage
• Identity Management

10. Good times
● 119 games shipped since 2005
● Smooth launch
● High concurrent user count
● Happy gamers

11. Not so good times

12. It’s just one small change ...

13. What was Skypilot ?
Goal : deliver and run game services for a
production title, Skylanders Imaginators.
How : through a containerised continuous
delivery pipeline.
Length : 9 months.

15. Current Challenges
Building, configuring, testing, deploying [and running] services
• requires specialist knowledge
• is not safe (incomplete and/or unreliable automation)
• is not efficient
• is not consistent across environments
• does not allow us to work in small batch sizes

16. Legacy Challenges
We had a few other issues that we needed to address
• Services had always run on bare metal
• Monolithic codebase
• Internal tooling didn’t support container deployment
• Processes were inconsistent across teams
• Processes were immature

17. Industry Challenges
• Shorter development cycles
• Increasingly complex services
• Supporting more platforms/customers/markets
• Exceeding customers expectations
• More innovation to meet customer diversity

18. The Solution
We want to work in small batch sizes in order to
• reduce iteration time
• increase quality
• fulfil our mandate: unlock engineering agility
We also wanted to
• codify our deployments
• become lean
• reduce the entry-level to service deployment

19. Skypilot principles
• Simple
• Fast
• Reliable
• Safe
@damo_marshall

20. Keep it simple
• git
• docker
• docker-registry
• docker-compose

21. Introduce something new
• Jenkins - Pipelines as Code
• Mesos/Marathon
• CoreOS

22. Workflows
• Building service containers
• Adding configuration
• Deploying service containers

23. Building service containers

24. Adding configuration
Shared Build Infrastructure
Configuration

25. Deploying service containers

26. So how did it work out?
Skylanders was successfully launched on
October 13th, 2016 without issue.

27. Efficiencies
Time to build a production cluster days → 15 - 20 minutes
Time to deploy a title environment hours → 10 - 15 minutes
Time to recover from a full outage hours → 8 - 15 minutes
Time to recover from database failure hours → 40 - 60 seconds

28. What’s next ?
● Provide a better UX for deployments
● Reduce new deployment time by
parallelizing the CD test stage
● Prepare for larger scale deployments
● Leverage container orchestration
● Educate

29. “With zero Ops experience, I was able to
create a Production cluster, on my own, in
minutes”
Anar Rahimli, Skypilot customer

30. “I was able to update a service from
scratch within 45 mins. This included
setup time, learning git and 30 mins of
user error debugging”
Lisa Reilly, Project Manager

32. Tips for container adoption
● Build a coalition of users
● Encourage the coalition to standardize
● Image Registry is a must (Check out Harbor by VMware)
● Demonstrate tooling in the Docker Ecosystem
● Organize Meetups, Bootcamps, Hackathons

33. "The most important thing about
containers is the process of using them,
not the things themselves"
Justin Warren, PivotNine

34. Thank You!
@tomwillfixit