Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Skypilot
Delivering Amazing
Game Experiences
through
Containerized
Pipelines
"Technologies that are going to affect our lives in
the next decade are being tested and developed in
the video game spher...
Thomas Shaw :
❏ Build Engineer, Demonware (Activision)
❏ Docker Captain
❏ Co-founder and organizer of ShipItCon 2017
❏ Hus...
Who are Demonware?
We provide online services and
infrastructure for some of the world’s most
popular video game franchise...
Demonware in numbers
• 170 employees
• 3 offices; Dublin, Vancouver and Shanghai
• 469 million registered gamers
• 4000 se...
A brief history of Demonware time
2009(MW2)
Company size
Concurrent users
Size of our monolith
Number of services
Operatio...
Demonware Stack
Popular Services
• Matchmaking
• Leaderboards
• Marketplace
• Storage
• Identity Management
Good times
● 119 games shipped since 2005
● Smooth launch
● High concurrent user count
● Happy gamers
Not so good times
It’s just one small change ...
What was Skypilot ?
Goal : deliver and run game services for a
production title, Skylanders Imaginators.
How : through a c...
Current Challenges
Building, configuring, testing, deploying [and running] services
• requires specialist knowledge
• is n...
Legacy Challenges
We had a few other issues that we needed to address
• Services had always run on bare metal
• Monolithic...
Industry Challenges
• Shorter development cycles
• Increasingly complex services
• Supporting more platforms/customers/mar...
The Solution
We want to work in small batch sizes in order to
• reduce iteration time
• increase quality
• fulfil our mand...
Skypilot principles
• Simple
• Fast
• Reliable
• Safe
@damo_marshall
Keep it simple
• git
• docker
• docker-registry
• docker-compose
Introduce something new
• Jenkins - Pipelines as Code
• Mesos/Marathon
• CoreOS
Workflows
• Building service containers
• Adding configuration
• Deploying service containers
Building service containers
Adding configuration
Shared Build Infrastructure
Configuration
Deploying service containers
So how did it work out?
Skylanders was successfully launched on
October 13th, 2016 without issue.
Efficiencies
Time to build a production cluster days → 15 - 20 minutes
Time to deploy a title environment hours → 10 - 15 ...
What’s next ?
● Provide a better UX for deployments
● Reduce new deployment time by
parallelizing the CD test stage
● Prep...
“With zero Ops experience, I was able to
create a Production cluster, on my own, in
minutes”
Anar Rahimli, Skypilot custom...
“I was able to update a service from
scratch within 45 mins. This included
setup time, learning git and 30 mins of
user er...
Tips for container adoption
● Build a coalition of users
● Encourage the coalition to standardize
● Image Registry is a mu...
"The most important thing about
containers is the process of using them,
not the things themselves"
Justin Warren, PivotNi...
Thank You!
@tomwillfixit
DevOpsDays Galway 2017 - Skypilot Project
DevOpsDays Galway 2017 - Skypilot Project
DevOpsDays Galway 2017 - Skypilot Project
Upcoming SlideShare
Loading in …5
×

DevOpsDays Galway 2017 - Skypilot Project

249 views

Published on

Delivering Amazing Game Experiences through Containerized Pipelines (Demonware)

Published in: Technology
  • Be the first to comment

DevOpsDays Galway 2017 - Skypilot Project

  1. 1. Skypilot Delivering Amazing Game Experiences through Containerized Pipelines
  2. 2. "Technologies that are going to affect our lives in the next decade are being tested and developed in the video game sphere" Keith Stuart, The Guardian
  3. 3. Thomas Shaw : ❏ Build Engineer, Demonware (Activision) ❏ Docker Captain ❏ Co-founder and organizer of ShipItCon 2017 ❏ Husband/Father Lesser known facts : Originally from a small town called Carrickfergus. Once worked as Santa. Enjoys coffee.
  4. 4. Who are Demonware? We provide online services and infrastructure for some of the world’s most popular video game franchises.
  5. 5. Demonware in numbers • 170 employees • 3 offices; Dublin, Vancouver and Shanghai • 469 million registered gamers • 4000 servers • 132 billion API calls per month • 15.1 million requests per second at peak • 1.2+ million metrics a minute • 25 million database queries per second
  6. 6. A brief history of Demonware time 2009(MW2) Company size Concurrent users Size of our monolith Number of services Operational overhead Gastric ulcer size 2013 (Ghosts) 2015 (BO3)
  7. 7. Demonware Stack
  8. 8. Popular Services • Matchmaking • Leaderboards • Marketplace • Storage • Identity Management
  9. 9. Good times ● 119 games shipped since 2005 ● Smooth launch ● High concurrent user count ● Happy gamers
  10. 10. Not so good times
  11. 11. It’s just one small change ...
  12. 12. What was Skypilot ? Goal : deliver and run game services for a production title, Skylanders Imaginators. How : through a containerised continuous delivery pipeline. Length : 9 months.
  13. 13. Current Challenges Building, configuring, testing, deploying [and running] services • requires specialist knowledge • is not safe (incomplete and/or unreliable automation) • is not efficient • is not consistent across environments • does not allow us to work in small batch sizes
  14. 14. Legacy Challenges We had a few other issues that we needed to address • Services had always run on bare metal • Monolithic codebase • Internal tooling didn’t support container deployment • Processes were inconsistent across teams • Processes were immature
  15. 15. Industry Challenges • Shorter development cycles • Increasingly complex services • Supporting more platforms/customers/markets • Exceeding customers expectations • More innovation to meet customer diversity
  16. 16. The Solution We want to work in small batch sizes in order to • reduce iteration time • increase quality • fulfil our mandate: unlock engineering agility We also wanted to • codify our deployments • become lean • reduce the entry-level to service deployment
  17. 17. Skypilot principles • Simple • Fast • Reliable • Safe @damo_marshall
  18. 18. Keep it simple • git • docker • docker-registry • docker-compose
  19. 19. Introduce something new • Jenkins - Pipelines as Code • Mesos/Marathon • CoreOS
  20. 20. Workflows • Building service containers • Adding configuration • Deploying service containers
  21. 21. Building service containers
  22. 22. Adding configuration Shared Build Infrastructure Configuration
  23. 23. Deploying service containers
  24. 24. So how did it work out? Skylanders was successfully launched on October 13th, 2016 without issue.
  25. 25. Efficiencies Time to build a production cluster days → 15 - 20 minutes Time to deploy a title environment hours → 10 - 15 minutes Time to recover from a full outage hours → 8 - 15 minutes Time to recover from database failure hours → 40 - 60 seconds
  26. 26. What’s next ? ● Provide a better UX for deployments ● Reduce new deployment time by parallelizing the CD test stage ● Prepare for larger scale deployments ● Leverage container orchestration ● Educate
  27. 27. “With zero Ops experience, I was able to create a Production cluster, on my own, in minutes” Anar Rahimli, Skypilot customer
  28. 28. “I was able to update a service from scratch within 45 mins. This included setup time, learning git and 30 mins of user error debugging” Lisa Reilly, Project Manager
  29. 29. Tips for container adoption ● Build a coalition of users ● Encourage the coalition to standardize ● Image Registry is a must (Check out Harbor by VMware) ● Demonstrate tooling in the Docker Ecosystem ● Organize Meetups, Bootcamps, Hackathons
  30. 30. "The most important thing about containers is the process of using them, not the things themselves" Justin Warren, PivotNine
  31. 31. Thank You! @tomwillfixit

×