Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

DevOpsDays Galway 2017 - Skypilot Project

264 views

Published on

Delivering Amazing Game Experiences through Containerized Pipelines (Demonware)

Published in: Technology
  • Be the first to comment

DevOpsDays Galway 2017 - Skypilot Project

  1. 1. Skypilot Delivering Amazing Game Experiences through Containerized Pipelines
  2. 2. "Technologies that are going to affect our lives in the next decade are being tested and developed in the video game sphere" Keith Stuart, The Guardian
  3. 3. Thomas Shaw : ❏ Build Engineer, Demonware (Activision) ❏ Docker Captain ❏ Co-founder and organizer of ShipItCon 2017 ❏ Husband/Father Lesser known facts : Originally from a small town called Carrickfergus. Once worked as Santa. Enjoys coffee.
  4. 4. Who are Demonware? We provide online services and infrastructure for some of the world’s most popular video game franchises.
  5. 5. Demonware in numbers • 170 employees • 3 offices; Dublin, Vancouver and Shanghai • 469 million registered gamers • 4000 servers • 132 billion API calls per month • 15.1 million requests per second at peak • 1.2+ million metrics a minute • 25 million database queries per second
  6. 6. A brief history of Demonware time 2009(MW2) Company size Concurrent users Size of our monolith Number of services Operational overhead Gastric ulcer size 2013 (Ghosts) 2015 (BO3)
  7. 7. Demonware Stack
  8. 8. Popular Services • Matchmaking • Leaderboards • Marketplace • Storage • Identity Management
  9. 9. Good times ● 119 games shipped since 2005 ● Smooth launch ● High concurrent user count ● Happy gamers
  10. 10. Not so good times
  11. 11. It’s just one small change ...
  12. 12. What was Skypilot ? Goal : deliver and run game services for a production title, Skylanders Imaginators. How : through a containerised continuous delivery pipeline. Length : 9 months.
  13. 13. Current Challenges Building, configuring, testing, deploying [and running] services • requires specialist knowledge • is not safe (incomplete and/or unreliable automation) • is not efficient • is not consistent across environments • does not allow us to work in small batch sizes
  14. 14. Legacy Challenges We had a few other issues that we needed to address • Services had always run on bare metal • Monolithic codebase • Internal tooling didn’t support container deployment • Processes were inconsistent across teams • Processes were immature
  15. 15. Industry Challenges • Shorter development cycles • Increasingly complex services • Supporting more platforms/customers/markets • Exceeding customers expectations • More innovation to meet customer diversity
  16. 16. The Solution We want to work in small batch sizes in order to • reduce iteration time • increase quality • fulfil our mandate: unlock engineering agility We also wanted to • codify our deployments • become lean • reduce the entry-level to service deployment
  17. 17. Skypilot principles • Simple • Fast • Reliable • Safe @damo_marshall
  18. 18. Keep it simple • git • docker • docker-registry • docker-compose
  19. 19. Introduce something new • Jenkins - Pipelines as Code • Mesos/Marathon • CoreOS
  20. 20. Workflows • Building service containers • Adding configuration • Deploying service containers
  21. 21. Building service containers
  22. 22. Adding configuration Shared Build Infrastructure Configuration
  23. 23. Deploying service containers
  24. 24. So how did it work out? Skylanders was successfully launched on October 13th, 2016 without issue.
  25. 25. Efficiencies Time to build a production cluster days → 15 - 20 minutes Time to deploy a title environment hours → 10 - 15 minutes Time to recover from a full outage hours → 8 - 15 minutes Time to recover from database failure hours → 40 - 60 seconds
  26. 26. What’s next ? ● Provide a better UX for deployments ● Reduce new deployment time by parallelizing the CD test stage ● Prepare for larger scale deployments ● Leverage container orchestration ● Educate
  27. 27. “With zero Ops experience, I was able to create a Production cluster, on my own, in minutes” Anar Rahimli, Skypilot customer
  28. 28. “I was able to update a service from scratch within 45 mins. This included setup time, learning git and 30 mins of user error debugging” Lisa Reilly, Project Manager
  29. 29. Tips for container adoption ● Build a coalition of users ● Encourage the coalition to standardize ● Image Registry is a must (Check out Harbor by VMware) ● Demonstrate tooling in the Docker Ecosystem ● Organize Meetups, Bootcamps, Hackathons
  30. 30. "The most important thing about containers is the process of using them, not the things themselves" Justin Warren, PivotNine
  31. 31. Thank You! @tomwillfixit

×