Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

DockerCon EU 2015: It's in the game: the path to micro-services at Electronic Arts with Docker


Published on

Presented by Andrew Hately, CTO - Cloud Architecture, IBM and Scott Porter, Sr. Developer, Electronic Arts

Learn how Docker can be used to achieve near bare-metal performance and a scale-out architecture that enables game backends to scale and stay responsive during load spikes. Game popularity can change with every feature and content pack release, and IBM and Electronic Arts have transitioned a mobile game engine to leverage Docker to enable rapid rollouts while handling more game users. In this session you'll learn design tips from the development of this next-gen gaming platform in an industry where user loyalty and performance are everything. Docker packaging of the game services is enabling a transition to a more flexible, micro-service based architecture, and this session will discuss the development lessons learned during that transition as well as the transition to using Docker in production.

Published in: Technology

DockerCon EU 2015: It's in the game: the path to micro-services at Electronic Arts with Docker

  1. 1. It's in the game: The path to micro-services at Electronic Arts with Docker Andrew Hately IBM CTO Cloud Architecture Scott Porter EA Cloudcell Server Lead Melbourne Firemonkeys
  2. 2. GAMING MARKET GROWING ACROSS ALL PLATFORMS AND REGIONS Mobile PC/F2P Console NORTH AMERICA LATAM EUROPE ASIA/ROW GROWTH YoY 12% REVENUES 2015est. 21B GROWTH YoY 19% REVENUES 2015est. 4B GROWTH YoY 7% REVENUES 2015est. 17B GROWTH YoY 9% REVENUES 2015est. 40B 17% 46% 37%41% 16% 43% 195M 182M 553M 817M 33% 24% 43% 51% 44% 5% 2015 est. Revenues Consist of Software Gaming Market
  3. 3. MOBILE FUELING INDUSTRY GROWTH $3.3 $4.3 $6.1 $9.9 $17.0 $27.8 $34.0 2009 2010 2011 2012 2013 2014 2015 Mobile games industry has grown 10x in less than a decade Mobile Gaming Revenue, WW $B
  4. 4. Mobile Game Server architecture/requirements • Mobile Gaming Back-end is a very typical Web application • Frequent updates to game content - infrequent updates to game engine • Daily peak users 150,000 to 250,000 simultaneous users and 1-2 million daily • Need to scale-up/down around release and content pack releases • Stateless front-ends with memory caching and both SQL and no-SQL DB • 50TB per day of data - Increasing use of analytics on user experience, recorded game telemetry, replays, etc. Firewall Load Balancer NGNIX Game Server REDIS Memcache MySQL cluster PHP PHP PHP PHP PHP Object Storage
  5. 5. Demo – Docker for Gaming
  6. 6. • Our current Ops complexity • Partnership with IBM • Why we believe Docker can help • How we integrated Docker
  7. 7. Scott Porter Cloudcell Server Lead EA Melbourne Firemonkeys @sl_porter
  8. 8. EA MELBOURNE FIREMONKEYS • Moble studio based in Melbourne, Australia • Formed in 2012 from two EA Mobile studios, Iron Monkey and Firemint • Focused on developing the highest quality mobile titles
  9. 9. • Firemonkeys in house Client/Server Tech Stack • Mobile focused connected features • Originally developed by Firemint for Flight Control
  10. 10. Integrated into all Firemonkeys live titles
  11. 11. • Live Service Games • Bi monthly client releases • No downtime What do we do?
  12. 12. • 13.5k Peak Requests/s • ~900M Requests/day • ~430k Peak Concurrent Players • 5M – 8M Daily Active Players • > 100 TB Binary Data • 170 Hosts (60 Live Hosts) • Team of 7 Current Scale
  13. 13. • Baremetal Hosts • Nginx, PHP, Memcache, MySQL, Redis • Batched client requests, short-lived, stateless • Requests can touch any datastore Architecture
  14. 14. Game Logic Data StorageLoad Balancing Architecture External Services StatsD / Diamond / Graphite / Graphana Object Storage Datadog
  15. 15. • Automated Development / Manual Live process • In house database migration and code deployment system • Bash & rsync • Rollbacks are handled via code duplication Current Deployment Methodology
  16. 16. • Asynchronous processes (workers) manually configured ( services, cron jobs ) • Brittle. • SaltStack is awesome! • But still slow, complex, no HA Current Orchestration Methodology
  17. 17. • Moved binary data to SoftLayer Object Storage (Open Stack Swift) • Currently provision for Peak • Shard live environments per game title • Interested in transitioning to microservices Current Scalability
  18. 18. • Improved deployments • Growing ecosystem • Scalable • Easy transition, run side by side with existing stack Why Docker?
  19. 19. • Focused on the Game Logic Layer • Decomposition • Nginx • Admin • Application • Workers Packaging Cloudcell
  20. 20. • Generic Containers • Run multiple containers, not processes • Tradeoff considerations Packaging Cloudcell
  21. 21. Code Building Images Jenkins Docker Registry Base Images nginx php-fpm php-worker Built Images php-fpm-nfs-0001 php-fpm-nfs-0002 php-worker-a-1 nfs-0002.tar worker-a-1.tar nfs-0001.tar Dev Env Jenkins Application Code (SVN)
  22. 22. • Fleet • Unit files per service • Nginx • Admin • Application • Each worker type Orchestration
  23. 23. Load Testing Host Baremetal nginx Application Worker A Load Generation LTG LTG LTG LTG LTG LTG Host Docker Docker nginx php-fpm-nfs-0001 php-worker-a-1 etcd Fleet
  24. 24. • Overall 5 – 10% overhead running Load Test with Docker Load Testing Results PLACEHOLDER NEED A REAL GRAPH HERE!
  25. 25. • Docker Private Registry • Jenkins • Studio Jenkins ships code to build machine • Build machine builds and commits images to Registry • Generates Fleet configuration from templates • Executes Fleet deployments Deployments
  26. 26. • Config Editor • Bridges Cloudcell Application deployments to environment state • Provides a simple UI for building, deployments, and state via Jenkins and Fleet Deployments
  27. 27. Deployments Docker Build Machine Host Docker etcd Jenkins Docker Registry Config Editor nginx php-fpm-nfs-0001 php-worker-a-1 Fleet
  28. 28. Docker enhanced architecture on SoftLayer CentOS 7 host Discovery/Config Fleet managed Docker services nginx- confd.service NGINX-confd container Orchestration Docker php- fpm@1.service ea/php- fpm:latest 1 php- fpm@2.service ea/php- fpm:latest 2 php-fpm-discovery @2.service php-worker- A@2.service ea/php- worker:latest 2 php-worker- B@1.service ea/php- worker:latest 1 php-worker-A-discovery @2.service php-worker-B-discovery @1.service php-worker- A@1.service ea/php- worker:latest 1 php-worker-A-discovery @1.service php-fpm-discovery @1.service Redis Externaldataservices MySQLMemcached LoadBalancer
  29. 29. Configuration, orchestration and deployment host servi ce servi ce servi ce servi ce host servi ce servi ce servi ce servi ce host servi ce servi ce servi ce servi ce host servi ce servi ce servi ce servi ce Configure topology, maintain state. Build images with code, deploy to or scale target environments. Run containers in staging, load test, and production. Capture metrics.
  30. 30. • Centralised Logging • Monitoring - processes inside containers Future Work
  31. 31. — Scott Porter “Using IBM and Docker container solutions, we have been able to build a gaming platform that is massively scalable, highly performant and meets our deployment velocity needs.” 33
  32. 32. Thank you! @cloudtroll Andrew Hately Scott Porter @sl_porter