Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

AWS re:Invent 2016| GAM401 | Riot Games: Standardizing Application Deployments Using Amazon ECS and Terraform

2,084 views

Published on

This session covers how the data team at Riot Games utilizes ECS to consolidate and improve disparate deployment and hosting strategies across a wide range of applications deployed via Docker containers. The team will share how cluster management and container orchestration through ECS enables the team to quickly adopt and evolve shared service hosting solutions as Riot continues its journey towards becoming a multi-game studio.

Published in: Technology

AWS re:Invent 2016| GAM401 | Riot Games: Standardizing Application Deployments Using Amazon ECS and Terraform

  1. 1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Adam Rozumek November 28, 2016 Riot Games: Standardizing Application Deployments Using Amazon ECS and Terraform GAM401
  2. 2. RIOT GAMES | GAM 401 Standardizing Application Deployments Using Amazon ECS and Terraform
  3. 3. SYSTEMS ENGINEER ADAM ROZUMEK INTRODUCTIONS WHO AM I?
  4. 4. AMAZON ECS. TERRAFORM. VIDEO GAMES. WHAT TO EXPECT ECS CONSOLIDATION WINS & LESSONS Adapting existing deployments & infrastructure TERRAFORM ENCAPSULATION STRATS Object Oriented Development Operations MULTI-GAME MODULAR SERVICE DESIGN A different kind of scaling 1 2 3
  5. 5. AMAZON ECS. TERRAFORM. VIDEO GAMES. WHAT TO EXPECT ECS CONSOLIDATION WINS & LESSONS Adapting existing deployments & infrastructure TERRAFORM ENCAPSULATION STRATS Object Oriented Development Operations MULTI-GAME MODULAR SERVICE DESIGN A different kind of scaling 1 2 3
  6. 6. AMAZON ECS. TERRAFORM. VIDEO GAMES. WHAT TO EXPECT ECS CONSOLIDATION WINS & LESSONS Adapting existing deployments & infrastructure TERRAFORM ENCAPSULATION STRATS Object Oriented Development Operations MULTI-GAME MODULAR SERVICE DESIGN A different kind of scaling 1 2 3
  7. 7. AMAZON ECS. TERRAFORM. VIDEO GAMES. WHAT TO EXPECT ECS CONSOLIDATION WINS & LESSONS Adapting existing deployments & infrastructure TERRAFORM ENCAPSULATION STRATS Object Oriented Development Operations MULTI-GAME MODULAR SERVICE DESIGN A different kind of scaling 1 2 3
  8. 8. 7.5MILLION PEAK CONCURRENT PLAYERS 100MILLION MONTHLY ACTIVE PLAYERS MORE THAN 27MILLION DAILY ACTIVE PLAYERS MORE THAN MORE THAN 2016 LEAGUE OF LEGENDS STATS
  9. 9. DATA PRODUCTS & SERVICES OUR MISSION Empower teams at Riot to make timely, data-informed products by maintaining a scalable and reliable data platform AWS re:Invent 2015 | (GAM303) Riot Games: Migrating Mountains of Big Data to AWS Sean Maloney
  10. 10. PROBLEM
  11. 11. PROBLEM SCALING TOTAL OWNERSHIP ON AWS Total ownership We want to empower developers to: • Provision their own infrastructure • Execute their own deployments • Monitor their own metrics
  12. 12. PROBLEM SCALING TOTAL OWNERSHIP ON AWS Resource attribution • Who owns these EBS volumes? • What applications depend on these security groups? • Can these AMIs be deleted?
  13. 13. PROBLEM SCALING TOTAL OWNERSHIP ON AWS Security
  14. 14. PROBLEM SCALING TOTAL OWNERSHIP ON AWS Security • Auditing is important, but it’s reactive • Operational time sink Security Monkey AWS Trusted Advisor
  15. 15. PROBLEM SCALING TOTAL OWNERSHIP ON AWS Enforcing conventions • Common syntax for tags • Simplify reservations • Standardize networks AWS re:Invent 2014 | (GAM304) How Riot Games re:Invented Their AWS Model Jonathan McCaffrey Marty Chong
  16. 16. PROBLEM SCALING TOTAL OWNERSHIP ON AWS Enforcing conventions • Common syntax for tags • Simplify reservations • Standardize networks AWS re:Invent 2014 | (GAM304) How Riot Games re:Invented Their AWS Model Jonathan McCaffrey Marty Chong
  17. 17. PROBLEM SCALING TOTAL OWNERSHIP ON AWS Amazon EC2 Container Service Terraform
  18. 18. CONTAINERS
  19. 19. CONTAINERS STANDARDIZED APPLICATION UNITS… ON AWS • Dockerfiles capture application dependencies
  20. 20. CONTAINERS STANDARDIZED APPLICATION UNITS… ON AWS • Dockerfiles capture application dependencies • Common use cases have great community support
  21. 21. CONTAINERS STANDARDIZED APPLICATION UNITS… ON AWS • Dockerfiles capture application dependencies • Common use cases have great community support • Profit from our own engineering community
  22. 22. CONTAINERS STANDARDIZED APPLICATION UNITS… ON AWS
  23. 23. CONTAINERS STANDARDIZED APPLICATION UNITS… ON AWS Embrace the abstraction • Plan for failure at all levels • Avoid manual intervention whenever possible • It’s ephemeral all the way down
  24. 24. CONTAINERS STANDARDIZED APPLICATION UNITS… ON AWS Scheduling is hard We need to: • Quickly and “fairly” run tasks • Prevent resource conflicts • Provide reasonable fault tolerance
  25. 25. CONTAINERS STANDARDIZED APPLICATION UNITS… ON AWS We need AWS hardware • Enable total ownership of the AWS resources backing our containers • Avoid the security, resource attribution, and convention degradation pitfalls
  26. 26. HISTORY
  27. 27. E CONTAINERS STANDARDIZED APPLICATION UNITS… ON AWS COS First iteration (2013) • Traditional ASGs • AMIs configured to launch a single Docker container • Managed by Netflix’s Asgard
  28. 28. E CONTAINERS STANDARDIZED APPLICATION UNITS… ON AWS COS First iteration (2013) • Easy to adopt • Familiar AWS concepts
  29. 29. E CONTAINERS STANDARDIZED APPLICATION UNITS… ON AWS COS First iteration (2013) • No increased resource utilization • No elasticity improvements
  30. 30. CONTAINERS STANDARDIZED APPLICATION UNITS… ON AWS Second iteration (2013) • Third party container orchestration • Production scale had unexpected challenges
  31. 31. CONTAINERS STANDARDIZED APPLICATION UNITS… ON ECS! Third iteration: Amazon EC2 Container Service • ECS AMI provides all necessary software • Designed with AWS integration in mind • Free!
  32. 32. AMAZON EC2 CONTAINER SERVICE KEY FEATURE TIMELINE Nov 2014 ECS ANNOUNCED Re:Invent 2014
  33. 33. AMAZON EC2 CONTAINER SERVICE KEY FEATURE TIMELINE Nov 2014 April 2015 Dec 2015 August 2016 ECS ANNOUNCED Re:Invent 2014 ECS GENERALLY AVAILABLE ECR & NEW REGIONS ECS becomes available in the final missing region we need (Frankfurt) for our globally deployed applications
  34. 34. AMAZON EC2 CONTAINER SERVICE KEY FEATURE TIMELINE Nov 2014 April 2015 Dec 2015 May 2016 July 2016 August 2016 ECS ANNOUNCED re:Invent 2014 ECS GENERALLY AVAILABLE ECR & NEW REGIONS ECS becomes available in the final missing region we need (Frankfurt) for our globally deployed applications SERVICE SCALING Automatic task count scaling based on CloudWatch metrics introduced
  35. 35. AMAZON EC2 CONTAINER SERVICE KEY FEATURE TIMELINE Nov 2014 April 2015 Dec 2015 May 2016 July 2016 August 2016 ECS ANNOUNCED Re:Invent 2014 ECS GENERALLY AVAILABLE ECR & NEW REGIONS ECS becomes available in the final missing region we need (Frankfurt) for our globally deployed applications SERVICE SCALING Automatic task count scaling based on CloudWatch metrics introduced TASK SPECIFIC IAM ROLES Unlocked a lot of cluster sharing potential
  36. 36. AMAZON EC2 CONTAINER SERVICE KEY FEATURE TIMELINE Nov 2014 April 2015 Dec 2015 May 2016 July 2016 August 2016 ECS ANNOUNCED re:Invent 2014 ECS GENERALLY AVAILABLE ECR & NEW REGIONS ECS becomes available in the final missing region we need (Frankfurt) for our globally deployed applications SERVICE SCALING Automatic task count scaling based on CloudWatch metrics introduced TASK SPECIFIC IAM ROLES Unlocked a lot of cluster sharing potential APPLICATION LOAD BALANCERS Several key improvements for ECS
  37. 37. BEYOND ECS INFRASTRUCTURE AS CODE • At scale, orchestrating infrastructure in a consistent, reproducible way is key
  38. 38. BEYOND ECS INFRASTRUCTURE AS CODE • At scale, orchestrating infrastructure in a consistent, reproducible way is key Total ownership
  39. 39. TERRAFORM
  40. 40. TERRAFORM INFRASTRUCTURE AS OBJECT-ORIENTED CODE • Intelligent & flexible flow control
  41. 41. TERRAFORM INFRASTRUCTURE AS OBJECT-ORIENTED CODE • Intelligent & flexible flow control • Configuration drift managed
  42. 42. TERRAFORM INFRASTRUCTURE AS OBJECT-ORIENTED CODE • Intelligent & flexible flow control • Configuration drift managed • AWS resources documented
  43. 43. TERRAFORM INFRASTRUCTURE AS OBJECT-ORIENTED CODE • Intelligent & flexible flow control • Configuration drift managed • AWS resources documented • Share common use cases
  44. 44. VPC NAT Gateway Route 53 Hosted Zone Route Tables VPN Gateway VPC Internet Gateway Application Subnets Tools Instances Instances Instances Availability Zone C Availability Zone B Availability Zone A TERRAFORM INFRASTRUCTURE AS OBJECT-ORIENTED CODE
  45. 45. TERRAFORM INFRASTRUCTURE AS OBJECT-ORIENTED CODE Simplified infrastructure definitions
  46. 46. CASE STUDIES
  47. 47. ECS CLUSTER TERRAFORM BUILDING BLOCKS VPC NAT Gateway Route 53 Hosted Zone Route Tables VPN Gateway VPC Internet Gateway Application Subnets Tools Instances Instances Instances Availability Zone C Availability Zone B Availability Zone A ECS Cluster Auto Scaling Group Security Group Security Group Security Group Instance Instance Instance Instance Instance Instance Launch Configuration User Data IAM Role CloudWatch Alarms
  48. 48. ECS CLUSTER TERRAFORM BUILDING BLOCKS Module 1 Module 2 Module 3
  49. 49. ECS CLUSTER TERRAFORM BUILDING BLOCKS
  50. 50. MICROSERVICES WITHOUT SERVICE ENDPOINTS ECS Cluster Autoscaling Group Security Group Security Group Security Group Instance Instance Instance Instance Instance Instance Launch Configuration User Data IAM Role CloudWatch Alarm ECS Service Task Definition CloudWatch Alarms IAM Role
  51. 51. MICROSERVICES WITH SERVICE ENDPOINTS ECS Cluster ECS Service Task Definition CloudWatch Alarms IAM Role Application Load Balancer Monitoring CloudWatch Alarms SNS Topics Security Group Security Group Listeners Target Groups Route 53
  52. 52. PERSISTENT DATA LOSE ECS HOSTS WITHOUT LOSING DATA ECS Cluster ECS Service Application Load Balancer Monitoring CloudWatch Alarms SNS Topics Security Group Security Group Listeners Target Groups Route 53 Attachment Group EBS EBS EBS Elastic Network Interface Elastic IP
  53. 53. LESSONS LEARNED
  54. 54. LESSONS WHAT WORKED FOR US • Socialize your templates & take advantage of the community! • Terraform community modules • Terraform AWS blog
  55. 55. LESSONS WHAT WORKED FOR US • Break apart your stacks ECS Cluster
  56. 56. LESSONS WHAT WORKED FOR US • Break apart your stacks, but don’t overdo it ECS Service ALB SNS Topics Route 53 IAM Role Task Definition CloudWatch Alarms
  57. 57. LESSONS WHAT WORKED FOR US • Break apart your stacks, but don’t overdo it • Use modules to reduce code duplication CloudWatch Alarms SNS Topics ECS Cluster ALB
  58. 58. LESSONS WHAT WORKED FOR US • Use remote state files
  59. 59. LESSONS WHAT WORKED FOR US • Be liberal with your cluster provisioning • Don’t risk resource contention in production • With good orchestration, additional clusters != additional operational overhead
  60. 60. LESSONS WHAT WORKED FOR US • Tag everything all of the time • Keep your tags organized in your Terraform templates • Have top level variables that get applied to every resource • Create a tag for every dimension that is useful to your business
  61. 61. LESSONS WHAT WORKED FOR US • Centralize your logs or Amazon CloudWatch Logs
  62. 62. LESSONS WHAT WORKED FOR US • Stay up to date with release blogs and application updates • ECS updates on the AWS blog • Terraform’s open source GitHub repository changelog
  63. 63. LESSONS WHAT WORKED FOR US • Capture your AMI generation process
  64. 64. LESSONS WHAT WORKED FOR US • Profile your memory requirements, monitor for scheduling issues
  65. 65. LESSONS WHAT WORKED FOR US • Take advantage of flow control in Terraform • Use the Terraform graph tool to visualize your dependencies
  66. 66. RIOT ENGINEERING BLOG THESE PROBLEMS AND MORE https://engineering.riotgames.com/ Riot engineering How we use data http://na.leagueoflegends.com/en/tag/insights
  67. 67. JOIN US! ENJOY NOMS. DRINK DRINKS. PLAY GAMES. Tuesday November 29th, 2016 6:30 – 10:30PM Paiza Lounge The Venetian 36th Floor
  68. 68. Thank you!
  69. 69. Remember to complete your evaluations!

×