Deploying the 'League of Legends' Data Pipeline with Chef (ARC205) | AWS re:Invent 2013

  • 3,539 views
Uploaded on

Over the past year, the data team at Riot Games has been using Chef to both configure instances in Amazon Elastic Compute Cloud (EC2) and build AMIs. With Chef as an integral part of the workflow, …

Over the past year, the data team at Riot Games has been using Chef to both configure instances in Amazon Elastic Compute Cloud (EC2) and build AMIs. With Chef as an integral part of the workflow, we've autoscaled thousands of instances in support of the data pipeline for League of Legends and have found that Chef doesn't always play perfectly in the world of autoscaling groups and ephemeral instances. In this talk, we cover what's worked and what's failed and explain how to best utilize Chef in the world of Amazon Web Services.

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
3,539
On Slideshare
0
From Embeds
0
Number of Embeds
5

Actions

Shares
Downloads
48
Comments
0
Likes
5

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. DEPLOYING THE LEAGUE OF LEGENDS DATA PIPELINE WITH CHEF TROTTER CASHION ENGINEERING MANAGER| NOVEMBER 2013
  • 2. 2009 LAUNCH TEAM ORIENTED ONLINE PC GAME 100+ CHAMPS LIVE PLAYERS VS. LIVE PLAYERS
  • 3. Image  source:  herocomplex.la2mes.com/  
  • 4. WE WORK WITH BIG DATA 7+ BILLION EVENTS PER DAY TESTED @ 70+ BILLION EVENTS PER DAY 100+ TABLES 10+ TABLES 100MM TO 1B ROWS/DAY 7+ PETABYTE GAME EVENT DATASET SEMI-GLOBAL DEPLOYMENT 0 DOWNTIME RUNS IN CLOUD (AWS) + DATACENTER
  • 5. OVERVIEW CORE CONCEPTS AGENDA CHEF SOLO CHEF SERVER GOLDEN IMAGES
  • 6. OVERVIEW BASED ON REAL WORLD USAGE Big Data Team from April 2012 to June 2013 Also incorporates subsequent work with Player Accounts team
  • 7. OVERVIEW WHAT DO WE WANT FROM AUTOMATED INFRASTRUCTURE?
  • 8. OVERVIEW WHAT DO WE WANT TO BE ABLE TO DO FOR PLAYERS?
  • 9. WHAT PLAYERS WANT OVERVIEW TO PLAY LOL ‣  ‣  ‣  Minimize  Unplanned   Down2me   Rolling  Deploys   Expand  Capacity   NEW FEATURES ‣  ‣  Low  Maintenance   Low  Barrier  for  New  Services  
  • 10. OVERVIEW CORE: ALWAYS UP, EASY TO DEPLOY
  • 11. OVERVIEW CORE CONCEPTS AGENDA CHEF SOLO CHEF SERVER GOLDEN IMAGES
  • 12. UNDERLYING COMPONENTS CORE CONCEPTS
  • 13. UNDERLYING COMPONENTS ELB CORE CONCEPTS
  • 14. UNDERLYING COMPONENTS EC2 ELB CORE CONCEPTS
  • 15. UNDERLYING COMPONENTS CORE CONCEPTS EC2 ELB VPC VPC
  • 16. UNDERLYING COMPONENTS CORE CONCEPTS EC2 ELB VPC VPC VGW
  • 17. UNDERLYING COMPONENTS CORE CONCEPTS EC2 ELB VPC VPC VGW VPN DC
  • 18. UNDERLYING COMPONENTS CORE CONCEPTS EC2 ELB VPC VPC VGW S3 DATA STORAGE
  • 19. DATA CENTER VS. CLOUD CORE CONCEPTS
  • 20. DATA CENTER VS. CLOUD CORE CONCEPTS Min. 2W & 3G
  • 21. DATA CENTER VS. CLOUD CORE CONCEPTS DATA CENTER MODEL Min. 2W & 3G
  • 22. DATA CENTER VS. CLOUD CORE CONCEPTS DATA CENTER MODEL W Min. 2W & 3G
  • 23. DATA CENTER VS. CLOUD CORE CONCEPTS DATA CENTER MODEL W W Min. 2W & 3G
  • 24. DATA CENTER VS. CLOUD CORE CONCEPTS DATA CENTER MODEL W W G Min. 2W & 3G
  • 25. DATA CENTER VS. CLOUD CORE CONCEPTS DATA CENTER MODEL W W G G Min. 2W & 3G
  • 26. DATA CENTER VS. CLOUD CORE CONCEPTS DATA CENTER MODEL W G W G G Min. 2W & 3G
  • 27. DATA CENTER VS. CLOUD CORE CONCEPTS DATA CENTER MODEL W G W G G Min. 2W & 3G 85%
  • 28. DATA CENTER VS. CLOUD CORE CONCEPTS DATA CENTER MODEL W G W G G Min. 2W & 3G 85%
  • 29. DATA CENTER VS. CLOUD CORE CONCEPTS DATA CENTER MODEL W W W G G G Min. 2W & 3G 85%
  • 30. DATA CENTER VS. CLOUD CORE CONCEPTS DATA CENTER MODEL W W W G G G Min. 2W & 3G MANUAL manual scaling no Chef required 85%
  • 31. DATA CENTER VS. CLOUD CORE CONCEPTS AUTOSCALING CLOUD DATA CENTER MODEL W W W G G G Min. 2W & 3G MANUAL manual scaling no Chef required 85%
  • 32. DATA CENTER VS. CLOUD CORE CONCEPTS AUTOSCALING CLOUD DATA CENTER MODEL W W W G G G Min. 2W & 3G MANUAL manual scaling no Chef required 85%
  • 33. DATA CENTER VS. CLOUD CORE CONCEPTS AUTOSCALING CLOUD DATA CENTER MODEL W W W W W W W G G G G G G G Min. 2W & 3G MANUAL manual scaling no Chef required 85% G
  • 34. DATA CENTER VS. CLOUD CORE CONCEPTS AUTOSCALING CLOUD DATA CENTER MODEL W W W W W W W G G G G G G G Min. 2W & 3G MANUAL manual scaling no Chef required 85% G
  • 35. DATA CENTER VS. CLOUD CORE CONCEPTS AUTOSCALING CLOUD DATA CENTER MODEL W W W W W W W G G G G G G G Min. 2W & 3G MANUAL manual scaling no Chef required 85% AUTO automatic scaling automated provisioning needed G
  • 36. userdata.sh START AMI OPERATING SYSTEM CORE CONCEPTS
  • 37. userdata.sh CORE CONCEPTS START AMI OPERATING SYSTEM GOAL Publish Hello World •  Instantly •  Automatically
  • 38. userdata.sh CORE CONCEPTS START AMI OPERATING SYSTEM 1st BOOT GOAL Publish Hello World •  Instantly •  Automatically
  • 39. userdata.sh CORE CONCEPTS START AMI OPERATING SYSTEM 1st BOOT Userdata.sh GOAL Publish Hello World •  Instantly •  Automatically
  • 40. userdata.sh CORE CONCEPTS START AMI OPERATING SYSTEM 1st BOOT Userdata.sh GOAL #!/bin/bash yum install apache Publish Hello World •  Instantly •  Automatically wget http://hw/hello_world.html…
  • 41. userdata.sh CORE CONCEPTS START OPERATING SYSTEM Userdata.sh REPLACED W/ AMI 1st BOOT GOAL #!/bin/bash yum install apache wget http://hw/first-boot.json Publish Hello World •  Instantly •  Automatically wget s3://hw/hello_world.html… chef-client
  • 42. IMMUTABLE SERVERS LIMITED TO 2 ACTIONS: 1)  Start & provision server 2)  Kill it CORE CONCEPTS
  • 43. IMMUTABLE SERVERS CORE CONCEPTS LIMITED TO 2 ACTIONS: 1)  Start & provision server 2)  Kill it WITH IMMUTABLE SERVERS, YOU CAN’T SSH INTO BOX & CHANGESTHINGS PREVENT UNEXPECTED CHANGE TO SERVERS
  • 44. IMMUTABLE SERVERS CORE CONCEPTS LIMITED TO 2 ACTIONS: 1)  Start & provision server 2)  Kill it WITH IMMUTABLE SERVERS, YOU CAN’T SSH INTO BOX & CHANGESTHINGS PREVENT UNEXPECTED CHANGE TO SERVERS W1 W1 W1 MUTABLE SERVER: CHANGE EACH BOX TO NEW VERSION W1 W1
  • 45. IMMUTABLE SERVERS CORE CONCEPTS LIMITED TO 2 ACTIONS: 1)  Start & provision server 2)  Kill it WITH IMMUTABLE SERVERS, YOU CAN’T SSH INTO BOX & CHANGESTHINGS PREVENT UNEXPECTED CHANGE TO SERVERS W2 W1 W1 MUTABLE SERVER: CHANGE EACH BOX TO NEW VERSION W1 W1
  • 46. IMMUTABLE SERVERS CORE CONCEPTS LIMITED TO 2 ACTIONS: 1)  Start & provision server 2)  Kill it WITH IMMUTABLE SERVERS, YOU CAN’T SSH INTO BOX & CHANGESTHINGS PREVENT UNEXPECTED CHANGE TO SERVERS W2 W2 W1 MUTABLE SERVER: CHANGE EACH BOX TO NEW VERSION W1 W1
  • 47. IMMUTABLE SERVERS CORE CONCEPTS LIMITED TO 2 ACTIONS: 1)  Start & provision server 2)  Kill it WITH IMMUTABLE SERVERS, YOU CAN’T SSH INTO BOX & CHANGESTHINGS PREVENT UNEXPECTED CHANGE TO SERVERS W2 W2 W2 MUTABLE SERVER: CHANGE EACH BOX TO NEW VERSION W1 W1
  • 48. IMMUTABLE SERVERS CORE CONCEPTS LIMITED TO 2 ACTIONS: 1)  Start & provision server 2)  Kill it WITH IMMUTABLE SERVERS, YOU CAN’T SSH INTO BOX & CHANGESTHINGS PREVENT UNEXPECTED CHANGE TO SERVERS W2 W2 W2 MUTABLE SERVER: CHANGE EACH BOX TO NEW VERSION W2 W1
  • 49. IMMUTABLE SERVERS CORE CONCEPTS LIMITED TO 2 ACTIONS: 1)  Start & provision server 2)  Kill it WITH IMMUTABLE SERVERS, YOU CAN’T SSH INTO BOX & CHANGESTHINGS PREVENT UNEXPECTED CHANGE TO SERVERS W2 W2 W2 MUTABLE SERVER: CHANGE EACH BOX TO NEW VERSION W2 W2
  • 50. IMMUTABLE SERVERS CORE CONCEPTS LIMITED TO 2 ACTIONS: 1)  Start & provision server 2)  Kill it WITH IMMUTABLE SERVERS, YOU CAN’T SSH INTO BOX & CHANGESTHINGS PREVENT UNEXPECTED CHANGE TO SERVERS W1 W1 W1 IMMUTABLE SERVER: FLIP ON BOXES WITH NEW VERSION, SHUT OFF EXISTING BOXES W1 W1
  • 51. IMMUTABLE SERVERS CORE CONCEPTS LIMITED TO 2 ACTIONS: 1)  Start & provision server 2)  Kill it WITH IMMUTABLE SERVERS, YOU CAN’T SSH INTO BOX & CHANGESTHINGS PREVENT UNEXPECTED CHANGE TO SERVERS W1 W1 W1 W2 W2 W2 IMMUTABLE SERVER: FLIP ON BOXES WITH NEW VERSION, SHUT OFF EXISTING BOXES W1 W1 W2 W2
  • 52. IMMUTABLE SERVERS CORE CONCEPTS LIMITED TO 2 ACTIONS: 1)  Start & provision server 2)  Kill it WITH IMMUTABLE SERVERS, YOU PREVENT UNEXPECTED CHANGES TO SERVERS W1 W1 W1 W2 W2 W2 IMMUTABLE SERVER: FLIP ON BOXES WITH NEW VERSION, SHUT OFF EXISTING BOXES W1 W1 W2 W2
  • 53. IMMUTABLE SERVERS LIMITED TO 2 ACTIONS: 1)  Start & provision server 2)  Kill it WITH IMMUTABLE SERVERS, YOU CAN’T SSH INTO BOX & CHANGE THINGS No snowflakes BENEFITS: Easy rollbacks Cleaner deploys CORE CONCEPTS
  • 54. OVERVIEW CORE CONCEPTS AGENDA CHEF SOLO CHEF SERVER GOLDEN IMAGES
  • 55. WHAT’S REQUIRED •  All cookbooks in one tarball •  Somewhere to put the tarball •  Tell instances how to provision with the tarball CHEF SOLO
  • 56. WHAT’S REQUIRED •  All cookbooks in one tarball •  Somewhere to put the tarball •  Tell instances how to provision with the tarball PACKAGING •  Use Berkshelf •  `berkshelf package COOKBOOK_NAME` •  `tar czvf cookbooks-VERSION.tgz ./cookbooks` CHEF SOLO
  • 57. WHAT’S REQUIRED •  All cookbooks in one tarball •  Somewhere to put the tarball •  Tell instances how to provision with the tarball PACKAGING •  Use Berkshelf •  `berkshelf package COOKBOOK_NAME` •  `tar czvf cookbooks-VERSION.tgz ./cookbooks` STORAGE OPTIONS •  S3 •  `s3cmd` makes it easy to upload to S3 •  Internal asset server CHEF SOLO
  • 58. CHEF SOLO WHAT’S REQUIRED PROVISIONING •  All cookbooks in one tarball •  Somewhere to put the tarball •  Tell instances how to provision with the tarball •  userdata.sh •  Get cookbooks off s3 and untar to /var/chef/ cookbooks •  Get first-boot.json off s3 -> /etc/chef/firstboot.json •  Get solo.rb off s3 -> /etc/chef/solo.rb •  Run `chef-solo` PACKAGING •  Use Berkshelf •  `berkshelf package COOKBOOK_NAME` •  `tar czvf cookbooks-VERSION.tgz ./cookbooks` STORAGE OPTIONS •  S3 •  `s3cmd` makes it easy to upload to S3 •  Internal asset server
  • 59. CHEF SOLO: PROs VS CONs CHEF SOLO WHERE IT WORKS WHERE IT FAILS •  Good when your Chef run is fast •  Easy to set up •  No single point of failure •  When Chef runs occasionally fail •  Configuration changes •  Service discovery
  • 60. WHAT PLAYERS WANT TO PLAY LOL ü  Minimize  Unplanned   Down2me   ü  Rolling  Deploys   ü  Expand  Capacity   CHEF SOLO NEW FEATURES q  Low  Maintenance   q  Low  Barrier  for  New  Services  
  • 61. CHEF SOLO RECOMMENDATION: •  Use chef-solo approach to get started •  Stop when you have multiple applications
  • 62. OVERVIEW CORE CONCEPTS AGENDA CHEF SOLO CHEF SERVER AMIs
  • 63. WHAT’S REQUIRED •  Chef Server •  Cookbooks uploaded to Chef Server •  Tell instances how to provision with Chef Server CHEF SERVER
  • 64. WHAT’S REQUIRED •  Chef Server •  Cookbooks uploaded to Chef Server •  Tell instances how to provision with Chef Server RUNNING CHEF SERVER •  Run  your own •  Hosted Chef CHEF SERVER
  • 65. WHAT’S REQUIRED •  Chef Server •  Cookbooks uploaded to Chef Server •  Tell instances how to provision with Chef Server RUNNING CHEF SERVER •  Run your own •  Hosted Chef GETTING COOKBOOKS TO CHEF SERVER •  Use berkshelf •  `berks upload COOKBOOK_NAME` CHEF SERVER
  • 66. WHAT’S REQUIRED PROVISIONING •  Chef Server •  Cookbooks uploaded to Chef Server •  Tell instances how to provision with Chef Server •  userdata.sh •  Fetch validation.pem •  Fetch first-boot.json •  Fetch client.rb •  Run `chef-client …` RUNNING CHEF SERVER •  Run your own •  Hosted Chef GETTING COOKBOOKS TO CHEF SERVER •  Use berkshelf •  `berks upload COOKBOOK_NAME` CHEF SERVER
  • 67. CHEF SERVER: PROs VS CONs CHEF SERVER WHERE IT WORKS WHERE IT FAILS •  Updating feature flag configuration •  Warning: Can affect rolling deploys! •  Single point of failure in Chef Server •  Long Chef runs cause problems •  Can cause problems in large organizations where multiple teams depend on the same cookbook
  • 68. WHAT PLAYERS WANT TO PLAY LOL q  Minimize  Unplanned   Down2me   q  Rolling  Deploys   q  Expand  Capacity   CHEF SERVER NEW FEATURES ü  Low  Maintenance   q  Low  Barrier  for  New  Services  
  • 69. CHEF SERVER RECOMMENDATION: •  Use chef-server when managing feature flags •  Not bad when you have multiple applications to deploy •  Pro tips: •  Use a shutdown script to remove instances from the chef server •  Only use chef in daemon mode if you’re certain only feature flags will change
  • 70. OVERVIEW CORE CONCEPTS AGENDA CHEF SOLO CHEF SERVER GOLDEN IMAGES
  • 71. GOLDEN IMAGES WHAT’S REQUIRED •  Means to create Golden Image •  Configuration management
  • 72. GOLDEN IMAGES WHAT’S REQUIRED •  Means to create Golden Image •  Configuration management Creating Image •  We use Chef-Solo •  Build an RPM •  Cookbook installs RPMs
  • 73. GOLDEN IMAGES WHAT’S REQUIRED Configuration Management •  Means to create Golden Image •  Configuration management •  Archaius (on disk config) •  Chef Creating Image •  We use Chef-Solo •  Build an RPM •  Cookbook installs RPMS
  • 74. CHEF SOLO: PROs VS CONs GOLDEN IMAGES WHERE IT WORKS WHERE IT FAILS •  Rollbacks are perfect and instances always boot •  Boot times are very fast •  Need another method for configuration management •  Need another method for service discovery •  Releasing patches requires entire new build
  • 75. WHAT PLAYERS WANT TO PLAY LOL ü  Minimize  Unplanned   Down2me   ü  Rolling  Deploys   ü  Expand  Capacity   GOLDEN IMAGES NEW FEATURES ü  Low  Maintenance   q  Low  Barrier  for  New  Services  
  • 76. GOLDEN IMAGES RECOMMENDATION: •  Use when auto scaling is a must •  Requires an up-front effort to get going •  Make sure you can afford it •  Pro tip: •  Use the Netflix stack: Asgard, Aminator, and Archaius
  • 77. THANK YOU! QUESTIONS? TROTTER CASHION ENGINEERING MANAGER| NOVEMBER 2013 c