Successfully reported this slideshow.
Your SlideShare is downloading. ×

Honu/Big Data @ Riot Games

More Related Content

Similar to Honu/Big Data @ Riot Games

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Honu/Big Data @ Riot Games

  1. 1. RIOT GAMES USING HADOOP TO IMPROVE THE PLAYER EXPERIENCE (AND TRACK HOW MANY TEEMOS DIE PER DAY) BARRY LIVINGSTON & JEROME BOULON February 2013
  2. 2. 1 2 3 4 INTRODUCTION 5 6 7
  3. 3. 1 INTRO ABOUT THE SPEAKERS 2 3 4 5 6 7
  4. 4. 1 INTRO THIS PRESENTATION IS ABOUT… 2 • A quick history of our data warehouse 3 • Our high level architecture • Player experience use cases 4 • How Hadoop has enabled these use cases • Changes we have made to our architecture 5 to facilitate deeper insight at velocity • Where we’re headed 6 7
  5. 5. 1 INTRO WHO is RIOT GAMES? 2 • Developer and publisher of League of Legends 3 • Founded 2006 by gamers for gamers • Player experience focused 4 – Needless to say, data is pretty important to understanding the player experience! 5 6 7
  6. 6. 1 LEAGUE OF LEGENDS INTRO 2 12 MILLION 70 MILLION DAILY ACTIVE PLAYERS REGISTERED PLAYERS 3 4 5 3 MILLION 32 MILLION PEAK CONCURRENT MONTHLY ACTIVE PLAYERS PLAYERS 6 7 Numbers based on Riot Games data published October 2012.
  7. 7. 1 2 3 4 A QUICK HISTORY 5 6 7
  8. 8. 1 INITIAL LAUNCH / SCRAPPY START UP PHASE 2 HISTORY START-UP 3 • Had a single, dedicated MySQL instance for the DW • Data was ETL’d from production slaves into this instance 4 • Queries were run in MySQL • Reporting was done in Excel • All ETLs, Queries, and Reporting were done by one person! 5 6 This worked great! 7
  9. 9. 1 AND THEN – CRAZY GROWTH!!!! CRAZY 2 HISTORY START-UP GROWTH 3 # unique logins TOTAL ACTIVE PLAYERS 4 June 2012 4.2M 5 NOV. 2011 6 7 time
  10. 10. 1 THE BREAKING POINT CRAZY BREAKING 2 HISTORY START-UP GROWTH POINT 3 • Data Warehouse reached a breaking point – 24 hours of data took 24.5 hours to ETL • We couldn’t handle… 4 – multiple environments in a vertical MySQL instance – a single environment in a vertical MySQL instance 5 • We needed to change! 6 7
  11. 11. 1 INTRODUCTION OF HADOOP CRAZY BREAKING 2 HISTORY START-UP GROWTH POINT HADOOP! 3 • Hadoop has a number of great qualities! – Cost Effective 4 – Scalable – Open Source 5 – We could execute quickly! 6 7
  12. 12. 1 HIGH LEVEL ARCHITECTURE, JUNE 2012 Business 2 HISTORY Audit Plat Analyst LoL Tableau 3 NORTH AMERICA Pentaho Audit Plat + Custom ETL Hive Data Pentaho MySQL 4 + Warehouse LoL Sqoop EUROPE 5 Audit Plat LoL 6 Analysts KOREA 7
  13. 13. 1 BUT, THIS WASN’T GOOD ENOUGH 2 HISTORY • We needed to improve on many levels – Shorten time to insight 3 – Increase depth of insight – Enable data analysis for client-side features 4 – Flexible auditing framework – Log ingestion and analysis 5 – International data infrastructure 6 7
  14. 14. 1 BUT, THIS WASN’T GOOD ENOUGH 2 HISTORY • We needed to improve on many levels – Shorten time to insight 3 – Increase depth of insight – Enable data analysis for client-side features 4 – Flexible auditing framework – Log ingestion and analysis 5 – International data infrastructure 6 7
  15. 15. 1 2 3 4 CONTEXT 5 6 7
  16. 16. 1 LEAGUE OF LEGENDS GAMEPLAY BASICS 2 3 CONTEXT 4 5 6 7
  17. 17. 1 2 3 CONTEXT 4 5 6 7
  18. 18. 1 2 3 CONTEXT 4 5 6 7
  19. 19. 1 CLIENT FOOTPRINT 2 • As a AAA video game, a significant portion of our software runs directly on players’ machines – High performance graphics 3 CONTEXT – Responsiveness 4 • There is logic in these components that is ONLY exercised on the client-side 5 • Understanding the performance, reliability, and stability of these features is paramount to improving the player 6 experience! 7
  20. 20. 1 “THE PATCHER” 2 3 CONTEXT 4 5 6 7
  21. 21. 1 “LOBBY CLIENT” 2 3 CONTEXT 4 5 6 7
  22. 22. 1 “GAME CLIENT” 2 3 CONTEXT 4 5 6 7
  23. 23. 1 2 3 4 GAME CLIENT 5 PERFORMANCE 6 7
  24. 24. 1 CHALLENGE: THE GAME IS ALIVE! 2 • The game is a living, breathing service that’s always in motion 3 • Updated every 2-3 weeks USECASE 4 #1 – New champions – New items – New effects/particles 5 – Changes in environment – Changes in design and design balance 6 7
  25. 25. 1 CHALLENGE: WE’RE GLOBAL! 2 3 USECASE 4 #1 5 6 7
  26. 26. 1 CHALLENGE: PC VARIABILITY 2 • Hardware and OS profiles are significantly different even within regions! 3 • OS and Patch Level • CPU 4 USECASE #1 • Memory • Video Card • Video Card Memory 5 • Drivers! 6 7
  27. 27. 1 CHALLENGE: GRAPHICS SETTINGS 2 3 USECASE 4 #1 5 6 7
  28. 28. 1 CHALLENGE: CLIENT-SIDE LOGIC 2 3 USECASE 4 #1 5 6 7
  29. 29. 1 IMPROVING THE PLAYER EXPERIENCE 2 • We need to gather information across all of these dimensions in order to UNDERSTAND the player experience 3 • We use this information to: 4 USECASE – React quickly to changes #1 – Optimize performance – Optimize designs 5 – Improve our testing • Like Creating our Compatibility Testing Lab! 6 7
  30. 30. 1 REACTING QUICKLY 2 3 USECASE 4 #1 5 6 7
  31. 31. 1 GAME LOAD SCREEN 2 3 USECASE 4 #1 5 6 7
  32. 32. 1 IMPROVING LOAD TIMES 2 3 USECASE 4 #1 5 6 7
  33. 33. 1 OPTIMIZING DESIGN AND PERFORMANCE 2 3 USECASE 4 #1 5 6 7
  34. 34. 1 HOW DID WE SOLVE THIS? 2 • We have an ARMY of TEEMOs watching players’ machines through their telescopes?!?!?! 3 – Not really, but we DID consider it! USECASE 4 #1 5 6 7
  35. 35. 1 HONU PIPELINE 2 Generate Collect Organize 3 Honu Honu Honu Client Collectors Demux 4 SDK 5 6 7
  36. 36. 1 HONU-CLIENT SDK 2 3 4 5 GAME_CLIENT_STATS timestamp source app pingAvg serverId system 6 1234567890 99.123.456.78 game_client 220.9542 12.345.678.90 Intel64 … Select avg(f[‘pingAVG’]) from game_client_stats group by f[‘serverId’]; 7
  37. 37. 1 HONU-TOOLS 2 • DradisTestUI: WEB UI to send a message to Dradis 3 directly without any coding • EchoService: WEB UI to easily and immediately 4 visualize the data that has been sent to Honu Collectors 5 6 7
  38. 38. 1 HONU-COLLECTORS 2 • Each collector: – Collect events from multiple SQS 3 clients (Thrift/NIO) – Save all events to one S3 4 compressed file locally – Upload that file every XX minutes to S3 5 – Send a message to SQS for Honu Collectors Demux 6 7
  39. 39. 1 HONU-DEMUX 2 SQS S3 3 S3 S S3 3 4 Standalone SS tandalone tandalone SDemux tandalone Demux Standalone Demux Demux S3 S3 5 Demux HIVE MERGE 6 7
  40. 40. 1 HONU-DEMUX S3 2 SQS Demux 3 4 S3 S3 S3 5 HIVE MERGE SS tandalone tandalone 6 Standalone Demux Standalone Demux Demux S3 S3 Demux 7
  41. 41. 1 JUNE 2012 WWW Client 2 Mobile 3 4 5 6 7
  42. 42. 1 WWW Mobile Client 2 3 4 5 6 7
  43. 43. 1 2 3 4 MATCHMAKING 5 6 7
  44. 44. 1 MATCHMAKING 2 • One of the most important features outside of gameplay • Like a dating service, the objective is to match people up; 3 • Number of different queues that players can line up in, depending on the type of match they’re looking for 4 USECASE 5 #2 6 Critical that this system is balanced balanced and able to create good matches quickly 7
  45. 45. 1 MATCHING THE RIGHT PLAYERS 2 3 4 USECASE 5 #2 6 7
  46. 46. 1 IMPROVING THE EXPERIENCE 2 • We want to ensure that all players are having the best possible experience getting the matches they want – This is VERY challenging! 3 – We’re obviously always studying this and trying to improve it 4 • Recently, we’ve started combining client data with data we have about many other dimensions – Queue times USECASE 5 #2 – Match quality – Player Skill/Matchmaking Rating 6 • Obviously, we hope this will lead us to improvements we haven’t identified as of yet 7
  47. 47. 1 HOW DO WE ACCOMPLISH THAT? 2 3 4 USECASE 5 #2 6 7
  48. 48. 1 2 Tools & Business Process Tools & Business Process Tools & Business Process 3 4 Audit Plat Audit Plat Audit Plat LoL LoL LoL 5 NORTH AMERICA EUROPE KOREA 6 MySQL 7 JUNE 2012
  49. 49. 1 Audit Plat Audit Plat Audit Plat LoL LoL LoL 2 NORTH AMERICA EUROPE KOREA 3 4 5 6 7 Dashboards Tools & Business Process
  50. 50. 1 FLOWS 2 3 4 USECASE 5 #2 6 7
  51. 51. 1 WWW Mobile Client 2 3 4 5 6 7
  52. 52. 1 2 3 4 Data Flows 5 6 7
  53. 53. 1 Data Flows 2 Audit Plat Audit Plat Audit Plat LoL LoL LoL 3 NORTH AMERICA EUROPE KOREA 4 5 6 Workflows 7
  54. 54. 1 Data Flows 2 Audit Plat Audit Plat LoL 3 Audit Plat LoL EUROPE LoL KOREA NORTH AMERICA 4 5 6 Workflows 7
  55. 55. 1 WWW Mobile Client 2 3 4 5 6 Workflows 7
  56. 56. 1 Enterprise Workflow Management Using Oozie @ Riot Games 2 3 Matt Goeke and 4 Barry Livingston Tomorrow : 11:40 12:30 5 6 Workflows 7
  57. 57. 1 2 3 4 THE FUTURE 5 6 7
  58. 58. 1 DECREASE TEEMO DEATHS?!?!?! 2 3 4 5 6 THE 7 FUTURE
  59. 59. 1 CONTINUE INCREASING VELOCITY June 2012 February 2013 2 MySQL tables 180 1200 Pipeline Events/day 0 2.5+ Billion 3 Workflows Cronjob + Pentaho Oozie Environment Datacenter DC + AWS SLA 1 day 2 hours 4 Event tracking • 2+ weeks (DB • 10 minutes update) • Dependencies: DBA • Self-Service 5 teams + ETL teams + Tools teams • Downtime (3h min.) • No downtime 6 THE 7 FUTURE
  60. 60. 1 OUR IMMEDIATE GOALS 2 • Self-Service reporting • Metadata Management Service 3 • Real-time aggregation pipeline • Real-time slicing/dicing for non-critical data 4 • Log ingestion and analysis • International data infrastructure 5 6 THE 7 FUTURE
  61. 61. 1 CHALLENGE: MAKE IT GLOBAL 2 • Data centers across the globe since latency has huge effect on gameplay  log data scattered around the world 3 • Large presence in Asia -- some areas (e.g., PH) have bandwidth challenges or bandwidth is expensive 4 5 6 THE 7 FUTURE
  62. 62. 1 CHALLENGE: WE HAVE BIG DATA STRUCTURED DATA 2 500G DAILY APPLICATION AND OPERATIONAL LOGS 3 4.5TB DAILY 4 RIOT YOUTUBE CHANNEL 3MM SUBSCRIBERS 448+MM VIEWS 5 + chat logs + detailed gameplay event tracking 6 + so on…. THE 7 FUTURE
  63. 63. 1 OUR AUDACIOUS GOALS 2 Build a world-class data and analytics organization • Deeply understand players across the globe • Apply that understanding to improve games for players 3 • Deeply understand our entire ecosystem, including social media 4 Have ability to identify, understand and react to meaningful trends in real time 5 Have deep, real-time understanding of our systems from player experience and operational standpoints 6 THE 7 FUTURE
  64. 64. 1 SHAMELESS HIRING PLUG 2 • Like most everybody else at this conference… we’re hiring! 3 • The Riot Manifesto Player experience first 4 Challenge convention Focus on talent and team 5 Take play seriously Stay hungry, stay humble 6 THE 7 FUTURE
  65. 65. 1 SHAMELESS HIRING PLUG 2 3 4 5 6 THE And yes, you can play games at work. 7 FUTURE It’s encouraged!
  66. 66. THANK YOU! QUESTIONS? BARRY LIVINGSTON & JEROME BOULON blivingston@riotgames.com jboulon@riotgames.com

Editor's Notes

  • Times where there were 20% month over month growth in a single environment2 environments w/~200K CCU to 16 environments and 1.3million CCU in the space of 12 monthsResources were focused on getting our operational systems to scale along with demand
  • Today, we’re going to focus on a few of these.
  • Before we talk about our first usecase, we need to give you a little bit of context about the game and gameplay (super high level), as well as a quick overview of some of the pieces of the LoL architectureSession Based Team play - basic idea is like “kill the other team’s nexus (base)” – MOBA!If you die, you re-spawn after a certain amount of time (that time grows as the game progresses)Lots of strategy to the game
  • Each player “summons” a Champion that he playsEach champion has very different abilities
  • Each player “summons” a Champion that he playsEach champion has very different abilities
  • “Typical” Hardware profiles differ significantly from region to regionWe want to make sure that all regions have a good experienceWe also want to understand latency from various locations in the world to our installs
  • There are lots of customizable settings around resolution and graphics quality that players can changeAll of these settings have a potential impact on performance
  • So given that we have all of these challenges, what are we doing to improve the
  • So given that we have all of these challenges, what are we doing to improve the

×