Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
RIOT GAMESUSING HADOOP TO IMPROVE THE PLAYER           EXPERIENCE     (AND TRACK HOW MANY TEEMOS DIE PER DAY)     BARRY LI...
1234    INTRODUCTION567
1   INTRO            ABOUT THE SPEAKERS234567
1   INTRO            THIS PRESENTATION IS ABOUT…2            • A quick history of our data warehouse3           • Our high...
1   INTRO                    WHO is RIOT GAMES?2      • Developer and publisher of League of Legends3     • Founded 2006 b...
1                                   LEAGUE OF LEGENDS    INTRO2               12 MILLION                                  ...
1234    A QUICK HISTORY567
1    INITIAL LAUNCH / SCRAPPY START UP PHASE2   HISTORY        START-UP3             •   Had a single, dedicated MySQL ins...
1                      AND THEN – CRAZY GROWTH!!!!                                      CRAZY2   HISTORY            START-...
1                           THE BREAKING POINT                                  CRAZY           BREAKING2   HISTORY      S...
1                  INTRODUCTION OF HADOOP                                   CRAZY         BREAKING2   HISTORY       START-...
1             HIGH LEVEL ARCHITECTURE, JUNE 2012                                                                       Bus...
1          BUT, THIS WASN’T GOOD ENOUGH2   HISTORY              • We needed to improve on many levels                –   S...
1          BUT, THIS WASN’T GOOD ENOUGH2   HISTORY              • We needed to improve on many levels                –   S...
1234    CONTEXT567
1     LEAGUE OF LEGENDS GAMEPLAY BASICS23   CONTEXT4567
123   CONTEXT4567
123   CONTEXT4567
1                        CLIENT FOOTPRINT2             • As a AAA video game, a significant portion of our software       ...
1              “THE PATCHER”23   CONTEXT4567
1              “LOBBY CLIENT”23   CONTEXT4567
1              “GAME CLIENT”23   CONTEXT4567
1234    GAME CLIENT5   PERFORMANCE67
1              CHALLENGE: THE GAME IS ALIVE!2              • The game is a living, breathing service that’s always in moti...
1              CHALLENGE: WE’RE GLOBAL!23    USECASE4      #1567
1                CHALLENGE: PC VARIABILITY2              • Hardware and OS profiles are significantly different even withi...
1              CHALLENGE: GRAPHICS SETTINGS23    USECASE4      #1567
1              CHALLENGE: CLIENT-SIDE LOGIC23    USECASE4      #1567
1        IMPROVING THE PLAYER EXPERIENCE2              • We need to gather information across all of these dimensions     ...
1              REACTING QUICKLY23    USECASE4      #1567
1              GAME LOAD SCREEN23    USECASE4      #1567
1              IMPROVING LOAD TIMES23    USECASE4      #1567
1       OPTIMIZING DESIGN AND PERFORMANCE23    USECASE4      #1567
1                 HOW DID WE SOLVE THIS?2              • We have an ARMY of TEEMOs watching players’ machines             ...
1                 HONU PIPELINE2      Generate     Collect    Organize3    Honu                   Honu         Honu    Cli...
1                 HONU-CLIENT SDK2345                GAME_CLIENT_STATS                timestamp      source           app ...
1                     HONU-TOOLS2    • DradisTestUI: WEB UI to send a message to Dradis3     directly without any coding  ...
1                HONU-COLLECTORS2   • Each collector:      – Collect events from multiple                     SQS3       c...
1          HONU-DEMUX2    SQS                   S33                           S3                                 S S3     ...
1          HONU-DEMUX                                 S32    SQS    Demux34                  S3                           ...
1    JUNE 2012                         WWW                               Client2               Mobile34567
1             WWW    Mobile         Client234567
1234    MATCHMAKING567
1                           MATCHMAKING2         • One of the most important features outside of gameplay          • Like ...
1              MATCHING THE RIGHT PLAYERS234    USECASE5      #267
1                IMPROVING THE EXPERIENCE2             • We want to ensure that all players are having the best           ...
1          HOW DO WE ACCOMPLISH THAT?234    USECASE5      #267
12    Tools & Business Process   Tools & Business Process Tools & Business Process34             Audit    Plat            ...
1     Audit    Plat           Audit   Plat                 Audit    Plat      LoL                     LoL                 ...
1              FLOWS234    USECASE5      #267
1             WWW    Mobile         Client234567
1234    Data Flows567
1                                  Data Flows2                  Audit    Plat       Audit   Plat    Audit   Plat          ...
1                               Data Flows2                                  Audit   Plat                                 ...
1                         WWW                Mobile         Client23456   Workflows7
1        Enterprise Workflow Management Using Oozie                       @ Riot Games23                    Matt Goeke and...
1234    THE FUTURE567
1             DECREASE TEEMO DEATHS?!?!?!23456      THE7   FUTURE
1         CONTINUE INCREASING VELOCITY                                    June 2012             February 20132        MySQ...
1                   OUR IMMEDIATE GOALS2             •   Self-Service reporting             •   Metadata Management Servic...
1             CHALLENGE: MAKE IT GLOBAL2        • Data centers across the globe since latency has huge effect on          ...
1             CHALLENGE: WE HAVE BIG DATA                   STRUCTURED DATA2                   500G DAILY                 ...
1               OUR AUDACIOUS GOALS2             Build a world-class data and analytics organization             • Deeply ...
1                  SHAMELESS HIRING PLUG2            • Like most everybody else at this conference… we’re               hi...
1             SHAMELESS HIRING PLUG23456      THE                   And yes, you can play games at work.7   FUTURE        ...
THANK YOU!QUESTIONS?     BARRY LIVINGSTON        &      JEROME BOULON blivingston@riotgames.com       jboulon@riotgames.com
Upcoming SlideShare
Loading in …5
×

Honu/Big Data @ Riot Games

2,902 views

Published on

Big Data @ Riot Games – Using Honu & Hadoop to Understand Player Experience.

Published in: Technology

Honu/Big Data @ Riot Games

  1. 1. RIOT GAMESUSING HADOOP TO IMPROVE THE PLAYER EXPERIENCE (AND TRACK HOW MANY TEEMOS DIE PER DAY) BARRY LIVINGSTON & JEROME BOULON February 2013
  2. 2. 1234 INTRODUCTION567
  3. 3. 1 INTRO ABOUT THE SPEAKERS234567
  4. 4. 1 INTRO THIS PRESENTATION IS ABOUT…2 • A quick history of our data warehouse3 • Our high level architecture • Player experience use cases4 • How Hadoop has enabled these use cases • Changes we have made to our architecture5 to facilitate deeper insight at velocity • Where we’re headed67
  5. 5. 1 INTRO WHO is RIOT GAMES?2 • Developer and publisher of League of Legends3 • Founded 2006 by gamers for gamers • Player experience focused4 – Needless to say, data is pretty important to understanding the player experience!567
  6. 6. 1 LEAGUE OF LEGENDS INTRO2 12 MILLION 70 MILLION DAILY ACTIVE PLAYERS REGISTERED PLAYERS345 3 MILLION 32 MILLION PEAK CONCURRENT MONTHLY ACTIVE PLAYERS PLAYERS67 Numbers based on Riot Games data published October 2012.
  7. 7. 1234 A QUICK HISTORY567
  8. 8. 1 INITIAL LAUNCH / SCRAPPY START UP PHASE2 HISTORY START-UP3 • Had a single, dedicated MySQL instance for the DW • Data was ETL’d from production slaves into this instance4 • Queries were run in MySQL • Reporting was done in Excel • All ETLs, Queries, and Reporting were done by one person!56 This worked great!7
  9. 9. 1 AND THEN – CRAZY GROWTH!!!! CRAZY2 HISTORY START-UP GROWTH3 # unique logins TOTAL ACTIVE PLAYERS4 June 2012 4.2M5 NOV. 201167 time
  10. 10. 1 THE BREAKING POINT CRAZY BREAKING2 HISTORY START-UP GROWTH POINT3 • Data Warehouse reached a breaking point – 24 hours of data took 24.5 hours to ETL • We couldn’t handle…4 – multiple environments in a vertical MySQL instance – a single environment in a vertical MySQL instance5 • We needed to change!67
  11. 11. 1 INTRODUCTION OF HADOOP CRAZY BREAKING2 HISTORY START-UP GROWTH POINT HADOOP!3 • Hadoop has a number of great qualities! – Cost Effective4 – Scalable – Open Source5 – We could execute quickly!67
  12. 12. 1 HIGH LEVEL ARCHITECTURE, JUNE 2012 Business2 HISTORY Audit Plat Analyst LoL Tableau3 NORTH AMERICA Pentaho Audit Plat + Custom ETL Hive Data Pentaho MySQL4 + Warehouse LoL Sqoop EUROPE5 Audit Plat LoL6 Analysts KOREA7
  13. 13. 1 BUT, THIS WASN’T GOOD ENOUGH2 HISTORY • We needed to improve on many levels – Shorten time to insight3 – Increase depth of insight – Enable data analysis for client-side features4 – Flexible auditing framework – Log ingestion and analysis5 – International data infrastructure67
  14. 14. 1 BUT, THIS WASN’T GOOD ENOUGH2 HISTORY • We needed to improve on many levels – Shorten time to insight3 – Increase depth of insight – Enable data analysis for client-side features4 – Flexible auditing framework – Log ingestion and analysis5 – International data infrastructure67
  15. 15. 1234 CONTEXT567
  16. 16. 1 LEAGUE OF LEGENDS GAMEPLAY BASICS23 CONTEXT4567
  17. 17. 123 CONTEXT4567
  18. 18. 123 CONTEXT4567
  19. 19. 1 CLIENT FOOTPRINT2 • As a AAA video game, a significant portion of our software runs directly on players’ machines – High performance graphics3 CONTEXT – Responsiveness4 • There is logic in these components that is ONLY exercised on the client-side5 • Understanding the performance, reliability, and stability of these features is paramount to improving the player6 experience!7
  20. 20. 1 “THE PATCHER”23 CONTEXT4567
  21. 21. 1 “LOBBY CLIENT”23 CONTEXT4567
  22. 22. 1 “GAME CLIENT”23 CONTEXT4567
  23. 23. 1234 GAME CLIENT5 PERFORMANCE67
  24. 24. 1 CHALLENGE: THE GAME IS ALIVE!2 • The game is a living, breathing service that’s always in motion3 • Updated every 2-3 weeks USECASE4 #1 – New champions – New items – New effects/particles5 – Changes in environment – Changes in design and design balance67
  25. 25. 1 CHALLENGE: WE’RE GLOBAL!23 USECASE4 #1567
  26. 26. 1 CHALLENGE: PC VARIABILITY2 • Hardware and OS profiles are significantly different even within regions!3 • OS and Patch Level • CPU4 USECASE #1 • Memory • Video Card • Video Card Memory5 • Drivers!67
  27. 27. 1 CHALLENGE: GRAPHICS SETTINGS23 USECASE4 #1567
  28. 28. 1 CHALLENGE: CLIENT-SIDE LOGIC23 USECASE4 #1567
  29. 29. 1 IMPROVING THE PLAYER EXPERIENCE2 • We need to gather information across all of these dimensions in order to UNDERSTAND the player experience3 • We use this information to:4 USECASE – React quickly to changes #1 – Optimize performance – Optimize designs5 – Improve our testing • Like Creating our Compatibility Testing Lab!67
  30. 30. 1 REACTING QUICKLY23 USECASE4 #1567
  31. 31. 1 GAME LOAD SCREEN23 USECASE4 #1567
  32. 32. 1 IMPROVING LOAD TIMES23 USECASE4 #1567
  33. 33. 1 OPTIMIZING DESIGN AND PERFORMANCE23 USECASE4 #1567
  34. 34. 1 HOW DID WE SOLVE THIS?2 • We have an ARMY of TEEMOs watching players’ machines through their telescopes?!?!?!3 – Not really, but we DID consider it! USECASE4 #1567
  35. 35. 1 HONU PIPELINE2 Generate Collect Organize3 Honu Honu Honu Client Collectors Demux4 SDK567
  36. 36. 1 HONU-CLIENT SDK2345 GAME_CLIENT_STATS timestamp source app pingAvg serverId system6 1234567890 99.123.456.78 game_client 220.9542 12.345.678.90 Intel64 … Select avg(f[‘pingAVG’]) from game_client_stats group by f[‘serverId’];7
  37. 37. 1 HONU-TOOLS2 • DradisTestUI: WEB UI to send a message to Dradis3 directly without any coding • EchoService: WEB UI to easily and immediately4 visualize the data that has been sent to Honu Collectors567
  38. 38. 1 HONU-COLLECTORS2 • Each collector: – Collect events from multiple SQS3 clients (Thrift/NIO) – Save all events to one S34 compressed file locally – Upload that file every XX minutes to S35 – Send a message to SQS for Honu Collectors Demux67
  39. 39. 1 HONU-DEMUX2 SQS S33 S3 S S3 34 Standalone SS tandalone tandalone SDemux tandalone Demux Standalone Demux Demux S3 S35 Demux HIVE MERGE67
  40. 40. 1 HONU-DEMUX S32 SQS Demux34 S3 S3 S35 HIVE MERGE SS tandalone tandalone6 Standalone Demux Standalone Demux Demux S3 S3 Demux7
  41. 41. 1 JUNE 2012 WWW Client2 Mobile34567
  42. 42. 1 WWW Mobile Client234567
  43. 43. 1234 MATCHMAKING567
  44. 44. 1 MATCHMAKING2 • One of the most important features outside of gameplay • Like a dating service, the objective is to match people up;3 • Number of different queues that players can line up in, depending on the type of match they’re looking for4 USECASE5 #26 Critical that this system is balanced balanced and able to create good matches quickly7
  45. 45. 1 MATCHING THE RIGHT PLAYERS234 USECASE5 #267
  46. 46. 1 IMPROVING THE EXPERIENCE2 • We want to ensure that all players are having the best possible experience getting the matches they want – This is VERY challenging!3 – We’re obviously always studying this and trying to improve it4 • Recently, we’ve started combining client data with data we have about many other dimensions – Queue times USECASE5 #2 – Match quality – Player Skill/Matchmaking Rating6 • Obviously, we hope this will lead us to improvements we haven’t identified as of yet7
  47. 47. 1 HOW DO WE ACCOMPLISH THAT?234 USECASE5 #267
  48. 48. 12 Tools & Business Process Tools & Business Process Tools & Business Process34 Audit Plat Audit Plat Audit Plat LoL LoL LoL5 NORTH AMERICA EUROPE KOREA6 MySQL7 JUNE 2012
  49. 49. 1 Audit Plat Audit Plat Audit Plat LoL LoL LoL2 NORTH AMERICA EUROPE KOREA34567 Dashboards Tools & Business Process
  50. 50. 1 FLOWS234 USECASE5 #267
  51. 51. 1 WWW Mobile Client234567
  52. 52. 1234 Data Flows567
  53. 53. 1 Data Flows2 Audit Plat Audit Plat Audit Plat LoL LoL LoL3 NORTH AMERICA EUROPE KOREA456 Workflows7
  54. 54. 1 Data Flows2 Audit Plat Audit Plat LoL3 Audit Plat LoL EUROPE LoL KOREA NORTH AMERICA456 Workflows7
  55. 55. 1 WWW Mobile Client23456 Workflows7
  56. 56. 1 Enterprise Workflow Management Using Oozie @ Riot Games23 Matt Goeke and4 Barry Livingston Tomorrow : 11:40 12:3056 Workflows7
  57. 57. 1234 THE FUTURE567
  58. 58. 1 DECREASE TEEMO DEATHS?!?!?!23456 THE7 FUTURE
  59. 59. 1 CONTINUE INCREASING VELOCITY June 2012 February 20132 MySQL tables 180 1200 Pipeline Events/day 0 2.5+ Billion3 Workflows Cronjob + Pentaho Oozie Environment Datacenter DC + AWS SLA 1 day 2 hours4 Event tracking • 2+ weeks (DB • 10 minutes update) • Dependencies: DBA • Self-Service5 teams + ETL teams + Tools teams • Downtime (3h min.) • No downtime6 THE7 FUTURE
  60. 60. 1 OUR IMMEDIATE GOALS2 • Self-Service reporting • Metadata Management Service3 • Real-time aggregation pipeline • Real-time slicing/dicing for non-critical data4 • Log ingestion and analysis • International data infrastructure56 THE7 FUTURE
  61. 61. 1 CHALLENGE: MAKE IT GLOBAL2 • Data centers across the globe since latency has huge effect on gameplay  log data scattered around the world3 • Large presence in Asia -- some areas (e.g., PH) have bandwidth challenges or bandwidth is expensive456 THE7 FUTURE
  62. 62. 1 CHALLENGE: WE HAVE BIG DATA STRUCTURED DATA2 500G DAILY APPLICATION AND OPERATIONAL LOGS3 4.5TB DAILY4 RIOT YOUTUBE CHANNEL 3MM SUBSCRIBERS 448+MM VIEWS5 + chat logs + detailed gameplay event tracking6 + so on…. THE7 FUTURE
  63. 63. 1 OUR AUDACIOUS GOALS2 Build a world-class data and analytics organization • Deeply understand players across the globe • Apply that understanding to improve games for players3 • Deeply understand our entire ecosystem, including social media4 Have ability to identify, understand and react to meaningful trends in real time5 Have deep, real-time understanding of our systems from player experience and operational standpoints6 THE7 FUTURE
  64. 64. 1 SHAMELESS HIRING PLUG2 • Like most everybody else at this conference… we’re hiring!3 • The Riot Manifesto Player experience first4 Challenge convention Focus on talent and team5 Take play seriously Stay hungry, stay humble6 THE7 FUTURE
  65. 65. 1 SHAMELESS HIRING PLUG23456 THE And yes, you can play games at work.7 FUTURE It’s encouraged!
  66. 66. THANK YOU!QUESTIONS? BARRY LIVINGSTON & JEROME BOULON blivingston@riotgames.com jboulon@riotgames.com

×