RIOT GAMESUSING HADOOP TO IMPROVE THE PLAYER           EXPERIENCE     (AND TRACK HOW MANY TEEMOS DIE PER DAY)     BARRY LI...
1234    INTRODUCTION567
1   INTRO            ABOUT THE SPEAKERS234567
1   INTRO            THIS PRESENTATION IS ABOUT…2            • A quick history of our data warehouse3           • Our high...
1   INTRO                    WHO is RIOT GAMES?2      • Developer and publisher of League of Legends3     • Founded 2006 b...
1                                   LEAGUE OF LEGENDS    INTRO2               12 MILLION                                  ...
1234    A QUICK HISTORY567
1    INITIAL LAUNCH / SCRAPPY START UP PHASE2   HISTORY        START-UP3             •   Had a single, dedicated MySQL ins...
1                      AND THEN – CRAZY GROWTH!!!!                                      CRAZY2   HISTORY            START-...
1                           THE BREAKING POINT                                  CRAZY           BREAKING2   HISTORY      S...
1                  INTRODUCTION OF HADOOP                                   CRAZY         BREAKING2   HISTORY       START-...
1             HIGH LEVEL ARCHITECTURE, JUNE 2012                                                                       Bus...
1          BUT, THIS WASN’T GOOD ENOUGH2   HISTORY              • We needed to improve on many levels                –   S...
1          BUT, THIS WASN’T GOOD ENOUGH2   HISTORY              • We needed to improve on many levels                –   S...
1234    CONTEXT567
1     LEAGUE OF LEGENDS GAMEPLAY BASICS23   CONTEXT4567
123   CONTEXT4567
123   CONTEXT4567
1                        CLIENT FOOTPRINT2             • As a AAA video game, a significant portion of our software       ...
1              “THE PATCHER”23   CONTEXT4567
1              “LOBBY CLIENT”23   CONTEXT4567
1              “GAME CLIENT”23   CONTEXT4567
1234    GAME CLIENT5   PERFORMANCE67
1              CHALLENGE: THE GAME IS ALIVE!2              • The game is a living, breathing service that’s always in moti...
1              CHALLENGE: WE’RE GLOBAL!23    USECASE4      #1567
1                CHALLENGE: PC VARIABILITY2              • Hardware and OS profiles are significantly different even withi...
1              CHALLENGE: GRAPHICS SETTINGS23    USECASE4      #1567
1              CHALLENGE: CLIENT-SIDE LOGIC23    USECASE4      #1567
1        IMPROVING THE PLAYER EXPERIENCE2              • We need to gather information across all of these dimensions     ...
1              REACTING QUICKLY23    USECASE4      #1567
1              GAME LOAD SCREEN23    USECASE4      #1567
1              IMPROVING LOAD TIMES23    USECASE4      #1567
1       OPTIMIZING DESIGN AND PERFORMANCE23    USECASE4      #1567
1                 HOW DID WE SOLVE THIS?2              • We have an ARMY of TEEMOs watching players’ machines             ...
1                 HONU PIPELINE2      Generate     Collect    Organize3    Honu                   Honu         Honu    Cli...
1                 HONU-CLIENT SDK2345                GAME_CLIENT_STATS                timestamp      source           app ...
1                     HONU-TOOLS2    • DradisTestUI: WEB UI to send a message to Dradis3     directly without any coding  ...
1                HONU-COLLECTORS2   • Each collector:      – Collect events from multiple                     SQS3       c...
1          HONU-DEMUX2    SQS                   S33                           S3                                 S S3     ...
1          HONU-DEMUX                                 S32    SQS    Demux34                  S3                           ...
1    JUNE 2012                         WWW                               Client2               Mobile34567
1             WWW    Mobile         Client234567
1234    MATCHMAKING567
1                           MATCHMAKING2         • One of the most important features outside of gameplay          • Like ...
1              MATCHING THE RIGHT PLAYERS234    USECASE5      #267
1                IMPROVING THE EXPERIENCE2             • We want to ensure that all players are having the best           ...
1          HOW DO WE ACCOMPLISH THAT?234    USECASE5      #267
12    Tools & Business Process   Tools & Business Process Tools & Business Process34             Audit    Plat            ...
1     Audit    Plat           Audit   Plat                 Audit    Plat      LoL                     LoL                 ...
1              FLOWS234    USECASE5      #267
1             WWW    Mobile         Client234567
1234    Data Flows567
1                                  Data Flows2                  Audit    Plat       Audit   Plat    Audit   Plat          ...
1                               Data Flows2                                  Audit   Plat                                 ...
1                         WWW                Mobile         Client23456   Workflows7
1        Enterprise Workflow Management Using Oozie                       @ Riot Games23                    Matt Goeke and...
1234    THE FUTURE567
1             DECREASE TEEMO DEATHS?!?!?!23456      THE7   FUTURE
1         CONTINUE INCREASING VELOCITY                                    June 2012             February 20132        MySQ...
1                   OUR IMMEDIATE GOALS2             •   Self-Service reporting             •   Metadata Management Servic...
1             CHALLENGE: MAKE IT GLOBAL2        • Data centers across the globe since latency has huge effect on          ...
1             CHALLENGE: WE HAVE BIG DATA                   STRUCTURED DATA2                   500G DAILY                 ...
1               OUR AUDACIOUS GOALS2             Build a world-class data and analytics organization             • Deeply ...
1                  SHAMELESS HIRING PLUG2            • Like most everybody else at this conference… we’re               hi...
1             SHAMELESS HIRING PLUG23456      THE                   And yes, you can play games at work.7   FUTURE        ...
THANK YOU!QUESTIONS?     BARRY LIVINGSTON        &      JEROME BOULON blivingston@riotgames.com       jboulon@riotgames.com
Upcoming SlideShare
Loading in …5
×

Honu/Big Data @ Riot Games

2,629 views

Published on

Big Data @ Riot Games – Using Honu & Hadoop to Understand Player Experience.

Published in: Technology
1 Comment
7 Likes
Statistics
Notes
No Downloads
Views
Total views
2,629
On SlideShare
0
From Embeds
0
Number of Embeds
94
Actions
Shares
0
Downloads
0
Comments
1
Likes
7
Embeds 0
No embeds

No notes for slide
  • Times where there were 20% month over month growth in a single environment2 environments w/~200K CCU to 16 environments and 1.3million CCU in the space of 12 monthsResources were focused on getting our operational systems to scale along with demand
  • Today, we’re going to focus on a few of these.
  • Before we talk about our first usecase, we need to give you a little bit of context about the game and gameplay (super high level), as well as a quick overview of some of the pieces of the LoL architectureSession Based Team play - basic idea is like “kill the other team’s nexus (base)” – MOBA!If you die, you re-spawn after a certain amount of time (that time grows as the game progresses)Lots of strategy to the game
  • Each player “summons” a Champion that he playsEach champion has very different abilities
  • Each player “summons” a Champion that he playsEach champion has very different abilities
  • “Typical” Hardware profiles differ significantly from region to regionWe want to make sure that all regions have a good experienceWe also want to understand latency from various locations in the world to our installs
  • There are lots of customizable settings around resolution and graphics quality that players can changeAll of these settings have a potential impact on performance
  • So given that we have all of these challenges, what are we doing to improve the
  • So given that we have all of these challenges, what are we doing to improve the
  • Honu/Big Data @ Riot Games

    1. 1. RIOT GAMESUSING HADOOP TO IMPROVE THE PLAYER EXPERIENCE (AND TRACK HOW MANY TEEMOS DIE PER DAY) BARRY LIVINGSTON & JEROME BOULON February 2013
    2. 2. 1234 INTRODUCTION567
    3. 3. 1 INTRO ABOUT THE SPEAKERS234567
    4. 4. 1 INTRO THIS PRESENTATION IS ABOUT…2 • A quick history of our data warehouse3 • Our high level architecture • Player experience use cases4 • How Hadoop has enabled these use cases • Changes we have made to our architecture5 to facilitate deeper insight at velocity • Where we’re headed67
    5. 5. 1 INTRO WHO is RIOT GAMES?2 • Developer and publisher of League of Legends3 • Founded 2006 by gamers for gamers • Player experience focused4 – Needless to say, data is pretty important to understanding the player experience!567
    6. 6. 1 LEAGUE OF LEGENDS INTRO2 12 MILLION 70 MILLION DAILY ACTIVE PLAYERS REGISTERED PLAYERS345 3 MILLION 32 MILLION PEAK CONCURRENT MONTHLY ACTIVE PLAYERS PLAYERS67 Numbers based on Riot Games data published October 2012.
    7. 7. 1234 A QUICK HISTORY567
    8. 8. 1 INITIAL LAUNCH / SCRAPPY START UP PHASE2 HISTORY START-UP3 • Had a single, dedicated MySQL instance for the DW • Data was ETL’d from production slaves into this instance4 • Queries were run in MySQL • Reporting was done in Excel • All ETLs, Queries, and Reporting were done by one person!56 This worked great!7
    9. 9. 1 AND THEN – CRAZY GROWTH!!!! CRAZY2 HISTORY START-UP GROWTH3 # unique logins TOTAL ACTIVE PLAYERS4 June 2012 4.2M5 NOV. 201167 time
    10. 10. 1 THE BREAKING POINT CRAZY BREAKING2 HISTORY START-UP GROWTH POINT3 • Data Warehouse reached a breaking point – 24 hours of data took 24.5 hours to ETL • We couldn’t handle…4 – multiple environments in a vertical MySQL instance – a single environment in a vertical MySQL instance5 • We needed to change!67
    11. 11. 1 INTRODUCTION OF HADOOP CRAZY BREAKING2 HISTORY START-UP GROWTH POINT HADOOP!3 • Hadoop has a number of great qualities! – Cost Effective4 – Scalable – Open Source5 – We could execute quickly!67
    12. 12. 1 HIGH LEVEL ARCHITECTURE, JUNE 2012 Business2 HISTORY Audit Plat Analyst LoL Tableau3 NORTH AMERICA Pentaho Audit Plat + Custom ETL Hive Data Pentaho MySQL4 + Warehouse LoL Sqoop EUROPE5 Audit Plat LoL6 Analysts KOREA7
    13. 13. 1 BUT, THIS WASN’T GOOD ENOUGH2 HISTORY • We needed to improve on many levels – Shorten time to insight3 – Increase depth of insight – Enable data analysis for client-side features4 – Flexible auditing framework – Log ingestion and analysis5 – International data infrastructure67
    14. 14. 1 BUT, THIS WASN’T GOOD ENOUGH2 HISTORY • We needed to improve on many levels – Shorten time to insight3 – Increase depth of insight – Enable data analysis for client-side features4 – Flexible auditing framework – Log ingestion and analysis5 – International data infrastructure67
    15. 15. 1234 CONTEXT567
    16. 16. 1 LEAGUE OF LEGENDS GAMEPLAY BASICS23 CONTEXT4567
    17. 17. 123 CONTEXT4567
    18. 18. 123 CONTEXT4567
    19. 19. 1 CLIENT FOOTPRINT2 • As a AAA video game, a significant portion of our software runs directly on players’ machines – High performance graphics3 CONTEXT – Responsiveness4 • There is logic in these components that is ONLY exercised on the client-side5 • Understanding the performance, reliability, and stability of these features is paramount to improving the player6 experience!7
    20. 20. 1 “THE PATCHER”23 CONTEXT4567
    21. 21. 1 “LOBBY CLIENT”23 CONTEXT4567
    22. 22. 1 “GAME CLIENT”23 CONTEXT4567
    23. 23. 1234 GAME CLIENT5 PERFORMANCE67
    24. 24. 1 CHALLENGE: THE GAME IS ALIVE!2 • The game is a living, breathing service that’s always in motion3 • Updated every 2-3 weeks USECASE4 #1 – New champions – New items – New effects/particles5 – Changes in environment – Changes in design and design balance67
    25. 25. 1 CHALLENGE: WE’RE GLOBAL!23 USECASE4 #1567
    26. 26. 1 CHALLENGE: PC VARIABILITY2 • Hardware and OS profiles are significantly different even within regions!3 • OS and Patch Level • CPU4 USECASE #1 • Memory • Video Card • Video Card Memory5 • Drivers!67
    27. 27. 1 CHALLENGE: GRAPHICS SETTINGS23 USECASE4 #1567
    28. 28. 1 CHALLENGE: CLIENT-SIDE LOGIC23 USECASE4 #1567
    29. 29. 1 IMPROVING THE PLAYER EXPERIENCE2 • We need to gather information across all of these dimensions in order to UNDERSTAND the player experience3 • We use this information to:4 USECASE – React quickly to changes #1 – Optimize performance – Optimize designs5 – Improve our testing • Like Creating our Compatibility Testing Lab!67
    30. 30. 1 REACTING QUICKLY23 USECASE4 #1567
    31. 31. 1 GAME LOAD SCREEN23 USECASE4 #1567
    32. 32. 1 IMPROVING LOAD TIMES23 USECASE4 #1567
    33. 33. 1 OPTIMIZING DESIGN AND PERFORMANCE23 USECASE4 #1567
    34. 34. 1 HOW DID WE SOLVE THIS?2 • We have an ARMY of TEEMOs watching players’ machines through their telescopes?!?!?!3 – Not really, but we DID consider it! USECASE4 #1567
    35. 35. 1 HONU PIPELINE2 Generate Collect Organize3 Honu Honu Honu Client Collectors Demux4 SDK567
    36. 36. 1 HONU-CLIENT SDK2345 GAME_CLIENT_STATS timestamp source app pingAvg serverId system6 1234567890 99.123.456.78 game_client 220.9542 12.345.678.90 Intel64 … Select avg(f[‘pingAVG’]) from game_client_stats group by f[‘serverId’];7
    37. 37. 1 HONU-TOOLS2 • DradisTestUI: WEB UI to send a message to Dradis3 directly without any coding • EchoService: WEB UI to easily and immediately4 visualize the data that has been sent to Honu Collectors567
    38. 38. 1 HONU-COLLECTORS2 • Each collector: – Collect events from multiple SQS3 clients (Thrift/NIO) – Save all events to one S34 compressed file locally – Upload that file every XX minutes to S35 – Send a message to SQS for Honu Collectors Demux67
    39. 39. 1 HONU-DEMUX2 SQS S33 S3 S S3 34 Standalone SS tandalone tandalone SDemux tandalone Demux Standalone Demux Demux S3 S35 Demux HIVE MERGE67
    40. 40. 1 HONU-DEMUX S32 SQS Demux34 S3 S3 S35 HIVE MERGE SS tandalone tandalone6 Standalone Demux Standalone Demux Demux S3 S3 Demux7
    41. 41. 1 JUNE 2012 WWW Client2 Mobile34567
    42. 42. 1 WWW Mobile Client234567
    43. 43. 1234 MATCHMAKING567
    44. 44. 1 MATCHMAKING2 • One of the most important features outside of gameplay • Like a dating service, the objective is to match people up;3 • Number of different queues that players can line up in, depending on the type of match they’re looking for4 USECASE5 #26 Critical that this system is balanced balanced and able to create good matches quickly7
    45. 45. 1 MATCHING THE RIGHT PLAYERS234 USECASE5 #267
    46. 46. 1 IMPROVING THE EXPERIENCE2 • We want to ensure that all players are having the best possible experience getting the matches they want – This is VERY challenging!3 – We’re obviously always studying this and trying to improve it4 • Recently, we’ve started combining client data with data we have about many other dimensions – Queue times USECASE5 #2 – Match quality – Player Skill/Matchmaking Rating6 • Obviously, we hope this will lead us to improvements we haven’t identified as of yet7
    47. 47. 1 HOW DO WE ACCOMPLISH THAT?234 USECASE5 #267
    48. 48. 12 Tools & Business Process Tools & Business Process Tools & Business Process34 Audit Plat Audit Plat Audit Plat LoL LoL LoL5 NORTH AMERICA EUROPE KOREA6 MySQL7 JUNE 2012
    49. 49. 1 Audit Plat Audit Plat Audit Plat LoL LoL LoL2 NORTH AMERICA EUROPE KOREA34567 Dashboards Tools & Business Process
    50. 50. 1 FLOWS234 USECASE5 #267
    51. 51. 1 WWW Mobile Client234567
    52. 52. 1234 Data Flows567
    53. 53. 1 Data Flows2 Audit Plat Audit Plat Audit Plat LoL LoL LoL3 NORTH AMERICA EUROPE KOREA456 Workflows7
    54. 54. 1 Data Flows2 Audit Plat Audit Plat LoL3 Audit Plat LoL EUROPE LoL KOREA NORTH AMERICA456 Workflows7
    55. 55. 1 WWW Mobile Client23456 Workflows7
    56. 56. 1 Enterprise Workflow Management Using Oozie @ Riot Games23 Matt Goeke and4 Barry Livingston Tomorrow : 11:40 12:3056 Workflows7
    57. 57. 1234 THE FUTURE567
    58. 58. 1 DECREASE TEEMO DEATHS?!?!?!23456 THE7 FUTURE
    59. 59. 1 CONTINUE INCREASING VELOCITY June 2012 February 20132 MySQL tables 180 1200 Pipeline Events/day 0 2.5+ Billion3 Workflows Cronjob + Pentaho Oozie Environment Datacenter DC + AWS SLA 1 day 2 hours4 Event tracking • 2+ weeks (DB • 10 minutes update) • Dependencies: DBA • Self-Service5 teams + ETL teams + Tools teams • Downtime (3h min.) • No downtime6 THE7 FUTURE
    60. 60. 1 OUR IMMEDIATE GOALS2 • Self-Service reporting • Metadata Management Service3 • Real-time aggregation pipeline • Real-time slicing/dicing for non-critical data4 • Log ingestion and analysis • International data infrastructure56 THE7 FUTURE
    61. 61. 1 CHALLENGE: MAKE IT GLOBAL2 • Data centers across the globe since latency has huge effect on gameplay  log data scattered around the world3 • Large presence in Asia -- some areas (e.g., PH) have bandwidth challenges or bandwidth is expensive456 THE7 FUTURE
    62. 62. 1 CHALLENGE: WE HAVE BIG DATA STRUCTURED DATA2 500G DAILY APPLICATION AND OPERATIONAL LOGS3 4.5TB DAILY4 RIOT YOUTUBE CHANNEL 3MM SUBSCRIBERS 448+MM VIEWS5 + chat logs + detailed gameplay event tracking6 + so on…. THE7 FUTURE
    63. 63. 1 OUR AUDACIOUS GOALS2 Build a world-class data and analytics organization • Deeply understand players across the globe • Apply that understanding to improve games for players3 • Deeply understand our entire ecosystem, including social media4 Have ability to identify, understand and react to meaningful trends in real time5 Have deep, real-time understanding of our systems from player experience and operational standpoints6 THE7 FUTURE
    64. 64. 1 SHAMELESS HIRING PLUG2 • Like most everybody else at this conference… we’re hiring!3 • The Riot Manifesto Player experience first4 Challenge convention Focus on talent and team5 Take play seriously Stay hungry, stay humble6 THE7 FUTURE
    65. 65. 1 SHAMELESS HIRING PLUG23456 THE And yes, you can play games at work.7 FUTURE It’s encouraged!
    66. 66. THANK YOU!QUESTIONS? BARRY LIVINGSTON & JEROME BOULON blivingston@riotgames.com jboulon@riotgames.com

    ×