Riot Games uses Hadoop to improve the player experience and track game metrics like Teemo deaths. They initially used a single MySQL database that struggled as the player base grew exponentially. Hadoop allowed them to collect vast amounts of structured and unstructured data from multiple sources globally. This data helps optimize matchmaking, improve client performance across varying hardware, and react quickly to game changes. Their goals are to provide self-service analytics, ingest more real-time data, and gain deeper global insights to continuously enhance the player experience.

1. RIOT GAMES
USING HADOOP TO IMPROVE THE PLAYER
EXPERIENCE
(AND TRACK HOW MANY TEEMOS DIE PER DAY)
BARRY LIVINGSTON & JEROME BOULON
February 2013

2. 1
2
3
4
INTRODUCTION
5
6
7

3. 1 INTRO
ABOUT THE SPEAKERS
2
3
4
5
6
7

4. 1 INTRO
THIS PRESENTATION IS ABOUT…
2
• A quick history of our data warehouse
3 • Our high level architecture
• Player experience use cases
4 • How Hadoop has enabled these use cases
• Changes we have made to our architecture
5
to facilitate deeper insight at velocity
• Where we’re headed
6
7

5. 1 INTRO
WHO is RIOT GAMES?
2
• Developer and publisher of League of Legends
3 • Founded 2006 by gamers for gamers
• Player experience focused
4 – Needless to say, data is pretty important to
understanding the player experience!
5
6
7

6. 1
LEAGUE OF LEGENDS
INTRO
2
12 MILLION 70 MILLION
DAILY ACTIVE PLAYERS REGISTERED PLAYERS
3
4
5
3 MILLION 32 MILLION
PEAK CONCURRENT MONTHLY ACTIVE
PLAYERS PLAYERS
6
7
Numbers based on Riot Games data published October 2012.

7. 1
2
3
4
A QUICK HISTORY
5
6
7

8. 1 INITIAL LAUNCH / SCRAPPY START UP PHASE
2 HISTORY START-UP
3 • Had a single, dedicated MySQL instance for the DW
• Data was ETL’d from production slaves into this instance
4 • Queries were run in MySQL
• Reporting was done in Excel
• All ETLs, Queries, and Reporting were done by one person!
5
6 This worked great!
7

9. 1 AND THEN – CRAZY GROWTH!!!!
CRAZY
2 HISTORY START-UP
GROWTH
3
# unique logins
TOTAL ACTIVE PLAYERS
4
June 2012
4.2M
5 NOV. 2011
6
7
time

10. 1 THE BREAKING POINT
CRAZY BREAKING
2 HISTORY START-UP
GROWTH POINT
3 • Data Warehouse reached a breaking point
– 24 hours of data took 24.5 hours to ETL
• We couldn’t handle…
4 – multiple environments in a vertical MySQL instance
– a single environment in a vertical MySQL instance
5 • We needed to change!
6
7

11. 1 INTRODUCTION OF HADOOP
CRAZY BREAKING
2 HISTORY START-UP
GROWTH POINT
HADOOP!
3
• Hadoop has a number of great qualities!
– Cost Effective
4
– Scalable
– Open Source
5 – We could execute quickly!
6
7

12. 1 HIGH LEVEL ARCHITECTURE, JUNE 2012
Business
2 HISTORY Audit Plat
Analyst
LoL
Tableau
3 NORTH AMERICA
Pentaho
Audit Plat +
Custom ETL Hive Data Pentaho MySQL
4 +
Warehouse
LoL
Sqoop
EUROPE
5
Audit Plat
LoL
6 Analysts
KOREA
7

13. 1
BUT, THIS WASN’T GOOD ENOUGH
2 HISTORY
• We needed to improve on many levels
– Shorten time to insight
3
– Increase depth of insight
– Enable data analysis for client-side features
4
– Flexible auditing framework
– Log ingestion and analysis
5 – International data infrastructure
6
7

14. 1
BUT, THIS WASN’T GOOD ENOUGH
2 HISTORY
• We needed to improve on many levels
– Shorten time to insight
3
– Increase depth of insight
– Enable data analysis for client-side features
4
– Flexible auditing framework
– Log ingestion and analysis
5 – International data infrastructure
6
7

15. 1
2
3
4
CONTEXT
5
6
7

16. 1
LEAGUE OF LEGENDS GAMEPLAY BASICS
2
3 CONTEXT
4
5
6
7

17. 1
2
3 CONTEXT
4
5
6
7

18. 1
2
3 CONTEXT
4
5
6
7

19. 1
CLIENT FOOTPRINT
2 • As a AAA video game, a significant portion of our software
runs directly on players’ machines
– High performance graphics
3 CONTEXT
– Responsiveness
4 • There is logic in these components that is ONLY exercised
on the client-side
5
• Understanding the performance, reliability, and stability of
these features is paramount to improving the player
6 experience!
7

20. 1
“THE PATCHER”
2
3 CONTEXT
4
5
6
7

21. 1
“LOBBY CLIENT”
2
3 CONTEXT
4
5
6
7

22. 1
“GAME CLIENT”
2
3 CONTEXT
4
5
6
7

23. 1
2
3
4 GAME CLIENT
5 PERFORMANCE
6
7

24. 1
CHALLENGE: THE GAME IS ALIVE!
2
• The game is a living, breathing service that’s always in motion
3
• Updated every 2-3 weeks
USECASE
4 #1
– New champions
– New items
– New effects/particles
5 – Changes in environment
– Changes in design and design balance
6
7

25. 1
CHALLENGE: WE’RE GLOBAL!
2
3
USECASE
4 #1
5
6
7

26. 1
CHALLENGE: PC VARIABILITY
2
• Hardware and OS profiles are significantly different even within
regions!
3 • OS and Patch Level
• CPU
4
USECASE
#1
• Memory
• Video Card
• Video Card Memory
5
• Drivers!
6
7

27. 1
CHALLENGE: GRAPHICS SETTINGS
2
3
USECASE
4 #1
5
6
7

28. 1
CHALLENGE: CLIENT-SIDE LOGIC
2
3
USECASE
4 #1
5
6
7

29. 1
IMPROVING THE PLAYER EXPERIENCE
2
• We need to gather information across all of these dimensions
in order to UNDERSTAND the player experience
3
• We use this information to:
4
USECASE – React quickly to changes
#1
– Optimize performance
– Optimize designs
5 – Improve our testing
• Like Creating our Compatibility Testing Lab!
6
7

30. 1
REACTING QUICKLY
2
3
USECASE
4 #1
5
6
7

31. 1
GAME LOAD SCREEN
2
3
USECASE
4 #1
5
6
7

32. 1
IMPROVING LOAD TIMES
2
3
USECASE
4 #1
5
6
7

33. 1
OPTIMIZING DESIGN AND PERFORMANCE
2
3
USECASE
4 #1
5
6
7

34. 1
HOW DID WE SOLVE THIS?
2
• We have an ARMY of TEEMOs watching players’ machines
through their telescopes?!?!?!
3 – Not really, but we DID consider it!
USECASE
4 #1
5
6
7

35. 1
HONU PIPELINE
2
Generate Collect Organize
3
Honu
Honu Honu
Client
Collectors Demux
4 SDK
5
6
7

36. 1
HONU-CLIENT SDK
2
3
4
5
GAME_CLIENT_STATS
timestamp source app pingAvg serverId system
6 1234567890 99.123.456.78 game_client 220.9542 12.345.678.90 Intel64 …
Select avg(f[‘pingAVG’]) from game_client_stats group by f[‘serverId’];
7

37. 1
HONU-TOOLS
2
• DradisTestUI: WEB UI to send a message to Dradis
3 directly without any coding
• EchoService: WEB UI to easily and immediately
4
visualize the data that has been sent to Honu
Collectors
5
6
7

38. 1
HONU-COLLECTORS
2 • Each collector:
– Collect events from multiple SQS
3 clients (Thrift/NIO)
– Save all events to one S3
4
compressed file locally
– Upload that file every XX
minutes to S3
5
– Send a message to SQS for Honu Collectors
Demux
6
7

39. 1
HONU-DEMUX
2
SQS
S3
3
S3
S S3
3
4 Standalone
SS
tandalone
tandalone
SDemux
tandalone
Demux
Standalone
Demux
Demux S3 S3
5 Demux
HIVE
MERGE
6
7

40. 1
HONU-DEMUX
S3
2
SQS Demux
3
4 S3
S3 S3
5
HIVE
MERGE
SS
tandalone
tandalone
6 Standalone
Demux
Standalone
Demux
Demux S3 S3
Demux
7

41. 1
JUNE 2012
WWW
Client
2 Mobile
3
4
5
6
7

42. 1
WWW
Mobile Client
2
3
4
5
6
7

43. 1
2
3
4
MATCHMAKING
5
6
7

44. 1
MATCHMAKING
2 • One of the most important features outside of gameplay
• Like a dating service, the objective is to match people up;
3 • Number of different queues that players can line up in, depending
on the type of match they’re looking for
4
USECASE
5 #2
6 Critical that this system is balanced
balanced
and able to create good matches quickly
7

45. 1
MATCHING THE RIGHT PLAYERS
2
3
4
USECASE
5 #2
6
7

46. 1
IMPROVING THE EXPERIENCE
2 • We want to ensure that all players are having the best
possible experience getting the matches they want
– This is VERY challenging!
3
– We’re obviously always studying this and trying to improve it
4 • Recently, we’ve started combining client data with data we
have about many other dimensions
– Queue times
USECASE
5 #2 – Match quality
– Player Skill/Matchmaking Rating
6
• Obviously, we hope this will lead us to improvements we
haven’t identified as of yet
7

47. 1
HOW DO WE ACCOMPLISH THAT?
2
3
4
USECASE
5 #2
6
7

48. 1
2
Tools & Business Process Tools & Business Process Tools & Business Process
3
4
Audit Plat Audit Plat Audit Plat
LoL LoL LoL
5 NORTH AMERICA EUROPE KOREA
6 MySQL
7 JUNE 2012

49. 1 Audit Plat Audit Plat Audit Plat
LoL LoL LoL
2 NORTH AMERICA EUROPE KOREA
3
4
5
6
7 Dashboards Tools & Business Process

50. 1
FLOWS
2
3
4
USECASE
5 #2
6
7

51. 1
WWW
Mobile Client
2
3
4
5
6
7

52. 1
2
3
4
Data Flows
5
6
7

53. 1
Data Flows
2
Audit Plat Audit Plat Audit Plat
LoL LoL LoL
3
NORTH AMERICA EUROPE KOREA
4
5
6 Workflows
7

54. 1
Data Flows
2
Audit Plat
Audit Plat
LoL
3 Audit Plat
LoL
EUROPE
LoL
KOREA
NORTH AMERICA
4
5
6 Workflows
7

55. 1
WWW
Mobile Client
2
3
4
5
6 Workflows
7

56. 1
Enterprise Workflow Management Using Oozie
@ Riot Games
2
3
Matt Goeke and
4 Barry Livingston
Tomorrow : 11:40 12:30
5
6 Workflows
7

57. 1
2
3
4
THE FUTURE
5
6
7

58. 1
DECREASE TEEMO DEATHS?!?!?!
2
3
4
5
6
THE
7 FUTURE

59. 1
CONTINUE INCREASING VELOCITY
June 2012 February 2013
2
MySQL tables 180 1200
Pipeline Events/day 0 2.5+ Billion
3 Workflows Cronjob + Pentaho Oozie
Environment Datacenter DC + AWS
SLA 1 day 2 hours
4
Event tracking • 2+ weeks (DB • 10 minutes
update)
• Dependencies: DBA • Self-Service
5 teams + ETL teams +
Tools teams
• Downtime (3h min.) • No downtime
6
THE
7 FUTURE

60. 1
OUR IMMEDIATE GOALS
2
• Self-Service reporting
• Metadata Management Service
3
• Real-time aggregation pipeline
• Real-time slicing/dicing for non-critical data
4
• Log ingestion and analysis
• International data infrastructure
5
6
THE
7 FUTURE

61. 1
CHALLENGE: MAKE IT GLOBAL
2 • Data centers across the globe since latency has huge effect on
gameplay  log data scattered around the world
3
• Large presence in Asia -- some areas (e.g., PH) have bandwidth
challenges or bandwidth is expensive
4
5
6
THE
7 FUTURE

62. 1
CHALLENGE: WE HAVE BIG DATA
STRUCTURED DATA
2
500G DAILY
APPLICATION AND OPERATIONAL LOGS
3
4.5TB DAILY
4 RIOT YOUTUBE CHANNEL
3MM SUBSCRIBERS
448+MM VIEWS
5
+ chat logs
+ detailed gameplay event tracking
6 + so on….
THE
7 FUTURE

63. 1
OUR AUDACIOUS GOALS
2
Build a world-class data and analytics organization
• Deeply understand players across the globe
• Apply that understanding to improve games for players
3
• Deeply understand our entire ecosystem, including social media
4 Have ability to identify, understand and react to
meaningful trends in real time
5
Have deep, real-time understanding of our systems
from player experience and operational standpoints
6
THE
7 FUTURE

64. 1
SHAMELESS HIRING PLUG
2 • Like most everybody else at this conference… we’re
hiring!
3
• The Riot Manifesto
Player experience first
4 Challenge convention
Focus on talent and team
5
Take play seriously
Stay hungry, stay humble
6
THE
7 FUTURE

65. 1
SHAMELESS HIRING PLUG
2
3
4
5
6
THE
And yes, you can play games at work.
7 FUTURE
It’s encouraged!

66. THANK YOU!
QUESTIONS?
BARRY LIVINGSTON & JEROME BOULON
blivingston@riotgames.com jboulon@riotgames.com