1. RIOT GAMES
USING HADOOP TO IMPROVE THE PLAYER
EXPERIENCE
(AND TRACK HOW MANY TEEMOS DIE PER DAY)
BARRY LIVINGSTON & JEROME BOULON
February 2013

2. 1
2
3
4
INTRODUCTION
5
6
7

3. 1 INTRO
ABOUT THE SPEAKERS
2
3
4
5
6
7

4. 1 INTRO
THIS PRESENTATION IS ABOUT…
2
• A quick history of our data warehouse
3 • Our high level architecture
• Player experience use cases
4 • How Hadoop has enabled these use cases
• Changes we have made to our architecture
5
to facilitate deeper insight at velocity
• Where we’re headed
6
7

5. 1 INTRO
WHO is RIOT GAMES?
2
• Developer and publisher of League of Legends
3 • Founded 2006 by gamers for gamers
• Player experience focused
4 – Needless to say, data is pretty important to
understanding the player experience!
5
6
7

6. 1
LEAGUE OF LEGENDS
INTRO
2
12 MILLION 70 MILLION
DAILY ACTIVE PLAYERS REGISTERED PLAYERS
3
4
5
3 MILLION 32 MILLION
PEAK CONCURRENT MONTHLY ACTIVE
PLAYERS PLAYERS
6
7
Numbers based on Riot Games data published October 2012.

7. 1
2
3
4
A QUICK HISTORY
5
6
7

8. 1 INITIAL LAUNCH / SCRAPPY START UP PHASE
2 HISTORY START-UP
3 • Had a single, dedicated MySQL instance for the DW
• Data was ETL’d from production slaves into this instance
4 • Queries were run in MySQL
• Reporting was done in Excel
• All ETLs, Queries, and Reporting were done by one person!
5
6 This worked great!
7

9. 1 AND THEN – CRAZY GROWTH!!!!
CRAZY
2 HISTORY START-UP
GROWTH
3
# unique logins
TOTAL ACTIVE PLAYERS
4
June 2012
4.2M
5 NOV. 2011
6
7
time

10. 1 THE BREAKING POINT
CRAZY BREAKING
2 HISTORY START-UP
GROWTH POINT
3 • Data Warehouse reached a breaking point
– 24 hours of data took 24.5 hours to ETL
• We couldn’t handle…
4 – multiple environments in a vertical MySQL instance
– a single environment in a vertical MySQL instance
5 • We needed to change!
6
7

11. 1 INTRODUCTION OF HADOOP
CRAZY BREAKING
2 HISTORY START-UP
GROWTH POINT
HADOOP!
3
• Hadoop has a number of great qualities!
– Cost Effective
4
– Scalable
– Open Source
5 – We could execute quickly!
6
7

12. 1 HIGH LEVEL ARCHITECTURE, JUNE 2012
Business
2 HISTORY Audit Plat
Analyst
LoL
Tableau
3 NORTH AMERICA
Pentaho
Audit Plat +
Custom ETL Hive Data Pentaho MySQL
4 +
Warehouse
LoL
Sqoop
EUROPE
5
Audit Plat
LoL
6 Analysts
KOREA
7

13. 1
BUT, THIS WASN’T GOOD ENOUGH
2 HISTORY
• We needed to improve on many levels
– Shorten time to insight
3
– Increase depth of insight
– Enable data analysis for client-side features
4
– Flexible auditing framework
– Log ingestion and analysis
5 – International data infrastructure
6
7

14. 1
BUT, THIS WASN’T GOOD ENOUGH
2 HISTORY
• We needed to improve on many levels
– Shorten time to insight
3
– Increase depth of insight
– Enable data analysis for client-side features
4
– Flexible auditing framework
– Log ingestion and analysis
5 – International data infrastructure
6
7

15. 1
2
3
4
CONTEXT
5
6
7

16. 1
LEAGUE OF LEGENDS GAMEPLAY BASICS
2
3 CONTEXT
4
5
6
7

17. 1
2
3 CONTEXT
4
5
6
7

18. 1
2
3 CONTEXT
4
5
6
7

19. 1
CLIENT FOOTPRINT
2 • As a AAA video game, a significant portion of our software
runs directly on players’ machines
– High performance graphics
3 CONTEXT
– Responsiveness
4 • There is logic in these components that is ONLY exercised
on the client-side
5
• Understanding the performance, reliability, and stability of
these features is paramount to improving the player
6 experience!
7

20. 1
“THE PATCHER”
2
3 CONTEXT
4
5
6
7

21. 1
“LOBBY CLIENT”
2
3 CONTEXT
4
5
6
7

22. 1
“GAME CLIENT”
2
3 CONTEXT
4
5
6
7

23. 1
2
3
4 GAME CLIENT
5 PERFORMANCE
6
7

24. 1
CHALLENGE: THE GAME IS ALIVE!
2
• The game is a living, breathing service that’s always in motion
3
• Updated every 2-3 weeks
USECASE
4 #1
– New champions
– New items
– New effects/particles
5 – Changes in environment
– Changes in design and design balance
6
7

25. 1
CHALLENGE: WE’RE GLOBAL!
2
3
USECASE
4 #1
5
6
7

26. 1
CHALLENGE: PC VARIABILITY
2
• Hardware and OS profiles are significantly different even within
regions!
3 • OS and Patch Level
• CPU
4
USECASE
#1
• Memory
• Video Card
• Video Card Memory
5
• Drivers!
6
7

27. 1
CHALLENGE: GRAPHICS SETTINGS
2
3
USECASE
4 #1
5
6
7

28. 1
CHALLENGE: CLIENT-SIDE LOGIC
2
3
USECASE
4 #1
5
6
7

29. 1
IMPROVING THE PLAYER EXPERIENCE
2
• We need to gather information across all of these dimensions
in order to UNDERSTAND the player experience
3
• We use this information to:
4
USECASE – React quickly to changes
#1
– Optimize performance
– Optimize designs
5 – Improve our testing
• Like Creating our Compatibility Testing Lab!
6
7

30. 1
REACTING QUICKLY
2
3
USECASE
4 #1
5
6
7

31. 1
GAME LOAD SCREEN
2
3
USECASE
4 #1
5
6
7

32. 1
IMPROVING LOAD TIMES
2
3
USECASE
4 #1
5
6
7

33. 1
OPTIMIZING DESIGN AND PERFORMANCE
2
3
USECASE
4 #1
5
6
7

34. 1
HOW DID WE SOLVE THIS?
2
• We have an ARMY of TEEMOs watching players’ machines
through their telescopes?!?!?!
3 – Not really, but we DID consider it!
USECASE
4 #1
5
6
7

35. 1
HONU PIPELINE
2
Generate Collect Organize
3
Honu
Honu Honu
Client
Collectors Demux
4 SDK
5
6
7

36. 1
HONU-CLIENT SDK
2
3
4
5
GAME_CLIENT_STATS
timestamp source app pingAvg serverId system
6 1234567890 99.123.456.78 game_client 220.9542 12.345.678.90 Intel64 …
Select avg(f[‘pingAVG’]) from game_client_stats group by f[‘serverId’];
7

37. 1
HONU-TOOLS
2
• DradisTestUI: WEB UI to send a message to Dradis
3 directly without any coding
• EchoService: WEB UI to easily and immediately
4
visualize the data that has been sent to Honu
Collectors
5
6
7

38. 1
HONU-COLLECTORS
2 • Each collector:
– Collect events from multiple SQS
3 clients (Thrift/NIO)
– Save all events to one S3
4
compressed file locally
– Upload that file every XX
minutes to S3
5
– Send a message to SQS for Honu Collectors
Demux
6
7

39. 1
HONU-DEMUX
2
SQS
S3
3
S3
S S3
3
4 Standalone
SS
tandalone
tandalone
SDemux
tandalone
Demux
Standalone
Demux
Demux S3 S3
5 Demux
HIVE
MERGE
6
7

40. 1
HONU-DEMUX
S3
2
SQS Demux
3
4 S3
S3 S3
5
HIVE
MERGE
SS
tandalone
tandalone
6 Standalone
Demux
Standalone
Demux
Demux S3 S3
Demux
7

41. 1
JUNE 2012
WWW
Client
2 Mobile
3
4
5
6
7

42. 1
WWW
Mobile Client
2
3
4
5
6
7

43. 1
2
3
4
MATCHMAKING
5
6
7

44. 1
MATCHMAKING
2 • One of the most important features outside of gameplay
• Like a dating service, the objective is to match people up;
3 • Number of different queues that players can line up in, depending
on the type of match they’re looking for
4
USECASE
5 #2
6 Critical that this system is balanced
balanced
and able to create good matches quickly
7

45. 1
MATCHING THE RIGHT PLAYERS
2
3
4
USECASE
5 #2
6
7

46. 1
IMPROVING THE EXPERIENCE
2 • We want to ensure that all players are having the best
possible experience getting the matches they want
– This is VERY challenging!
3
– We’re obviously always studying this and trying to improve it
4 • Recently, we’ve started combining client data with data we
have about many other dimensions
– Queue times
USECASE
5 #2 – Match quality
– Player Skill/Matchmaking Rating
6
• Obviously, we hope this will lead us to improvements we
haven’t identified as of yet
7

47. 1
HOW DO WE ACCOMPLISH THAT?
2
3
4
USECASE
5 #2
6
7

48. 1
2
Tools & Business Process Tools & Business Process Tools & Business Process
3
4
Audit Plat Audit Plat Audit Plat
LoL LoL LoL
5 NORTH AMERICA EUROPE KOREA
6 MySQL
7 JUNE 2012

49. 1 Audit Plat Audit Plat Audit Plat
LoL LoL LoL
2 NORTH AMERICA EUROPE KOREA
3
4
5
6
7 Dashboards Tools & Business Process

50. 1
FLOWS
2
3
4
USECASE
5 #2
6
7

51. 1
WWW
Mobile Client
2
3
4
5
6
7

52. 1
2
3
4
Data Flows
5
6
7

53. 1
Data Flows
2
Audit Plat Audit Plat Audit Plat
LoL LoL LoL
3
NORTH AMERICA EUROPE KOREA
4
5
6 Workflows
7

54. 1
Data Flows
2
Audit Plat
Audit Plat
LoL
3 Audit Plat
LoL
EUROPE
LoL
KOREA
NORTH AMERICA
4
5
6 Workflows
7

55. 1
WWW
Mobile Client
2
3
4
5
6 Workflows
7

56. 1
Enterprise Workflow Management Using Oozie
@ Riot Games
2
3
Matt Goeke and
4 Barry Livingston
Tomorrow : 11:40 12:30
5
6 Workflows
7

57. 1
2
3
4
THE FUTURE
5
6
7

58. 1
DECREASE TEEMO DEATHS?!?!?!
2
3
4
5
6
THE
7 FUTURE

59. 1
CONTINUE INCREASING VELOCITY
June 2012 February 2013
2
MySQL tables 180 1200
Pipeline Events/day 0 2.5+ Billion
3 Workflows Cronjob + Pentaho Oozie
Environment Datacenter DC + AWS
SLA 1 day 2 hours
4
Event tracking • 2+ weeks (DB • 10 minutes
update)
• Dependencies: DBA • Self-Service
5 teams + ETL teams +
Tools teams
• Downtime (3h min.) • No downtime
6
THE
7 FUTURE

60. 1
OUR IMMEDIATE GOALS
2
• Self-Service reporting
• Metadata Management Service
3
• Real-time aggregation pipeline
• Real-time slicing/dicing for non-critical data
4
• Log ingestion and analysis
• International data infrastructure
5
6
THE
7 FUTURE

61. 1
CHALLENGE: MAKE IT GLOBAL
2 • Data centers across the globe since latency has huge effect on
gameplay  log data scattered around the world
3
• Large presence in Asia -- some areas (e.g., PH) have bandwidth
challenges or bandwidth is expensive
4
5
6
THE
7 FUTURE

62. 1
CHALLENGE: WE HAVE BIG DATA
STRUCTURED DATA
2
500G DAILY
APPLICATION AND OPERATIONAL LOGS
3
4.5TB DAILY
4 RIOT YOUTUBE CHANNEL
3MM SUBSCRIBERS
448+MM VIEWS
5
+ chat logs
+ detailed gameplay event tracking
6 + so on….
THE
7 FUTURE

63. 1
OUR AUDACIOUS GOALS
2
Build a world-class data and analytics organization
• Deeply understand players across the globe
• Apply that understanding to improve games for players
3
• Deeply understand our entire ecosystem, including social media
4 Have ability to identify, understand and react to
meaningful trends in real time
5
Have deep, real-time understanding of our systems
from player experience and operational standpoints
6
THE
7 FUTURE

64. 1
SHAMELESS HIRING PLUG
2 • Like most everybody else at this conference… we’re
hiring!
3
• The Riot Manifesto
Player experience first
4 Challenge convention
Focus on talent and team
5
Take play seriously
Stay hungry, stay humble
6
THE
7 FUTURE

65. 1
SHAMELESS HIRING PLUG
2
3
4
5
6
THE
And yes, you can play games at work.
7 FUTURE
It’s encouraged!

66. THANK YOU!
QUESTIONS?
BARRY LIVINGSTON & JEROME BOULON
blivingston@riotgames.com jboulon@riotgames.com