The document discusses Riot Games' transition to using Hadoop for managing their big data and improving player experience in League of Legends due to rapid growth and performance challenges. It highlights the development of their 'Honu' big data pipeline which allows for self-service data analysis and enables quicker insights into player behavior and game performance. Key themes include the democratization of data access, the challenges of a constantly evolving game environment, and the need for scalable and efficient data management solutions.

1. BIG DATA @
RIOT GAMES
USING HADOOP TO IMPROVE THE PLAYER EXPERIENCE
BARRY LIVINGSTON & SANDEEP SHRESTHA | JULY 2013

2. SPEAKERS

3. CONTEXT
HIGH LEVEL ARCHITECTURE
PLAYER EXPERIENCE USE CASES
SUMMARY
QUICK DATA WAREHOUSE HISTORY

4. FIRST, A BIT OF CONTEXT…

6. WHAT IS LEAGUE OF LEGENDS?
2009
LAUNCH
TEAM
ORIENTED
100+
CHAMPS
MODERN
FANTASY

7. WHAT IS LEAGUE OF LEGENDS?

8. LEAGUE OF LEGENDS GAMEPLAY - CHAMPIONS

9. LEAGUE OF LEGENDS GAMEPLAY - GAMEPLAY

10. A QUICK HISTORY

11. INITIAL LAUNCH / SCRAPPY START UP PHASE
‣ Had
a
single,
dedicated
MySQL
instance
for
the
DW
‣ Data
was
ETL’d
from
produc@on
slaves
into
this
instance
‣ Queries
were
run
in
MySQL
‣ Repor@ng
was
done
in
Excel
▾ All
ETLs,
queries
and
repor@ng
were
done
by
one
person
HISTORY
START-­‐UP
THIS WORKED GREAT!

12. THEN – CRAZY GROWTH
HISTORY
START-­‐UP
@me
#
unique
logins
TOTAL
ACTIVE
PLAYERS
June
2012
CRAZY
GROWTH

13. THE BREAKING POINT
HISTORY
START-­‐UP
CRAZY
GROWTH
BREAKING
POINT
‣ Data
warehouse
reached
a
breaking
point
▾ 24
hours
of
data
took
24.5
hours
to
ETL
‣ We
couldn’t
handle…
▾ Mul@ple
environments
in
a
ver@cal
MySQL
instance
▾ A
single
environment
in
a
ver@cal
MySQL
instance
‣ We
needed
to
change

14. INTRODUCTION OF HADOOP
HISTORY
START-­‐UP
CRAZY
GROWTH
BREAKING
POINT
‣ Hadoop
has
a
number
of
great
quali@es
▾ Cost
effec@ve
▾ Scalable
▾ Open
source
▾ We
could
execute
quickly
HADOOP

15. HIGH LEVEL ARCHITECTURE – JUNE 2012
Tableau
Hive
Data
Warehouse
Pentaho
+
Custom
ETL
+
Sqoop
MySQL
Pentaho
Analysts
EUROPE
Audit
Plat
LoL
KOREA
Audit
Plat
LoL
NORTH
AMERICA
Audit
Plat
LoL
Business
Analyst

16. BUT, THIS WASN’T GOOD ENOUGH
‣ The
@me
to
arrive
at
insight
was
too
long!
‣ Our
solu@on
required
too
much
data
team
involvement
▾ Schema
changes
▾ ETL
tweaks
▾ Hive
metadata
updates
‣ Hive
is
painful
for
ad-­‐hoc
or
interac@ve
analysis
▾ Especially
for
non-­‐technical
folks

17. GOALS
‣ Democra@ze
data
access
▾ Enable
Self-­‐service
Data
Collec@on
and
Analysis
‣ Create
ac@onable
insights
‣ Increase
speed
to
insight

18. USE CASE:
GAME CLIENT PERFORMANCE

19. CLIENT FOOTPRINT
‣ Significant
por@on
of
our
soware
runs
directly
on
players’
machines
▾ High
performance
graphics
▾ Responsiveness
‣ There
is
logic
in
these
components
that's
ONLY
exercised
on
the
client-­‐side
‣ Understanding
the
performance,
reliability
and
stability
of
these
features
is
paramount
to
improving
the
player
experience

20. PATCHER

21. LOBBY CLIENT

22. GAME CLIENT

23. ITEM SHOP

24. CHALLENGE: THE GAME IS ALIVE
The
game
is
a
living,
breathing
service
that’s
always
in
mo@on
‣ New
champions
‣ New
items
‣ New
effects/par@cles
‣ Changes
in
environment
‣ Changes
in
design
and
design
balance
UPDATE
2-3WEEKS

25. CHALLENGE: WE’RE GLOBAL

26. CHALLENGE: PC VARIABILITY
‣ Hardware
and
OS
profiles
are
significantly
different
even
within
regions
▾ OS
and
patch
level
▾ CPU
▾ Memory
▾ Video
card
▾ Video
card
memory
▾ Drivers

27. CHALLENGE: GRAPHIC SETTINGS

28. CHALLENGE: CLIENT-SIDE LOGIC

29. IMPROVING THE PLAYER EXPERIENCE
‣ We
need
to
gather
informa@on
across
all
of
these
dimensions
in
order
to
UNDERSTAND
the
player
experience
‣ We
use
this
info
to:
▾ React
quickly
to
changes
▾ Op@mize
performance
▾ Op@mize
designs
▾ Improve
our
tes@ng
• Like
crea@ng
our
compa@bility
tes@ng
lab

30. REACTING QUICKLY

31. GAME LOAD SCREEN

32. IMPROVING LOAD TIME

33. OPTIMIZING DESIGN AND PERFORMANCE

34. OPTIMIZING DESIGN AND PERFORMANCE

35. OPTIMIZING DESIGN AND PERFORMANCE

36. OPTIMIZING DESIGN AND PERFORMANCE

37. HOW DID WE SOLVE THIS
WE HAVE AN ARMY OF TEEMOS WATCHING PLAYERS’ MACHINES THROUGH THEIR TELESCOPES?!
(NOT REALLY, BUT WE DID CONSIDER IT)

38. HONU: GENERATE - COLLECT - ANALYZE
‣ Riot’s
self-­‐service
end-­‐to-­‐end
Big
Data
pipeline
▾ Cloud-­‐ready
(AWS
compa@ble)
▾ Internal
data-­‐center
ready
▾ Persistent
storage:
HDFS/S3
▾ Batch
processing:
Apache
Hadoop/AWS
EMR
▾ Data
publish:
Apache
Hive

39. EVENT GENERATION
‣ Honu
SDKs:
Java,
C++,
Erlang
‣ Collector
discovery
‣ Failover
‣ Load
balancing
‣ Buffering/Batching
‣ Dispatching
‣ Thri
transport

40. HONU CLIENT SDK
Select
avg(f[‘pingAVG’])
from
game_client_stats
group
by
f[‘serverId’];
pingAvg
serverId
system
source
app
@mestamp
1234567890
99.123.456.78
game_client
220.9542
12.345.678.90
Intel64
…
GAME_CLIENT_STATS

41. EVENT COLLECTION
‣ Honu
collector
‣ Online
system
‣ High
availability
–
100%
up@me
‣ Horizontally
scalable
‣ Elas@c
‣ Fault
tolerant
‣ Neulix
OSS
Eureka
discovery
service

42. HONU COLLECTOR
‣ Collect
events
from
mul@ple
clients
(Thri/NIO)
‣ Save
all
events
to
one
compressed
file
locally
‣ Upload
that
file
every
XX
minutes
to
HDFS/S3
‣ Send
a
message
to
Queue/SQS
for
Demux
H
o
n
u
C
o
l
l
e
c
t
o
r
s
S
Q
S
S
3

43. EVENT ORGANIZATION
‣ Honu
demux
‣ Mul@-­‐stage
batch
processing
pipeline
‣ Elas@c
producer-­‐consumer
‣ Apache
Hadoop
map
reduce
‣ Standalone
map
reduce
mode
‣ Apache
Hive
integra@on

44. HONU DEMUX
‣ Mul@-­‐Stage
batch
processing
pipeline
‣ Bucket
events
to
separate
tables
‣ Write
Hive
par@@on
files
‣ Add
par@@ons
to
Hive
metastore
‣ Merge
par@@ons
Demux
SQS
S3
S3
Standalone
Demux
Standalone
Demux
Standalone
Demux
Standalone
Demux
S3 S3
S3 S3
HIVE
MERGE

45. HONU PIPELINE
HONU
CLIENT
SDK
HONU
COLLECTORS
HONU
DEMUX
ORGANIZECOLLECTGENERATE

46. USE CASE:
PLAYER BEHAVIOR

47. PLAYER BEHAVIOR

48. PLAYER BEHAVIOR INITIATIVES
TRIBUNAL JUSTICE
‣ Community
regulated
‣ In-­‐game
chat
log
‣ Player
stats
‣ Inventory
‣ Game
Info

49. PLAYER BEHAVIOR INITIATIVES
HONOR SYSTEM
‣ Recognize
posi@ve
experience
‣ Improve
sportsmanship

50. STARTUP TIPS
TEAMS THAT USE SMART PINGS TO ALERT OTHER PLAYERS TO THREATS ARE MORE LIKELY TO WIN GAME
PLAYERS WHO FOLLOW THE SUMMONER'S CODE WIN 27% MORE GAMES
THE TRIBUNAL BANS PLAYERS FOR NEGATIVE BEHAVIOR SUCH AS VERBAL HARASSMENT
PLAYERS WHO COOPERATE WITH THEIR TEAM WIN 31% MORE GAMES

51. HOW WE SOLVED IT – EXTEND HONU
HONU
CLIENT
SDK
HONU
COLLECTORS
HONU
DEMUX
ORGANIZECOLLECTGENERATE

52. HONU TOOLS: DRADIS
‣ Hwp
based
data
collec@on
‣ Large
volume
of
data
from
untrusted
source
‣ C10K
‣ Nginx
+
Newy
‣ 4+
billion
API
calls/day
‣ Peak
100K+
calls/sec

53. HONU TOOLS: DRADIS
‣ Json
Messages:
▾ curl
-­‐d
’[
{"messageType":
"Foo",
"@mestamp":
1369064555,
"fact":
"Hello
World!"},
{"messageType":
"Foo",
"@mestamp":
1369064555,
"fact":
"Hello
Dradis!",
"fic@on":
"Hello
Honu!"}]’
‣ Hive
Query:
▾ Select
*
from
foo
where
f[‘fact’]
=
‘Hello
Dradis!’
Table:
Foo

54. HONU TOOLS: ECHO SERVICE
‣ Web
UI
to
easily
and
immediately
visualize
the
data
that
has
been
sent
to
Honu
collectors
‣ Self-­‐service
end-­‐to-­‐end
pipeline

55. HONU TOOLS: ECHO SERVICE
‣ Web
UI
to
easily
and
immediately
visualize
the
data
that
has
been
sent
to
Honu
collectors
‣ Self-­‐service
end-­‐to-­‐end
pipeline

56. HONU TOOLS: ECHO SERVICE
‣ Web
UI
to
easily
and
immediately
visualize
the
data
that
has
been
sent
to
Honu
collectors
‣ Self-­‐service
end-­‐to-­‐end
pipeline

57. HONU TOOLS: METADATA SERVICE
‣ Data
discovery
‣ Schema
management
‣ Counter,
@me

58. HONU TOOLS: REAL-TIME SLICING/DICING
‣ Integration with Platfora
‣ End-user ad-hoc analysis tool
‣ Interactive visual feedback
‣ Realtime exploration/graphing @ 109 data points

59. HONU TOOLS: REAL-TIME SLICING/DICING

60. HONU TOOLS: WORKFLOW MANAGEMENT
ENTERPRISE WORKFLOW
MANAGEMENT
MATT GOEKE
@ LATER TODAY
ClientMobile
WWW

61. HONU STATS
‣ 7+ billion events/day
‣ Tested @ 70+ billion events/day
‣ 100+ tables
▾ 10+ tables @ 100M – 1B rows/day
‣ 7 Petabytes Game Event Dataset
‣ Semi-global deployment
‣ 0 downtime
‣ Runs in cloud (AWS) +
datacenter

62. SUMMARY

63. GOALS
ü Democra@ze
Data
Access
ü Enable
Self-­‐service
Data
Collec@on
and
Analysis
ü Create
Ac@onable
Insights
ü Increase
Speed
to
Insight
HONU
HONU
CLIENT
SDK

64. FUTURE
‣ Improve
self-­‐service
workflow
&
tooling
▾ Metadata
management
▾ Discovery
of
captured
data
▾ Workflow
management
▾ Plauora
to
all
teams
‣ Real@me
event
aggrega@on
‣ Global
data
infrastructure
‣ Replace
legacy
audit/event
logging
services

65. HANDLE INCREASING DATA VELOCITY
JUNE 2012 JULY 2013
MySQL
tables
180
1200
Pipeline
Events/day
0
7+
Billion
Workflows
Cronjob
+
Pentaho
Oozie
Environment
Datacenter
DC
+
AWS
SLA
1
day
2
hours
Event
tracking
• 2+
weeks
(DB
update)
• Dependencies:
DBA
teams
+
ETL
teams
+
Tools
teams
• Down@me
(3h
min.)
• 10
minutes
• Self-­‐Service
• No
down@me

66. DECREASE TEEMO DEATHS?

67. SHAMELESS HIRING PLUG
Like most everybody else at this conference… we’re hiring!
PLAYER EXPERIENCE FIRST
CHALLENGE CONVENTION
FOCUS ON TALENT AND TEAM
TAKE PLAY SERIOUSLY
STAY HUNGRY, STAY HUMBLE
THE RIOT MANIFESTO

68. SHAMELESS HIRING PLUG
AND YES, YOU CAN PLAY GAMES AT WORK
IT’S ENCOURAGED!

69. THANK YOU! QUESTIONS?
BARRY LIVINGSTON
blivingston@riotgames.com
SANDEEP SHRESTHA
sshrestha@riotgames.com