Kafka Summit SF 2017 - Riot's Journey to Global Kafka Aggregation

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

RIOT’S JOURNEY TO
GLOBAL KAFKA AGGREGATION

SINGE GRAHAM
SYSTEMS ENGINEER
DATA PRODUCTS & SOLUTIONS
SGRAHAM @
RIOTGAMES.COM
INTRODUCTION
SYSTEMS ENGINEER
DATA PRODUCT...

AGENDA

RIOT

WHERE WE CAME FROM

WHERE WE’RE AT

WHERE WE’RE GOING

2009
LAUNCH
TEAM
ORIENTED
130+
CHAMPS
MODERN
FANTASY
LEAGUE OF LEGENDS

STATS RELEASED SEPTEMBER 2016
27 MILLION
DAILY ACTIVE
PLAYERS
MORE THAN
LEAGUE PLAYER BASE
100 MILLION
MONTHLY ACTIVE
PLAY...

WHERE WE CAME FROM

COMPLEX ARCHITECTURE
Dozens of different services maintained by independent teams
CHALLENGES

COMPLEX ARCHITECTURE
Dozens of different services maintained by independent teams
OPERATIONAL DATA
Teams were often on the...

COMPLEX ARCHITECTURE
Dozens of different services maintained by independent teams
OPERATIONAL DATA
Teams were often on the...

HISTORICAL OPS DATA
➢ Powered by Zabbix

HISTORICAL OPS DATA
➢ Powered by Zabbix
➢ No common logs aggregation

HISTORICAL OPS DATA
➢ Powered by Zabbix
➢ No common logs aggregation
➢ Occasional ELK stacks, duplicated effort

PERFORMANCE ISSUES
➢ Delayed publishing to warehouse due to batch-driven nature of collector

PERFORMANCE ISSUES
➢ Delayed publishing to warehouse due to batch-driven nature of collector
➢ Slow chain of derived batch...

PERFORMANCE ISSUES
➢ Delayed publishing to warehouse due to batch-driven nature of collector
➢ Slow chain of derived batch...

PROBLEMS AMPLIFIED

WHAT NOW?
➢ Over-siloed Operational Data (Metrics)
➢ Non-collected / aggregated Operational Data (Logs)
➢ Slow / Untimely ...

ENTER KAFKA
➢ Massively scalable
➢ Easy aggregation from multiple clusters
➢ Operationally straightforward

WHERE WE’RE AT

REGIONAL COLLECTION
➢ Regional data buffers

REGIONAL COLLECTION
➢ Regional data buffers
➢ 6 Node Clusters per region
➢
➢
○ 12 x 3.4Ghz
○ 192GB RAM
○ 6TB NVMe

REGIONAL COLLECTION
➢ Regional data buffers
➢ 6 Node Clusters per region
○ 12 x 3.4Ghz
○ 192GB RAM
○ 6TB NVMe
➢ Global agg...

GLOBAL AGGREGATION
➢ Initially planned to replicate over the public Internet

GLOBAL AGGREGATION
➢ Initially planned to replicate over the public Internet
➢ Not as terrible as it sounds!

GLOBAL AGGREGATION
➢ Initially planned to replicate over the public Internet
➢ Not as terrible as it sounds!
➢ But not gre...

GLOBAL AGGREGATION
➢ Initially planned to replicate over the public Internet
➢ Not as terrible as it sounds!
➢ But not gre...

REPLICATION FLOW
KAFKA
KAFKA KAFKA KAFKA
KAFKA CONNECT
REPLICATOR
AWS
DC #1 DC #2 DC #3

MESSAGE REPLICATION
➢ Started out with MirrorMaker

MESSAGE REPLICATION
➢ Started out with MirrorMaker
➢ Performed well, but prone to frequent crashing

MESSAGE REPLICATION
➢ Started out with MirrorMaker
➢ Performed well, but prone to frequent crashing
➢ Kafka Connect Replic...

PROTIP: REPLICATION
MIRRORMAKER
‣ Offsets stored in ZooKeeper
KAFKA CONNECT REPLICATOR
‣ Offsets stored in Kafka
‣ No easy...

PROTIP: REPLICATION
MIRRORMAKER
‣ Offsets stored in ZooKeeper
KAFKA CONNECT REPLICATOR
‣ Offsets stored in Kafka
‣ No easy...

MESSAGE COLLECTION
➢ Abstract away Kafka from producers to minimize integration friction
PRODUCER COLLECTOR
KAFKA
BROKERS
...

MESSAGE COLLECTION
➢ Abstract away Kafka from producers to minimize integration friction
➢ Synchronous message processing
...

WATCHING THE WATCHER

PROTIP: PARTITION REASSIGNMENT
KAFKA
‣ Scales horizontally extremely well
‣ Has no ‘smart’ logic when it comes to partitio...

PROTIP: PARTITION REASSIGNMENT
KAFKA
‣ Scales horizontally extremely well
‣ Has no ‘smart’ logic when it comes to partitio...

LEAGUE OF (LOW) LATENCY
League of Legends requires low latency between the player and servers for the best experience.
FAC...

LEAGUE OF (LOW) LATENCY
League of Legends requires low latency between the player and servers for the best experience.
FAC...

LEAGUE OF (LOW) LATENCY
FACT
Milliseconds matter
League of Legends requires low latency between the player and servers for...

RIOT DIRECT
➢ Monitor network stats from game clients

RIOT DIRECT
➢ Monitor network stats from game clients
➢ Work with ISPs directly to resolve issues

RIOT DIRECT
➢ Monitor network stats from game clients
➢ Work with ISPs directly to resolve issues
➢ Data timeliness issues

Interesting fact.
OH WOW, THAT IS REALLY COOL.
your image here

WHERE WE’RE GOING

REAL-TIME ANALYTICS
➢ Grouping batches of data to the Warehouse? Gone.

REAL-TIME ANALYTICS
➢ Grouping batches of data to the Warehouse? Gone.
➢ Waiting for derived processing jobs to complete? ...

REAL-TIME ANALYTICS
➢ Grouping batches of data to the Warehouse? Gone.
➢ Waiting for derived processing jobs to complete? ...

BI-DIRECTIONAL MESSAGING
GLOBAL
DC DC DC

BI-DIRECTIONAL MESSAGING
DCGLOBAL
DC DC DC

BI-DIRECTIONAL MESSAGING
DCGLOBAL
DC DC DCDC DC

STREAMING METRICS AGGREGATION
➢ For trends, not triage

STREAMING METRICS AGGREGATION
➢ For trends, not triage
➢ KStreams + KTables = Easy stateful processing

PERSONAL INFORMATION (PI) HANDLING
➢ Not acceptable to blindly ingest data

PERSONAL INFORMATION (PI) HANDLING
➢ Not acceptable to blindly ingest data
➢ Regional stream processors apply local data s...

PERSONAL INFORMATION (PI) HANDLING
➢ Not acceptable to blindly ingest data
➢ Regional stream processors apply local data s...

https://engineering.riotgames.comENGINEERING
BLOG
WRAPPING IT UP
https://riotgames.com/careersWE’RE
HIRING
Come pick our b...

THANK YOU!

Kafka Summit SF 2017 - Riot's Journey to Global Kafka Aggregation

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

2024 Trend Updates: What Really Wor... by Search Engine Jou... 1026982 views
Storytelling For The Web: Integrate... by Chiara Aliotta 937145 views
Artificial Intelligence, Data and C... by OECD Directorate ... 853259 views
How to Leverage AI to Boost Employe... by SocialHRCamp 357183 views
2024 State of Marketing Report – by... by Marius Sescu 203199 views
Everything You Need To Know About C... by Expeed Software 224364 views

View on Slideshare

1 of 65 Ad

View on Slideshare

1 of 65 Ad