Apache Kafka in Gaming Industry (Games, Mobile, Betting, Gambling, Bookmaker, Video Streaming)

Apache Kafka and Event Streaming
in the Gaming Industry
Use Cases, Architectures and Technologies
Kai Waehner
Technology Evangelist
contact@kai-waehner.de
LinkedIn
@KaiWaehner
www.confluent.io
www.kai-waehner.de

Apache Kafka in the Gaming Industry – @KaiWaehner - www.kai-waehner.de
1. The Evolution of the Gaming Industry
2. Event Streaming
3. Infrastructure Operations
4. Game Telemetry
5. Monetization Network
6. Payment Engine
7. Fraud Detection
8. Customer 360
Agenda

The Evolution of the Games Industry
Triple A
Casual /
mid-core
games
Mobile
MOBA

https://www.bloomberg.com/news/articles/2019-01-23/peak-video-game-top-analyst-sees-industry-slumping-in-2019

• Professional eSports
• $$$ Tournaments
• Live video streaming
• Real-time betting
https://www.esportsearnings.com/games

Various Business Models
• Hardware sales
• Game sales
• Free-to-play + in-game purchases
One-time in-game purchases (skins, champions, misc.)
Gambling (Loot boxes)
• Game-as-a-service (Subscription)
Seasonal in-game purchases like passes for theme
events, mid-season invitational & world
championship, passes for competitive play
• Game-Infrastructure-as-a-Service
• Merchandise sales
• Community
eSports broadcast, ticket sales, franchising fees)
• Live betting
• Video streaming
including ads, rewards, etc.
• …

In-Game AI
NPCs
Environments
Features
Fraud
Detection
Cheating
Financial
Child abuse
Game
Analytics
Retention
Game changes
(Real-time or next patch)
Research
Find new algorithms
Improve AI
Convert to business problems
Evolution of “AI” (Artificial Intelligence) in Gaming
https://us.forums.blizzard.com/en/wow/t/actions-taken-to-address-exploitative-gameplay/558339
https://www.youtube.com/watch?v=qPGPpcgY5Rk
https://www.researchgate.net/publication/258832679_Behavior_Evolution_in_Tomb_Raider_Underworld
https://www.youtube.com/watch?v=A89ZDjF51Nk
1980s: Business rules
2000s: Finite state machines
2010s: Behaviour trees
2020s: Machine Learning
Gold Farming
2020: 74,000 Bots banned

Huge demand to build an open, flexible, scalable platform
• Real time
• Scalability
• High availability
• Decoupling
• Cost reduction
• Flexibility
• Elasticity
• Standards-based
• Extendibility
• Security
• Infrastructure-independent
• Multi-region / global

Event Streaming for
Traditional and Innovative New Use Cases
in the Gaming Industry
Real Time Processing
Digital Transformation
Strategic Goals
Live events and
tournaments
Real time
advertising
Integration with
video streaming
platforms
Financial fraud
detection
Infrastructure
monitoring
(data center, network,
Game, player)
Cross-platform
chat function
Analytics of game
telemetry
(real-time + batch)
Live operations
(e.g. changes
during the game)
Automated
and intelligent
match making
Marketing
campaigns
based on
meta data
Recommendation of
in-game purchases
Payment
processing
Rewards
(e.g. in-game
content)
Cheating Ban
Gaming
infrastructure as
a Service
Social network
integration
IoT integration
(e.g. casinos,
consumer IoT)
Geolocation-
based
services
Customer 360
(e.g. loyalty
system)

STREAM
PROCESSING
Create and store
materialized views
Filter
Analyze in-flight
Time
C CC
Event Streaming

An Event Streaming Platform
is the Underpinning of an Event-driven Architecture
Microservices
Mainframes
SaaS apps
Mobile
Customer 360
Real-time fraud
detection
Data warehouse
Producers
Consumers
Database
change
Microservices
events
SaaS
data
Customer
experiences
Streams of real time events
Stream processing
apps
Connectors
Connectors
Stream processing
apps

Hybrid Architecture - On Premise vs. Public Cloud
VM
SELF MANAGED
FULLY MANAGED
Cloud-first (greenfield) vs.
Hybrid architecture vs.
Strategic move from on premise to cloud
Regional vs. global deployments
Betting is more regional (mainly because of laws)
Games are more global

Infrastructure operations
Live monitoring and troubleshooting
Monitor the results of live operations
• Game clients, game servers, game services
• Service health 24/7
• Special events such as weekend tournaments, limited
time offers and user acquisition campaigns
Real-time monitoring
• Troubleshooting
• Who creates the problem? Client? ISP? Game itself?
Example: New marketing campaign
• “Play for free over the weekend”
• Scalability - Huge extra traffic
• Monitoring – Was the marketing campaign successful?
How profitable is the game / business?
• Real time (e.g. alerting)
• Batch (e.g. analytics and reporting of success with
Snowflake)

Elasticity is key for the games industry
“We have an almost 10 times
difference in workloads
between peak and low-peak.
Elasticity is really, really
important for us in any
particular region at the cloud
providers.”
Chris Dyl,
Director of Platform at Epic Games
(@ AWS Summit 2018)
https://www.youtube.com/watch?v=MCLrA401vHw

https://www.confluent.io/blog/scaling-kafka-to-10-gb-per-second-in-confluent-cloud/
Elastic scalability in the cloud
Throughput/DataTraffic
demand capacity

Confluent Cloud
Milliseconds Minutes
Basic, Standard [0-100Mbps]
Do Nothing
Elastic Scaling with fully-managed Confluent Cloud
*Even in public clouds provider quotas for VMs, disks, security groups can be encountered causing delays. Confluent has these limits raised already.
Dedicated [Mbps - Gbps]
1 Click—Select CKU from drop down in
cluster management UI and click Apply
Changes
Other Kafka Services
Days - Weeks
Determine how much capacity is needed
Procure capacity*
Configure new brokers
a. Disks b. OS c. Network d. Kafka (application)
Identify partitions on specific brokers to
rebalance & topics they are part of
For each Topic: migrate partitions
a. Increase ISR +1 b. Wait for new replica to sync
c. Failover master d. Reduce ISR -1 e. Delete old replica

Dynamic Scaling with Confluent Platform
T3, P0
T1, P0
T2, P1
T3, P0
T1, P1
T2, P0
T3, P1
Broker 1 Broker 2 New: Broker 3
T1, P0 T2, P1T1, P1
T2, P0T3, P1
Broker 1 Broker 2 Broker 3
Scaling with OSS Kafka
T3, P0T1, P0 T2, P1
Broker 1 Broker 2 New: Broker 3
Object Store
Broker 1 Broker 2 Broker 3
Object Store
T3, P1T1, P1 T2, P0
T3, P0T1, P0 T2, P1T3, P1 T1, P1 T2, P0
Scaling with Tiered Storage +
Self-balancing Clusters + Kubernetes Operator
Rebalance in Hours... Rebalance in Seconds...

Game Telemetry
Real-time analytics and data correlation
• Game telemetry
• = How the player plays the game
• Business logic: user actions (button clicks, shooting, use
item), game environment metrics (quests, level up)
• Technical information: login from a specific server, IP
address, location
• Global Gaming
• Proxies all over the world à Regional latency, millions of
clients
• Central analytics cluster à Anonymize data
• Game monitoring
• How well do players progress through the game and what
problems occurred
• Live operations – Adjust the gameplay
• Server-side changes while the player is playing the game
(e.g. time-limited event à give reward)
• Real-time updates to improve the game or align to
audience needs (or in other words: Recommend an item /
upgrade / skin / other in-game purchase

Real time game telemetry analytics
gRPC
HTTP
TCP
Normalize telemetry streams to
a common schema
Apply real time analytics across ALL sources
via the normalized telemetry stream
Single source of insight
into the performance and
availability of the entire
network infrastructure.

Big Fish Games
Real-time analytics of game telemetry data for live operations
(aka increasing revenue while the player plays the game)
https://www.confluent.io/kafka-summit-sf18/how-big-fish-games-developed-real-time-analytics/
Casual and mid-core games.
2.5 billion games to customers in
150 countries, representing over
450 unique mobile games and
over 3,500 unique PC games.
https://www.confluent.io/kafka-summit-sf18/how-big-fish-games-developed-real-time-analytics/

Automated matchmaking
• Request matchmaking
• Discover potential matches
• Get player acceptance
• Find resources to host match
• Start new game session
• Connect players to new game session
https://www.researchgate.net/figure/Typical-multiplayer-cloud-gaming-architecture_fig1_336167521
http://patft.uspto.gov/netacgi/nph-Parser?Sect2=PTO1&Sect2=HITOFF&p=1&u=/netahtml/PTO/search-
bool.html&r=1&f=G&l=50&d=PALL&RefSrch=yes&Query=PN/9789406
https://www.pcgamer.com/activision-wins-patent-that-uses-matchmaking-to-make-you-want-to-buy-stuff/
https://www.kotaku.com.au/2017/10/activision-patents-matchmaking-that-encourages-players-to-buy-microtransactions/
“System and method for driving
microtransactions in multiplayer
video games”
Data Correlation
and Analytics
(as part of the story)

Monetization network
• In-game advertising
• Micro-transactions and in-game purchases
Sell Skins, Upgrade to next level, …
• Game-Infrastructure-as-a-Service
Multi-platform-and-store-integration, matchmaking, advertising,
player identity and friends, cross-play, lobbies, leader boards,
achievements, game analytics, ...
• Partner network
Cross-sell game data, game SDK, game analytics, ...

Monetization network
Mobile
CRM
Payment
Integration
Player
Live Data
Player
History
Database
change
User
data
Game
data
Payment
data
Real time
advertising
Game activities
Player retention
Game-Infrastructure-
as-a-Service
Payment
Ad

Unity Ads - Monetization network
• In 2019, content installed 33 billion times reaching 3
billion devices worldwide
• Real-time 3D development platform
• One of the largest monetization network in the world
• Reward players for watching ads
• Incorporate banner ads
• Incorporate Augmented Reality (AR) ads
• Playable ads
• Cross-Promotions
• Unity is a data-driven company;
• Average about half a million events per second
• Handles millions of dollars of monetary transactions
• Data infrastructure based on Confluent Platform, Confluent Cloud
and Apache Kafka
• Single common data pipeline for analytics, R&D,
monetization, cloud services, etc. for real-time and
batch processing
• Real-time monetization network
• Feed machine learning models in real-time
• Data lake went from two-day latency down to 15 minutes
https://www.confluent.io/blog/how-unity-uses-confluent-for-real-time-event-streaming-at-scale/

Chat function within games and cross-platform
Send text, in-game screenshots, items, etc.
Player
Player
Database
change
Microservices
events
SaaS
data
Customer
experiences
Text message
In-game
screenshot
Confluent
REST Proxy
Game analytics
Reporting
C++
C# + Lua / Python
HTTP
Java
JavaScript
+
Swift
Kafka Connect

Payment infrastructure
Real-time + scalable + reliable + technology-independent
• Games industry
• Betting
• Casino
• 3D game engines
• Video streaming
• 3rd services

Betting / Bookmaking
In the loop in real time… To make new bets quickly… Get another margin…
https://qconlondon.com/system/files/presentation-slides/ian_thomas_-_designing-a-global-sportsbook-final.pdf

Real time betting infrastructure
Customer
Customer
Profile
Database
change
Microservices
events
SaaS
data
Customer
experiences
Live Bet
Historical
Transactions
Confluent
REST Proxy
Batch Analytics
(Initial Odds)
Real Time Analytics
(Live Odds)
Real-time
Compliance checks
Risk
Calculation

Betting delay and approval in live bets
Synthetic delay to calculate risk, adjust odds, increase margin, reject bet if “too good”
Stateful Correlation of Events
Live Bet
(HTTP from Mobile App)
Time
Betting Engine
(ksqlDB)

William Hill
From a Monolith to a flexible, scalable microservice architecture
• Kafka as central, reliable
streaming infrastructure
• Kafka for messaging,
storage, cache and
processing of data
• Independent decoupled
microservices
• Decoupling and
replayability
• Technology independence
• High throughput + low
latency + real time
https://www.codemesh.io/codemesh2015/peter-morgan
https://www.confluent.io/kafka-summit-london18/building-low-latency-high-throughput-pipelines-with-kafka-from-scratch/

William Hill
The trading platform for millions of bets every day
• Kafka is the heart of all
events and transactions
• “process-to-process”
execution in real time
• Integration with analytic
models for real time
machine learning
• Various data sources and
data sinks (real time,
batch, request-response)
https://www.codemesh.io/codemesh2015/peter-morgan
https://www.confluent.io/kafka-summit-london18/building-low-latency-
high-throughput-pipelines-with-kafka-from-scratch/

Bookmaker Business == Banking Business
Not everyone can start from greenfield
à Mainframe integration, offloading and replacement
https://www.kai-waehner.de/blog/2020/04/24/mainframe-offloading-replacement-apache-kafka-connect-ibm-db2-mq-cdc-cobol/
Regulated market, PII data à Data processing on premise
Offloading non-PII data to cloud à Analytics
Legacy technologies like mainframe à Key cost factor, monolithic and inflexible

Kafka Clusters
can Stretch over Regions
Zero Downtime + Zero Data loss
(RPO=0 and RTO=0)
e.g. Stretched over US East + Mid + West
Automate Disaster Recovery
Sync or Async Replication per Topic
Offset Preserving
Automated Client Failover without
Custom Code
Multi-Region Cluster
(Only available in Confluent Platform)

Global gaming
Aggregate Small Footprint
Edge Deployments with
Replication (Aggregation)
Simplify Disaster Recovery
Operations with
Multi-Region Clusters
with RPO=0 and RTO=0
Stream Data Globally with
Replication and Cluster Linking

A global gaming platform
Central data infrastructure and application clusters in geographically diverse locations
• Gambling at the edge + analytics in the cloud
• Data quality is key for legal compliance
• Data locality for PII data
• Responsible gaming compliance
• Client retention is key - keep engagement
• Real-time streaming analytics use cases
• Player winning / losing streak
• Player conversion
• Player game achievement
• Fraud detection - e.g. payment windows
• Long running windows per player over days / months
• Tournaments
• Incentive unhappy players with additional free credit
• Reports to regulator - replay old events in guaranteed order
• Geolocation to enable features, limitations or commissions

Fraud in the gaming industry
“49 percent of the
7,200 companies
they surveyed had
experienced fraud
of some kind”
www.kai-waehner.de | @KaiWaehner
Real time analytics for detecting anomalies
• Cheating: Fake accounts, bots, …
• Financial fraud: match-fixing, stolen credit cards, …

Apache Kafka as Infrastructure for ML

Apache Kafka’s Open Ecosystem as Infrastructure for ML
Kafka
Streams/
ksqlDB
Kafka
Connect
Confluent REST Proxy
Confluent Schema Registry
Go/.NET/Python
Kafka Producer
ksqlDB
Python
Client

Fraud Department
C++
Edge
Gateway
MQTT
Analytics
Department
Spark
BI
Tableau
Event
Streaming
Kafka Cluster
Integration
Kafka Connect
Stream
Processing
ksqlDB
Live Bet via Mobile App
WebSocket
Event Streaming Platform
Other Components
Real Time
Application
Java
(6b) All Data
(7) Potential Fraud
(3)
Read Data
Model Training (5)
Deploy
Fraud Model
(8b) Alert Fraud Department (e.g. Mobile App)
(2)
Correlate
Data
(6a) Consume bet data
Fraud Model
Integration
Kafka Connect
(8a) Block user
(4)
Train Fraud Model
(1)
Ingest Data
Database
Oracle
Integration
GoldenGate
Consumer
Python
Streaming Analytics for Fraud Detection at Scale

”Legal fraud” in tennis games
https://www.youtube.com/watch?v=kvT_yIVI2Qw
Bet on the next point
• before result of the actual play reaches
the betting engine
• after the play happens
Challenge #1: How to detect this “fraud”?
Challenge #2: How to solve it (without
killing this revenue stream)?

Detection of financial fraud and cheating
Jupyter notebooks analyzing data pre-processed with ksqlDB
56
SELECT bet_id, event_id, bet_sport_id, feed_content
FROM bet_sensor c
LEFT JOIN bet_sport m ON c.bet_sport_id = m.bet_sport_id
WHERE m.bet_sport_type =‘soccer';

Customer 360
Real-time and context-specific acquisition, engagement and retention
• Real Time Event Streaming
• Game event triggers
• Personalized statistics and odds
• Player segmentation
• Campaign orchestration (“player journey”)
• Loyalty system
• Rewards
e.g. upgrade, exclusive in-game content, beta keys for
announcement event
• Avoid customer churn
• Cross-selling
• Social Network integration
• Twitter, Facebook, …
• Example: Candy Crush
• Partner integration
• API Management
https://xtremepush.com/in-play-betting-and-real-time-engagement-for-sportsbooks/

Customer 360, loyalty and rewards
Streamer
(MOBA Players / eSports Studio)
Spectator
Database
change
Microservices
events
SaaS
data
Customer
experiences
Game telemetry
Video stream
and chat
Twitch
Reward
Drops
API
CRM

Sky Betting & Gaming
Realtime streaming architecture with Kafka’s ecosystem
for customer 360 use cases
“Most of our streaming data is in the form of topics on a Kafka cluster. This means we can
use tooling designed around Kafka instead of general streaming solutions with Kafka
plugins/connectors.
Kafka itself is a fast-moving target, with client libraries constantly being updated; waiting
for these new libraries to be included in an enterprise distribution of Hadoop or any off
the shelf tooling is not really an option. Finally, the data in our first use-case is user-
generated and needs to be presented back to the user as quickly as possible.”
https://sbg.technology/2017/01/23/streaming-architectures/
https://www.youtube.com/watch?v=P7iY3EPP_TY

Hotstar – OTT for millions of cricket fans in India
https://www.confluent.io/kafka-summit-san-francisco-2019/scaling-for-indias-cricket-hungry-population/

IoT integration
• Live eSports events, TV, video streaming and news stations
• Fan engagement
• Audience communication

Cross-Company Kafka Integration
Streaming Replication and API Management
MirrorMaker 2
Confluent Replicator
Bookmaker
Video
Streaming
Streaming integration
between companies
API Management
(REST et al) are not
appropriate for
streaming data
Infosec and politics are
your biggest hurdle
Game
Publisher

Confluent Platform
Fully Managed Cloud ServiceSelf Managed Software FREEDOM OF
CHOICE
COMMITTER-DRIVEN
EXPERTISE
PartnersTrainingProfessional
Services
Enterprise
Support
Apache Kafka
EFFICIENT
OPERATIONS AT SCALE
PRODUCTION-
STAGE PREREQUISITES
UNRESTRICTED
DEVELOPER PRODUCTIVITY
SQL-based
Stream Processing
KSQL (ksqlDB)
Rich Pre-built Ecosystem
Connectors | Hub | Schema Registry
Multi-language Development
non-Java clients | REST Proxy
GUI-driven Mgmt & Monitoring
Control Center
Flexible DevOps Automation
Operator | Ansible
Dynamic Performance &
Elasticity
Auto Data Balancer | Tiered Storage
Enterprise-grade Security
RBAC | Secrets | Audit logs
Data Compatibility
Schema Registry | Schema Validation
Global Resilience
Multi-Region Clusters | Replicator
Developer Operator Architect
Open Source | Community licensed
PARTNERSHIP
FOR BUSINESS SUCCESS
Complete
Engagement Model
Revenue / Cost / Risk
Impact
TCO / ROI
Executive Buyer

I N V E S T M E N T & T I M E
VALUE
3
4
5
1
2
Event Streaming Maturity Model
Initial Awareness /
Pilot (1 Kafka Cluster)
Start to Build Pipeline /
Deliver 1 New Outcome
(1 Kafka Cluster)
Mission-Critical
Deployment
(Stretched, Hybrid,
Multi-Region)
Build Contextual
Event-Driven Apps
(Stretched, Hybrid,
Multi-Region)
Central Nervous
System
(Global Kafka)
Product, Support, Training, Partners, Technical Account Management...

Kai Waehner
Technology Evangelist
contact@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
www.confluent.io
LinkedIn
Questions? Feedback?
Let’s connect!

Apache Kafka in Gaming Industry (Games, Mobile, Betting, Gambling, Bookmaker, Video Streaming)

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Apache Kafka in Gaming Industry (Games, Mobile, Betting, Gambling, Bookmaker, Video Streaming)

Similar to Apache Kafka in Gaming Industry (Games, Mobile, Betting, Gambling, Bookmaker, Video Streaming) (20)

More from Kai Wähner

More from Kai Wähner (20)

Recently uploaded

Recently uploaded (20)

Apache Kafka in Gaming Industry (Games, Mobile, Betting, Gambling, Bookmaker, Video Streaming)