This document discusses building an analytics pipeline with AWS services for purposes such as improving games and increasing player engagement. It recommends using services like Kinesis, Redshift, EMR, Athena, and QuickSight for real-time, batch, and interactive analytics. Examples are given of Fortnite's analytics pipeline and how streaming, batch, and speed layers can be implemented using Kinesis, S3, Elasticsearch, and other AWS services. Templates and a reference architecture are provided to help get started with a gaming analytics solution.

1. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Building an Analytics Pipeline with a small team.
Mark Bate,
Solutions Architect (Games)

2. Redshift
Data warehousing
EMR
Hadoop + Spark
Athena
Interactive analytics
Kinesis Data Analytics
Real time
Elasticsearch Service
Operational Analytics
RDS
MySQL, PostgreSQL, MariaDB,
Oracle, SQL Server
Aurora
MySQL, PostgreSQL
API-driven AI ServicesQuickSight SageMaker
DynamoDB
Key value
ElastiCache
Redis, Memcached
Neptune
Graph
Timestream
Time Series
QLDB
Ledger Database
S3/ Glacier
Glue
ETL & Data Catalog
Lake Formation
Data Lakes
Database Migration Service | Snowball | Snowmobile | Kinesis Data Firehose | Kinesis Data Streams | Managed Streaming for Kafka
Data Movement
Analytics Databases
Business Intelligence & Machine Learning
Data Lake
Managed
Blockchain
Blockchain
Templates
Blockchain
RDS on VMware
DocumentDB
Document
AWS analytics portfolio
Broad and deep portfolio, purpose-built for builders

3. Attract more players
Increase their enjoyment
Longer play sessions
Encourage engagement
“Analytics can’t make a hit game -
but can improve it!”
Analytics flywheel for games

4. Timely decisions require new data in
minutes
Real time Seconds Minutes Hours Days Months
Valueofdatatodecision-making
Preventive/Predictive
Actionable Reactive Historical
Time critical decisions Traditional “batch” business intelligence
Data loses value quickly over time
Source: Perishable insights, Mike Gualtieri, Forrester

5. How do I build content players truly enjoy?
Know what players
are doing and when
• Sessions
• Progress
• Location
• Blockers
• Feedback
Measure and
validate core game
mechanics
• Gameplay
• Player choices
• Goals
• Performance
Experiment
• Scientific method
• Validate feedback

6. Analytic: Level design

7. Analytic: Level design

8. Telemetry
(events)
Metrics
(answers)
Anatomy of an analytics pipeline

9. Telemetry
(events)
{
‘event_id’ : ‘05b00439-6a07-4112-9c8d-165f1643e5d1’,
‘event_type’ : ‘player_death’,
‘event_timestamp’ : ‘2017-05-01T21:05:18.000Z’,
‘arena’ : ‘el_dorado’,
‘position_x’ : 507.12,
‘position_y’ : 551.61
}
Ingest/
Collect
Store
Process/
analyze
Consume/
visualize
Metrics
(answers)
Avoid wasted effort here
Not adding differentiated value
Traditionally difficult to scale
Unpredictable utilization
Anatomy of analytics pipelines

10. Analytics pipeline behind Fortnite
Game
clients
Game
servers
Launcher
Game
services
Grafana
Scoreboards API
Limited raw data
(real time ad-hoc SQL)User ETL
(metric definition)
Spark on EMR Amazon
DynamoDB
NEAR REAL-TIME
PIPELINE
BATCH PIPELINE
ETL
using EMR
Tableau/BI
Ad-hoc SQLS3
(Data lake)
Amazon
Kinesis
APIs
Databases
S3
Other
sources
Analytics platform runs onAWS
125M events/min ingested at peak
Amazon S3 leveraged as a data lake, >35 petabytes
Amazon Kinesis used to collect telemetry data
Real-time analytics done through Spark on Amazon EMR
Amazon EMR for large batch data processing
Game designers use data to inform their decisions

11. Streaming with Amazon Kinesis
SQL
Analyze data streams with SQL or
Java (Flink)
Kinesis Data Analytics
Capture, process, and store data
streams
Kinesis Data Streams
Load data streams into data
stores
Kinesis Data Firehose
Easily collect, process, and analyze data and video streams in real time

12. Streaming real-time data with AWS

13. Hadoop with Amazon EMR
Your own code on Amazon EC2
Centralized Data lake / Single
source of truth /
Data Warehouse
(Amazon S3,
Amazon Redshift)
Ingest
data streams
Kinesis Data
Streams
Kinesis Data
Analytics
Analyze data
streams with SQL
or Java
Kinesis Data
Firehose
Load into
data stores
Serverless w/ AWS Lambda
Amazon Kinesis for analytics pipelines
Collect, process, and analyze data streams in real time
Flexible
processing options

14. Batch Layer
Game
Backend
Amazon
Kinesis Data
Firehose
Game
Clients
Visualization
Dashboard
Amazon S3 Amazon AthenaAWS Glue

15. Speed Layer
Amazon
Kinesis Data
Firehose
Game
Clients
Amazon
Elasticsearch
Service
Kibana
Dashboard
Game
Backend

16. Authentication options
Reference: Serverless Real-Time Analytics for Mobile Gaming

17. Get started!
Gaming Analytics Pipeline Solution
• Reference architecture with pre-built templates,
instructions
• Launch into your account with AWS CloudFormation
• Learn more:
https://aws.amazon.com/solutions/gaming-analytics-
pipeline/
• Reference:
https://d1.awsstatic.com/architecture-
diagrams/ArchitectureDiagrams/serverless-analytics-
for-mobile-gaming.pdf