Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

GAM310_Build a Telemetry and Analytics Pipeline for Game Balancing

272 views

Published on

In this workshop, we will together build telemetry/analytics data processing pipelines to assist game developers/architects, designers and producers. We will use a fictitious RPG and ingest data from in-game events. We will then analyze the data to help with game balancing, troubleshooting and other relevant recommendations for game developers and designers. As a participant, you will use Amazon Kinesis, Amazon Kinesis Firehose, Amazon Analytics, Amazon EMR, Amazon Redshift, Amazon S3, Amazon Athena and Amazon QuickSight. Prerequisites include having your own laptop and an interest in big data services, game data processing & analytics.

In order to make your time in the workshop as productive as possible, please make sure to check out the additional information below.


AWS account: Fully functional AWS Account with administrative access. Participant should have the ability to create & destroy resources in the us-west-2 and eu-west-1 regions via API, CLI & AWS Console.
Device/OS: A laptop computer – running Mac OS X, a Linux flavor or Windows. The computer will need a functional ssh/Remote Desktop client.
AWS service familiarity/experience:Familiarity/Experience with EC2, S3 & the AWS Console will be good. For the rest of the services, we will introduce each during the workshop.
Audience: Game Developers (server programmers), Architects, Game Producers/Designers, Game Marketing/Analytics team – hands-on members

  • Be the first to comment

  • Be the first to like this

GAM310_Build a Telemetry and Analytics Pipeline for Game Balancing

  1. 1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS re:INVENT Build a Telemetry and Analytics Pipeline for Game Balancing Nirav Doshi - niravdd@amazon.com - Sr. Technical Account Manager, AWS ANZ Brent Maxwell - braxwell@amazon.com - Solutions Architect, AWS ANZ G A M 3 1 0 N o v e m b e r 2 7 , 2 0 1 7
  2. 2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Workshop Agenda (2 hours, 30 mins) Prerequisites, preparation, setup Game data, analytics The game The bot 1. Analytics pipeline (45 mins) 2. Serverless analytics pipeline (15 mins) 3. Analytics on the stream (20 mins) 4. Big data analytics (40 mins) Learnings, Q&A
  3. 3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Prerequisites, Preparation & Setup AWS account and admin access setup The instructional guide - amzn.to/GAM310 - Start with the prerequisites section - Keep your ACCESS_KEY/SECRET_KEY handy Experimentation encouraged!… however - - Follow the instructions precisely, avoid digressing - Try out options, only when asked to Workshop support
  4. 4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Game Data & Analytics Real-time player and game data - Game balancing - Marketing - Fraud investigation - Debugging, other Gaming companies & data crunching Common practices Challenges Self-managed large on- premises data crunching environments - Scalability - Spiraling Costs (Infra, Resources) Third-party products and services - Data Ownership/Restrictions - Security
  5. 5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Game Data & Analytics Harvest your data! Using flexible, extensible, managed solutions Ingest — Store — Analyze — Consume Interact with your data any way you like Pick and choose tools, customize interactions Leverage the scalability of the cloud Take ownership of your data in a secure manner
  6. 6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. The Game Fictitious role-playing game Game client emits data - Player demographics - Player unit—attributes - Enemy units—attributes
  7. 7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. The Bot Simple data generator* - How a real-time streaming app behaves? - KPL, faker.js - Game-play data is generated by the bot Game-play data constitutes - Player demographics - Game-play data – game level, squad stats, boss stats, result * Thanks to Bashar Al-fallouji (AWS Solutions Architect, Sydney) for his excellent bot!
  8. 8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 1. Analytics Pipeline
  9. 9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Purpose & Outcome Basic pipeline, easily customizable Similar to existing analytics products/services in the market Store raw stream data to Amazon Redshift and visualize Game balancing, analysis
  10. 10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Kinesis Streams For developers Collect and stream data for ordered, re-playable, real- time processing Amazon Kinesis Firehose For developers, data scientists Easily load massive volumes of streaming data into Amazon S3, Amazon Redshift or Amazon EMR Amazon Kinesis Analytics For developers, data scientists Easily analyze data streams using standard SQL queries in real-time Amazon Kinesis
  11. 11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Kinesis Streams Storage AWS Global Infrastructure App services Deployment and administration Networking Analytics Compute Database • Ingest streaming data • Process data in real-time • Stream terabytes of data per hour
  12. 12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Kinesis Streams Store data as a continuous stream • Easy administration: simply create a new stream and set the desired level of capacity with shards. Scale to match your data throughput rate and volume • Build real-time applications: perform continual processing on streaming big data using Amazon Kinesis Client Library (KCL), Apache Spark/Storm, AWS Lambda, and more • Low cost: cost-efficient for workloads of any scale Send clickstream data to Kinesis Streams Kinesis Streams stores and exposes clickstream data for processing Custom application built on Kinesis Client Library makes real-time content recommendations Readers see personalized content suggestions
  13. 13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Load massive volumes of streaming data into destinations • Zero administration: capture and deliver streaming data into Amazon Simple Storage Service (S3), Amazon Redshift, and other destinations without writing an application or managing infrastructure • Direct-to-data store integration: batch, compress, and encrypt streaming data for delivery into data destinations in as little as 60 seconds using simple configurations • Seamless elasticity: seamlessly scale to match data throughput without intervention Amazon Kinesis Firehose Capture and submit streaming data to Kinesis Firehose Kinesis Firehose loads streaming data continuously into Amazon S3 and Amazon Redshift Analyze streaming data using your favourite BI Tools
  14. 14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Storage AWS Global Infrastructure App services Deployment and administration Networking Compute Database Analytics Amazon S3 • Scalable object storage • 1 byte to 5 TB in size per object, unlimited number of objects • 99.999999999% durability, 99.9% availability • Regional service, no single points of failure • Server Side Encryption (SSE)
  15. 15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Deployment and administration Compute Storage AWS Global Infrastructure Database App services Networking Analytics Amazon Redshift • Managed massively parallel petabyte scale data warehouse • Streaming backup/restore to Amazon S3 • Load data from Amazon S3, Amazon DynamoDB and Amazon EMR • Extensive security features • Online scaling from 160 GB to 2 PB
  16. 16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Storage AWS Global Infrastructure App services Deployment and administration Networking Analytics Compute Database Amazon QuickSight • BI service, perform impromptu analysis • Build visualizations • Share and collaborate via storyboards • Native access on major mobile platforms
  17. 17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Analytics Pipeline
  18. 18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Analytics Pipeline amzn.to/GAM310
  19. 19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 2. Serverless Analytics Pipeline
  20. 20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Purpose & Outcome Use Amazon S3 for data storage and Amazon Athena to query the data Powerful and easy tool for data scientists, game designers, game producers, and server developers Perform standard SQL queries on stored data You may also use Amazon Elasticsearch (Amazon ES) and Kibana to process and visualize data as part of another serverless analytics pipeline Suitable for server/client log processing and querying
  21. 21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Storage AWS Global Infrastructure App services Deployment and administration Networking Analytics Compute Database Amazon Athena • Query and analyze data stored on Amazon S3 with standard (ANSI) SQL queries • No ETL required • Serverless and simple • Pay per query, on the amount of data scanned
  22. 22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Storage AWS Global Infrastructure App services Deployment and administration Networking Analytics Compute Database Amazon Elasticsearch Service • Setup an Amazon Elasticsearch Service cluster in minutes • Integrated with Logstash and Kibana • Scale Amazon ES clusters seamlessly
  23. 23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Serverless Analytics Pipeline amzn.to/GAM310
  24. 24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 3. Analytics on the Stream
  25. 25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Purpose & Outcome Execute real-time SQL queries on the streaming data Create alerts, get notified, and act/respond quicker
  26. 26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. • SQL queries on streams: easily connect to a Kinesis stream or Firehose delivery stream and run SQL queries against the data • Build real-time applications: perform continual processing on streaming big data with sub-second processing latencies • Easy scalability: elastically scales to match data throughput Amazon Kinesis Analytics Connect to Kinesis Streams, Kinesis Firehose delivery streams Run standard SQL queries against the data streams Kinesis Analytics can send processed data to analytics tools so you can create alerts and act in real time
  27. 27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Analytics on the Stream amzn.to/GAM310
  28. 28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4. Big Data Analytics
  29. 29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Purpose & Outcome Process large amount of data using a Hadoop cluster (Amazon EMR) Analyse processed data to investigate game imbalance
  30. 30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Storage AWS Global Infrastructure App services Deployment and administration Networking Analytics Compute Database Amazon EMR • Scalable Hadoop/Spark clusters as a service • Launch a cluster in minutes • Easy to use: fully managed • HDFS, Amazon EBS, and Amazon S3 file systems
  31. 31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Scalability and elasticity - Resize a running cluster based on work to be done Durability and availability - Fault-tolerant for slave node (HDFS) - Backup to Amazon S3 for resilience against master node failures Standard interfaces - Hive, Pig, Spark, HBase, Impala, Hunk, Presto, Zeppelin, and other popular tools Amazon EMR Amazon EMR cluster Amazon EMR cluster Amazon EMR cluster
  32. 32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Big Data Analytics amzn.to/GAM310
  33. 33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Learnings Takeaways: • Set up and take control of your flexible, extensible, and secure analytics pipeline • Ingest, process, and refine own game-play data to improve game operations • Extract powerful insights from your own data, act quicker • Make the best of streaming tech and managed services for smarter operations
  34. 34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Thank you! Workshop team: Nirav Doshi - niravdd@amazon.com Brent Maxwell - braxwell@amazon.com Keith Lafaso - lafaso@amazon.com Beomseok Lee - beomseok@amazon.com

×