Be the first to like this
Scopely’s portfolio of social and mid-core games
generates billions of events each day, covering everything from in-game actions to advertising to game engine performance.As this portfolio grew in the past two years, Scopely moved all event analysis from third-party hosted solutions to a new
event analytics pipeline on top of Redis and Kinesis, dramatically reducing operating costs and enabling new real-time analysis and more efficient warehousing. Our solution receives events over HTTP and SQS and provides real-time aggregation using a
custom Redis-backed application, as well as prompt loads into HDFS for batch analyses. Recently, we migrated our real-time layer from a pure Redis datastore to a hybrid datastore with recent data in Redis and older data in DynamoDB, retaining
performance while further reducing costs. In this session we will describe our experience building, tuning and monitoring this pipeline, and the role of Redis in supporting handling of Kinesis worker failover, deployment, and idempotence, in addition to its more visible role in data aggregation. This session is intended be helpful for those building streaming data systems and intended be helpful for those building streaming data systems and those looking for solutions for aggregation and idempotence.