Ingest, store, analyze, and monitor a large volume of real-time streaming data using cloud-based managed services. The reference architecture will showcase the following solution areas using managed services/serverless technologies.- Setup and Manage Mult-Region infrastructure via CI/CD - Ingest and Store a large volume of streaming data (Ex: 100K records/second) - Analyze and derive insights in near real-time - Monitor the infrastructure and pipelines

1. Large Scale Streaming Analytics
using Cloud Services (AWS)
Jay Palaniappan
Principal Solutions Architect
Agilisium

2. Real-time Data
• App Events
• Game Telemetry
• Ad Tracking
• Social Media
• IoT

3. Challenges
• Performance
• Scalability
• Reliability
• Operational Overhead

4. AWS - Managed Services
Ingest
Kinesis Streams
Kinesis Firehose
Kinesis Analytics
Transform
Lambda
EMR
(Spark/Flink)
Store
S3
Redshift
ElasticSearch
DynamoDB
Analyze
Redshift
ElasticSearch
Athena
QuickSight

5. Ingest
Kinesis
(Streams)
Firehose
(pipeline)
SDKs/
Agents
Services/
Tools
• Methods
• SDKs
• Agent
• 3rd Party Services

6. Transform / Store
Kinesis
(Streams)
Lambda
Any DestinationEMR
Firehose
KCL

7. Transform / Store
Firehose
Transform
S3
Redshift
ElasticSearch

8. Real-time Analytics
FirehoseKinesis Analytics
(SQL/Flink)
Streams
Lambda
Destinations
Destinations
EMR
(Spark/Flink)
Destinations

9. Analytics & Data Science
S3
Redshift
ElasticSearch
Catalog Athena
Reporting
Tools
SageMaker
Kibana

10. Real-time
Data Feed
Raw
Data Store
Batch Analytics
Aggregated
Data Lake
Real-time
Analytics
Central Cloud Region
Events
Data Consumers
(Analysts / Data Scientists)
Player In
Game
Data Feed
Data Pond
Replication
Events
Remote Cloud Region
Solution Architecture (Logical)

11. Solution Architecture (System)
us-west-2
sa-east-1
Cross Region Replication
Kinesis
(Streams)
Firehose
(Parquet Conversion)
S3
(Storage)
Crawler
(Table Updates)
Catalog
(Tables)
Athena
(Queries)
Lambda
(Create Aggregates)
Redshift
(Aggregates)
Firehose
(Persistence)
Kinesis Analytics
(Realtime Aggregations)
Streams
(Consolidated)
Kinesis
(In Game Streams)
Firehose
(Parquet Conversion)
Stream Aggregator
Stream Aggregator
S3
(Storage)
Data Analysts
Events
1 Minute
5 Mins
Events

12. CI / CD
CodeCommit CodeBuild CloudFormation
Deployment
CodePipeline

13. Well Architected Framework
• Security
• Reliability
• Performance Efficiency
• Cost Optimization
• Operational Excellence

14. Thank You!

15. Questions?