Downloaded 16 times
![AWS KINESIS ANALYTICS
STREAMING SQL
▸ Tumbling Window
[...] GROUP BY
FLOOR((“SOURCE_SQL_STREAM_001”.ROWTIME – TIMESTAMP
‘1970-01-01 00:00:00’) SECOND / 10 TO SECOND)
▸ Sliding Window
SELECT AVG(change) OVER W1 as avg_change
FROM "SOURCE_SQL_STREAM_001"
WINDOW W1 AS (PARTITION BY ticker_symbol RANGE INTERVAL
'10' SECOND PRECEDING)](https://image.slidesharecdn.com/awskinesis-170505071201/75/Aws-Kinesis-26-2048.jpg)
Uploaded bySzilveszter Molnár
Aws Kinesis
This document provides an overview of AWS Kinesis and its components for streaming data. It discusses Kinesis Streams for ingesting and processing streaming data at scale. Kinesis Streams uses shards to provide throughput capacity. To ingest 10,000 records per second of 512 byte size each would require a Kinesis stream configured with 10 shards. Kinesis Firehose is for delivering streaming data to destinations like S3 or Redshift. Kinesis Analytics allows running SQL queries on streaming data and processing it in real-time.
Aws Kinesis
- 1. AWS KINESIS SCRATCHING THESURFACE OF
- 2.
- 3. QUESTION What (Kafka?) clusterdo you need to ingest: - 10,000 record/sec - record size 512Bytes
- 4.
- 5. AWS KINESIS WHAT ISKINESIS? ▸ Platform for streaming data on AWS ▸ "sometimes TBs per hour"
- 6. AWS KINESIS MOTIVATION ▸ HighlyScalable ▸ Durable ▸ Elastic ▸ Replay-able reads
- 7. AWS KINESIS KINESIS COMPONENTS FirehoseAnalyticsStreams
- 8.
- 9.
- 10. AWS KINESIS STREAMS TERMINOLOGY ▸Streams - ordered sequence of data records ▸ Data record - Sequence Number, Partition Key, Data Blob ▸ 1MB max ▸ Retention period - 24h - 7d ▸ Producers, Consumers ▸ Shards
- 11. AWS KINESIS STREAMS KINESISSTREAMS - SHARDS ▸ Fixed unit of capacity ▸ Read ▸ 5 transaction / sec ▸ 2MB / sec ▸ Write ▸ 1000 records / sec ▸ 1MB / sec
- 12. QUESTION What cluster doyou need to ingest - 10,000 records/sec - record size 512Bytes
- 13. AWS KINESIS STREAMS KINESISSTREAMS - SHARD CALCULATION Requirement Kinesis Stream Write Capacity 10,000 records/sec 1,000 Record/sec 512 Bytes/rec (5MB/sec) 1MB/sec
- 14. AWS KINESIS STREAMS KINESISSTREAMS - SHARD CALCULATION ▸ 10 Shards ▸ 10,000 records / sec ▸ 10MB / sec
- 15.
- 16. AWS KINESIS STREAMS DEMO ▸Create a stream with 1 shard ▸ Put 10 records / sec ▸ Read in batches every sec { RecordId: 11, KeyPressCount: 22, UserId: 33 }
- 17.
- 18.
- 19. AWS KINESIS FIREHOSE TERMINOLOGY ▸Firehose delivery stream ▸ record - 1MB max ▸ data producer ▸ buffer size, buffer interval
- 20. AWS KINESIS FIREHOSE DATADELIVERY ▸ KINESIS STREAM ▸ S3 ▸ Redshift ▸ Elasticsearch
- 21.
- 22.
- 23.
- 24.
- 25. AWS KINESIS ANALYTICS TERMINOLOGY ▸Input ▸ Application Code ▸ In-App-Streams ▸ Pumps ▸ Streaming SQL ▸ Output
- 26. AWS KINESIS ANALYTICS STREAMINGSQL ▸ Tumbling Window [...] GROUP BY FLOOR((“SOURCE_SQL_STREAM_001”.ROWTIME – TIMESTAMP ‘1970-01-01 00:00:00’) SECOND / 10 TO SECOND) ▸ Sliding Window SELECT AVG(change) OVER W1 as avg_change FROM "SOURCE_SQL_STREAM_001" WINDOW W1 AS (PARTITION BY ticker_symbol RANGE INTERVAL '10' SECOND PRECEDING)
- 27. AWS KINESIS ANALYTICS STREAMINGSQL - TUMBLING WINDOW
- 28.
- 29. AWS KINESIS ANALYTICS DEMO ▸Check BDMeetup Application in AWS Console ▸ Producer / Consumer PRODUCER APP BDMEETUP STREAM ANALYTICS APP BDMEETUP-OUTPUT STREAM CONSUMER APP
- 30.