Download as PDF, PPTX
![AWS KINESIS ANALYTICS
STREAMING SQL
▸ Tumbling Window
[...] GROUP BY
FLOOR((“SOURCE_SQL_STREAM_001”.ROWTIME – TIMESTAMP
‘1970-01-01 00:00:00’) SECOND / 10 TO SECOND)
▸ Sliding Window
SELECT AVG(change) OVER W1 as avg_change
FROM "SOURCE_SQL_STREAM_001"
WINDOW W1 AS (PARTITION BY ticker_symbol RANGE INTERVAL
'10' SECOND PRECEDING)](https://image.slidesharecdn.com/awsbigdatastack-170612103317/75/Big-Data-on-AWS-26-2048.jpg)
Uploaded bySzilveszter Molnár
Big Data on AWS
This document provides an introduction to various AWS big data services including Kinesis, Kinesis Firehose, Kinesis Analytics, SQS, IoT, Data Pipeline, DynamoDB, EMR, Lambda, Redshift, and QuickSight. It describes the core concepts and components of each service. For services like Kinesis, Kinesis Analytics, and EMR it provides details on architecture, terminology, pricing models, and best practices. The document aims to give readers an overview of the AWS big data landscape and how the different services can be used together in a big data context.
Big Data on AWS
- 1. BIG DATA WITHAWS INTRO TO Szilveszter Molnar, Senior Pre-Sales Engineer @moszinet
- 2.
- 3.
- 4.
- 5.
- 6. GENERIC AWS Amazon EC2 ElasticCompute Cloud Amazon S3 Simple Storage Service (Volume Types) (Pricing Model) IAM Identity And Access Management
- 8.
- 9. GLACIER CONCEPTS ▸ Keep allyour data at a much lower cost ▸ Move automatically from S3 to Glacier ▸ Compliance requirements to keep your data ▸ Vault Lock (based on IAM policies)
- 11.
- 12. AWS KINESIS WHAT ISKINESIS? ▸ Platform for streaming data on AWS ▸ "sometimes TBs per hour"
- 13. AWS KINESIS KINESIS COMPONENTS FirehoseAnalyticsStreams
- 14.
- 15.
- 16. AWS KINESIS STREAMS CONCEPTS ▸Streams - ordered sequence of data records ▸ Data record - Sequence Number, Partition Key, Data Blob ▸ 1MB max ▸ Retention period - 24h - 7d ▸ Producers, Consumers ▸ Shards
- 17. AWS KINESIS STREAMS KINESISSTREAMS - SHARDS ▸ Fixed unit of capacity ▸ Read ▸ 5 transaction / sec ▸ 2MB / sec ▸ Write ▸ 1000 records / sec ▸ 1MB / sec
- 18.
- 19.
- 20. AWS KINESIS FIREHOSE CONCEPTS ▸Firehose delivery stream ▸ record - 1MB max ▸ data producer ▸ buffer size, buffer interval
- 21. AWS KINESIS FIREHOSE DATADELIVERY ▸ KINESIS STREAM ▸ S3 ▸ Redshift ▸ Elasticsearch
- 22.
- 23.
- 24.
- 25. AWS KINESIS ANALYTICS TERMINOLOGY ▸Input ▸ Application Code ▸ In-App-Streams ▸ Pumps ▸ Streaming SQL ▸ Output
- 26. AWS KINESIS ANALYTICS STREAMINGSQL ▸ Tumbling Window [...] GROUP BY FLOOR((“SOURCE_SQL_STREAM_001”.ROWTIME – TIMESTAMP ‘1970-01-01 00:00:00’) SECOND / 10 TO SECOND) ▸ Sliding Window SELECT AVG(change) OVER W1 as avg_change FROM "SOURCE_SQL_STREAM_001" WINDOW W1 AS (PARTITION BY ticker_symbol RANGE INTERVAL '10' SECOND PRECEDING)
- 27. AWS KINESIS ANALYTICS STREAMINGSQL - TUMBLING WINDOW
- 29.
- 30. SQS CONCEPTS ▸ Simple QueueService ▸ Send, Store, Retrieve messages ▸ between applications ▸ Acts as a buffer ▸ At least once delivery ▸ FIFO is also supported
- 31. SQS CONCEPTS ▸ Queues arecreated in regions ▸ 14 days retention ▸ no message priority is supported (2 queues)
- 32. SQS EXAMPLE WEB SERVER REQUEST QUEUE RESPONSEQUEUE PROCESSOR NODES
- 34.
- 35. IOT CONCEPTS ▸ Managed CloudPlatform from Internet of Things ▸ Billions of devices, trillions of messages ▸ Messages can be routed to other AWS services and other devices
- 36. IOT IOT AND BIGDATA ▸ IoT devices produce data ▸ Analyze streaming data real time ▸ Process and store data ▸ Don't worry about capacity, scaling, infrastructure
- 37. IOT ARCHITECTURE GENERIC IOT THING AWS IOT Elasticsearch KinesisFirehose Kinesis Streams DynamoDB Machine Learning CloudWatch S3 SQS, SNS
- 38. IOT ARCHITECTURE AWS IOT SDK AUTHENTICATION AUTHORIZATION DEVICEGATEWAY RULE ENGINE DEVICE SHADOWS DEVICE REGISTRY
- 40.
- 41. DATA PIPELINE TERMINOLOGY ▸ Aweb service to process and move data between AWS compute and storage services or on-premise data sources ▸ ETL Workflow ▸ Runs on an EC2 instance or EMR cluster that are provisioned automatically
- 42.
- 43. DATA PIPELINE ARCHITECTURE ▸ DataNodes - destination for your data ▸ Activity ▸ Hive ▸ Pig ▸ Shell script ▸ etc ▸ Preconditions ▸ Schedules
- 45.
- 46. DYNAMO DB CONCEPTS ▸ Fullymanaged NoSQL database ▸ No visible "servers" ▸ Single digit latency ▸ Document & Key-Value models ▸ No storage limitations ▸ Runs on SSD
- 47. DYNAMO DB CONCEPTS ▸ Collectionof Tables ▸ Performance is set on the tables ▸ Write Capacity Units - count of 1KB blocks ▸ Read Capacity Units - count of 4KB blocks ▸ Eventually Consistent by default (data in 3 regions) ▸ Strong consistent reads supported
- 48. DYNAMO DB CONCEPTS ▸ Schemaless ▸Contains Items (rows) and Attributes (elements) ▸ Special Attributes ▸ Partition Key ▸ Sort Key
- 49. DYNAMO DB DATA TYPES ▸String ▸ Number ▸ Binary ▸ Bool ▸ Null ▸ Document (List/Map) ▸ Set
- 50. DYNAMO DB DYNAMO DBIN THE AWS ECOSYSTEM ▸ On EMR Dynamo DB is integrated with Hive ▸ Copy data to/from S3 with Data Pipeline ▸ Lambda can be used as triggers ▸ Move data into Redshift with the COPY command
- 52.
- 53. EMR CONCEPTS ▸ Managed ClusterPlatform ▸ Hadoop, Spark, Presto, HBase, etc... ▸ Master, Core and Task Nodes ▸ Single AZ concept
- 54.
- 55.
- 56. EMR CORE NODE ▸ Likea slave node ▸ Runs tasks ▸ HDFS ▸ Data node, Node Manager, Application Master
- 57. EMR TASK NODE ▸ Likea slave node ▸ No HDFS ▸ Can be added/remove from a running cluster ▸ Extra compute capacity
- 58. EMR STORAGE OPTIONS ▸ InstanceStore ▸ ephemeral - deleted when instance terminates/lost ▸ use when High I/O performance is necessary ▸ EBS for HDFS ▸ EMRFS
- 59. EMR EMRFS ▸ Implementation ofHDFS ▸ Wrapper over S3 ▸ Resize, terminate clusters without loosing data ▸ Multiple clusters can point to the same data in S3 ▸ EMRFS & HDFS ▸ S3DistCp
- 60. EMR EMRFS - CONSISTENTVIEWS ▸ S3 ▸ Read After Write consistent for new data ▸ Eventual consistent for overwrite & delete ▸ Solve it with Consistent Views switch
- 61. EMR INSTANCE TYPES ▸ Guidelinesby Amazon ▸ Map Reduce ▸ Batch Oriented - M3 & M4 instance types ▸ Machine Learning ▸ P2, C3, C4 ▸ Spark ▸ R3, R4 ▸ Large Instance store for HDFS ▸ I2, D2
- 63.
- 64. LAMBDA CONCEPTS ▸ Function asa Service ▸ Billed for the ms of use ▸ Isolated & Stateless ▸ Persistence - your responsibility ▸ Event-driven
- 65. LAMBDA LIMITS ▸ /tmp space- 512MB ▸ threads - 1,024 ▸ max execution duration - 300 seconds
- 66.
- 67. LAMBDA EXAMPLES S3 LAMBDA FILE WITH WATERMARK newfile Add Watermark DYNAMO DB LAMBDA change data EMAIL WITH REPORT
- 69.
- 70. REDSHIFT CONCEPTS ▸ Fully managedpetabyte scale data warehouse ▸ MPP (Massively Parallel Processing database) ▸ OLAP & BI applications ▸ ANSI SQL compatible ▸ Column oriented ▸ Single AZ ▸ Backups to S3
- 71. REDSHIFT ARCHITECTURE LEADER NODE COMPUTE NODECOMPUTE NODE COMPUTE NODE
- 72. REDSHIFT LEADER NODE ▸ SQLEndpoint ▸ Coordinates parallel query execution ▸ Stores metadata
- 73. REDSHIFT COMPUTE NODES ▸ Executequeries in parallel ▸ Scale out/in, up/down
- 74. REDSHIFT COLUMNAR DATABASE ▸ Storesdata on disk grouped by columns, not rows Row based storage Column based storage
- 75. REDSHIFT BENEFITS ▸ Queries usefewer columns - less disk I/O ▸ Data aggregation (eg. SUM(sales)) ▸ Compression
- 77.
- 78. QUICKSIGHT CONCEPTS ▸ Cloud basedanalytics service ▸ Build visualizations ▸ Ad-hoc analysis ▸ Connect to AWS services, EC2 or on-premises DB
- 79. QUICKSIGHT SUPPORTED DATA SOURCES ▸Redshift ▸ Aurora ▸ Athena ▸ RDS (Maria DB, SQL Server, MySQL, PostgreSQL) ▸ Files ▸ Salesforce
- 80.
- 82. AWS BIG DATA READINGMATERIALS ▸ Kinesis ▸ Kinesis Firehose Transformation with Lambda ▸ Implementing producers with Amazon Kinesis Producer Library ▸ EMR ▸ Best practices for Amazon EMR (2013) ▸ Lambda ▸ Big Data processing with serverless MapReduce ▸ Redshift ▸ Redshift Table Design ▸ Optimizing for Star Schemas and Interleaved Sorting
- 83. AWS BIG DATA HOMEWORK- OPTIONAL 1. Create a transient EMR cluster that automatically starts your application (Hive, Pig, Spark, etc...) 2. Create a Redshift cluster, experiment with the COPY command (bulk load rows) vs INSERT statement ‣ 100 rows at least Note: Make sure you do not forget to delete the created resources ...
- 84.