Streaming Data Analytics with Amazon Kinesis Firehose and Redshift
•
2 likes•1,513 views
by Ray Zhu, Product Manager, Amazon Kinesis
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Streaming Data Analytics with Amazon Kinesis Firehose and Redshift
Similar to Streaming Data Analytics with Amazon Kinesis Firehose and Redshift (20)
More from Amazon Web Services
More from Amazon Web Services (20)
Recently uploaded
Recently uploaded (10)
Streaming Data Analytics with Amazon Kinesis Firehose and Redshift
- 1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Ray Zhu, Sr. Product Manager, Amazon Kinesis 2/23/2017 Streaming Data Analytics with Amazon Kinesis Firehose and Redshift
- 2. Agenda • Kinesis Firehose and Redshift • Build a Streaming Solution for Log Analytics o Step 1 Set Up Redshift DB and Table o Step 2 Create Firehose Delivery Stream and Configure Data Transformation o Step 3 Send Data to Firehose Delivery Stream o Step 4 Query and Analyze the Data from Redshift o Step 5 Monitor Streaming Data Pipeline
- 3. Load streaming data into Amazon S3, Amazon Redshift, and Amazon Elasticsearch Service Kinesis Firehose
- 4. Petabyte-scale data warehouse Amazon Redshift
- 5. Stream Data to Redshift
- 6. Data Flow Overview Kinesis Producer UI Amazon Kinesis Firehose Amazon Redshift Generate web logs Deliver processed web logs to Redshift Run SQL queries on processed web logs Transform raw data to structured data
- 7. Step 1 Set Up Redshift DB and Table
- 11. Review
- 12. Configure VPC Security Group
- 13. Configure VPC Security Group US East (N. Virginia) 52.70.63.192/27 US West (Oregon) 52.89.255.224/27 EU (Ireland) 52.19.239.192/27
- 14. Connect to Redshift DB
- 15. Create a Redshift Table create table weblogs( host_address varchar(512), request_time timestamp, request_method varchar(6), request_path varchar(1024), request_protocol varchar(10), response_code integer, response_size integer, referrer_host varchar(1024), user_agent varchar(1024) );
- 16. Create a Redshift Table
- 17. Step 2 Set Up Firehose Delivery Stream and Configure Data Transformation
- 18. Destination
- 19. Configuration
- 20. Review
- 21. Step 3 Send Data to Firehose Delivery Stream
- 22. Sample Data 219.134.32.117 - - [16/Feb/2017:09:38:20 -0800] "GET /wp-content HTTP/1.1" 200 4521 "-" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/5.1; .NET CLR 3.8.23015.5)" 95.169.41.62 - - [16/Feb/2017:09:38:20 -0800] "PUT /app/main/posts HTTP/1.1" 200 3883 "-" "Mozilla/5.0 (Windows NT 6.2; Trident/7.0; rv:11.0) like Gecko" 221.147.191.247 - - [16/Feb/2017:09:38:20 -0800] "GET /explore HTTP/1.1" 200 6579 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1) AppleWebKit/538.0.1 (KHTML, like Gecko) Chrome/38.0.895.0 Safari/538.0.1" 179.96.123.130 - - [16/Feb/2017:09:38:20 -0800] "GET /list HTTP/1.1" 200 560 "-" "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:5.4) Gecko/20100101 Firefox/5.4.6" 132.119.12.76 - - [16/Feb/2017:09:38:20 -0800] "PUT /explore HTTP/1.1" 200 3131 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_0 rv:5.0; AZ) AppleWebKit/535.1.0 (KHTML, like Gecko) Version/4.0.3 Safari/535.1.0" 74.113.56.92 - - [16/Feb/2017:09:38:20 -0800] "DELETE /app/main/posts HTTP/1.1" 200 7069 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_9) AppleWebKit/532.1.0 (KHTML, like Gecko) Chrome/15.0.877.0 Safari/532.1.0"
- 23. After Data Transformation 1.133.158.104,16/Feb/2017:10:26:46 -0800,GET,/search/tag/list,HTTP/1.1,200,9523,- ,"Mozilla/5.0 (Windows; U; Windows NT 5.3) AppleWebKit/531.1.1 (KHTML, like Gecko) Chrome/24.0.827.0 Safari/531.1.1" 194.189.242.208,16/Feb/2017:10:26:46 -0800,GET,/explore,HTTP/1.1,200,8202,- ,"Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 5.0; Trident/5.1)" 210.104.234.68,16/Feb/2017:10:26:46 -0800,GET,/wp-content,HTTP/1.1,200,6523,- ,"Mozilla/5.0 (Windows; U; Windows NT 5.0) AppleWebKit/538.0.2 (KHTML, like Gecko) Chrome/19.0.804.0 Safari/538.0.2" 12.140.32.105,16/Feb/2017:10:26:46 -0800,PUT,/wp-admin,HTTP/1.1,200,9273,- ,"Mozilla/5.0 (compatible; MSIE 7.0; Windows NT 6.0; Trident/6.0)" 208.53.124.37,16/Feb/2017:10:26:46 -0800,GET,/explore,HTTP/1.1,200,5187,- ,"Mozilla/5.0 (Windows; U; Windows NT 5.2) AppleWebKit/531.2.1 (KHTML, like Gecko) Chrome/36.0.842.0 Safari/531.2.1“ 113.80.90.8,16/Feb/2017:10:26:46 -0800,PUT,/wp-content,HTTP/1.1,200,4431,- ,"Mozilla/5.0 (Windows; U; Windows NT 5.2) AppleWebKit/534.1.1 (KHTML, like Gecko) Chrome/23.0.886.0 Safari/534.1.1"
- 24. Send Data
- 25. Step 4 Query and Analyze the Data from Redshift
- 26. Query Data • Find distribution of response codes over days SELECT TRUNC(request_time), response_code, COUNT(*) FROM weblogs GROUP BY 1,2 ORDER BY 1,3 DESC; • Count the number of 404 response codes SELECT COUNT(*) FROM weblogs WHERE response_code = 404; • Show all requests paths with status “PAGE NOT FOUND” SELECT TOP 1 request_path, COUNT(*) FROM weblogs WHERE response_code = 404 GROUP BY 1 ORDER BY 2 DESC;
- 27. Step 5 Monitor Streaming Data Pipeline
- 28. Monitor with CloudWatch Metrics
- 29. Monitor with CloudWatch Logs
- 30. Q & A
- 31. Thank you!
Editor's Notes
- 'use strict'; console.log('Loading function'); /* Combined Apache Log format parser */ const parser = /^([\d.]+) (\S+) (\S+) \[([\w:\/]+\s[+\-]\d{4})\] \"(.+?)\" (\d{3}) (\d+) \"([^\"]+)\" \"([^\"]+)\"/; exports.handler = (event, context, callback) => { let success = 0; // Number of valid entries found let failure = 0; // Number of invalid entries found /* Process the list of records and transform them */ const output = event.records.map((record) => { const entry = (new Buffer(record.data, 'base64')).toString('utf8'); const match = parser.exec(entry); if (match) { /* Prepare CSV version from Apache log data */ const requestParts = match[5].split(' '); const result = `${match[1]},${match[4]},${requestParts[0]},${requestParts[1]},${requestParts[2]},${match[6]},${match[7]},${match[8]},"${match[9]}"\n`; const payload = (new Buffer(result, 'utf8')).toString('base64'); success++; return { recordId: record.recordId, result: 'Ok', data: payload, }; } else { /* Failed event, notify the error and leave the record intact */ failure++; return { recordId: record.recordId, result: 'ProcessingFailed', data: record.data, }; } }); console.log(`Processing completed. Successful records ${success}, Failed records ${failure}.`); callback(null, { records: output }); };
- {{internet.ip}} - - [{{date.now("DD/MMM/YYYY:HH:mm:ss ZZ")}}] "{{random.weightedArrayElement({"weights":[0.6,0.1,0.1,0.2],"data":["GET","POST","DELETE","PUT"]})}} {{random.arrayElement(["/list","/wp-content","/wp-admin","/explore","/search/tag/list","/app/main/posts","/posts/posts/explore"])}} HTTP/1.1" {{random.weightedArrayElement({"weights": [0.9,0.04,0.02,0.04], "data":["200","404","500","301"]})}} {{random.number(10000)}} "-" "{{internet.userAgent}}"