Uploaded byTetsuya Mase
AWSサービスで実現するEightの行動ログ活用基盤
The document outlines the agenda and details of the AWS Dev Day Tokyo 2018, focusing on data management and processing using various AWS services. Key elements include the integration of services like S3, Redshift, and Glue for analytics and ETL processes. The content also emphasizes the architecture involving Kinesis, Lambda, and DynamoDB for enhancing database performance and operational efficiency.
More Related Content
![[20220126] JAWS-UG 2022初頭までに葬ったAWSアンチパターン大紹介](https://cdn.slidesharecdn.com/ss_thumbnails/20220126-anti-220126190603-thumbnail.jpg?width=640&height=640&fit=bounds)
![[JAWS Days 2015 LT]使い始めて3年半、ようやくテスト始めました](https://cdn.slidesharecdn.com/ss_thumbnails/jasw-days-lt-150323200941-conversion-gate01-thumbnail.jpg?width=640&height=640&fit=bounds)
AWSサービスで実現するEightの行動ログ活用基盤
- 1. AWS Eight Sansan Eight AWS DevDay Tokyo 2018 Database Track #04 Nov. 01, 2018
- 2. Sansan Eight SRE Amazon WebServices (2015) JAWS-UG (10 ) 2014
- 4. Your Business Network EightDSOCSansan
- 6.
- 10.
- 11.
- 12.
- 13.
- 18.
- 19.
- 20. Eight = DevDay Tokyo 2017 https://d1.awsstatic.com/events/jp/2017/summit/devday/D4T8-5.pdf 💡
- 22.
- 23.
- 24. Aggregator S3 (store) Elasticsearch (view) CloudService Redshift (view/store) Redash (view) Servers / Applications Kinesis Data Streams Lambda DynamoDB Lambda SQS ElastiCache EC2 (Poll) DynamoDB Lambda ElastiCache Athena (view)
- 25.
- 26. Cloud Service Redshift (view/store)Redash (view) Servers / Applications Aggregator S3 (store) Elasticsearch (view) Kinesis Data Streams Lambda DynamoDB Lambda SQS ElastiCache EC2 (Poll) DynamoDB Lambda ElastiCache Athena (view)
- 27. fluentd Redshift Redshift or Redash ApplicationsCloud Service Redshift Redash Analyst
- 28. fluentd + KinesisData Firehose S3 + Redshift Glue + Redshift + Redash + Aggregators Kinesis Data Firehose S3 (Delivered logs) Lambda (Classify by name) S3 (Classified Logs) Glue Redshift Redash Analyst
- 29. Aggregators Kinesis Data Firehose S3 (Delivered logs) Lambda (Classifyby name) S3 (Classified Logs) Glue Redshift fluent-plugin-kinesis Kinesis Data Firehose S3 Firehose Lambda Firehose S3 Redshift S3
- 30. Kinesis Data Firehose Lambda (Classify byname) Glue Redshift Kinesis Data Streams Kinesis Data Firehose S3 Lambda 1 -1Firehose DeliveryStream Lambda Redshift COPY OK Glue S3 Athena Redshift Spectrum
- 32. AWS Glue Full ManagedETL Service
- 33.
- 35. Glue DPU 0.44 1ETL 10 5DPU 2DPU DPU 0.44 1 10 2DPU 1 0.44(USD/DPU hour) * 10/60(hour) * 2(DPU) = 0.147(USD) 50 50 * 0.44(USD/DPU hour) * 10/60(hour) * 2(DPU) = 7.333(USD) 30 $5,280
- 37. 76 1 1 2025 IOwait S3 Read / Redshift Write Python concurrent.futures.ThreadPoolExecutor 4DPU15 5 5 1
- 39.
- 41. choice Glue int/long 100 34583958001 string/ value JSON long/string ID ’undefined’ Glue (DynamicFrame) choice Redshift NULL information_schema.columns resolveChoice {“user_id”: 3234567890, “device_type”: “iOS”, “logged_in”: 1234567890} {“user_id”: “123”, “device_type”: “iOS”, “logged_in”: 1234567890} {“user_id”: 12345, “device_type”: “Android”, “logged_in”: 1234567890} {“user_id”: {“int”: null, “long”: 3234567890}, “device_type”: “iOS”, “logged_in”: 1234567890} {“user_id”: {“int”: “123”, “long”: null}, “device_type”: “iOS”, “logged_in”: 1234567890} {“user_id”: {“int”: 12345, “long”: null}, “device_type”: “Android”, “logged_in”: 1234567890}
- 42. timestamp UNIX timestamp “YYYY-mm-ddHH:MM:SS” NG or OK “YYYY-mm-dd HH:MM:SS” JSON string COLUMN_NAME_string COLUMN_NAME SELECT S3 Glue timestamp ApplyMapping source - target COLUMN_NAME_string {“user_id”: 1234567, “device_type”: “iOS”, “logged_in”: ”2018-11-01 01:00:00"} {“user_id”: “123”, “device_type”: “iOS”, “logged_in”: ”2018-11-01 03:00:00”} {“user_id”: 12345, “device_type”: “Android”, “logged_in”: ”2018-11-01 05:20:00"} id | bigint | not null default… user_id | bigint | not null device_type | integer | not null logged_in | timestamp without time zone | not null id | bigint | not null default… user_id | bigint | not null device_type | integer | not null logged_in | timestamp without time zone | not null logged_in_string | character varying(255) |
- 43. Not Null Redshift NotNull information_schema.columns NULL
- 44. S3 to Glue GlueContext.create_dynamic_frame.from_catalog Glue GlueContext.create_dynamic_frame.from_options S3 S3key pyspark.sql.functions.input_file_name() from_catalog
- 45. Glue to Redshift GlueContext.write_dynamic_frame.from_jdbc_conf COPY spark-redshift DataFrame information_schema DynamicFrameDropNullFields COPY TRUNCATECOLUMNS Job Bookmark
- 46.
- 47.
- 48.
- 50. information_schema.columns Redshift Redshift SELECT Job Bookmark 11 Max concurrency = 1
- 51. Aggregator S3 (store) Elasticsearch (view) Redshift(view/store) Redash (view) Servers / Applications Kinesis Data Streams Lambda DynamoDB Lambda SQS ElastiCache EC2 (Poll) DynamoDB Lambda ElastiCache Athena (view) Kinesis Data Firehose S3 Lambda S3 Glue
- 52.