Downloaded 32 times
Uploaded byAmazon Web Services Korea
AWS Summit Seoul 2023 | 스타트업의 서버리스 기반 SaaS 데이터 처리 및 데이터웨어하우스 구축 사례
에이슬립은 AI 모델로 수면을 정확하게 분석하여 맞춤형 수면 서비스를 제공하는 SaaS 서비스를 제공합니다. 본 세션에서는 SaaS 서비스 인증 및 권한제어, 사용량 측정 등 필수적인 기능을 AWS 기반 서버리스 서비스를 통해 제공하여 데이터베이스 부하 없이 대규모 트래픽에도 견딜 수 있는 아키텍처를 소개합니다. 매드업은 디지털 마케팅 자동화 솔루션을 개발 및 운영하면서 대용량 데이터를 수집, 가공해서 데이터웨어하우스를 만드는 도전이 있습니다. 본 세션에서는 기존의 Amazon Athena에서 어떻게 Amazon Redshift Serverless를 통해 데이터가 필요한 조직에 정확하게 공급했는지 그 사례와 장단점을 소개합니다.
More Related Content
Similar to AWS Summit Seoul 2023 | 스타트업의 서버리스 기반 SaaS 데이터 처리 및 데이터웨어하우스 구축 사례
![[Keynote] 슬기로운 AWS 데이터베이스 선택하기 - 발표자: 강민석, Korea Database SA Manager, WWSO, A...](https://cdn.slidesharecdn.com/ss_thumbnails/d3s01aws-230704014400-3eeae447-thumbnail.jpg?width=640&height=640&fit=bounds)
![[AWS Dev Day] 앱 현대화 | 실시간 데이터 처리를 위한 현대적 애플리케이션 개발 방법 - 김영진 AWS 솔루션즈 아키텍트, 이세...](https://cdn.slidesharecdn.com/ss_thumbnails/realtime-190930044254-thumbnail.jpg?width=640&height=640&fit=bounds)
![[Games on AWS 2019] AWS 입문자를 위한 초단기 레벨업 트랙 | AWS 레벨업 하기! : 데이터베이스 - 박주연 AWS 솔...](https://cdn.slidesharecdn.com/ss_thumbnails/04gamesonawsawsdatabase-191014082829-thumbnail.jpg?width=640&height=640&fit=bounds)
![[D3T1S01] Gen AI를 위한 Amazon Aurora 활용 사례 방법](https://cdn.slidesharecdn.com/ss_thumbnails/d3t1s01genaiamazonaurora-240702042912-516e67f4-thumbnail.jpg?width=640&height=640&fit=bounds)
![[D3T1S06] Neptune Analytics with Vector Similarity Search](https://cdn.slidesharecdn.com/ss_thumbnails/d3t1s06neptuneanalyticsvectorsilimliaritysearch-240702042912-94c41309-thumbnail.jpg?width=640&height=640&fit=bounds)
![[D3T1S03] Amazon DynamoDB design puzzlers](https://cdn.slidesharecdn.com/ss_thumbnails/d3t1s03amazondynamodbdesignpuzzlers-240702042912-ad6df881-thumbnail.jpg?width=640&height=640&fit=bounds)
![[D3T1S04] Aurora PostgreSQL performance monitoring and troubleshooting by use...](https://cdn.slidesharecdn.com/ss_thumbnails/d3t1s04aurorapostgresqlperformancemonitoringandtroubleshooting-240702042912-5df626e3-thumbnail.jpg?width=640&height=640&fit=bounds)
![[D3T1S07] AWS S3 - 클라우드 환경에서 데이터베이스 보호하기](https://cdn.slidesharecdn.com/ss_thumbnails/d3t1s07-240702042911-cb134cd6-thumbnail.jpg?width=640&height=640&fit=bounds)
![[D3T1S05] Aurora 혼합 구성 아키텍처를 사용하여 예상치 못한 트래픽 급증 대응하기](https://cdn.slidesharecdn.com/ss_thumbnails/d3t1s05aurora-240702042911-c7f3f22d-thumbnail.jpg?width=640&height=640&fit=bounds)
![[D3T1S02] Aurora Limitless Database Introduction](https://cdn.slidesharecdn.com/ss_thumbnails/d3t1s02auroralimitlessdatabaseintroduction-240702042911-cb5552b7-thumbnail.jpg?width=640&height=640&fit=bounds)
![[D3T2S01] Amazon Aurora MySQL 메이저 버전 업그레이드 및 Amazon B/G Deployments 실습](https://cdn.slidesharecdn.com/ss_thumbnails/d3t2s01amazonaurorabluegreendeployment-240702042226-3ae36566-thumbnail.jpg?width=640&height=640&fit=bounds)
![[D3T2S03] Data&AI Roadshow 2024 - Amazon DocumentDB 실습](https://cdn.slidesharecdn.com/ss_thumbnails/d3t2s03documentdbhandson-240702042224-047bbc2c-thumbnail.jpg?width=640&height=640&fit=bounds)
![[Keynote] Accelerating Business Outcomes with AWS Data - 발표자: Saeed Gharadagh...](https://cdn.slidesharecdn.com/ss_thumbnails/d2s01keynoteacceleratingbusinessoutcomeswithawsdata-230704011737-e053cfca-thumbnail.jpg?width=640&height=640&fit=bounds)
AWS Summit Seoul 2023 | 스타트업의 서버리스 기반 SaaS 데이터 처리 및 데이터웨어하우스 구축 사례
- 1. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. SEOUL | MAY 4, 2023
- 2. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. SaaS Backend Developer
- 3. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. / Liam Park System Softerware Engineer Backend Engineer AWS infra, Node JS And Python Developer TmaxTibero Software Engineer Carillon Backend Engineer Asleep, Senior Backend Engineer
- 4. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Agenda 1. Sleeptrack API 2. API 3. Sleeptrack API - / / 4. Sleeptrack API – 5.
- 5. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 1. Sleeptrack API
- 6. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Sleeptrack API
- 7. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Sleeptrack API
- 8. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 2. API
- 9. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Saas API 01 AUTOMATION API 02 MINIMAL DEPENDENCY Live . 03 ACCURACY API . 04 SCALABILITY .
- 10. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. client ECS server log RDS cloud watch API Gateway update per request ELB 1. ECS API 2. API DB 3. DB ,
- 11. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 3. Sleeptrack API / /
- 12. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Amazon Athena • SQL S3 • csv, json, parquet • • RDS DocumentDB DynamoDB S3 Glue Athena
- 13. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Athena 1. SERVERLESS 2. Athena 3. ( / / ) Spark/Flink Presto/Hive Athena RedShift 운영 효율성 Low Low High Medium Learning Curve High Medium Low Low 운영 비용 High Medium Low Medium
- 14. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. API Firehose API Gateway API Firehose 01 02 03 04 Access log Parquet Data Athena Parquet Data Athena Firehose S3 JSON JSON Access Log Parquet Data
- 15. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. API Client API Gateway ECS S3 access log Firehose RDS Athena Glue Schema ELB (1) / /
- 16. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Firehose API Gateway Firehose
- 17. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Access Log Firehose S3
- 18. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Access Log Firehose S3
- 19. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Parquet Data • • Athena S3 • 23MB JSON Data $25 JSON Data 500KB Parquet Data $5 Parquet Data Before After
- 20. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Athena : 2023 5 4 2 Create Parquet access log Glue Schema Partitioned by Datetime parquet Schema For Parquet Athena CTAS Query Add Next Partition
- 21. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. DB • Athena Jenkins DB Parquet Athena insert hourly Client API Gateway Jenkins aggregate query ECS RDS ELB
- 22. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 4. Sleeptrack API
- 23. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Amazon Kinesis SQL Application • SQL • Kinesis Stream Kinesis SQL Application Firehose S3
- 24. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. SQL Application
- 25. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. API Firehose API Gateway API Firehose 01 02 03 SQL Application Lambda SQL Application lambda Firehose SQL Application
- 26. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. API – Client API Gateway Firehose SQL Application Lambda RDS update per minute
- 27. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. SQL Application Firehose SQL Application
- 28. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. SQL Application SQL Application Lambda RDS
- 29. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. SQL Application SQL Application Lambda RDS
- 30. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 5.
- 31. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. DB API query string, header access log API access log SQL Application
- 32. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. DW Jaeyeon Baek Tech Lead MADUP
- 33. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. ❏ Adtech(Advertising technology) ❏ Tech Lead at Data Platform Team ❏ Cloud GDE ❏ HashiCorp Ambassador 2023 ❏ : jybaek.tistory.com
- 34. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. ❏ ❏ / / ❏ Datashare ❏ Redshift Serverless ❏ Wrap up Agenda
- 35. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. ❏ ❏ / / ❏ Datashare ❏ Redshift Serverless ❏ Wrap up Agenda
- 36. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. , ,
- 37. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. 05.06 install : 2,522 install : 2,599 install : 2,871 05.01 install : 2,501 06.06 06.06 . install 04 20 05.02 install : 2,872
- 38. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. DW manifest delete copy commit download •data freshness •transaction size •distkey, sortkey
- 39. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. ❏ ❏ / / ❏ Datashare ❏ Redshift Serverless ❏ Wrap up Agenda
- 40. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. , / / EKS , Pod . MWAA Redshift . ECS Fargate JupyterHub ( ) .
- 41. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. / / • • Amazon S3 , Redshift COPY bz2 ▪ raw data bucket: ▪ compressed data bucket : Redshift COPY ○ • MWAA DAG manifest Amazon Redshift Delete & COPY Internal Micro Services Amazon S3 Amazon Simple Queue Service Amazon EKS Amazon MWAA Amazon Athena Amazon Redshift
- 42. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. / / average 30% • Average 30%, Maximum 90% • Counting only COPY queries. Redshift dc2.large
- 43. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. / / average 100% Normal Query • DM Load •
- 44. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. / / Redshift ETL Data Scientist Data Mart ETC
- 45. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. / / dc2.large to RA3 AQUA(Advanced Query Accelerator) 10 Amazon Managed Storage . ra3.xlplus 32TB Datashare Amazon Redshift
- 46. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. ❏ ❏ / / ❏ Datashare ❏ Redshift Serverless ❏ Wrap up Agenda
- 47. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Datashare Datashare AWS AWS . , Amazon Redshift
- 48. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Datashare Datashare … •inbound shares / outbound shares ▪ •producer cluster / consumer cluster ▪ •same AWS account or different AWS accounts ▪ . •Encryption ▪ producer, consumers ▪ .
- 49. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Datashare Datashare Amazon Redshift Managed Storage Producer Cluster (write data) consumer cluster (read shared data) consumer cluster (read shared data) Amazon Redshift Amazon Redshift Amazon Redshift
- 50. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. ❏ ❏ / / ❏ Datashare ❏ Redshift Serverless ❏ Wrap up Agenda
- 51. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Redshift Serverless DW , Amazon Redshift Serverless (2022-07 GA, ) (RPUs) , Amazon Redshift Serverless
- 52. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Redshift Serverless Datashare Redshift Serverless Amazon Redshift Managed Storage Producer Cluster (write data) consumer cluster (read shared data) consumer cluster (read shared data) Amazon Redshift Amazon Redshift Serverless Amazon Redshift Serverless
- 53. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Redshift Serverless Datashare Redshift Serverless Amazon Redshift Managed Storage Producer Cluster (write data) consumer cluster (read shared data) consumer cluster (read shared data) Amazon Redshift Amazon Redshift Serverless Amazon Redshift Serverless
- 54. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. 1 transaction Redshift Serverless - File (1) Table (1) x10 DELETE & COPY File (2) Table (2) DELETE & COPY File (32) Table (32) … … DELETE & COPY
- 55. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Redshift Serverless 타입 노드 스펙 월 비용 Delete & Insert 평균 시간 dc2.large x 8 - vCPU : 2 - Memory : 15 GIB - Storage : 0.16 TB 1,728 USD 224s ra3.xlplus x 2 - vCPU : 4 - Memory : 32 GIB - Storage : 32 TB 1,840.32 USD 201s ra3.xlplus x 3 - vCPU : 4 - Memory : 32 GIB - Storage : 32 TB 2,760 USD 145s Serverless x 32 RPU - RPU 시간 당 0.438 USD 매일 6시간 30일 사용 2,522.88 USD 102s Serverless x 64 RPU - RPU 시간 당 0.438 USD 매일 6시간 30일 사용 5,045.76 USD 95s - 32 (80MB) x 10
- 56. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Redshift Serverless DW Amazon MWAA Amazon Redshift Redshift Serverless Redshift Serverless RPUs Data Mart Data Scientist
- 57. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. ❏ ❏ / / ❏ Datashare ❏ Redshift Serverless ❏ Wrap up Agenda
- 58. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Wrap up • ▪ : DB scale-up. ▪ •Redshift Serveless RPUs ▪ ▪ / / • . ▪ ▪ ▪
- 59. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Wrap up Redshift • ▪ Athena •Athena ▪ ( Redshift ) ▪ (DAG) ▪ ETL •Athena ▪ Hive , , , and so on..
- 60. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Wrap up jybaek@madup.com :-) ?
- 61. © 2023, AmazonWeb Services, Inc. or its affiliates. All rights reserved. 감사합니다 © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.