Download as PDF, PPTX
![Use Cases & Characteristics
27
•Must be able to ingest and store all types of data
• Internal: transactional data, application logs, operational data
• External: programmatically and manually extracted meta data
•[Close-to-]real-time representation (for platform)
•Must be mapped to [loose]schema and queryable
•Must be source for further ETL and analytics operations
•Must have full production: secure, testing, logging, etc](https://image.slidesharecdn.com/hyunjoong-181004103244/75/Hyun-joong-27-2048.jpg)
![Future Work: E[CWL]K Framework
36](https://image.slidesharecdn.com/hyunjoong-181004103244/75/Hyun-joong-36-2048.jpg)
Hyun joong
- 2. How AWS savedmy first data lake journey Hyun Joong Kim − A story of a college student in the Korean startup scene − Presented by
- 3. ABOUT ME Hyun JoongKim 3 - Member of the KRUG/ AUSG - Senior at Hanyang University - Department of Information System - Former Intern at MyMusicTaste - Data team - Former Intern at Ebay Korea - Data Platform team
- 4. AGENDA About AUSG1. 2. 3. MyMusicTaste Data Lake 4 4.Retro 5. Closing
- 5. About AUSG AWSKrug UniversityStudent Group 1
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
- 14.
- 15.
- 16.
- 17.
- 18.
- 19.
- 20.
- 21.
- 22.
- 23.
- 24.
- 25. Motivation 25 •In-development: lighter, modular,less barriers • System-wide dependencies —> Per Service dependencies • Decoupled parallel development/deployment • More cool stuff! •In-production: lighter, scalable, fault tolerant • Single Point of Failure —> Distributed fault tolerance • Whole application auto-scaling —> scale individual services as needed
- 26. Motivation 26 •Diversified data sources •ElasticSearch • DynamoDB • PostgreSQL • S3 • Redis • … ..and more!
- 27. Use Cases &Characteristics 27 •Must be able to ingest and store all types of data • Internal: transactional data, application logs, operational data • External: programmatically and manually extracted meta data •[Close-to-]real-time representation (for platform) •Must be mapped to [loose]schema and queryable •Must be source for further ETL and analytics operations •Must have full production: secure, testing, logging, etc
- 28. Data Lake @MMT 3 Overall Architecture
- 29. Data Lake QueryLayer 29 Amazon S3 (storage for streams, snapshots, raw, pre- & post-transformed, and final query layer data) Amazon Glue Catalog (Hive metastore) Amazon Athena (Presto SQL queries) Periscope Data (dashboards, charts, visualizations)
- 30. So what isthis 'Glue' 30
- 31. Architecture at aGlance
- 32.
- 33. Retro: what wentwell 33 •Glue catalog & Hive QL •Automated schema discovery with Glue crawlers •Read & write to S3 + Parquet
- 34. Retro: What couldbe improved 34 •Glue :( • Bookmarks are black-boxed and demonstrate some non-deterministic issues • Development and maintenance of Glue scripts is clunky • Cost!!! …minimal monthly cost for running one job every 30mins: •Streaming from Aurora PostgreSQL 2 DPUs 5 DPUs 10 DPUs $211.20048 $528.00048 $1055.99952
- 35. Future Work: Spark+ EMR 35 •Control the scaling & lifecycle of our EMR resources, and reduce cost drastically WHILE increasing load •Faster development •Better management options
- 36. Future Work: E[CWL]KFramework 36
- 37. Future Work: Warehousing& Analytics 37
- 38.
- 39. SERVICE ISSUES 39 - Exposureto services for college students - Seminars, hands-on-labs that helped - Expose of architecture and services - Not a 100% fit but did help know where the pieces belonged to - Slack page for KRUG - Community members that are interested in such projects - Active QnA that helped in situations
- 40. Personal thoughts 40 - Wouldlove to have more members join - Make a bigger pool of enthusiastic students - AWS is great, but there is no magic - A bigger network of people who are interested in technology in general - Study the core of how things inside AWS work the way they do
- 41. Reference 41 1. https://datafloq.com/read/what-is-a-data-lake-what-are-the-benefits/2589 2. https://aws.amazon.com/big-data/datalakes-and-analytics/what-is-a-data-lake/ 3.https://docs.aws.amazon.com/aws-technical-content/latest/building-data-lakes/amazon-s3-data-lake-storage- platform.html 4. https://aws.amazon.com/glue/ 5. http://calculator.s3.amazonaws.com/index.html
- 42. www.mymusictaste.com Special Thanks ToPaul Elliot, data lead at MyMusicTaste