MongoDB에 적재되어있는 사용자의 데이터를 AWS 위에서 효율적으로 분석한 경험에 대해 공유합니다.

12. 1TB > 1b+ documents
wow
much documents

24. Amazon Athena

36. ! "

38. AWS Glue

49. WITH latest_summaries AS
(SELECT user_id AS uid,
max(base_datetime) AS mdt
FROM user_asset_summaries
GROUP BY (user_id))
SELECT user_id
FROM user_asset_summaries s
JOIN latest_summaries ls
ON s.user_id = ls.uid
AND s.base_datetime = ls.mdt
WHERE cardinality(s.cards) > 0

50. WITH latest_summaries AS
(SELECT user_id AS uid,
max(base_datetime) AS mdt
FROM user_asset_summaries
GROUP BY (user_id))
SELECT user_id
FROM user_asset_summaries s
JOIN latest_summaries ls
ON s.user_id = ls.uid
AND s.base_datetime = ls.mdt
WHERE cardinality(s.cards) > 0

57. glueContext
.getCatalogSource(
database = "mart",
tableName = "raw_user_asset_summaries",
redshiftTmpDir = "",
transformationContext = "ds"
)
.getDynamicFrame()

58. dynamicFrame
.toDF()
.withColumn(
"base_datetime",
to_timestamp(
col("base_datetime"),
"yyyy-MM-d'T'HH:mm:ss'Z'"
)
)

59. dataFrame
.write
.mode(SaveMode.Append)
.option("compression", "snappy")
.parquet(targetOutput)

62. [리빙 포인트] Glue ETL Job을 돌릴땐

63. [리빙 포인트] Glue ETL Job을 돌릴땐

67. !"
#

68. !"
#

69. !

70. ❤