Uploaded by@yuzutas0 Yokoyama
データ基盤の3分類と進化的データモデリング #DPCT
1. The document discusses concepts related to data pipelines and data management including data lakes, data warehouses, data marts, and data operations. 2. Common approaches involved moving data from sources to data lakes, then to warehouses and marts using tools like Hadoop, BigQuery, and Tableau. 3. Questions were asked and answered about specific implementation options and challenges around moving from existing Excel-based approaches to more scalable pipeline infrastructure.
More Related Content
Similar to データ基盤の3分類と進化的データモデリング #DPCT
データ基盤の3分類と進化的データモデリング #DPCT
- 1. 3 2019-02-13 Data PipelineCasual Talk #DPCT @ presented by @yuzutas0 https://www.pexels.com/photo/grayscale-photo-of-metal-building-185039/
- 2. WEB #DPCT KKD (/ ) Tableau
- 3.
- 4.
- 5.
- 6.
- 7.
- 8. DB id idid etc… ≠ ? x : y :SUM( ) ≠ ? is ( )
- 9. ※ 1 PdM& TL
- 10.
- 11.
- 12.
- 13.
- 14.
- 15.
- 16. Data Lake S3 GCS DB Dump
- 17.
- 18.
- 19.
- 20. Data Warehouse Hadoop hdfsRedshift BigQuery
- 21.
- 22.
- 23.
- 24. Data Mart Re:dash TableauBI Excel Excel ※ Excel ※ / BI ※
- 25.
- 26.
- 27.
- 28.
- 29.
- 30.
- 31. Airflow 1 Airflow ” ” Data Excel Ops
- 32.
- 35. Data Ops ※n=1
- 36. …… → Hadoop → Fluentd …… →Excel →
- 37.
- 38.
- 39.
- 40.
- 41.
- 42.
- 43.
- 44.
- 45.
- 46.
- 47. BigQuery BigQuery 3 BigQuery -Google Cloud Platform Source Warehouse Mart IF
- 48.
- 49. WITH 1 1 WITH SQL
- 50. SQL on SQL + +SQL → grep :Cloud Composer (GCP Full Managed Airflow)
- 51. 1. Tableau 2.SQL on
- 52. 3 No → → Yes → → BigQuery + SQL + Composer
- 53.
- 54.
- 55.
- 56. Tableau Tableau TableauServer CSV → S3 - “Big ball of mud”
- 57.
- 58.
- 59. AsIs S3 → Hadoop→ Tableau 1
- 60.
- 61.
- 62. 10 SQL https://www.sbcr.jp/products/4797376272.html https://www.shoeisha.co.jp/book/detail/9784798145143 NealFord ― https://www.oreilly.co.jp/books/9784873118567/ 2018 version by @fetarodc https://www.slideshare.net/tetsutarowatanabe/2018version-115735455
- 63.
- 64.