Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

BigQueryで作る分析環境

1,168 views

Published on

bq_sushi#6

Published in: Data & Analytics
  • Be the first to comment

BigQueryで作る分析環境

  1. 1. BigQuery @tyakachika
  2. 2. • (Masachika Yamaguchi) • twitter @tyakachika • • • Innova Structure(2015/04 ~ ) • DATUM STUDIO(2016/12 ~ ) • Toreta(2017/07 ) <-
  3. 3. • Toreta • •
  4. 4. Toreta
  5. 5. MySQL BigQuery Access log iOS MySQL to BigQuery
  6. 6. ER ER
  7. 7. … esa
  8. 8. • • •
  9. 9. • BigQuery console • Tableau • Python(Jupyter notebook) • R(Rmarkdown)
  10. 10. Python(Jupyter Notebook) R Tableau
  11. 11. • • • •
  12. 12. • • • •
  13. 13. 2
  14. 14. Airflow …
  15. 15. digdag • embulk (mysql postgresql ) • python ruby •
  16. 16. & task digdag-Server local
  17. 17. Schedule.dig TOP Parent Child import_to_bigquery/main.dig bigquery_to_salesforce/main.dig execute_query/main.dig backup/main.dig ga/main.dig incremental/main.dig sync_event/main.dig call …… ※ call digdag-Server digdag
  18. 18. 🎊 digdag
  19. 19. • Change log •
  20. 20. • • • • • etc… event (create/update) Change log &Reservation
  21. 21. Reservation create/update Sidekiq job kinesis event (json) put CloudWatch lambda S3 Schedule Records schedule Records 1 put event gcs
  22. 22. GCP • GCS • GCS BigQuery load
  23. 23. GCS • S3 • AWS access_key secret_key • GCS
  24. 24. • Partition table Partition • Append Delete/ Update GCS BigQuery load
  25. 25. Partition table
  26. 26. (table_name_YYYYMMDD) (Partition table) BigQuery
  27. 27. (table_name_YYYYMMDD) • • Alter Table • • Web Console •
  28. 28. (Partition table) • • Web Console • • Alter Table …
  29. 29. Alter Table
  30. 30. • Web Console Add New Fields • Patch table or update table • … • Alter Table
  31. 31. 1. MySQL 2.BigQuery 3. 5. or SQL 4. tmp 6. SQL 7. 8.tmp 9.tmp Python Operater
  32. 32. • BigQuery API Error • ->digdag retry retry • digdag retry • ->yml
  33. 33. • Table Detail Description • Overwrite Description • API Description • markdown esa
  34. 34. … BI
  35. 35. redash … …
  36. 36. • • • • redash
  37. 37. metabase
  38. 38. • SQL metabase
  39. 39. • metrics • segment • description metabase
  40. 40. • redash • • • https://github.com/metabase/metabase/issues/6275 •
  41. 41. • BigQuery gcp • • • BigQuery
  42. 42. • … •

×