This document discusses various tools used for data pipelines including Digdag for workflow automation, BigQuery for data storage and analysis, and Tableau, Python, and R for visualization and reporting. It provides examples of using these tools together including loading data from MySQL to BigQuery, partitioning BigQuery tables, and altering tables. It also briefly mentions other related tools like Airflow, Redash, and Metabase.

1. BigQuery
@tyakachika

2. • (Masachika Yamaguchi)
• twitter @tyakachika
•
•
• Innova Structure(2015/04 ~ )
• DATUM STUDIO(2016/12 ~ )
• Toreta(2017/07 ) <-

3. • Toreta
•
•

10. Toreta

12. MySQL
BigQuery
Access log
iOS
MySQL to
BigQuery

14. ER
ER

18. …
esa

20. •
•
•

22. • BigQuery console
• Tableau
• Python(Jupyter notebook)
• R(Rmarkdown)

23. Python(Jupyter Notebook)
R
Tableau

25. •
•
•
•

26. •
•
•
•

27. 2

29. Airflow
…

31. digdag
• embulk (mysql postgresql )
• python ruby
•

32. & task
digdag-Server
local

33. Schedule.dig
TOP Parent Child
import_to_bigquery/main.dig
bigquery_to_salesforce/main.dig
execute_query/main.dig
backup/main.dig
ga/main.dig
incremental/main.dig
sync_event/main.dig
call
……
※
call
digdag-Server
digdag

34. 🎊
digdag

36. • Change log
•

37. •
•
•
•
• etc…
event
(create/update)
Change log
&Reservation

38. Reservation create/update
Sidekiq job
kinesis
event (json) put
CloudWatch lambda
S3
Schedule
Records
schedule
Records
1 put
event
gcs

39. GCP
• GCS
• GCS BigQuery load

40. GCS
• S3
• AWS access_key secret_key
• GCS

41. • Partition table Partition
• Append Delete/
Update
GCS BigQuery load

42. Partition table

43. (table_name_YYYYMMDD)
(Partition table)
BigQuery

44. (table_name_YYYYMMDD)
•
• Alter Table
•
• Web Console
•

45. (Partition table)
•
• Web Console
•
• Alter Table …

46. Alter Table

47. • Web Console Add New Fields
• Patch table or update table
• …
•
Alter Table

48. 1. MySQL
2.BigQuery
3.
5. or SQL
4. tmp
6. SQL
7.
8.tmp
9.tmp
Python Operater

49. • BigQuery API Error
• ->digdag retry retry
• digdag retry
• ->yml

53. • Table Detail Description
• Overwrite Description
• API Description
• markdown esa

54. …
BI

55. redash …
…

57. •
•
•
•
redash

59. metabase

60. • SQL
metabase

61. • metrics
• segment
• description
metabase

62. • redash
•
•
• https://github.com/metabase/metabase/issues/6275
•

63. • BigQuery gcp
•
•
• BigQuery

64. • …
•