This document provides an overview of a plan to automate ingesting data from various sources into Google services for analysis. The plan involves: 1) Taking random data from sources using APIs; 2) Scheduling those sources to put data in Google Drive; 3) Scheduling BigQuery to ingest all data from Drive; and 4) Linking Data Studio to BigQuery for visualization and reporting. Google Apps Scripts will be used to help move data between the different services.

1. Automate some google
things..
www.tribalism.com.au | info@tribalism.com.au | +61404085046

2. Overview of the Plan
1. Take
Random Data
sources….
2. Schedule
sources to put
their data in
Google Drive
using their APIs
3. Schedule
BigQuery to
ingest
everything
from Drive
4. Link Data
Studio to
BigQuery

3. And we’ll need a little help on the way
(sometimes)...
Moves data from
Sheets or Drive to
BQ
Helps to store and
then move really
big files to BQ

4. Get Data into Google Drive/Sheets

5. Openweather
=IMPORTXML("http://api.openweathermap.org/
data/2.5/weather?q="&$A2&",au&mode=xml&u
nits=metric&appid=5af34bbf4676e3e763aec727
ed05d284", "/*/temperature/@value")

6. Supermetrics

7. Shopify or similar… use their API or Add-on

8. Get Data into BigQuery from Sheets/Drive

9. Get Data into BigQuery from Sheets/Drive

10. Sheets - Use Google Scripts
● They are awesome
● They link/have APIs to all Google platforms - Sheets, Drive, BigQuery, Adwords
○ (well they are the only ones Ive tried but I know they go to gmail and everything
too).

12. BQ is extremely sensitive with Date
Formats...
Hence this:
Feel free to copy this and
save yourself some
hair/teeth/eyeballs..
Note: Needs to be:
YYYY-MM-DD
Or
YYYY-MM-DD
HH:MM:SS.00

13. What it looks - like lets dive in..

14. What About CSV files in Drive??? Eg Shopify
We Save them on Drive and then
use scripts to upload them to BQ
and then move to processed once
done...

15. Scripts for CSV! Lets dive in

16. From BQ to Data Studio

17. BigQuery
1. All you need is:
a. a Google ID
b. A credit card (you need to enter billing details to start using)
c. To set up a project in cloud console. More details here:
https://cloud.google.com/bigquery/quickstart-web-ui
Then go here: https://bigquery.cloud.google.com

18. BQ is essentially database infrastructure in the
cloud...

19. Cool Things about it
1. You can connect to it through Google Scripts API
2. Its supposed to be really fast. And Id agree with that.
3. Its cheap $400 free credit on signing up. Ive used $3 of that in 3 months.
For more on pricing see here: https://cloud.google.com/bigquery/pricing
Summary - its cheap and quick and reliable

20. How do I sign up?
1. Anyone with a google account can set up for free. See here
https://cloud.google.com/bigquery/quickstart-web-ui
2. Need to go to Google Console above, set up an account with Billing (ie you
need a credit card).
3. How it works: you create a project in cloud console. This automatically has BQ
enabled and the project will appear in BQ when you go here:
https://bigquery.cloud.google.com/

21. Data Studio
1. All you need is a Google ID.
Go here: https://datastudio.google.com

22. BQ is a Datasource for Data Studio

23. So heres my Dashboard...
HERES MY awesome DASHBOARD

24. Some Horriffic Queries
Select
w1.Date2 as Date
,w1.Temp as Temp
,w1.Weather as Weather
,w2.Campaign_name
,w2.Ad_set_ID
,w2.Impressions
,w2.Link_clicks
from WAWTalk.ZW1 w1
join WAWTalk.ZW2 w2
on w1.Date2=w2.Date1
Select
CAST(DATE(Date) AS DATE) as
Date2
,Temp as Temp
,Weather as Weather
from WAWTalk.WeatherHist
Select
CAST(DATE(Date) AS DATE) as
Date1
,Campaign_name
,Ad_set_ID
,Impressions
,Link_clicks
from WAWTalk.Facebook