Hands on lab session to get started with Google BigQuery..

1. Baltimore

2. BIGQUERY
Code Labs
DevFest DC - 2017

3. SELECT tweet.@dineshMD,email.dinesh@acarin.net from Contacts
Dinesh Ganesan

4. Why do we need Big Data tools?
BigQuery - Show the power
Getting Started
Let’s Play - Analyzing USA Census Data within few seconds.
Loading data into BigQuery
Talking to BigQuery
Q&A
4
Agenda?

5. Why Big Data?
5
3. Why “Big Data” Is a Big Deal - Harvard Magazine
Enabled Biologist to compute the genome sequence within 24 hours rather waiting for a full year.
- Reducing Cost & Time
Enabled NASA to provide Climate Modeling as Service(MERRA) using data since 1979 ( few hundred peta bytes )
- Better Living and New business opportunities.
2012 – 100 Billion Google Searches every month. Leverages 30 trillion URLs for data collection
- Optimal Knowledge from huge data size.
2. MERRA: MODERN ERA-RETROSPECTIVE ANALYSIS FOR RESEARCH AND APPLICATIONS
1. Google Search - Wikipedia

6. Getting Started - GCloud Registration
1. https://console.cloud.google.com
1
2 3
6

7. Getting Started - Project Creation
1
3
7
2
4
Refresh the page if you don’t see the project name

8. Getting Started - Enable API
8
5

9. Getting Started - Lab
6 7 8
9

10. Getting Started - Lab
9
10

11. Query Editor
11
Query Editor
UDF Editor

12. Query Execution
select year, SUM(record_weight) as Births from
publicdata:samples.natality where (mother_race = 18 and
father_race = 18) group by year order by year;
Asian Indian Kids born
The race of the child. One of the following numbers:
1 - White
2 - Black
3 - American Indian
4 - Chinese
5 - Japanese
6 - Hawaiian
7 - Filipino
9 - Unknown/Other
18 - Asian Indian
28 - Korean
39 - Samoan
48 - Vietnamese

13. SOURCE DATA Formats
13
CSV
JSON
Avro
Cloud datastore backups

14. SOURCE DATA Formats
14
http://www.ssa.gov/OACT/babynames/names.zip
https://goo.gl/qZW9sx

15. Loading data - CSV Direct Upload
15
1
2
3

16. Loading data - CSV Direct Upload
16
4

17. Loading data - CSV Direct Upload
17
4
name:string,gender:string,count:integer

18. Loading Data - GCP Storage
18
gs://ssa-babynames/names/yob2013.txt
How about loading multiple files into a single
table?

19. Running Queries
19
Standard SQL

20. Simple Aggregates
20
Select COUNT(GoogleDevs),MAX(GoogleDevs),
STDDEV(GoogleDevs) FROM …

21. Complex Processing
21
Select … FROM … where
REGEXP_MATCH(url,”Acarin$”) AND user
CONTAINS ‘Mr.Foo’

22. Nested Select
22
Select count(*) from ( Select devGurus ….) Group
BY Name

23. Small JOIN
23
Select tbl_school.student_id from tbl_school JOIN
tbl_classroom ON tbl_classroom.student_id =
tbl_shool.student_id

24. Talking to BigQuery

25. Accessing data through API
25
https://developers.google.com/api-client-library/

26. Validating API
26
https://developers.google.com/apis-explorer/#p/bigquery/v2/

27. Validating API
27

28. Integrating with Google Sheet
28
https://developers.google.com/apps-script/advanced/bigquery
https://goo.gl/evaRRC
1 2

29. Q&A
T H A N K Y O U
SELECT @dineshMD from Contact.tweet