Google Cloud BigQuery is a serverless, highly scalable, and cost-effective cloud data warehouse that allows users to analyze massive datasets. BigQuery can process queries involving terabytes or petabytes of data in seconds by distributing the work across thousands of processors. It demonstrates its capabilities through examples that count words in Shakespeare, sum requests from Wikipedia data over time, and filter and aggregate multi-terabyte datasets to find total page views matching a regular expression, all in under 30 seconds.

1. Google Cloud Bigquery

2. +Michel Pereira
@michelpereira
Cloud Solutions Engineer
Google Cloud

3. Enterprise
Google Cloud is
The Datacenter as a Computer

4. Photo credit: Matt Chan

5. Google Research Publications

6. Open Source Implementations
Bigtable
Flume
Dremel

7. Managed Cloud Versions
Bigtable Bigtable
Flume Dataflow
Dremel BigQuery

8. Google BigQueryGoogle BigQuery

9. 02 Count some stuff

10. SELECT count(word)
FROM publicdata:samples.shakespeare
Words in Shakespeare

11. Query complete (1.3s elapsed, 1.27 MB processed)

12. SELECT sum(requests) as total
FROM [fh-bigquery:wikipedia.pagecounts_20150212_01]
Wikipedia hits over 1 hour

13. Query complete (1.1s elapsed, 46.7 MB processed)

14. SELECT
SUM(requests) AS total
FROM
TABLE_QUERY(
[fh-bigquery:wikipedia],
'REGEXP_MATCH(
table_id,
r"pagecounts_2015[0-9]{2}$")')
Several years of Wikipedia data

15. Query complete (6.7s elapsed, 460 GB processed)

16. How about a RegExp
SELECT
SUM(requests) AS total
FROM
TABLE_QUERY(
[fh-bigquery:wikipedia],
'REGEXP_MATCH(
table_id,
r"pagecounts_2015[0-9]{2}$")')
WHERE
(REGEXP_MATCH(title, '.*[dD]inosaur.*'))

17. Query complete (21.0s elapsed, 3.31 TB processed)

18. 03 How did it do that?
o_O

19. Qualities of a good RDBMS

20. Qualities of a good RDBMS
● Inserts & locking
● Indexing
● Cache
● Query planning

21. Qualities of a good RDBMS
● Inserts & locking
● Indexing
● Cache
● Query planning

25. Storing data - Break it, Compress it, Spread it
-- -- -- --
-- -- -- --
-- -- -- --
Table
Columns
Disks

26. Reading data: Life of a BigQuery
SELECT sum(requests) as sum
FROM (
SELECT requests, title
FROM [fh-bigquery:wikipedia.
pagecounts_201501]
WHERE
(REGEXP_MATCH(title, '[Jj]en.+'))
)

27. Life of a BigQuery - Nodes
L L
MMixer
Leaf
Storage

28. L L L L
M M
M
Life of a BigQuery - Nodes
Root Mixer
Mixer
Leaf
Storage

29. Life of a BigQuery - Execution Plan
Query
L L L L
M M
MRoot Mixer
Mixer
Leaf
Storage

30. Life of a BigQueryLife of a BigQuery - Map
Root Mixer
Mixer
Leaf
Storage
SELECT requests, title
L L L L
M M
M

31. Life of a BigQueryLife of a BigQuery - Filter, Group and Count
Root Mixer
Mixer
Leaf
Storage
5.4 Bil
SELECT requests, title
WHERE
(REGEXP_MATCH(title, '[Jj]en.+'))L L L L
M M
M

32. Life of a BigQueryLife of a BigQuery - Group and Count
Root Mixer
Mixer
Leaf
Storage
5.4 Bil
SELECT sum(requests)
5.8 Mil
WHERE
(REGEXP_MATCH(title, '[Jj]en.+'))
SELECT requests, title
L L L L
M M
M

33. Life of a BigQueryLife of a BigQuery - Aggregate at the Root
Root Mixer
Mixer
Leaf
Storage
5.4 Bil
SELECT sum(requests)
5.8 Mil
WHERE
(REGEXP_MATCH(title, '[Jj]en.+'))
SELECT requests, title
SELECT sum(requests)
L L L L
M M
M