This document discusses powering interactive data analysis with Google BigQuery. It begins with an introduction of the speaker and an overview of the World Telecommunication and Information Society Day theme of using big data for development. The rest of the document discusses challenges with traditional business reporting systems, how BigQuery provides a solution with its scalability, cost effectiveness and ease of use. It provides examples of the types of analyses that can be done with BigQuery and benefits like no maintenance or manual scaling required.

1. Powering Interactive Data
Analysis with Google
BigQuery
Márton Kodok / @martonkodok
Google Developer Expert at REEA.net, Targu Mures, Romania17 May 2017
Cluj Napoca, Romania

2. ● Geek. Hiker. Do-er.
● Among the Top3 romanians on Stackoverflow
● Google Developer Expert on Cloud technologies
● Crafting Web/Mobile backends at REEA.net Targu Mures
● BigQuery and database engine expert
● Active in mentoring
Twitter: @martonkodok
StackOverflow: pentium10
Slideshare: martonkodok
GitHub: pentium10
Powering Interactive Data Analysis with Google BigQuery @martonkodok
About me

3. Theme 2017: Big Data for Big Impact
The theme for WTISD-17, "Big Data for Big Impact," focuses on
the power of Big Data for development and aims to explore
how to turn imperfect, complex, often unstructured data into
actionable information in a development context.
Powering Interactive Data Analysis with Google BigQuery @martonkodok
17 May 2017, World Telecommunication and Information Society Day

4. Romania needs to
accelerateopenBigDataadoption
Powering Interactive Data Analysis with Google BigQuery @martonkodok

5. Let Education/Healthcare
benefitfromtalented BigData experts
Powering Interactive Data Analysis with Google BigQuery @martonkodok

6. <Start-up Nation> projects should focus on going
-from paperless to openendpoints
-from smalldatatoBigData
-sustain a City by building BigDataPlatformHubs
Powering Interactive Data Analysis with Google BigQuery @martonkodok

7. Everycompany,
no matter how far from the tech they are,
isevolvingintoasoftwarecompany,
and by extension a datacompany.
Powering Interactive Data Analysis with Google BigQuery @martonkodok

8. For a small company it’simportant
to have access to modernBigDatatools
withoutrunningadedicatedteam for it.
Powering Interactive Data Analysis with Google BigQuery @martonkodok

9. Powering Interactive Data Analysis with Google BigQuery @martonkodok
Agenda
The
Challenge
Architecture
Overview
Strategy &
Tricks
Winning
Solution

10. ❏ Need backend/database to STORE, QUERY, EXTRACT data
❏ Deep analytics - large, multi-source, complex, unstructured
❏ Be real time
❏ Terabyte scale - Cost effective
❏ Run Ad-Hoc reports - Without Developer - interactive
❏ Minimal engineering efforts - no dedicated BigData team
❏ Simple Query language (prefered SQL / Javascript)
Powering Interactive Data Analysis with Google BigQuery @martonkodok
Making analytics accessible to more companies

11. Powering Interactive Data Analysis with Google BigQuery @martonkodok
Legacy Business Reporting System
Web
Mobile
Web Server
Database
SQL
Cached
Platform Services
CMS/Framework
Report & Share
Business Analysis
Scheduled
Tasks
Batch Processing
Compute Engine
Multiple Instances

12. Powering Interactive Data Analysis with Google BigQuery @martonkodok
Web
Mobile
Web Server
Database
SQL
Cached
Platform Services
CMS/Framework
Report & Share
Business Analysis
Scheduled
Tasks
Batch Processing
Compute Engine
Multiple Instances
BehindtheScenes:
DaysToInsights

13. Powering Interactive Data Analysis with Google BigQuery @martonkodok
Legacy Business Reporting System
Web
Mobile
Web Server
Database
SQL
Cached
Platform Services
CMS/Framework
Report & Share
Business Analysis
Scheduled
Tasks
Batch Processing
Compute Engine
Multiple Instances
Minutes
to kick in
Hours to Run
Batch Processing
Hours to Clean and
Aggregate
DAYS TO
INSIGHTS

14. ● Terabyte scalable storage
● Real-time row ingestion
● Ask sophisticated queries
● Query-performance
● Low-maintenance
● Cost effective
● Wire them up easily
Goal: Store everything accessible by SQL immediately.
Powering Interactive Data Analysis with Google BigQuery @martonkodok
Desired system/platform
Engines:
● MongoDB, Riak, Redis
● ELK Stack (Elasticsearch-Logstash-Kibana)
● Cassandra, Hive, Hadoop...
● Amazon Athena, Google BigQuery...

15. Powering Interactive Data Analysis with Google BigQuery @martonkodok

16. ● Analytics-as-a-Service - Data Warehouse in the Cloud
● Fully-Managed by Google (US or EU zone)
● Scales into Petabytes
● Ridiculously fast
● SQL 2011 Standard + Javascript UDF (User Defined Functions)
● Familiar DB Structure (table, views, record, nested, JSON)
● Open Interfaces (REST, ODBC, Web UI, BQ command line tool)
● Integrates with Google Sheets + Google Cloud Storage + Pub/Sub connectors
● Client libraries available in YFL (your favorite languages)
● Decent pricing (queries $5/TB, storage: $20/TB cold: $10/TB) *May 2017
Powering Interactive Data Analysis with Google BigQuery @martonkodok
What is BigQuery?

17. Powering Interactive Data Analysis with Google BigQuery @martonkodok
Architecting for The Cloud
BigQuery
On-Premises Servers
Pipelines
ETL
Engine
Event Sourcing
Frontend
Platform Services
Metrics / Logs/
Streaming

18. Powering Interactive Data Analysis with Google BigQuery @martonkodok
Data Pipeline Integration
Analytics Backend
BigQuery
On-Premises Servers
Pipelines
FluentD
Event Sourcing
Frontend
Platform Services
Metrics / Logs/
Streaming
Development
Team
Data Analysts
Report & Share
Business Analysis
Tools
Tableau
QlikView
Data Studio
Internal
Dashboard
Database
SQL
Application
ServersServers
Cloud Storage
archive
Load
Export
Replay
Standard
Devices
HTTPS

19. Powering Interactive Data Analysis with Google BigQuery @martonkodok
<filter frontend.user.*>
@type record_transformer
enable_ruby
remove_keys host
<record>
bq {"insert_id":"${uid}","host":"${host}","created":"${time.to_i}"}
</record>
</filter>
<match frontend.user.*>
@type copy
<store>
@type forest
subtype file
<template>
path /tank/storage/${tag}.*.log
time_slice_format %Y%m%d
time_slice_wait 10m
</template>
</store>
<store>
@type bigquery
method insert
...
</store>
</match>
….bigquery section continued….
auth_method json_key
json_key /etc/td-agent/keys/key-31da042be48c.json
project project_id
dataset dataset_name
time_field timestamp
time_slice_format %Y%m%d
table user$%{time_slice}
ignore_unknown_values
schema_path /etc/td-agent/schema/user_login.json
1
2
3
4

20. ● On data that it is difficult to process/analyze using traditional databases
● On exploring unstructured data
● Not a replacement to traditional DBs, but it compliments the system
● Applying Javascript UDF on columnar storage to resolve complex tasks
(eg: JS for natural language processing)
● On streams (form wizard ...)
● On IoT streams
● Major strength is handling Large datasets
Powering Interactive Data Analysis with Google BigQuery @martonkodok
Where to use BigQuery?

21. Powering Interactive Data Analysis with Google BigQuery @martonkodok
Romanian stations that record the most days of snow

22. Powering Interactive Data Analysis with Google BigQuery @martonkodok
Mentions of RO politicians since ‘16 Nov in GDELT articles

23. ● Optimize product pages
● Email engagement
● Funnel Analysis
Powering Interactive Data Analysis with Google BigQuery @martonkodok
Achievements

24. Powering Interactive Data Analysis with Google BigQuery @martonkodok
Funnel analysis: Time on upsell pages

25. Example HITS chain:
● article1 -> page2 -> page3 -> page4 -> orderpage1 -> thankyoupage1
● page1 -> article2-> page3 -> orderpage2 -> ...
Powering Interactive Data Analysis with Google BigQuery @martonkodok
Attribute credit to first article visited on purchase

26. ● No manual sharding
● No capacity guessing
● No idle resources
● No maintenance windows
● No manual scaling
● No file mgmt
Powering Interactive Data Analysis with Google BigQuery @martonkodok
BigQuery: Serverless Data Warehouse

27. ● no provisioning/deploy
● no running out of resources
● no more focus on large scale execution plan
● no need to re-implement tricky concepts
(time windows / join streams)
● pay only the columns we have in your queries
● run raw ad-hoc queries (either by analysts/sales or Devs)
● no more throwing away-, expiring-, aggregating old data.
Powering Interactive Data Analysis with Google BigQuery @martonkodok
Our benefits

28. Powering Interactive Data Analysis with Google BigQuery @martonkodok
Easily Build Custom Reports and Dashboards

29. Questions?
Thank you.
Slides available on: slideshare.net/martonkodok