This document provides an overview and summary of Google Cloud Platform's BigQuery and DataPrep services. For BigQuery, it covers what it is used for, pricing models, how to load and query data, best practices, and limits. For DataPrep, it discusses what it is used for, pricing, how to wrangle and transform data using recipes and flows, performance optimization tips, and limits. Live demos are provided of loading, querying data in BigQuery and using transformations in DataPrep. The presenter's contact information is also included at the end.

1. Google Cloud Platform
1. Dive into BiqQuery
2. An idea for ETL - DataPrep
Paweł Mitruś
Warsaw, 5th April 2018

2. About me
 Team Manager @
 Almost 4 years with Data
 Triathlon freak
 Softskills optimist
Human brain explorer

3. Agenda
BigQuery DataPrep
1. What BQ is for
2. Pricing
3. BQ in action
• Using
• Loading
• Querying
4. Best practises
5. Limits
6. Summary
1. What DataPrep is for
2. Pricing
3. Wrangling data with DataPrep
4. Creating / scheduling Flows
5. Limits
6. Summary
45min ~ 15min ~

4. What BQ is for - Overview BigQuery

5. What BQ is for - Overview BigQuery

6. What BQ is for - Overview BigQuery
• Storage usage – based on data consumption
• Query – estimates how much slots it needs for query (CPU & RAM)
• Default 2 000 slots
• Spend over 40k $ / month – increase
• Maintenance - fully managed
• Backup and recovery
• 7 day history changes
• Point-in-time snapshot (Legacy SQL)
• Monitoring
• Stackdriver (GCP service)
• Export to BQ
• Security – IAM & admin (GCP)

7. BQ Pricing BigQuery

8. BQ Pricing – DML non-partitioned tables BigQuery

9. BQ Pricing – DML non-partitioned tables BigQuery
DEMO

10. BQ Pricing – DML partitioned tables BigQuery

11. BQ in action BigQuery
Using BQ:
• Web
• Command line (SDK)
• REST API / Client libraries (C#, GO, Node.js, PHP, Python, Java, Ruby)
File types:
• CSV
• JSON (newline delimited)
• Avro
• Parquet
• Cloud Datastore Backup
• Google Cloud Bigtable
• Google Sheets
Sources:
• Google Cloud Storage
• Google Drive
• Google Cloud Bigtable
• Local File
• Streaming

12. BQ in action - demos BigQuery
DEMO

13. BQ in action – CSV, schema autodetection BigQuery
Encoding
• BigQuery expects CSV data to be UTF-8 encoded. If you have CSV files with data encoded in ISO-
8859-1 (also known as Latin-1) format, you should explicitly specify the encoding when you load
your data so it can be converted to UTF-8. – NOT IN WEB UI, only UTF-8 and ISO-8859-1 supported
• Delimiters in CSV files can be any ISO-8859-1 single-byte character. To use a character in the range
128-255, you must encode the character as UTF-8. BigQuery converts the string to ISO-8859-1
encoding and uses the first byte of the encoded string to split the data in its raw, binary state.
Schema autodetection
• Delimiters: comma (,), pipe(|), tab(t)
• Header – if 1st row contains only strings and 2nd does not, then 1st is header
• Quoted new lines – detects, but is not recognized as a row boundary

14. BQ in action – append, job demos BigQuery
DEMO

15. BQ in action – querying BigQuery
DEMO

16. BQ best practises – cost controll BigQuery
• Avoid SELECT *
• Don't run queries to explore or preview table data – use preview option
• Before running queries, preview them to estimate costs
• Use the maximum bytes billed setting to limit query costs – using LIMIT statement changes
NOTHING
• Partition your tables by date
• If possible, materialize your query results in stages
• Use expiration time to remove the data when it's no longer needed

17. BQ best practises – query performance BigQuery
• Denormalize whenever possible (use nested / repeated fields)
• _PARTITIONTIME pseudo column to filter the partitions
• Do not overuse wildcards
• BQ doesn’t like joins
• Use approximate aggregation functions

18. BQ Limits - querying BigQuery
• Concurrent rate limit for on-demand, interactive queries — 50 concurrent queries
• Concurrent rate limit for queries that contain user-defined functions (UDFs) — 6 concurrent
queries
• Maximum number of tables referenced per query — 1,000
• Maximum concurrent slots per project for on-demand pricing — 2,000

19. BQ Limits – load, copy, export jobs BigQuery
• Load jobs per project per day — 50,000 (including failures)
• Wildcard URIs — 500 wildcard URIs per export
• Exports per day — 1,000 exports per project and up to 10 TB per day (the 10TB data limit is
cumulative across all exports)

20. BQ Limits – DML BigQuery
• Maximum UPDATE/DELETE statements per day per table — 96
• Maximum UPDATE/DELETE statements per day per project — 10,000
• Maximum INSERT statements per day per table — 1,000

21. BQ Summary BigQuery
Pros:
• Ease of pricing
• Computing resources
• Documentation
• Price
Cons:
• Troubleshooting
• ETL (Matillion?)
• Production cases, competences
• Available datasources

22. DataPrep - Agenda DataPrep
1. What DataPrep is for
2. Pricing
3. Wrangling data with DataPrep
4. Creating / scheduling Flows
5. Limits
6. Summary

23. What DataPrep is for - Overview DataPrep
• ETL
• Extract
• Transform
• Load
• Data Cleansing
• Data Profiling
• Data Enrichment
• Data Flows

24. DataPrep - Pricing DataPrep

25. DataPrep in action - demos
DEMO
DataPrep

26. DataPrep Perforemance - optimize
• Limit rows / filter data / drop unused columns
• Late unions
• Joins:
• Join operations should be performed early in your recipe. These steps bring together your data into a single consistent
dataset. By doing them early in the process, you reduce the chance of having changes to your join keys impacting the
results of your join operations.
• Tip: You should perform your join operation as late as possible in your recipe steps. If your joined dataset has not been
completely transformed, subsequent steps might impact the data in the dataset to which it was joined. If needed, you
can modify your join after its creation.
DataPrep

27. DataPrep Limits
• Sampling - Sample sizes are 10 MB (All values displayed or generated in the application are based
on the currently displayed sample)
• Sampling - Random samples are derived from up to the first 1 GB of the source file.
• Encoding - Within the application, UTF-8 encodings are displayed
• User-defined functions are not supported
• Integrations with datastores other than BigQuery, Google Cloud Storage, and the local filesystem
are not supported
• The Command Line Interface is not supported
• Sharing is not supported
• If you're using a Free Trial project, your project has a maximum of 8 cores available. You must
specify a combination of numWorkers, workerMachineType, and maxNumWorkers that fits within
your trial limit
DataPrep

28. DataPrep Summary
Pros:
• Multiple encoding supported
• Profiling results
Cons:
• Integration
• SDK / Command Line not supported
• Dependencies between flows
• „Parametrized” jobs throught DataFlow
(code-driven)
DataPrep

29. Contact me!
www.linkedin.com/in/pawelmitrus
www.facebook.com/pawelmitrus
pawel.mitrus@gmail.com