In this data management session, Christopher describes how to build robust and reliable data products in BigQuery and dbt, for PPC and SEO use cases. After an introduction to the modern data stack, six principles of reliable data products are presented, followed by the following use cases: - Google Ads Conversion upload - SEO sitemap efficiency report - Google Shopping product rating sync - Large-Scale link checker with advertools - Inventory-based PPC campaigns with dbt Here is the referenced selection of gists on github: https://gist.github.com/ChrisGutknecht

1. How To Build Data Products
in BigQuery for PPC & SEO
Christopher Gutknecht | @chrisgutknecht | Bergzeit

2. 1. Intro
Our Plan: Build Data Products & Activate Data
3. PPC & SEO Use Cases
2. Product Principles

3. About Chris: Acquisition & Analytics at Bergzeit
Digital Marketer
Data Nerd
Climber
1997 2008 2010 2022
Dad of 2
Online Store for Mountain Gear
145 M Revenue in FY 21/22
14 Countries, 5 Languages
World-class, data-driven team 🔥
Hiring a PPC!

4. I’d like to Set Clear Expectations for This Session
No BigQuery Intro
Data Management-Talk
What this session IS What it’s NOT
No BigQuery tactics
No ML Focus
Google Cloud & dbt focus
Data Product mindset
75% PPC, 25% SEO

5. Large-Scale PPC is Becoming the
Science of Managing Data Pipelines
RECAP FROM SMX 2019
Recap of My SMX 2019 Talk

6. Is Your Data Scattered Across Dozens of Islands?

7. If You’re Done With This… Get A Data Warehouse
Connectors

8. Data Warehouse = The New Center of Gravity?
Source: https://mikkeldengsoe.substack.com/p/future-of-the-data-warehouse

9. You Need Clean & Reliable Data for Data Products

10. Everybody has a Data Horror Story To Tell

11. Modern Data Stack: BigQuery & dbt at the core
Data Products

12. dbt is a Programming Environment for SQL

13. Are there Alternatives to dbt? Not really
No extra setup
No control
Saved queries Google Dataprep
No SQL (no-code)
Transformation &
scheduling only
Google Dataform
Free for GCP
Smaller ecosystem
SQL framework

14. The dbt community is rapidly growing

15. Wait: What about Javascript and Python?
Easy to get started
Instantly ready
Javascript Python
Powerful libraries
Leading data tools
SQL
Super scalable
Ideal for production
Centralized code
Only for smaller
data tasks
Harder to centralize

16. 1. Intro
Plan: Build Data Products & Activate Your Data
3. PPC & SEO Use Cases
2. Product Principles

17. Principles for Robust Data Products (with dbt)

18. #1: Structured Data Transformation

19. Models
Break your Transformations into Small Steps
Sources Exposures
(Data Products)
Stage Intermediate Marts

20. Use Jinja Macros to Template Your SQL Code
Jinja-Macro
Compiled SQL
https://docs.getdbt.com/docs/building-a-dbt-project/jinja-macros

21. #2: Version Control

22. Only Write Production Code with Version Control

23. Use JIRA-Tickets to Create Feature Branches

24. #3: Sufficient Data Test Coverage

25. Typical Characteristics of Data Quality

26. Define Data Quality Through Automated Tests
dbt core (5) dbt utils (15) dbt expectations (~60) custom tests (*)

27. #4: Deployment & Environments

28. Separate Environments for Test and Production
Test = Paper cup Production = Porcelain

29. #5: Proper Documentation

30. Keep Your Documentation Close To Your Code
Documentation in .yml and .md Files
Document all Sources, Models and Exposures
Get documentation as HTML
YAML File for Model Metadata
Markdown File for Description
SQL Files with Model code

31. Get Auto-Generated Project Documentation

32. Example Documentation for Source & Exposure
Exposure (Output)
Source (Input)

33. #6: Logs & Alerts

34. Production Updates and Tests are run as Jobs

35. Job Failures are logged as Alerts to Slack/Teams
Example Alert Failure

36. Result: Our BigQuery-focused GCP Architecture
CSV File
Enrichment
select *
Transformation
Ingestion
Web APIs
ML Model
Dashboard
Storage

37. 1. Intro
Let’s Finally Dig Into Some Use Cases!
3. PPC & SEO Use Cases
2. Product Principles

38. PPC Use Case: No Consent, No Ads Conversions?
?

39. Our Simple Transaction Backend Tracking
gclid
Conversion value
Timestamp
orderIdentifier
Example Python Code (see SMX 2020): https://gist.github.com/ChrisGutknecht

40. Identify New Conversions Missing in GA
SQL transformation steps in dbt (called “DAGs”)
Final output for Conversion Import

41. Yes We can! Upload Missing Ads Conversions

42. Combining Tag and Import Conversions in GAds
Full conversion coverage,
no modeling 👍

43. SEO Use Case: Sitemap Efficiency Report
Elias Dabbas’ advertools Package and sitemap_to_df()

44. Combine Sitemap with Crawl and GSC Data
SQL transformation
Result of combined data (30 days)

45. Bonus: Spy on a Competitor Sitemap with 4 LOC

46. PPC Use Case: Sync Product Ratings to Shopping
How do I get these (for free)?

47. Datainput: You Need A Simple Review Feed
review-level data
sku-level data: GTIN, sku, url

48. The Flow Diagram for the Shopping Ratings Sync

49. The CTR results are not conclusive (but, hey ⭐ )

50. Want to Sync Your Ratings? Test my Notebook!
Notebook on Github: https://gist.github.com/ChrisGutknecht

51. Use Case: Custom Large-Scale Link Checker
Didn’t work for us!
(Errors, no control, little customisation, Java-based)

52. Multi-Threaded URL-Crawling with Advertools
Elias Dabbas’ advertools and adv.crawl(url_list, …)
Fast crawling
Customised input

53. Filter Relevant Columns From Crawl Dataframe
Elias Dabbas’ advertools and adv.crawl(url_list, …)

54. The Flow Diagram for Our Link Checker
Notebook on Github: https://gist.github.com/ChrisGutknecht

55. PPC Use Case: Inventory-Based Campaigns

56. PPC Use Case: Inventory-Based Campaigns
Turn product feed data into campaign structures

57. We went from This Complex Graph in Dataprep

58. To This SQL-based Transformation in dbt
better control for complex transformations
easier to extend and reuse parts of logic
lower cost in long run
automated testing and documentation

59. There Are More Data Products in the Lab…
(Join Our Team!)

60. In The End: Do You Want to be an Architect?

61. Your Takeaways from this Session
1. What the modern data stack is and why it’s exciting
3. Which principles to apply for data products in production
2. Why dbt is the best tool for data warehouse transformation
4. A few interesting PPC and SEO use cases for you to try

62. Thanks for Your Time.
Looking Forward To Questions!
Chris Gutknecht | Teamlead A&O | Hiring a PPC!