On 2020-12-09 Laurens Vijnck and Jonny Daenen gave a workshop at PXL. During this session, we collectively provisioned a streaming ingestion pipeline in mere minutes. The technology stack included Pub/Sub, Dataflow, and BigQuery. Hereafter, students had the opportunity to perform interactive queries on their own real-time data to answer a series of business questions. These questions were borrowed from real-life cases that we encountered at Selligent Marketing Cloud. Google Colab (Free Jupyter Notebooks) and Google Data Studio have proven to be excellent tools to facilitate these kinds of interactive sessions.

1. Data
Engineering
& Data
Science
• Please install/update to the latest zoom
app
• Otherwise you cannot join our
breakout rooms
• https://zoom.us/download
• Start at 09:05 🕘
WELCOME!

2. Intro

3. LAURENS VIJNCK - DATA SHEET
• Data Engineer/Data Scientist
• 1 year at Selligent
• Master thesis on Streaming Analytics
• Interests: Streaming, Distributed Computing

4. JONNY DAENEN - DATA SHEET
• Data Engineer/Scientist
• 4 years at Selligent
• PhD Computer Science
• Focus = Data, Cloud

5. TAKEAWAY
• Cloud Technology alleviates operational burden
• Devops is a state of mind
• Road to production: Devil is in the details

6. AGENDA
• Intro
• Selligent & Data
• Use Case 1
• Real-time Data Analysis
• Big Data Tech in Google Cloud
• Data Engineering @ Selligent
• Roles & Tools
• Use Case 2
• Visual Data Exploration
• Reports in Data Studio
• Short presentation

7. INTERACTIVITY
• Raise your hand
• Use the chat

8. Selligent

10. BUILDING CONTENT

12. Data

13. What is data?

14. What data types do you
know?

15. Why is data useful?

16. When is data big data?

17. Big Data - the 3 V’s
17

18. Which data tech do you
know?

19. DATA & AI LANDSCAPE 2020
SOCIAL ANALYTICSLOG ANALYTICS WEB / MOBILE /
COMMERCE ANALYTICS
SEARCH
BI PLATFORMS
COMPUTER VISION SPEECH & NLP
DATA ANALYST PLATFORMS
DATA SCIENCE
NOTEBOOKS
DATA SCIENCE
PLATFORMS
APPLICATIONS — INDUSTRY
APPLICATIONS — ENTERPRISE
HUMAN
CAPITAL
PARTNERSHIPS
GOV’T &
INTELLIGENCE
EDUCATION REAL ESTATE FINANCE -
LENDING
ADVERTISING
MARKETING -
B2B
SALES
INFRASTRUCTURE
SERVER-
LESS
GRAPH DBs MPP DBsNoSQL DATABASES
DATA GOVERNANCEETL / DATA
TRANSFORMATION
CLUSTERSVCS
HORIZONTAL AI
MACHINE LEARNING FINANCE AUTOMATION & RPA
CUSTOMER EXPERIENCE / SERVICE
LEGAL
MARKETING - B2C
REGTECH&
COMPLIANCE
SECURITY
AGRICULTURE
FINANCE - INVESTING
HEALTHCARE INDUSTRIAL
INSURANCE
LIFE SCIENCES OTHERTRANSPORTATIONAI OPSDATAGENERATION
&LABELLING
GPUDBs&
CLOUD
AI HARDWAREMGMT / MONITORING
NewSQL DATABASES
VISUALIZATION
ANALYTICS & MACHINE INTELLIGENCE
DATA
QUALITY
DATA INTEGRATION
DATA LAKES DATA
WAREHOUSES
STREAMING /
IN-MEMORY
HADOOPSTORAGE
OTHERLOCATION INTELLIGENCE
DATA SOURCES & APIs
DATA MARKETPLACES
& DISCOVERY
AIR / SPACE / SEAFINANCIAL & ECONOMIC DATA PEOPLE / ENTITIES RESEARCHINCUBATORS &
SCHOOLS
DATA RESOURCES
DATA SERVICES
OPEN SOURCE
QUERY / DATA FLOW STREAMING &
MESSAGING
STAT TOOLS &
LANGUAGES
AI OPS
& INFRA
COLLABORATION SECURITYFRAMEWORKS DATA ACCESS & DATABASES SEARCH VISUALIZATIONLOGGING & MONITORINGORCHESTRATION
&PIPELINES
AI / MACHINE LEARNING / DEEP LEARNING
© Matt Turck (@mattturck) & FirstMark (@firstmarkcap) mattturck.com/data2020Version 1.0 - September 2020
COMMERCE

20. Facts

21. 4k interactions per
second
250M PER DAY

22. 150M emails per DAY

23. 99.7% deliverability rate

25. 25

26. Image courtesy of Monica Rogati

27. Use Case 1
Analyze interaction data
in real time

28. DATA SOURCE: BEHAVIORAL DATA
Show interest
Receive notification
Walk in store
Buy in store
Visit homepage
Install app
Open app

30. Demo: Part 1

31. STORAGE - BIGQUERY
• Columnar Storage
• No ops (serverless)
• Pay for storage
• Pay per byte queried (columns & time touched)
• Data Market
BigQuery

32. INGEST - PUB/SUB
• Event enters system
• Event is sent to Pub/Sub
• No ops (serverless)
• Globally available
• Pay as you go
• 7 day retention
• No ordering (alpha feature)
• No server-side filtering
Pub/Sub

33. PROCESSING - DATAFLOW
• Aggregation of events per consumer per tenant
• Dataflow
• Managed (choose your machines)
• Serverless
• Auto-scaling
• In-flight pipeline updates
• Monitoring
• Exactly-once
• Batch and strEAMing (Apache Beam)
• SQL available
• Documentation Unclear
DataFlow

36. Assignment

37. DATA EXPLORATION
• How many teams and members are present?
• How large is the audience?
• When did every member become active?
• What is the activity timespan of every team?
• Which minute of the hour received the most clicks?

38. The Daily Life of a Data
Engineer @ Selligent

39. PROBLEM

40. 40

41. 41

42. HOW WOULD YOU SOLVE THIS
GIVEN THE PREVIOUS DATASET?

45. user nr. 3
12:07:00
CLICK
MAX

46. THE KEY DATA CHALLENGES

47. Team roles

48. DATA ENGINEER
• Fault-tolerance
• What if pipeline fails?
• Streaming means: Re-execute, Re-execute, Re-execute
• Bundle
• Out of order processing of successive windows
• Can you deal with it?
• Depends on use case
• Exactly-once?
• Use native dataflow/beam operators

49. THE INTEGRATION WIZARD
• "Legacy application"
• Changes needed
• Release cycle of 6 weeks
• Alignment with other teams

50. SEND-TIME OPTIMIZATION

51. THE BUTCHER
• Unit tests
• Dataflow test framework
• Integration test
• Between services
• External components
• Mocking?
• Performance test
• Does it scale
• Multi-tenancy

52. THE AUTOMATION KING
• Infrastructure as Code
• Terraform
• Testing & Deployment (CICD)
• CircleCi

53. TERRAFORM
• Resources
• Automatic inference of dependencies

54. TERRAFORM
• Dependency graph

55. CIRCLECI
• Everything as code
• Automation
• 1 file

56. THE MONITOR
• Create dashboards
• Create alerts
• Custom inspection app
• App Engine
• Simple Python Flask App
• Cost!
Stackdriver

58. THE BANDIT
• Security & Compliance
• GDPR
• Checking
• Handling
• Archival
• Audit Logs
• (DPIA, Threat analysis, ...)

59. THE MANAGER
• Onboarding & Clients
• How to create business value?
• How to measure success?
• Who does activations?
• Do we need initial data loads?
• Who triggers it?
• What documents need to be signed?
• What do clients expect?

60. THE DATA SCIENTIST
• Analysis
• Potential improvements
• Client feedback
• AI Notebooks
• JupyterLab
• One-click start

61. TEAM VALUES
Everything as code
• Traceable
• Reproducable
• Explicit
Cloud/Serverless
• Less management
• Devops becomes easier
• Pay as you go
Automation
• Less ops work
• Reliable releases
• Continuous delivery

62. TOOLS
Cloud/Serverless
• PubSub
• BigQuery
• Datastore
• Dataflow (managed)
• Airflow (managed)
Everything as code
• git
• cicd
• infrastructure
Automation
• Airflow
• Azure pipelines
• Robot

63. Hanne Van Briel
Product Management
Tom Artoos
Back-end
Yohan Laudelout
Scrum Master
Timo Naessens
Quality
Kirill Ismagulov
Data Engineer / Scientist
Dirk Dupont
Data Engineer / Scientist
Jonny Daenen
Data Engineer / Scientist
TEAM DeLorean
Laurens Vijnck
Data Engineer / Scientist

64. Use Case 2
Visual Data Exploration

66. Assignment Datastudio

67. BUSINESS QUESTIONS
• What hour of the day are most people active?
• Which users are active before 10am?
• What is the most clicks people have done in 1 hour?
• How many clicks does an average user do in 1 week?
• How is the channel usage distributed?
• ...?

68. VISUALIZATIONS
• Heatmap of user activity in a day
• Heatmap of activity per channel
• Year over year comparison of activity per month
• For a given user, the activity timeline
• For a given user, the average number of clicks per week

69. Presentation

71. Image courtesy of Kris Peeters (Data Minded)

72. TAKEAWAY
• Cloud Technology alleviates operational burden
• Devops is a state of mind
• Road to production: Devil is in the details

73. Feedback Form

74. Q&A