Presented at the 25th Data Science Leuven meetup on 2020/03/11 Jonny Daenen explains the steps they took at Selligent to create a multi-tenant real-time data pipeline. He discusses all challenges the team encountered as well as the tools they used. The benefits of using Google Cloud Platform to remove operational hurdles when moving data pipelines to production are demonstrated.

1. Building
real-time data analytics
on Google Cloud

2. JONNY DAENEN - DATA SHEET
• Data Engineer/Scientist
• 3 years at Selligent
• PhD Computer Science
• Focus = Data!

3. SPOILER
• Data Engineering and the flip side of the Data Science Pyramid
• Getting to Production is a long road
• Using GCP to focus on the important stuff

4. ?

6. BUILDING CONTENT

7. BUILDING JOURNEYS

8. BOOK 1
The story of
Send-Time Optimization

9. Little Jonny*Regular Jonny
* for legal reasons we are obliged to mention that "Little Jonny" is in no way related to "Little John" from the "Robin Hood" story.

11. SEND-TIME OPTIMIZATION

13. OUR KEY DATA CHALLENGES

14. CHAPTER 1
A new Idea!

19. CAPTURE - PUB/SUB
• Event enters system
• Event is sent to Pub/Sub
• No ops
• Globally available
• Pay as you go
• 7 day retention
• No ordering (alpha feature)
• No server-side filtering
Pub/Sub

21. KAFKA COMPARISON
• Flexibility
• Configurable retention
• Technical Knowledge
• Configuration
• Management
• Certificate Management
• Operational cost

25. PROCESSING - DATAFLOW
• Aggregation of events per consumer per tenant
• Dataflow
• Managed (choose your machines)
• Auto-scaling
• In-flight pipeline updates
• Monitoring
• Exactly-once
• Batch and strEAMing (Apache Beam)
• SQL available
• Documentation Unclear
DataFlow

28. PROBLEM!
• Aggregate per consumer per tenant
• Too much state data
• 1.000.000.000 users?
• how much memory do we need?
• what about inactive consumers?
• Offload to document store
• key-value access

29. STATE - DATASTORE
• DataStore
• No ops
• Namespaces
• Deal with Failures
• Costs can go up
DataStore

33. IDEMPOTENCY

34. FAULT-TOLERANCE
• Idempotency!
• What if pipeline fails?
• Streaming means: Re-execute, Re-execute, Re-execute
• Bundle
• Out of order processing of successive windows
• Can you deal with it?
• Depends on use case
• Exactly-once?
• Use native dataflow/beam operators
The Researcher

35. MULTITENANCY

36. OUR DATA CHALLENGES

38. STATUS
• Servers setup:
• 0
• Code:
• Dataflow
• Manual setup:
• GCP project
• service accounts
• topics

39. STEP 4: PROFIT

40. CHAPTER 2
Reality

41. INTEGRATION?!
• "Legacy application"
• Changes needed
• Release cycle of 6 weeks
• Alignment with other teams
"The Wizard"

42. TESTING
• Unit tests
• Dataflow test framework
• Integration test
• Between services
• External components
• Mocking?
• Performance test
• Does it scale
• Multi-tenancy
"The Butcher"

44. AUTOMATION
• Infrastructure
• Terraform
• Testing & Deployment (CICD)
• CircleCi
"The King"

45. TERRAFORM
• Service Accounts
• Pub/Sub topics & subscriptions
• Permissions
• (Dataflow jobs)
• etc.

46. PROJECT SETUP
• Flower-system
• Expose clean data

47. CIRCLECI
• Everything as code
• Automation
• 1 file

48. MONITORING
• Create dashboards
• Create alerts
• Custom inspection app
• App Engine
• Simple Python Flask App
"The Bookkeeper"
Stackdriver

50. DATADECK

51. SECURITY & COMPLIANCE
• GDPR
• Checking
• Handling
• Archival
• Audit Logs
• (DPIA, Threat analysis, ...)
"The Bandit"

53. BIGQUERY
• Dataflow to BigQuery
• Storage
• BigQuery
• No ops
• Pay for storage
• Pay per byte queried
• Data Market
BigQuery

54. ONBOARDING, CLIENTS & LEGAL
• How to create business value?
• How to measure success?
• Who does activations?
• Do we need initial data loads?
• Who triggers it?
• What documents need to be signed?
• What do clients expect?
"Ms. Heard"

58. STATUS
• Servers setup:
• 0
• Code:
• Dataflow
• CircleCi
• Terraform
• App Engine
• Manual setup:
• GCP project

59. CHAPTER 3
Reality Strikes Again

60. ANALYSIS
• Potential improvements
• Client feedback
• AI Notebooks
• JupyterLab
• One-click start

63. STATUS
• Servers setup:
• 1 (Notebook server)
• Code:
• Dataflow
• CircleCi
• Terraform
• App Engine
• Manual setup:
• GCP project

64. BOOK 2?

65. KEY FIGURES
• Currently:
• 30 of the 700 clients
• 300 events per second
• Upcoming
• 100M emails per day
• Mobile push deliveries
• SMS deliveries
• Website views
• Different Aggregations

66. CLOUD COMPOSER
• Orchestration
• Batch jobs
• Cloud Composer
• CGP integration
• Managed Airflow
• Sloooow startup (>30')
• Expensive (300 EUR /month)

67. AZURE DEVOPS
• Almost Everything as code
• Automation
• Multiple Files ✅
• Easier for multiple environments
• Still evolving

69. STRATEGY

70. STRATEGIES
Everything as code
• Traceable
• Reproducable
• Explicit
Cloud/Serverless
• Less management
• Devops becomes easier
• Pay as you go
Automation
• Less ops work
• Reliable releases
• Continuous delivery

71. STRATEGIES
Cloud/Serverless
• PubSub
• BigQuery
• Datastore
• Dataflow (managed)
• Airflow (managed)
Everything as code
• git
• cicd
• infrastructure
Automation
• Airflow
• Azure pipelines
• Robot

72. Hanne Van Briel
Product Management
Tom Artoos
Back-end
Yohan Laudelout
Scrum Master
Timo Naessens
Quality
Kirill Ismagulov
Data Engineer / Scientist
Dirk Dupont
Data Engineer / Scientist
Jonny Daenen
Data Engineer / Scientist
TEAM DeLorean
Laurens Vijnck
Data Engineer / Scientist

73. GCP TAKEAWAY
• High-level concepts -> Focus on Data
• Good console and cli
• No ops global services
• Lower cost (depends!)
• Documentation not always up-to-date