Juan Sebastián Urrego Escobar, profile picture
Uploaded byJuan Sebastián Urrego Escobar
PDF, PPTX313 views

Scaling Your Data: Data Democratisation and DataOps

The document discusses the implementation of DataOps at Travelstart, focusing on data democratization, efficient data management, and an event-driven architecture. It outlines the evolution of data processing practices, comparing traditional business intelligence methods to modern DataOps approaches that enhance collaboration and automation across data flows. A key aspect highlighted is the architecture that supports real-time data processing and governance within the organization.

Embed presentation

Download as PDF, PPTX
1 / 109
2 / 109
3 / 109
4 / 109
5 / 109
6 / 109
7 / 109
8 / 109
9 / 109
10 / 109
11 / 109
12 / 109
13 / 109
14 / 109
15 / 109
16 / 109
17 / 109
18 / 109
19 / 109
20 / 109
21 / 109
22 / 109
23 / 109
24 / 109
25 / 109
26 / 109
27 / 109
28 / 109
29 / 109
30 / 109
31 / 109
32 / 109
33 / 109
34 / 109
35 / 109
36 / 109
37 / 109
38 / 109
39 / 109
40 / 109
41 / 109
42 / 109
43 / 109
44 / 109
45 / 109
46 / 109
47 / 109
48 / 109
49 / 109
50 / 109
51 / 109
52 / 109
53 / 109
54 / 109
55 / 109
56 / 109
57 / 109
58 / 109
59 / 109
60 / 109
61 / 109
62 / 109
63 / 109
64 / 109
65 / 109
66 / 109
67 / 109
68 / 109
69 / 109
70 / 109
71 / 109
72 / 109
73 / 109
74 / 109
75 / 109
76 / 109
77 / 109
78 / 109
79 / 109
80 / 109
81 / 109
82 / 109
83 / 109
84 / 109
85 / 109
86 / 109
87 / 109
88 / 109
89 / 109
90 / 109
91 / 109
92 / 109
93 / 109
94 / 109
95 / 109
96 / 109
97 / 109
98 / 109
99 / 109
100 / 109
101 / 109
102 / 109
103 / 109
104 / 109
105 / 109
106 / 109
107 / 109
108 / 109
109 / 109
Scaling Your Data: Data Democratisation and DataOps Juan Urrego March 2020
Juan Urrego 2 DataOps Team Lead YOU-Anne JOO-Anne ONE Who-Anne Colombia Columbia
3 Who is Travelstart? SOUTH AFRICA UAE TAIPEI EGYPT NIGERIA TANZANIA KENYA MIDDLE EAST ASIAAFRICAEUROPE ATHENS PORTO Travelstart is a leading online travel agency that helps today’s business and leisure travellers search, compare and book the best flight, hotel and car options with all your favourite airline and accommodation suppliers. Based in Cape Town, Travelstart proudly employs a team of more than 850 staff from all walks of life who are passionate about one goal - making travel simple.
Context
At Google, for example, nearly 80% of employees use Dremel (the internal counterpart to Google BigQuery) every month...everyone touches data on a regular basis to inform their decisions” Valliappa Lakshmanan (2018) 5 What Are We Doing? Context “
6 What Are We Doing? Context https://www.statista.com/statistics/219333/number-of-google-employees-by-department/
7 What Are We Doing? Context https://www.statista.com/statistics/219333/number-of-google-employees-by-department/ ~38% ~62%
Ask someone a question and you are likely to receive a link to a BigQuery view or query rather than the actual answer” Valliappa Lakshmanan (2018) 8 What Are We Doing? Context “
9 How Did It Start? Context 2014 Inception Classic BI architecture (pull-based) Design to supply a short-term need
10 How Did It Start? Context 2014 2018 Inception Classic BI architecture (pull-based) Design to supply a short-term need Non-performant No test environments No version control Users don’t trust the data Shit Hits the Fan
11 How Did It Start? Context 2014 2018 2019 Inception Classic BI architecture (pull-based) Design to supply a short-term need Non-performant No test environments No version control Users don’t trust the data Shit Hits the Fan DataOps Approach Event-Driven Best practices of Software Engineering Remove the BI bottleneck DIY -> Data Democratisation
DataOps
First Generation Proprietary enterprise data warehouse and business intelligence platforms 13 Data Process Generations DataOps https://martinfowler.com/articles/data-monolith-to-mesh.html Second Generation Big Data ecosystem with a data lake as a silver bullet Third Generation Event-Driven, Near Real Time and Cloud Based
● Data Mesh: Embracing ubiquitous data ○ Distributed Domain Driven Architecture (DDD), Self-serve Platform Design, and Product Thinking with Data ○ Data is discoverable, addressable, trustworthy & truthful, self-describing semantics & syntax, inter-operable & governed by global standards, domain data cross-functional teams 14 Data Process Generations DataOps https://martinfowler.com/articles/data-monolith-to-mesh.html
DataOps is a collaborative data management practice focused on improving the communication, integration and automation of data flows between data managers and data consumers across an organization. The goal of DataOps is to deliver value faster by creating predictable delivery and change management of data, data models and related artifacts” Gartner Glossary 15 What is DataOps? DataOps https://www.gartner.com/en/information-technology/glossary/dataops “
DataOps uses technology to automate the design, deployment and management of data delivery with appropriate levels of governance, and it uses metadata to improve the usability and value of data in a dynamic environment.” Gartner Glossary 16 What is DataOps? DataOps https://www.gartner.com/en/information-technology/glossary/dataops “
17 What is DataOps? DataOps DevOps Agile Lean DataOps
18 What is DataOps? DataOps https://www.saagie.com/blog/dataops-devops-2-0/ ● A few practices are common to DevOps and DataOps: ○ Automation (CI/CD) ○ Unit tests (code coverage is relevant) ○ Environments management ○ Versions management ○ Monitoring
19 DataOps at Travelstart DataOps Do It Yourself (DIY) Data democratisation Data-Driven Decision Making Don’t Repeat Yourself (DRY) Values
20 DataOps at Travelstart DataOps Do It Yourself (DIY) Data democratisation Data-Driven Decision Making Don’t Repeat Yourself (DRY) Data Quality Availability Usability Reliability Performance Flexibility Quality Attributes Values
21 DataOps at Travelstart DataOps Do It Yourself (DIY) Data democratisation Data-Driven Decision Making Don’t Repeat Yourself (DRY) Data Quality Availability Usability Reliability Performance Flexibility Quality Attributes Data Lakes IaC & CI/CD Data Dictionary Data Catalog Values Descriptive Predictive Prescriptive Framework Data Pipelines Data Warehouse
The Architecture
23 Reference Model The Architecture Lambda ~ Kappa Architecture Event Driven Architecture + Inmutable master data Precompute Views New Data stream Process Stream Increment Views View 1 View 2 View N View 1 View 2 View N Query Batch recompute Real-Time Increment Batch views Real Time views Batch Layer Serving Layer Speed Layer Main Architectural Pattern: Publisher-Subscriber
Distribution Tier Clean Raw Serving TierTransform TierSource Tier External Internal 24 Reference Architecture The Architecture Aggregation Tier Ingestion Integration Quality Raw Data OLAP Analytics Monitoring & Automation Tier Data Lakes Dead Letters Backups Data Marts Ubiquitous Data Real Time Views Alerting Performance Monitoring Dashboarding CI/CD
Monitoring & Automation Tier Alerting Performance Monitoring Dashboarding CI/CD Distribution Tier Clean Raw Serving TierTransform TierSource Tier External Internal 25 Reference Architecture The Architecture Aggregation Tier Ingestion Integration Quality Raw Data OLAP Analytics Data Lakes Dead Letters Backups Data Marts Ubiquitous Data Real Time Views
Source Tier ● All the internal and/or external service/component/instance/process that will behave as publisher/producer of events ● An event is a data record expressing an occurrence and its context. ○ Slow Changing Dimensions (SCD) ■ Type 1: Last state ■ Type 2: Complete state ■ Type 3: What changed ● According to cloudevents 1.0 it MUST have: ○ Unique ID ○ Source ○ SpecVersion ○ Type ● According to cloudevents 1.0 it is NICE to have: ○ DataContentType ○ DataSchema ○ Subject ○ Time 26 Solution Architecture Architecture https://github.com/cloudevents/spec/blob/v1.0/spec.md
Source Tier ● All the internal and/or external service/component/instance/process that will behave as publisher/producer of events ● An event is a data record expressing an occurrence and its context. ○ Slow Changing Dimensions (SCD) ■ Type 1: Last state ■ Type 2: Complete state ■ Type 3: What changed ● According to cloudevents 1.0 it MUST have: ○ Unique ID ○ Source ○ SpecVersion ○ Type ● According to cloudevents 1.0 it is NICE to have: ○ DataContentType ○ DataSchema ○ Subject ○ Time 27 Solution Architecture Architecture https://github.com/cloudevents/spec/blob/v1.0/spec.md
Source Tier ● All the internal and/or external service/component/instance/process that will behave as publisher/producer of events ● An event is a data record expressing an occurrence and its context. ○ Slow Changing Dimensions (SCD) ■ Type 1: Last state ■ Type 2: Complete state ■ Type 3: What changed ● According to cloudevents 1.0 it MUST have: ○ Unique ID ○ Source ○ SpecVersion ○ Type ● According to cloudevents 1.0 it is NICE to have: ○ DataContentType ○ DataSchema ○ Subject ○ Time 28 Solution Architecture Architecture https://github.com/cloudevents/spec/blob/v1.0/spec.md
Source Tier ● All the internal and/or external service/component/instance/process that will behave as publisher/producer of events ● An event is a data record expressing an occurrence and its context. ○ Slow Changing Dimensions (SCD) ■ Type 1: Last state ■ Type 2: Complete state ■ Type 3: What changed ● According to cloudevents 1.0 it MUST have: ○ Unique ID ○ Source ○ SpecVersion ○ Type ● According to cloudevents 1.0 it is NICE to have: ○ DataContentType ○ DataSchema ○ Subject ○ Time 29 Solution Architecture Architecture https://github.com/cloudevents/spec/blob/v1.0/spec.md
30 Solution Architecture Architecture Source Tier Event Granularity Coarse-grained Fined-grained Aggregation Required
31 Solution Architecture Architecture Source Tier Event Granularity Extreme Aggregation Required Coarse-grained Fined-grained
32 Solution Architecture Architecture Source Tier Event Granularity Extreme Field or sub-property Aggregation Required Phone Email Coarse-grained Fined-grained
33 Solution Architecture Architecture Source Tier Event Granularity Extreme Domain Event Field or sub-property Aggregation Required Phone Email SearchEvent Composition vs Aggregation Coarse-grained Fined-grained
34 Solution Architecture Architecture Source Tier Format, Compression and Codecs Plain Text Compression: GZIP, BZIP2, LZ4 Binary Codec: Deflate, Snappy, LZO
35 Solution Architecture Architecture Source Tier Travelstart Event Template Payload { “uuid”: “34c86044-4b73-46ec-b766”, “timestamp”: 1581749960349, …. } ● Main Format: JSON ● Compression: GZIP JSON Minify + GZIP -> ~96% size reduction Header Type: search_event Format: json System: backend UUID: 34c86044-4b73-46ec-b766 Ip: 10.10.0.1 InstanceId: backend-01 Hostname: mybackend.travelstart Compression: gzip Timestamp: 1581749960349
Distribution Tier Clean Raw Serving TierTransform TierSource Tier External Internal 36 Reference Architecture The Architecture Aggregation Tier Ingestion Integration Quality Raw Data OLAP Analytics Data Lakes Dead Letters Backups Data Marts Ubiquitous Data Real Time Views Monitoring & Automation Tier Alerting Performance Monitoring Dashboarding CI/CD
Aggregation Tier ● All the services and components that aggregate Atomic Events into a Domain Event ● This aggregation can happen: ○ Inside the source ■ On Memory ■ Using Storage ○ Outside the source ■ Middleware dedicated to do this ■ Ingestion or Integration pipelines ● External sources use dedicated endpoints: Receive data from external sources, setup the canonical structure and send to the topic ○ Serverless functions ○ Stateless containers ● Schedulers can serve as trigger to start the ETL 37 Solution Architecture Architecture
Aggregation Tier ● All the services and components that aggregate Atomic Events into a Domain Event ● This aggregation can happen: ○ Inside the source ■ On Memory ■ Using Storage ○ Outside the source ■ Middleware dedicated to do this ■ Ingestion or Integration pipelines ● External sources use dedicated endpoints: Receive data from external sources, setup the canonical structure and send to the topic ○ Serverless functions ○ Stateless containers ● Schedulers can serve as trigger to start the ETL 38 Solution Architecture Architecture
Aggregation Tier ● All the services and components that aggregate Atomic Events into a Domain Event ● This aggregation can happen: ○ Inside the source ■ On Memory ■ Using Storage ○ Outside the source ■ Middleware dedicated to do this ■ Ingestion or Integration pipelines ● External sources use dedicated endpoints: Receive data from external sources, setup the canonical structure and send to the topic ○ Serverless functions ○ Stateless containers ● Schedulers can serve as trigger to start the ETL 39 Solution Architecture Architecture
Aggregation Tier ● All the services and components that aggregate Atomic Events into a Domain Event ● This aggregation can happen: ○ Inside the source ■ On Memory ■ Using Storage ○ Outside the source ■ Middleware dedicated to do this ■ Ingestion or Integration pipelines ● External sources use dedicated endpoints: Receive data from external sources, setup the canonical structure and send to the topic ○ Serverless functions ○ Stateless containers ● Schedulers can serve as trigger to start the ETL 40 Solution Architecture Architecture
41 Solution Architecture Architecture Aggregation Tier Areas of aggregation Source System Producer 2 Read Replica(s) or API Aggregation System Distribution & Transformation DWH/DLRead Replica(s) Orchestrator Orchestrator Producer n Event Delivery Producer 1 Map & Aggr Event Delivery Windowing [session] Map & Aggregator Event Producer Persistence Complexity
42 Solution Architecture Architecture Aggregation Tier External Sources
43 Solution Architecture Architecture Aggregation Tier Schedulers
Distribution Tier Clean Raw Serving TierTransform TierSource Tier External Internal 44 Reference Architecture The Architecture Aggregation Tier Ingestion Integration Quality Raw Data OLAP Analytics Data Lakes Dead Letters Backups Data Marts Ubiquitous Data Real Time Views Monitoring & Automation Tier Alerting Performance Monitoring Dashboarding CI/CD
Distribution Tier ● All the Topics/Queues in charge of receive raw and clean to be distributed to multiple subscribers ● Topic per event type ● Subscription per subscriber ● Communication between producer and topics could failed, so keep in mind: ○ Retries ○ Graveyard (dead letters) 45 Solution Architecture Architecture
Distribution Tier ● All the Topics/Queues in charge of receive raw and clean to be distributed to multiple subscribers ● Topic per event type ● Subscription per subscriber ● Communication between producer and topics could failed, so keep in mind: ○ Retries ○ Graveyard (dead letters) 46 Solution Architecture Architecture
Distribution Tier ● All the Topics/Queues in charge of receive raw and clean to be distributed to multiple subscribers ● Topic per event type ● Subscription per subscriber ● Communication between producer and topics could failed, so keep in mind: ○ Retries ○ Graveyard (dead letters) 47 Solution Architecture Architecture
Distribution Tier ● All the Topics/Queues in charge of receive raw and clean to be distributed to multiple subscribers ● Topic per event type ● Subscription per subscriber ● Communication between producer and topics could failed, so keep in mind: ○ Retries ○ Graveyard (dead letters) 48 Solution Architecture Architecture
Distribution Tier Topics & Subscriptions 49 Solution Architecture Architecture
Distribution Tier Topics & Subscriptions 50 Solution Architecture Architecture
Distribution Tier Clean Raw Serving TierTransform TierSource Tier External Internal 51 Reference Architecture The Architecture Aggregation Tier Ingestion Integration Quality Raw Data OLAP Analytics Data Lakes Dead Letters Backups Data Marts Ubiquitous Data Real Time Views Monitoring & Automation Tier Alerting Performance Monitoring Dashboarding CI/CD
Transformation Tier ● Stream/Batch Data Pipelines in charge of receiving domain events, processing and persisting them ● A Pipeline can be split by domain and/or resource consumption ● 3 different categories: ○ Ingestion ○ Integration ○ Quality ■ Data Lineage and Data Provenance ● Serverless functions can be used here, but be careful with them! 52 Solution Architecture Architecture
Transformation Tier ● Stream/Batch Data Pipelines in charge of receiving domain events, processing and persisting them ● A Pipeline can be split by domain and/or resource consumption ● 3 different categories: ○ Ingestion ○ Integration ○ Quality ■ Data Lineage and Data Provenance ● Serverless functions can be used here, but be careful with them! 53 Solution Architecture Architecture
Transformation Tier ● Stream/Batch Data Pipelines in charge of receiving domain events, processing and persisting them ● A Pipeline can be split by domain and/or resource consumption ● 3 different categories: ○ Ingestion ○ Integration ○ Quality ■ Data Lineage and Data Provenance ● Serverless functions can be used here, but be careful with them! 54 Solution Architecture Architecture
Transformation Tier ● Stream/Batch Data Pipelines in charge of receiving domain events, processing and persisting them ● A Pipeline can be split by domain and/or resource consumption ● 3 different categories: ○ Ingestion ○ Integration ○ Quality ■ Data Lineage and Data Provenance ● Serverless functions can be used here, but be careful with them! 55 Solution Architecture Architecture
56 Solution Architecture Architecture Transformation Tier Ingestion Pipelines In charge of receiving the raw events, process, clean, enrich and validate quality of them - Uncompress - Standardize - Cleansing - Enrichment - Schematization - Quality Checks - Alerting Directed Acyclic Graph (DAG) Canonize Events Process Quality Assurance Persist Raw (Data Lake) Persistence Publish Clean Event RAW CLEAN
57 Solution Architecture Architecture Transformation Tier Ingestion Pipelines
58 Solution Architecture Architecture Transformation Tier Ingestion Pipelines
59 Solution Architecture Architecture Transformation Tier Ingestion Pipelines Col 1 Col 2 Col 3 Quality Check
60 Solution Architecture Architecture Transformation Tier Ingestion Pipelines Col 1 Col 2 Col 3 Metadata
61 Solution Architecture Architecture Transformation Tier Integration Pipelines In charge of integrating multiple sources to create consolidated data martsDirected Acyclic Graph (DAG) Identify Integration Type Read Event Sources Join Sources Persistence RAW/CLEAN
62 Solution Architecture Architecture Transformation Tier Quality Pipelines In charge of reading deadletters, run reports to analyse quality issues and bring back to life (if possible) dead/lost events (necromancer) Directed Acyclic Graph (DAG) Identify Quality Check Read Event Sources Run Quality Check Report CLEAN
63 Solution Architecture Architecture Transformation Tier Quality Pipelines
64 Solution Architecture Architecture Transformation Tier Quality Pipelines
65 Solution Architecture Architecture Transformation Tier Quality Pipelines
66 Solution Architecture Architecture Transformation Tier Quality Pipelines
67 Solution Architecture Architecture Transformation Tier Quality Pipelines com.travelstart.bi.etl.obqm.exceptions.NonNullableFieldException: The field 'bookingUuid' can't be null | at com.travelstart.bi.etl.pipelines.booking.transforms.TsMessageToBookingFn.processElement(TsMe ssageToBookingFn.java:152) : 821270 com.travelstart.bi.etl.obqm.exceptions.NonNullableFieldException: The field 'reference' can't be null | at com.travelstart.bi.etl.pipelines.booking.transforms.TsMessageToBookingFn.processElement(TsMe ssageToBookingFn.java:152) : 7336 com.travelstart.bi.etl.obqm.exceptions.NonNullableFieldException: The field 'bookingId' can't be null | at com.travelstart.bi.etl.pipelines.booking.transforms.TsMessageToBookingFn.processElement(TsMe ssageToBookingFn.java:152) : 2482
Transformation Tier 68 Solution Architecture Architecture
Transformation Tier 69 Solution Architecture Architecture
Transformation Tier 70 Solution Architecture Architecture
Transformation Tier 71 Solution Architecture Architecture
Transformation Tier 72 Solution Architecture Architecture
Still Awake?
Distribution Tier Clean Raw Serving TierTransform TierSource Tier External Internal 74 Reference Architecture The Architecture Aggregation Tier Ingestion Integration Quality Raw Data OLAP Analytics Data Lakes Dead Letters Backups Data Marts Ubiquitous Data Real Time Views Monitoring & Automation Tier Alerting Performance Monitoring Dashboarding CI/CD
Serving Tier ● Store all the raw and clean data to be accessible for our stakeholders ● About raw data: ○ Data Lakes ○ Dead Letters ○ Backups ● About OLAP: ○ Data Marts ○ Ubiquitous Data/ Data Sourcing ○ Real Time Views ● About Analytics: ○ Whatever it makes you happy (of course with data governance) 75 Solution Architecture Architecture
Serving Tier ● Store all the raw and clean data to be accessible for our stakeholders ● About raw data: ○ Data Lakes ○ Dead Letters ○ Backups ● About OLAP: ○ Data Marts ○ Ubiquitous Data/ Data Sourcing ○ Real Time Views ● About Analytics: ○ Whatever it makes you happy (of course with data governance) 76 Solution Architecture Architecture
Serving Tier ● Store all the raw and clean data to be accessible for our stakeholders ● About raw data: ○ Data Lakes ○ Dead Letters ○ Backups ● About OLAP: ○ Data Marts ○ Ubiquitous Data/ Data Sourcing ○ Real Time Views ● About Analytics: ○ Whatever it makes you happy (of course with data governance) 77 Solution Architecture Architecture
Serving Tier ● Store all the raw and clean data to be accessible for our stakeholders ● About raw data: ○ Data Lakes ○ Dead Letters ○ Backups ● About OLAP: ○ Data Marts ○ Ubiquitous Data/ Data Sourcing ○ Real Time Views ● About Analytics: ○ Whatever it makes you happy (of course with data governance) 78 Solution Architecture Architecture
79 Solution Architecture Architecture Serving Tier Data Lakes, Dead Letters & Backups Plain Text / Schema-less Compression: GZIP, BZIP2, LZ4 Binary / Schema Codec: Deflate, Snappy, LZO, LZIP, GZIP
80 Solution Architecture Architecture Serving Tier Data Lakes, Dead Letters & Backups
81 Solution Architecture Architecture Serving Tier Data Lakes, Dead Letters & Backups Standard/ Hotline Nearline Coldline Archive 180 days Min Duration: 90 days Min Duration: 365 days365 days Min Duration: 30 days 43% cost reduction 69% cost reduction 89% cost reduction Deletion Cost
82 Solution Architecture Architecture Serving Tier Data Lakes vs Data Warehouses Data Lake Data Warehouse Raw data It’s dirty Transactional structure not analytical Clean/Skew data Quality processes applied Schema defined for analytical purposes
83 Solution Architecture Architecture Serving Tier Data Marts, Event Sourcing and Real Time Views HyperCube Pivot Drilldown Roll Up Dicing Slicing
84 Solution Architecture Architecture Serving Tier Data Marts, Event Sourcing and Real Time Views More than 300 columns Includes facts and aggregations Following mostly a star modeling and denormalisation
85 Solution Architecture Architecture Serving Tier Data Marts, Event Sourcing and Real Time Views Views Materialise d Views
86 Solution Architecture Architecture Serving Tier Data Marts, Event Sourcing and Real Time Views
Distribution Tier Clean Raw Serving TierTransform TierSource Tier External Internal 87 Reference Architecture The Architecture Aggregation Tier Ingestion Integration Quality Raw Data OLAP Analytics Data Lakes Dead Letters Backups Data Marts Ubiquitous Data Real Time Views Monitoring & Automation Tier Alerting Performance Monitoring Dashboarding CI/CD
Monitoring & Automation Tier ● The command and control area of the whole architecture ● It needs to monitor everything related with data ingestion, processing and infrastructure ● Alerting: ○ Policies ○ Multiple notification channels ● Performance and Stability Monitoring ○ Bottlenecks, Bugs, etc ○ Slow steps, underutilized infrastructure, memory issues, etc ● CI/CD + IaC ○ Development and Deployment lifecycle completely automated ○ Any architectural change should be track through a version control ● Dashboards! 88 Solution Architecture Architecture
89 Solution Architecture Architecture Monitoring & Automation Tier Monitoring
90 Solution Architecture Architecture Monitoring & Automation Tier Monitoring
91 Solution Architecture Architecture Monitoring & Automation Tier Monitoring
92 Solution Architecture Architecture Monitoring & Automation Tier Monitoring
93 Solution Architecture Architecture Monitoring & Automation Tier Monitoring
94 Solution Architecture Architecture Monitoring & Automation Tier Alerting Messages delivered rate Messages in queue Too Many Resources Error Rate
95 Solution Architecture Architecture Monitoring & Automation Tier Alerting
96 Solution Architecture Architecture Monitoring & Automation Tier Alerting
97 Solution Architecture Architecture Monitoring & Automation Tier Alerting
98 Solution Architecture Architecture Monitoring & Automation Tier CI/CD & IaC Jenkins (Pipelines) Cloud Deployment Manager Cloud Build GitHub Actions
99 Solution Architecture Architecture Monitoring & Automation Tier CI/CD & IaC
100 Solution Architecture Architecture Monitoring & Automation Tier CI/CD & IaC
101 Solution Architecture Architecture Monitoring & Automation Tier CI/CD & IaC Spotless Code Analysis PR Analysis
102 Solution Architecture Architecture Monitoring & Automation Tier CI/CD & IaC
103 Solution Architecture Architecture Monitoring & Automation Tier CI/CD & IaC
104 Solution Architecture Architecture Monitoring & Automation Tier CI/CD & IaC info: title: Buckets Template description: Creates buckets in GCS with the correct life cycle rules and security constraints version: 1.0 properties: name: type: string description: Name property of the bucket standard `ts-<env>-dataops-<name>` pattern: ^[a-z0-9]+$ labels: type: object description: User-provided bucket labels, in key/value pairs. YAML Schema
105 Solution Architecture Architecture Monitoring & Automation Tier CI/CD & IaC info: title: Buckets Template description: Creates buckets in GCS with the correct life cycle rules and security constraints version: 1.0 properties: name: type: string description: Name property of the bucket standard `ts-<env>-dataops-<name>` pattern: ^[a-z0-9]+$ labels: type: object description: User-provided bucket labels, in key/value pairs. YAML Schema
106 Solution Architecture Architecture Monitoring & Automation Tier CI/CD & IaC Jinja Template
107 Solution Architecture Architecture Monitoring & Automation Tier CI/CD & IaC Jinja/YAML Template - name: ts-dataops-batch type: buckets.jinja properties: name: batch environment: {{ properties['environment'] }} labels: description: temporary_files_to_insert data_type: raw deleteRule: age: 7 isLive: true
108 Solution Architecture Architecture Monitoring & Automation Tier CI/CD & IaC
Questions juancho088 @juancho088 @tech-travelstart juan-urrego jsurrego We love talent. We are recruiting!

More Related Content

PPTX
Accidental DataOps
bySteve Ross
 
PPTX
DataOps: Nine steps to transform your data science impact Strata London May 18
byHarvinder Atwal
 
PPTX
DataOps - Big Data and AI World London - March 2020 - Harvinder Atwal
byHarvinder Atwal
 
PPTX
The Importance of DataOps in a Multi-Cloud World
byDATAVERSITY
 
PPTX
DataOps @ Scale: A Modern Framework for Data Management in the Public Sector
byTamrMarketing
 
PDF
How to Streamline DataOps on AWS
byEnterprise Management Associates
 
PDF
Integrating the CDO Role Into Your Organization; Managing the Disruption (MIT...
byCaserta
 
PPTX
Architecting Data For The Modern Enterprise - Data Summit 2017, Closing Keynote
byCaserta
 
Accidental DataOps
bySteve Ross
 
DataOps: Nine steps to transform your data science impact Strata London May 18
byHarvinder Atwal
 
DataOps - Big Data and AI World London - March 2020 - Harvinder Atwal
byHarvinder Atwal
 
The Importance of DataOps in a Multi-Cloud World
byDATAVERSITY
 
DataOps @ Scale: A Modern Framework for Data Management in the Public Sector
byTamrMarketing
 
How to Streamline DataOps on AWS
byEnterprise Management Associates
 
Integrating the CDO Role Into Your Organization; Managing the Disruption (MIT...
byCaserta
 
Architecting Data For The Modern Enterprise - Data Summit 2017, Closing Keynote
byCaserta
 

What's hot

PDF
Moving Past Infrastructure Limitations
byCaserta
 
PDF
General Data Protection Regulation - BDW Meetup, October 11th, 2017
byCaserta
 
PPTX
SKOS as the focal point of linked data strategies
bySemantic Web Company
 
PDF
7 Big Data Challenges and How to Overcome Them
byQubole
 
PDF
Designing a Successful Governed Citizen Data Science Strategy
byDATAVERSITY
 
PDF
Creating a DevOps Practice for Analytics -- Strata Data, September 28, 2017
byCaserta
 
PPTX
Big and fast data strategy 2017 jr
byJonathan Raspaud
 
PDF
Building New Data Ecosystem for Customer Analytics, Strata + Hadoop World, 2016
byCaserta
 
PDF
The Rise of the CDO in Today's Enterprise
byCaserta
 
PPTX
Using Machine Learning & Spark to Power Data-Driven Marketing
byCaserta
 
PPTX
Predictive and Prescriptive Analytics Expert Session Webinar
byibi
 
PDF
Knowledge Graphs for Transformation: Dynamic Context for the Intelligent Ente...
byNeo4j
 
PPTX
Fasten you seatbelt and listen to the Data Steward
byJean-Pierre Riehl
 
PDF
Drive your business with predictive analytics
byThe Marketing Distillery
 
PPTX
Data Intelligence: How the Amalgamation of Data, Science, and Technology is C...
byCaserta
 
PPTX
Your Data Nerd Friends Need You!
byDataKitchen
 
PDF
You're the New CDO, Now What?
byCaserta
 
PDF
Building a New Platform for Customer Analytics
byCaserta
 
PDF
The Key to Big Data Modeling: Collaboration
byEmbarcadero Technologies
 
PPTX
How to Build a Successful Data Team - Florian Douetteau (@Dataiku)
byDataiku
 
Moving Past Infrastructure Limitations
byCaserta
 
General Data Protection Regulation - BDW Meetup, October 11th, 2017
byCaserta
 
SKOS as the focal point of linked data strategies
bySemantic Web Company
 
7 Big Data Challenges and How to Overcome Them
byQubole
 
Designing a Successful Governed Citizen Data Science Strategy
byDATAVERSITY
 
Creating a DevOps Practice for Analytics -- Strata Data, September 28, 2017
byCaserta
 
Big and fast data strategy 2017 jr
byJonathan Raspaud
 
Building New Data Ecosystem for Customer Analytics, Strata + Hadoop World, 2016
byCaserta
 
The Rise of the CDO in Today's Enterprise
byCaserta
 
Using Machine Learning & Spark to Power Data-Driven Marketing
byCaserta
 
Predictive and Prescriptive Analytics Expert Session Webinar
byibi
 
Knowledge Graphs for Transformation: Dynamic Context for the Intelligent Ente...
byNeo4j
 
Fasten you seatbelt and listen to the Data Steward
byJean-Pierre Riehl
 
Drive your business with predictive analytics
byThe Marketing Distillery
 
Data Intelligence: How the Amalgamation of Data, Science, and Technology is C...
byCaserta
 
Your Data Nerd Friends Need You!
byDataKitchen
 
You're the New CDO, Now What?
byCaserta
 
Building a New Platform for Customer Analytics
byCaserta
 
The Key to Big Data Modeling: Collaboration
byEmbarcadero Technologies
 
How to Build a Successful Data Team - Florian Douetteau (@Dataiku)
byDataiku
 

Similar to Scaling Your Data: Data Democratisation and DataOps

PDF
BD_Architecture and Charateristics.pptx.pdf
byeramfatima43
 
PPTX
Connected development data
byRob Worthington
 
PDF
Data Platform in the Cloud
byAmihay Zer-Kavod
 
PPTX
Debbie Wilson: Deliver More Efficient, Joined-Up Services through Improved Ma...
byAGI Geocommunity
 
PDF
It's All About the Data - Tia Dubuisson
byCatalina Arango
 
PDF
Microservices, containers and event driven architecture - key factors in agil...
byJudy Breedlove
 
PDF
Data Services and the Modern Data Ecosystem (Middle East)
byDenodo
 
PPTX
ACDKOCHI19 - Next Generation Data Analytics Platform on AWS
byAWS User Group Kochi
 
PDF
HLayer / Cloud Native Best Practices
byAymen EL Amri
 
PDF
Designing Cloud Data Platforms 1st Edition Danil Zburivsky all chapter instan...
byanickyruning
 
PPTX
Microservices, containers and event driven architecture - key factors in agil...
byJudy Breedlove
 
PDF
Microservices, containers and event driven architecture - key factors in agil...
byJudy Breedlove
 
PDF
Introdution to Dataops and AIOps (or MLOps)
byAdrien Blind
 
PDF
An eventful tour from enterprise integration to serverless and functions
byChristian Posta
 
PPTX
Data governance datalakes_multitenancy
bySathish K S
 
PPTX
How to add security in dataops and devops
byUlf Mattsson
 
PPTX
Data & analytics challenges in a microservice architecture
byNiels Naglé
 
PPTX
Analytics with unified file and object
bySandeep Patil
 
PPTX
Big Data_Architecture.pptx
bybetalab
 
PDF
Rethinking Streaming Analytics For Scale
byHelena Edelson
 
BD_Architecture and Charateristics.pptx.pdf
byeramfatima43
 
Connected development data
byRob Worthington
 
Data Platform in the Cloud
byAmihay Zer-Kavod
 
Debbie Wilson: Deliver More Efficient, Joined-Up Services through Improved Ma...
byAGI Geocommunity
 
It's All About the Data - Tia Dubuisson
byCatalina Arango
 
Microservices, containers and event driven architecture - key factors in agil...
byJudy Breedlove
 
Data Services and the Modern Data Ecosystem (Middle East)
byDenodo
 
ACDKOCHI19 - Next Generation Data Analytics Platform on AWS
byAWS User Group Kochi
 
HLayer / Cloud Native Best Practices
byAymen EL Amri
 
Designing Cloud Data Platforms 1st Edition Danil Zburivsky all chapter instan...
byanickyruning
 
Microservices, containers and event driven architecture - key factors in agil...
byJudy Breedlove
 
Microservices, containers and event driven architecture - key factors in agil...
byJudy Breedlove
 
Introdution to Dataops and AIOps (or MLOps)
byAdrien Blind
 
An eventful tour from enterprise integration to serverless and functions
byChristian Posta
 
Data governance datalakes_multitenancy
bySathish K S
 
How to add security in dataops and devops
byUlf Mattsson
 
Data & analytics challenges in a microservice architecture
byNiels Naglé
 
Analytics with unified file and object
bySandeep Patil
 
Big Data_Architecture.pptx
bybetalab
 
Rethinking Streaming Analytics For Scale
byHelena Edelson
 

Recently uploaded

PPTX
Social Media Audit of Starbucks Coffee Company.pptx
bydiviap21
 
PDF
LECTURE - Overcoming the AI Failure Rate - AIG, DQM.pdf
bySami Laine
 
PPTX
Lecture 14- Introduction to Hypothesis Testing.pptx
bygbadamosiayomide
 
PPT
data collection tool ppt.ppttttdataadfttg
byMuhammadodil
 
PPTX
250106204341-c238c4fadonebeforelastyear.pptx
byAbduRouf3
 
PPTX
Query Parsing: The SEO Gatekeeper That Decides Rankings
bySemantic SEO BD
 
PPTX
The Singularity Sprint (2026): Neuralink + the AI Arms Race — Ridwan Sassman
byridwansassman
 
PDF
US Digital Fan Engagement Index 2026: Benchmarking Online Demand for Sports T...
byHyperset Group Ltd
 
PPTX
End to End Supply Chain Management- Analytics Case Study.
byNandhuNandha1
 
PPTX
GENPHYSICS-2.pptx this about general physics. Grade 11 lesson
byGwynMorata
 
PDF
Trustworthy AI : Governance of AI through Ethics
byKarim Baïna
 
PDF
Khan Traders Fish Meal – MSDS Material Safety Data Sheet
byKhan Traders Karachi
 
PPTX
I need all your support to get the data for the sections like photo gallery o...
byhsharsha1
 
PDF
PathRAG rag pipeline PathRAG rag pipeline .pdf
byhotslix023
 
PDF
Analysis of Union Budget 2026-27: Major Announcements and their Impact
byDeeshiPavecha
 
PDF
Columbus Data and Analytics Wednesday - People Analytics 101: Making Sense of...
byTim Wilson
 
PDF
Understanding Data Analytics: Concepts, Types, and Use Cases
byThe IoT Academy
 
PDF
Step by Step Guide to Buying a Old or Aged Verified Paxful Accounts in US.pdf
byTopSelleriTDotCom
 
PPTX
Organizational Structure and Design Topic 4
bydavenieralayaay
 
PDF
chapter one introduction to Probability theory.pdf
bySeid42
 
Social Media Audit of Starbucks Coffee Company.pptx
bydiviap21
 
LECTURE - Overcoming the AI Failure Rate - AIG, DQM.pdf
bySami Laine
 
Lecture 14- Introduction to Hypothesis Testing.pptx
bygbadamosiayomide
 
data collection tool ppt.ppttttdataadfttg
byMuhammadodil
 
250106204341-c238c4fadonebeforelastyear.pptx
byAbduRouf3
 
Query Parsing: The SEO Gatekeeper That Decides Rankings
bySemantic SEO BD
 
The Singularity Sprint (2026): Neuralink + the AI Arms Race — Ridwan Sassman
byridwansassman
 
US Digital Fan Engagement Index 2026: Benchmarking Online Demand for Sports T...
byHyperset Group Ltd
 
End to End Supply Chain Management- Analytics Case Study.
byNandhuNandha1
 
GENPHYSICS-2.pptx this about general physics. Grade 11 lesson
byGwynMorata
 
Trustworthy AI : Governance of AI through Ethics
byKarim Baïna
 
Khan Traders Fish Meal – MSDS Material Safety Data Sheet
byKhan Traders Karachi
 
I need all your support to get the data for the sections like photo gallery o...
byhsharsha1
 
PathRAG rag pipeline PathRAG rag pipeline .pdf
byhotslix023
 
Analysis of Union Budget 2026-27: Major Announcements and their Impact
byDeeshiPavecha
 
Columbus Data and Analytics Wednesday - People Analytics 101: Making Sense of...
byTim Wilson
 
Understanding Data Analytics: Concepts, Types, and Use Cases
byThe IoT Academy
 
Step by Step Guide to Buying a Old or Aged Verified Paxful Accounts in US.pdf
byTopSelleriTDotCom
 
Organizational Structure and Design Topic 4
bydavenieralayaay
 
chapter one introduction to Probability theory.pdf
bySeid42
 

Scaling Your Data: Data Democratisation and DataOps

  • 1.
    Scaling Your Data: DataDemocratisation and DataOps Juan Urrego March 2020
  • 2.
    Juan Urrego 2 DataOps TeamLead YOU-Anne JOO-Anne ONE Who-Anne Colombia Columbia
  • 3.
    3 Who is Travelstart? SOUTHAFRICA UAE TAIPEI EGYPT NIGERIA TANZANIA KENYA MIDDLE EAST ASIAAFRICAEUROPE ATHENS PORTO Travelstart is a leading online travel agency that helps today’s business and leisure travellers search, compare and book the best flight, hotel and car options with all your favourite airline and accommodation suppliers. Based in Cape Town, Travelstart proudly employs a team of more than 850 staff from all walks of life who are passionate about one goal - making travel simple.
  • 4.
  • 5.
    At Google, forexample, nearly 80% of employees use Dremel (the internal counterpart to Google BigQuery) every month...everyone touches data on a regular basis to inform their decisions” Valliappa Lakshmanan (2018) 5 What Are We Doing? Context “
  • 6.
    6 What Are WeDoing? Context https://www.statista.com/statistics/219333/number-of-google-employees-by-department/
  • 7.
    7 What Are WeDoing? Context https://www.statista.com/statistics/219333/number-of-google-employees-by-department/ ~38% ~62%
  • 8.
    Ask someone aquestion and you are likely to receive a link to a BigQuery view or query rather than the actual answer” Valliappa Lakshmanan (2018) 8 What Are We Doing? Context “
  • 9.
    9 How Did ItStart? Context 2014 Inception Classic BI architecture (pull-based) Design to supply a short-term need
  • 10.
    10 How Did ItStart? Context 2014 2018 Inception Classic BI architecture (pull-based) Design to supply a short-term need Non-performant No test environments No version control Users don’t trust the data Shit Hits the Fan
  • 11.
    11 How Did ItStart? Context 2014 2018 2019 Inception Classic BI architecture (pull-based) Design to supply a short-term need Non-performant No test environments No version control Users don’t trust the data Shit Hits the Fan DataOps Approach Event-Driven Best practices of Software Engineering Remove the BI bottleneck DIY -> Data Democratisation
  • 12.
  • 13.
    First Generation Proprietary enterprisedata warehouse and business intelligence platforms 13 Data Process Generations DataOps https://martinfowler.com/articles/data-monolith-to-mesh.html Second Generation Big Data ecosystem with a data lake as a silver bullet Third Generation Event-Driven, Near Real Time and Cloud Based
  • 14.
    ● Data Mesh:Embracing ubiquitous data ○ Distributed Domain Driven Architecture (DDD), Self-serve Platform Design, and Product Thinking with Data ○ Data is discoverable, addressable, trustworthy & truthful, self-describing semantics & syntax, inter-operable & governed by global standards, domain data cross-functional teams 14 Data Process Generations DataOps https://martinfowler.com/articles/data-monolith-to-mesh.html
  • 15.
    DataOps is acollaborative data management practice focused on improving the communication, integration and automation of data flows between data managers and data consumers across an organization. The goal of DataOps is to deliver value faster by creating predictable delivery and change management of data, data models and related artifacts” Gartner Glossary 15 What is DataOps? DataOps https://www.gartner.com/en/information-technology/glossary/dataops “
  • 16.
    DataOps uses technologyto automate the design, deployment and management of data delivery with appropriate levels of governance, and it uses metadata to improve the usability and value of data in a dynamic environment.” Gartner Glossary 16 What is DataOps? DataOps https://www.gartner.com/en/information-technology/glossary/dataops “
  • 17.
  • 18.
    18 What is DataOps? DataOps https://www.saagie.com/blog/dataops-devops-2-0/ ●A few practices are common to DevOps and DataOps: ○ Automation (CI/CD) ○ Unit tests (code coverage is relevant) ○ Environments management ○ Versions management ○ Monitoring
  • 19.
    19 DataOps at Travelstart DataOps DoIt Yourself (DIY) Data democratisation Data-Driven Decision Making Don’t Repeat Yourself (DRY) Values
  • 20.
    20 DataOps at Travelstart DataOps DoIt Yourself (DIY) Data democratisation Data-Driven Decision Making Don’t Repeat Yourself (DRY) Data Quality Availability Usability Reliability Performance Flexibility Quality Attributes Values
  • 21.
    21 DataOps at Travelstart DataOps DoIt Yourself (DIY) Data democratisation Data-Driven Decision Making Don’t Repeat Yourself (DRY) Data Quality Availability Usability Reliability Performance Flexibility Quality Attributes Data Lakes IaC & CI/CD Data Dictionary Data Catalog Values Descriptive Predictive Prescriptive Framework Data Pipelines Data Warehouse
  • 22.
  • 23.
    23 Reference Model The Architecture Lambda~ Kappa Architecture Event Driven Architecture + Inmutable master data Precompute Views New Data stream Process Stream Increment Views View 1 View 2 View N View 1 View 2 View N Query Batch recompute Real-Time Increment Batch views Real Time views Batch Layer Serving Layer Speed Layer Main Architectural Pattern: Publisher-Subscriber
  • 24.
    Distribution Tier Clean Raw Serving TierTransformTierSource Tier External Internal 24 Reference Architecture The Architecture Aggregation Tier Ingestion Integration Quality Raw Data OLAP Analytics Monitoring & Automation Tier Data Lakes Dead Letters Backups Data Marts Ubiquitous Data Real Time Views Alerting Performance Monitoring Dashboarding CI/CD
  • 25.
    Monitoring & AutomationTier Alerting Performance Monitoring Dashboarding CI/CD Distribution Tier Clean Raw Serving TierTransform TierSource Tier External Internal 25 Reference Architecture The Architecture Aggregation Tier Ingestion Integration Quality Raw Data OLAP Analytics Data Lakes Dead Letters Backups Data Marts Ubiquitous Data Real Time Views
  • 26.
    Source Tier ● Allthe internal and/or external service/component/instance/process that will behave as publisher/producer of events ● An event is a data record expressing an occurrence and its context. ○ Slow Changing Dimensions (SCD) ■ Type 1: Last state ■ Type 2: Complete state ■ Type 3: What changed ● According to cloudevents 1.0 it MUST have: ○ Unique ID ○ Source ○ SpecVersion ○ Type ● According to cloudevents 1.0 it is NICE to have: ○ DataContentType ○ DataSchema ○ Subject ○ Time 26 Solution Architecture Architecture https://github.com/cloudevents/spec/blob/v1.0/spec.md
  • 27.
    Source Tier ● Allthe internal and/or external service/component/instance/process that will behave as publisher/producer of events ● An event is a data record expressing an occurrence and its context. ○ Slow Changing Dimensions (SCD) ■ Type 1: Last state ■ Type 2: Complete state ■ Type 3: What changed ● According to cloudevents 1.0 it MUST have: ○ Unique ID ○ Source ○ SpecVersion ○ Type ● According to cloudevents 1.0 it is NICE to have: ○ DataContentType ○ DataSchema ○ Subject ○ Time 27 Solution Architecture Architecture https://github.com/cloudevents/spec/blob/v1.0/spec.md
  • 28.
    Source Tier ● Allthe internal and/or external service/component/instance/process that will behave as publisher/producer of events ● An event is a data record expressing an occurrence and its context. ○ Slow Changing Dimensions (SCD) ■ Type 1: Last state ■ Type 2: Complete state ■ Type 3: What changed ● According to cloudevents 1.0 it MUST have: ○ Unique ID ○ Source ○ SpecVersion ○ Type ● According to cloudevents 1.0 it is NICE to have: ○ DataContentType ○ DataSchema ○ Subject ○ Time 28 Solution Architecture Architecture https://github.com/cloudevents/spec/blob/v1.0/spec.md
  • 29.
    Source Tier ● Allthe internal and/or external service/component/instance/process that will behave as publisher/producer of events ● An event is a data record expressing an occurrence and its context. ○ Slow Changing Dimensions (SCD) ■ Type 1: Last state ■ Type 2: Complete state ■ Type 3: What changed ● According to cloudevents 1.0 it MUST have: ○ Unique ID ○ Source ○ SpecVersion ○ Type ● According to cloudevents 1.0 it is NICE to have: ○ DataContentType ○ DataSchema ○ Subject ○ Time 29 Solution Architecture Architecture https://github.com/cloudevents/spec/blob/v1.0/spec.md
  • 30.
    30 Solution Architecture Architecture Source Tier EventGranularity Coarse-grained Fined-grained Aggregation Required
  • 31.
    31 Solution Architecture Architecture Source Tier EventGranularity Extreme Aggregation Required Coarse-grained Fined-grained
  • 32.
    32 Solution Architecture Architecture Source Tier EventGranularity Extreme Field or sub-property Aggregation Required Phone Email Coarse-grained Fined-grained
  • 33.
    33 Solution Architecture Architecture Source Tier EventGranularity Extreme Domain Event Field or sub-property Aggregation Required Phone Email SearchEvent Composition vs Aggregation Coarse-grained Fined-grained
  • 34.
    34 Solution Architecture Architecture Source Tier Format,Compression and Codecs Plain Text Compression: GZIP, BZIP2, LZ4 Binary Codec: Deflate, Snappy, LZO
  • 35.
    35 Solution Architecture Architecture Source Tier TravelstartEvent Template Payload { “uuid”: “34c86044-4b73-46ec-b766”, “timestamp”: 1581749960349, …. } ● Main Format: JSON ● Compression: GZIP JSON Minify + GZIP -> ~96% size reduction Header Type: search_event Format: json System: backend UUID: 34c86044-4b73-46ec-b766 Ip: 10.10.0.1 InstanceId: backend-01 Hostname: mybackend.travelstart Compression: gzip Timestamp: 1581749960349
  • 36.
    Distribution Tier Clean Raw Serving TierTransformTierSource Tier External Internal 36 Reference Architecture The Architecture Aggregation Tier Ingestion Integration Quality Raw Data OLAP Analytics Data Lakes Dead Letters Backups Data Marts Ubiquitous Data Real Time Views Monitoring & Automation Tier Alerting Performance Monitoring Dashboarding CI/CD
  • 37.
    Aggregation Tier ● Allthe services and components that aggregate Atomic Events into a Domain Event ● This aggregation can happen: ○ Inside the source ■ On Memory ■ Using Storage ○ Outside the source ■ Middleware dedicated to do this ■ Ingestion or Integration pipelines ● External sources use dedicated endpoints: Receive data from external sources, setup the canonical structure and send to the topic ○ Serverless functions ○ Stateless containers ● Schedulers can serve as trigger to start the ETL 37 Solution Architecture Architecture
  • 38.
    Aggregation Tier ● Allthe services and components that aggregate Atomic Events into a Domain Event ● This aggregation can happen: ○ Inside the source ■ On Memory ■ Using Storage ○ Outside the source ■ Middleware dedicated to do this ■ Ingestion or Integration pipelines ● External sources use dedicated endpoints: Receive data from external sources, setup the canonical structure and send to the topic ○ Serverless functions ○ Stateless containers ● Schedulers can serve as trigger to start the ETL 38 Solution Architecture Architecture
  • 39.
    Aggregation Tier ● Allthe services and components that aggregate Atomic Events into a Domain Event ● This aggregation can happen: ○ Inside the source ■ On Memory ■ Using Storage ○ Outside the source ■ Middleware dedicated to do this ■ Ingestion or Integration pipelines ● External sources use dedicated endpoints: Receive data from external sources, setup the canonical structure and send to the topic ○ Serverless functions ○ Stateless containers ● Schedulers can serve as trigger to start the ETL 39 Solution Architecture Architecture
  • 40.
    Aggregation Tier ● Allthe services and components that aggregate Atomic Events into a Domain Event ● This aggregation can happen: ○ Inside the source ■ On Memory ■ Using Storage ○ Outside the source ■ Middleware dedicated to do this ■ Ingestion or Integration pipelines ● External sources use dedicated endpoints: Receive data from external sources, setup the canonical structure and send to the topic ○ Serverless functions ○ Stateless containers ● Schedulers can serve as trigger to start the ETL 40 Solution Architecture Architecture
  • 41.
    41 Solution Architecture Architecture Aggregation Tier Areasof aggregation Source System Producer 2 Read Replica(s) or API Aggregation System Distribution & Transformation DWH/DLRead Replica(s) Orchestrator Orchestrator Producer n Event Delivery Producer 1 Map & Aggr Event Delivery Windowing [session] Map & Aggregator Event Producer Persistence Complexity
  • 42.
  • 43.
  • 44.
    Distribution Tier Clean Raw Serving TierTransformTierSource Tier External Internal 44 Reference Architecture The Architecture Aggregation Tier Ingestion Integration Quality Raw Data OLAP Analytics Data Lakes Dead Letters Backups Data Marts Ubiquitous Data Real Time Views Monitoring & Automation Tier Alerting Performance Monitoring Dashboarding CI/CD
  • 45.
    Distribution Tier ● Allthe Topics/Queues in charge of receive raw and clean to be distributed to multiple subscribers ● Topic per event type ● Subscription per subscriber ● Communication between producer and topics could failed, so keep in mind: ○ Retries ○ Graveyard (dead letters) 45 Solution Architecture Architecture
  • 46.
    Distribution Tier ● Allthe Topics/Queues in charge of receive raw and clean to be distributed to multiple subscribers ● Topic per event type ● Subscription per subscriber ● Communication between producer and topics could failed, so keep in mind: ○ Retries ○ Graveyard (dead letters) 46 Solution Architecture Architecture
  • 47.
    Distribution Tier ● Allthe Topics/Queues in charge of receive raw and clean to be distributed to multiple subscribers ● Topic per event type ● Subscription per subscriber ● Communication between producer and topics could failed, so keep in mind: ○ Retries ○ Graveyard (dead letters) 47 Solution Architecture Architecture
  • 48.
    Distribution Tier ● Allthe Topics/Queues in charge of receive raw and clean to be distributed to multiple subscribers ● Topic per event type ● Subscription per subscriber ● Communication between producer and topics could failed, so keep in mind: ○ Retries ○ Graveyard (dead letters) 48 Solution Architecture Architecture
  • 49.
    Distribution Tier Topics &Subscriptions 49 Solution Architecture Architecture
  • 50.
    Distribution Tier Topics &Subscriptions 50 Solution Architecture Architecture
  • 51.
    Distribution Tier Clean Raw Serving TierTransformTierSource Tier External Internal 51 Reference Architecture The Architecture Aggregation Tier Ingestion Integration Quality Raw Data OLAP Analytics Data Lakes Dead Letters Backups Data Marts Ubiquitous Data Real Time Views Monitoring & Automation Tier Alerting Performance Monitoring Dashboarding CI/CD
  • 52.
    Transformation Tier ● Stream/BatchData Pipelines in charge of receiving domain events, processing and persisting them ● A Pipeline can be split by domain and/or resource consumption ● 3 different categories: ○ Ingestion ○ Integration ○ Quality ■ Data Lineage and Data Provenance ● Serverless functions can be used here, but be careful with them! 52 Solution Architecture Architecture
  • 53.
    Transformation Tier ● Stream/BatchData Pipelines in charge of receiving domain events, processing and persisting them ● A Pipeline can be split by domain and/or resource consumption ● 3 different categories: ○ Ingestion ○ Integration ○ Quality ■ Data Lineage and Data Provenance ● Serverless functions can be used here, but be careful with them! 53 Solution Architecture Architecture
  • 54.
    Transformation Tier ● Stream/BatchData Pipelines in charge of receiving domain events, processing and persisting them ● A Pipeline can be split by domain and/or resource consumption ● 3 different categories: ○ Ingestion ○ Integration ○ Quality ■ Data Lineage and Data Provenance ● Serverless functions can be used here, but be careful with them! 54 Solution Architecture Architecture
  • 55.
    Transformation Tier ● Stream/BatchData Pipelines in charge of receiving domain events, processing and persisting them ● A Pipeline can be split by domain and/or resource consumption ● 3 different categories: ○ Ingestion ○ Integration ○ Quality ■ Data Lineage and Data Provenance ● Serverless functions can be used here, but be careful with them! 55 Solution Architecture Architecture
  • 56.
    56 Solution Architecture Architecture Transformation Tier IngestionPipelines In charge of receiving the raw events, process, clean, enrich and validate quality of them - Uncompress - Standardize - Cleansing - Enrichment - Schematization - Quality Checks - Alerting Directed Acyclic Graph (DAG) Canonize Events Process Quality Assurance Persist Raw (Data Lake) Persistence Publish Clean Event RAW CLEAN
  • 57.
  • 58.
  • 59.
  • 60.
  • 61.
    61 Solution Architecture Architecture Transformation Tier IntegrationPipelines In charge of integrating multiple sources to create consolidated data martsDirected Acyclic Graph (DAG) Identify Integration Type Read Event Sources Join Sources Persistence RAW/CLEAN
  • 62.
    62 Solution Architecture Architecture Transformation Tier QualityPipelines In charge of reading deadletters, run reports to analyse quality issues and bring back to life (if possible) dead/lost events (necromancer) Directed Acyclic Graph (DAG) Identify Quality Check Read Event Sources Run Quality Check Report CLEAN
  • 63.
  • 64.
  • 65.
  • 66.
  • 67.
    67 Solution Architecture Architecture Transformation Tier QualityPipelines com.travelstart.bi.etl.obqm.exceptions.NonNullableFieldException: The field 'bookingUuid' can't be null | at com.travelstart.bi.etl.pipelines.booking.transforms.TsMessageToBookingFn.processElement(TsMe ssageToBookingFn.java:152) : 821270 com.travelstart.bi.etl.obqm.exceptions.NonNullableFieldException: The field 'reference' can't be null | at com.travelstart.bi.etl.pipelines.booking.transforms.TsMessageToBookingFn.processElement(TsMe ssageToBookingFn.java:152) : 7336 com.travelstart.bi.etl.obqm.exceptions.NonNullableFieldException: The field 'bookingId' can't be null | at com.travelstart.bi.etl.pipelines.booking.transforms.TsMessageToBookingFn.processElement(TsMe ssageToBookingFn.java:152) : 2482
  • 68.
  • 69.
  • 70.
  • 71.
  • 72.
  • 73.
  • 74.
    Distribution Tier Clean Raw Serving TierTransformTierSource Tier External Internal 74 Reference Architecture The Architecture Aggregation Tier Ingestion Integration Quality Raw Data OLAP Analytics Data Lakes Dead Letters Backups Data Marts Ubiquitous Data Real Time Views Monitoring & Automation Tier Alerting Performance Monitoring Dashboarding CI/CD
  • 75.
    Serving Tier ● Storeall the raw and clean data to be accessible for our stakeholders ● About raw data: ○ Data Lakes ○ Dead Letters ○ Backups ● About OLAP: ○ Data Marts ○ Ubiquitous Data/ Data Sourcing ○ Real Time Views ● About Analytics: ○ Whatever it makes you happy (of course with data governance) 75 Solution Architecture Architecture
  • 76.
    Serving Tier ● Storeall the raw and clean data to be accessible for our stakeholders ● About raw data: ○ Data Lakes ○ Dead Letters ○ Backups ● About OLAP: ○ Data Marts ○ Ubiquitous Data/ Data Sourcing ○ Real Time Views ● About Analytics: ○ Whatever it makes you happy (of course with data governance) 76 Solution Architecture Architecture
  • 77.
    Serving Tier ● Storeall the raw and clean data to be accessible for our stakeholders ● About raw data: ○ Data Lakes ○ Dead Letters ○ Backups ● About OLAP: ○ Data Marts ○ Ubiquitous Data/ Data Sourcing ○ Real Time Views ● About Analytics: ○ Whatever it makes you happy (of course with data governance) 77 Solution Architecture Architecture
  • 78.
    Serving Tier ● Storeall the raw and clean data to be accessible for our stakeholders ● About raw data: ○ Data Lakes ○ Dead Letters ○ Backups ● About OLAP: ○ Data Marts ○ Ubiquitous Data/ Data Sourcing ○ Real Time Views ● About Analytics: ○ Whatever it makes you happy (of course with data governance) 78 Solution Architecture Architecture
  • 79.
    79 Solution Architecture Architecture Serving Tier DataLakes, Dead Letters & Backups Plain Text / Schema-less Compression: GZIP, BZIP2, LZ4 Binary / Schema Codec: Deflate, Snappy, LZO, LZIP, GZIP
  • 80.
  • 81.
    81 Solution Architecture Architecture Serving Tier DataLakes, Dead Letters & Backups Standard/ Hotline Nearline Coldline Archive 180 days Min Duration: 90 days Min Duration: 365 days365 days Min Duration: 30 days 43% cost reduction 69% cost reduction 89% cost reduction Deletion Cost
  • 82.
    82 Solution Architecture Architecture Serving Tier DataLakes vs Data Warehouses Data Lake Data Warehouse Raw data It’s dirty Transactional structure not analytical Clean/Skew data Quality processes applied Schema defined for analytical purposes
  • 83.
    83 Solution Architecture Architecture Serving Tier DataMarts, Event Sourcing and Real Time Views HyperCube Pivot Drilldown Roll Up Dicing Slicing
  • 84.
    84 Solution Architecture Architecture Serving Tier DataMarts, Event Sourcing and Real Time Views More than 300 columns Includes facts and aggregations Following mostly a star modeling and denormalisation
  • 85.
    85 Solution Architecture Architecture Serving Tier DataMarts, Event Sourcing and Real Time Views Views Materialise d Views
  • 86.
    86 Solution Architecture Architecture Serving Tier DataMarts, Event Sourcing and Real Time Views
  • 87.
    Distribution Tier Clean Raw Serving TierTransformTierSource Tier External Internal 87 Reference Architecture The Architecture Aggregation Tier Ingestion Integration Quality Raw Data OLAP Analytics Data Lakes Dead Letters Backups Data Marts Ubiquitous Data Real Time Views Monitoring & Automation Tier Alerting Performance Monitoring Dashboarding CI/CD
  • 88.
    Monitoring & AutomationTier ● The command and control area of the whole architecture ● It needs to monitor everything related with data ingestion, processing and infrastructure ● Alerting: ○ Policies ○ Multiple notification channels ● Performance and Stability Monitoring ○ Bottlenecks, Bugs, etc ○ Slow steps, underutilized infrastructure, memory issues, etc ● CI/CD + IaC ○ Development and Deployment lifecycle completely automated ○ Any architectural change should be track through a version control ● Dashboards! 88 Solution Architecture Architecture
  • 89.
  • 90.
  • 91.
  • 92.
  • 93.
  • 94.
    94 Solution Architecture Architecture Monitoring &Automation Tier Alerting Messages delivered rate Messages in queue Too Many Resources Error Rate
  • 95.
  • 96.
  • 97.
  • 98.
    98 Solution Architecture Architecture Monitoring &Automation Tier CI/CD & IaC Jenkins (Pipelines) Cloud Deployment Manager Cloud Build GitHub Actions
  • 99.
  • 100.
  • 101.
    101 Solution Architecture Architecture Monitoring &Automation Tier CI/CD & IaC Spotless Code Analysis PR Analysis
  • 102.
  • 103.
  • 104.
    104 Solution Architecture Architecture Monitoring &Automation Tier CI/CD & IaC info: title: Buckets Template description: Creates buckets in GCS with the correct life cycle rules and security constraints version: 1.0 properties: name: type: string description: Name property of the bucket standard `ts-<env>-dataops-<name>` pattern: ^[a-z0-9]+$ labels: type: object description: User-provided bucket labels, in key/value pairs. YAML Schema
  • 105.
    105 Solution Architecture Architecture Monitoring &Automation Tier CI/CD & IaC info: title: Buckets Template description: Creates buckets in GCS with the correct life cycle rules and security constraints version: 1.0 properties: name: type: string description: Name property of the bucket standard `ts-<env>-dataops-<name>` pattern: ^[a-z0-9]+$ labels: type: object description: User-provided bucket labels, in key/value pairs. YAML Schema
  • 106.
    106 Solution Architecture Architecture Monitoring &Automation Tier CI/CD & IaC Jinja Template
  • 107.
    107 Solution Architecture Architecture Monitoring &Automation Tier CI/CD & IaC Jinja/YAML Template - name: ts-dataops-batch type: buckets.jinja properties: name: batch environment: {{ properties['environment'] }} labels: description: temporary_files_to_insert data_type: raw deleteRule: age: 7 isLive: true
  • 108.
  • 109.