The document discusses the integration of service mesh into the modern data stack (MDS), highlighting the evolution towards cloud-based architectures that combine data lakes and data warehouses. It outlines the characteristics of MDS, including modular design and automation, while emphasizing the importance of service mesh for monitoring, security, and operational efficiency. The document also presents specific tools and components, such as Airbyte, dbt, and Apache Airflow, within the context of implementing a data mesh framework.

1. Fabian Hardt
SEPTEMBER 2022
HOW SERVICE MESH
FITS INTO THE
MODERN DATA
STACK

2. AGENDA
MOTIVATION
01
WHAT IS THE MDS
02
SUMMARY
04
ARCHITECTURE
03

3. MOTIVATION
01

4. Data Lake and DWH are combined
In the same way, BI and AI are growing together, see e.g.
Snowflake and Databricks. Counter-movement to
increasing specialization. In general: it moves a lot.
WHAT WE ARE CURRENTLY SEEING...
A lot of money in the market
With it increasing fragmentation of functionalities; each
new startup takes care of a special function. This
distribution of individual functionalities in separate tools.
Cloud is the new standard
Hardly anyone still builds analytical architectures on-
premises. But there are exceptions. The Hadoop
ecosystem is becoming less important; Data lakes
increasingly on object storage in the public cloud.
Use of software development best
practices
Also as a result of migration to the cloud. So infrastructure
as code, automation, CI/CD. Increasing sympathy for code-
first and open source, return of frameworks, DIY, SQL only
(skills!).

5. WHAT IS THE MODERN DATA STACK
02

6. CORE CHARACTERISTICS OF THE MODERN DATA STACK (MDS)
Automation and
operationalization
Basic paradigms of modern software
development are being introduced, including
GitOps, CI/CD, containers and automated
testing.
Best of Breed and Modular
A Modern Data Stack has a modular
structure. Individual components can be
exchanged. EL+T are separated. The best tool
is selected for each discipline.
Cloud DWH
Central data storage component of the
Modern Data Stack. Combines advantages
of data lake and data warehouse.
SaaS / IaC
Focus on maintainability and low time to
market. This can be achieved using SaaS
services from cloud providers or automation
using IaC.

7. WHY TO USE MODERN DATA STACK?
¢ Data Mesh
¢ Total hype at the moment
¢ Organizational Framework for Data Driven Companies
¢ Data Products with APIs - similar to microservices
¢ Domain Driven Design from software development as a basis
¢ Clear responsibilities for Data Products
¢ More flexibility for developers in tool selection
¢ Modern Data Stack as a technical framework to implement data mesh (organizational)
¢ Flexible architecture to support “free choice of weapons” – Modern Data Platform
¢ APIs for intern and extern purposes
¢ Focus: Shorter “Time to Market”

8. DATA MESH – DATA PRODUCTS
¢ In direct connection with microservices from the classic SD environment
¢ Operational applications vs. analytical applications

9. USAGE OF DATA PRODUCTS
Data API

10. OUR SELECTION: COMPONENTS

11. AIRBYTE
¢ Data Ingest
¢ Many standard connectors available
¢ Saas, Cloud, APIs, Databases,…
¢ Facebook, Google, Salesforce, Redshift, Snowflake, BigQuery, …
¢ Own connectors with Python Connector Development Kit
¢ Simple transformations possible
¢ SaaS (just in US) und Open Source for own installations
¢ Container based operation
¢ Separation of platform/connectors (server, UI, scheduler,
...)
¢ New container for each connector
¢ Possible alternatives: Stitch, Fivetran, Singer, Meltano, …

12. DBT
¢ Data Transformation („Data Build Tool“)
¢ Just „T“ in EL+T – Extraction separately
¢ ELT approach, so-called models are compiled for the target platform
(e.g. Cloud DWH, Snowflake) and executed there
¢ Code-first, SQL with Jinja (Templating)
¢ There is a growing community, extensions can be downloaded
¢ SaaS and Open Source (Python)
¢ DEV environment cloud.getdbt.com
¢ Any editor can be used locally, CLI available (dbt-core)
¢ Deployment
¢ VM, Docker container, can be integrated almost anywhere
¢ Possible alternatives: Azure Data Factory, Talend, Informatica, …

13. APACHE AIRFLOW
¢ Workflow management system
¢ Originally developed by Airbnb
¢ Running a DAG (Directed Acyclic Graph)
¢ Nodes contain operators, can execute code, but also control other
tools
¢ Popular for building/running data pipelines
¢ Best suited for GitOps / integration into pipelines
¢ Managed variants available and open source
¢ Astronomer, Managed Airflow bei AWS, Google
¢ Consists of: Scheduler, Worker, UI, DB, Flower (Celery, Redis)
¢ Parallel processing on several workers possible
¢ Scales thanks to container technology
¢ Possible alternatives: Dagster, Luigi, Prefect, …

14. ARCHITECTURE
03

15. MDS ARCHITECTURE WITHOUT SERVICE MESH

16. TYPICAL SERVICE MESH WITHOUT MDS

17. AND BOTH TOGEHTER

18. KUMA IN ACTION
¢ All internal MDS services get
sidecars
¢ Central overview over
services of all domains
¢ Status of services
¢ Metrics of services
¢ Traffic between components
can be controlled

19. METRICS & TRACING
Metrics & Tracing of internal MDS components:

20. WHAT PROBLEMS DOES A SERVICE MESH SOLVE IN MDS
¢ Centralized Service Mesh implementaion
¢ Centralized overview over all services
¢ Internal – MDS
¢ External – Data APIs
¢ Centralized monitoring over all services
¢ Monitoring
¢ Logging
¢ Tracing
¢ Decrease time to market
¢ Developers don't have to worry about recurring problems
¢ Security – TLS
¢ Authentication & Authorization
¢ Support for exporting data & APIs

21. SUMMARY
04

22. ¢ Modern Data Stack as a distributed data platform
¢ One possible architecture to build support Data Mesh
implementation
¢ Service Mesh helps to
¢ Secure…
¢ Monitor…
¢ Trace…
¢ …this Modern Data Stack Architecture
¢ But: The complexity of the system is additionally increased
¢ The team must have a deep understanding of service mesh
SUMMARY

23. Q & A

24. Fabian Hardt
CONTACT
Solution Architect
Fabian.hardt@opitz-consulting.com
https://twitter.com/fabian_hardt
www.linkedin.com/in/fabian-hardt