Transforming Data Processing with Kubernetes: Journey Towards a Self-Serve Data Mesh - Rakesh Subramanian Suresh & Jainik Vora, Intuit This presentation explores how Intuit uses Kubernetes with Domain-Driven Design and Data Mesh principles to transform its data processing landscape, crucial for its AI-driven expert platform. We will discuss the importance of clean data in developing robust generative artificial intelligence and how Intuit is addressing this through the creation of paved paths for data platforms running on Kubernetes. We'll examine the challenges and solutions in managing 100,000 data pipelines and 1000+ engineers interacting with data, highlighting the need for scalable solutions. We'll also discuss how Intuit uses Kubernetes to build its batch and stream processing platform, overcoming hurdles in data pipeline deployment, scheduling, orchestration, and dependency management. We'll conclude by emphasizing how this transformation, based on treating data as a product, has improved decision-making speed and accuracy across the organization and fostered a more efficient, collaborative data culture.

1. Rakesh Suresh
Jainik Vora
Transforming Data Processing with
Kubernetes: Journey Towards a
Self-Serve Data Mesh
Nov 6, 2023

2. © 2023 Intuit Inc. All rights reserved. 2
Speaker Introduction
Jainik Vora
Sr. Staff Software Engineer
Rakesh Suresh
Sr. Staff Software Engineer
jainiksvora
rakeshsuresh

3. 3
© 2023 Intuit Inc. All rights reserved.
Agenda
Intuit
About Intuit and its mission
Data Mesh
What is Data Mesh and the problems it addresses
Data Lake & Data Mesh
How Intuit implements data mesh with a real example
Intuit’s Data Mesh Concepts
Foundational concepts defined for Data Mesh
Self-Serve Data Processing on Kubernetes
Architecture of Batch and Stream Processing Platform

4. ©2022 Intuit Inc. All rights reserved. 4
100%
services on
Modern SaaS
65B
machine learning
predictions per day
24k
financial institutions
[+50 crypto]
$560B
money moved
3.6B
requests during
peak season (no
customer failures)
Data
Integration
Fintech
Infrastructure
Identity
AI
Infrastructure
Modern Dev
Experience
AI-driven expert platform
Intuit is leading the way in building an AI-native development platform using cloud native open source
technology. We’re committed to building tools that scale and giving back to the open source community.

5. ©2022 Intuit Inc. All rights reserved.
We believe in open source
and open collaboration
bit.ly/intuit-oss
Created, open-sourced,
used, and maintained
by Intuit
Recipient of the
End User Award
in 2019 & 2022
End user of Cloud
Native and mobile
open source tech

6. © 2023 Intuit Inc. All rights reserved. 6
Data Mesh

7. © 2023 Intuit Inc. All rights reserved. 7
What is Data Mesh?

8. A data mesh is a decentralized data
architecture that organizes data by a
specific business domain.

9. Instead of data acting as a
by-product of a process, it becomes
the product, where data producers
act as data product owners.

10. © 2023 Intuit Inc. All rights reserved. 10
Why Data Mesh?
Improve value
of Data
Smart Product
Experiences
using Data
Power AI
Power Generative AI Applications
like Intuit Assist
Reduce time
to discover &
access Data
Serve variety
of Data
Personas

11. © 2023 Intuit Inc. All rights reserved. 11
Data Mesh Principles
Lorem ipsum
congue
Data Mesh
Domain
Driven
Ownership
Data
Product
Data Access
Self Serve
Infrastructure

12. © 2023 Intuit Inc. All rights reserved. 12
Data Lake & Data
Mesh

13. Small Business Owner has unpaid
Invoices
The small business owner logins to quickbooks and
realizes there are unpaid invoices from customers.
Options provided by the system:
◆ (A) The system reminds the owner about unpaid invoices
◆ (B) The system also offers an add-on feature that
automatically sends invoice reminders to customers
Small Biz & Quickbooks

14. © 2023 Intuit Inc. All rights reserved. 14
Invoice Business Technical Requirement
❖ Notification: Track &
Remind Business
Owner and their
Customer
❖ Get unpaid invoices
by Business
❖ Get unpaid invoices
for each Customer
grouped by
Business

15. © 2023 Intuit Inc. All rights reserved. 15
Traditional Data Lake Architecture

16. © 2023 Intuit Inc. All rights reserved. 16
Traditional Data Lake Architecture for Invoice

17. © 2023 Intuit Inc. All rights reserved. 17
Invoice is our Data Domain

18. © 2023 Intuit Inc. All rights reserved. 18
Lets ask some questions on Invoice Data..
◆ How do I find Invoice data for my use
case?
◆ Who is the domain expert for Invoice
data?
◆ What is the schema of the Invoice data?
◆ Where is Invoice data located for
consumption?
◆ How can I get access to Invoice data
and who can approve?
◆ Is there derived data from Invoice? How
do I derive data from Invoice?

19. © 2023 Intuit Inc. All rights reserved. 19
Data Mesh
Concepts for Intuit

20. © 2023 Intuit Inc. All rights reserved. 20
Organization & Discovery of Data
How do I find Invoice data for my use case?
Data Map
Organization of data using domain, sub-domain and
bounded context

21. © 2023 Intuit Inc. All rights reserved. 21
Organization & Discovery of Data
How do I find Invoice data for my use case?
Data Map
Organization of data using domain, sub-domain and
bounded context
Data Product
Foundational unit of data map, organized by data map

22. © 2023 Intuit Inc. All rights reserved. 22
Ownership of Data
Who is the domain expert for Invoice data?
Data Product
Consolidates essential information to enable data
consumers
Data Steward
Defines the data product and responsible for its contract

23. © 2023 Intuit Inc. All rights reserved. 23
Data Contract
What is the schema of the Invoice data?
Semantic Model
Consolidates essential modeling and schema
information enabling data consumers understand the
data
SLA
Defines the data product and responsible for its contract
like data quality, data freshness etc…

24. © 2023 Intuit Inc. All rights reserved. 24
Data Ports
Where is Invoice data located for consumption?
Data Assets
Provides location and medium through which data can
be consumed
Tag
Additional context for optimal discovery

25. © 2023 Intuit Inc. All rights reserved. 25
Access & Governance
How can I get access to Invoice data and who can approve?
Access Control List
Track explicit read and write access control
Access approved by Data Steward

26. © 2023 Intuit Inc. All rights reserved. 26
Data Concepts - Data Processing & Lineage
Is there derived data from Invoice? How do I derive data from Invoice?

27. © 2023 Intuit Inc. All rights reserved. 27
Self-Serve
Data Processing on
Kubernetes

28. © 2023 Intuit Inc. All rights reserved. 28
Scope of Data Processing At Intuit
Scale
2000+ users
100,000+ pipelines (batch
and streaming)
Variety of Users
Data Engineers
Data Scientists
Machine Learning Engineers
Data Analysts
Variety of Use Cases
Type
◆ Batch
◆ Streaming
Categories
◆ Model Training & Feature Gen
◆ Derivation & Enrichments
◆ Data Movement

29. © 2023 Intuit Inc. All rights reserved. 29
Self Serve Data Processing as Paved Path
Operate & Monitor
◆ Log forwarding & Metrics
reporting
◆ Alert & Notification
◆ DR Failover & Failback
Provision & Deploy
◆ Infrastructure provisioning
◆ Deployment of processing
artifacts
◆ Data Map Registration
◆ Lineage
Author & Define
◆ Authoring tools geared towards
user persona and expertise
◆ Access to input and output
◆ Scheduling & Orchestration

30. © 2023 Intuit Inc. All rights reserved. 30
Batch Processing Platform Architecture

31. © 2023 Intuit Inc. All rights reserved. 31
Stream Processing Platform Architecture

32. © 2023 Intuit Inc. All rights reserved. 32
Kubernetes Power Data Processing
Intuit Kubernetes Service
◆ Core Infrastructure Layer
◆ Runs
– Control plane APIs
– Processing jobs
Argo Workflow & Events
◆ Scheduling & Orchestration
◆ Deployment Workflow

33. © 2023 Intuit Inc. All rights reserved. 33
Learn more
Data Mesh
◆ Data Mesh Principles & Logical Architecture by Zhamak Dehghani
◆ Intuit’s Data Mesh Strategy
◆ Intuit’s Data Mesh Concepts
Data Processing
◆ How Intuit Built Stream Processing Platform with Flink
◆ Large Scale Batch Processing with Argo Workflow and Events

34. Q&A
© 2023 Intuit Inc. All rights reserved. 34