"Idempotence is a mathematical requirement of particular operations where the operation can be applied multiple times without changing the result beyond the initial application. The main driver behind the idempotency requirement is often to handle duplicated messages. As developers and architects, we need to pay close attention to how we deal with our production data during new deployments to ensure we are not losing any data, duplicating messages, or introducing malformed data into our system. Furthermore, we need to figure out how to automate the process and add testing guarantees to prevent any potential human error. In this session, you will learn about the idempotent Kafka Producer & Consumer architecture and how to automate the CI/CD process with open-source tools."

1. CI/CD with an
Idempotent Kafka
Producer & Consumer
Eden Ohana
SW Engineer, Treeverse
eden.ohana@treeverse.io
April 2022

2. Table of Contents
Data lakes Continuous data
streaming
Git for data Kafka with lakeFS
Demo

3. Data Lakes Today - Video

4. Deployment
DATA
Data Source
Data Source
Data Consumer
Data Consumer
ETL ETL
Data Lakes Today

5. Streaming Data with Kafka Connect

6. Kafka In Production

7. Challenges with Continuous Data Streaming
● In-ability to experiment, compare and reproduce
● Troubleshoot producer issues
● Data quality tests
● Cross collection consistency, synchronize kafka topics

8. Idempotency and data lakes problems
● The idempotent producer solves the problem of duplicate messages and
provides the 'exactly-once-delivery'
● Exactly once delivery requires tight operation of multiple parts in the data
pipeline
● Data quality is at risk during CI/CD of one component of the data pipeline

9. How is it solved today?
● Duplicate data, maintain sandbox
● Real time data quality tests
● Object level versioning
● Readers Writers separation

10. Production
Deployment Development
DATA
How do you
test new /
modified
pipelines?
What
happens
when
production
data gets
corrupted?
How do you
guarantee
data quality?
Data Lakes Today

11. we would manage data
from dev to production
the way we manage code
In a perfect world

12. s3://data-repo/collections/foo
s3://data-repo/main/collections/foo
lakectl branch create
lakefs://repo@testing-spark-3
--source lakefs://repo@main
# output:
# created branch 'testing-spark-3.0',
# pointing to commit ID: 'd1e9adc71c10a’
Object Store
Manage Data like Code With lakeFS

13. Why Git for Data
Challenge Solution
$ lakectl revert main^1
$ lakectl branch create my-branch
$ spark.read.parquet
(‘s3://my-repo/<commit_id>’)
$ lakectl merge my-branch main
$ lakectl branch create new-logic
$ lakectl merge new-logic main
Quickly recover from an error
Develop in Isolation
Troubleshoot
Atomic Update
Reprocess

14. Production
Deployment Development
DATA
Safely test
your pipelines
on an
production
identical
isolated
branch
Easily revert
corrupted data.
Time travel to
debug a snapshot
Production
identical
isolated
branch and
hooks to test
your data
Data Lake With lakeFS

15. How it Works

16. lakeFS Architecture

17. Data Model
Commit
Meta range ID: ...
Master
Commit ID: ...
8-12MB .sst files,
containing continuous
sorted keys
.sst file,
pointing to
range files
Further Reading

18. How does lakeFS Work?

19. Kafka with lakeFS

20. What Does A Typical Environment Look Like?

21. In Dev
In Development
● Experimentation - try tools and code in isolation
● Debug - create an isolated snapshot of the data at the time of the failure
● Collaborate - tools, code or different versions of your data

22. In Deployment
In Deployment
● Version control - manage data interfaces with clear versions and easily point
the consumers to newly deployed data.
● Test - define pre-merge and pre-commit hooks to run tests that enforce best
practices.
prevent breaking changes from entering your production data environment

23. In Production
In Production
● Roll Back - recover from errors by instantly reverting data to a former,
consistent snapshot of the data lake.
● Troubleshoot - investigate production errors by starting with a snapshot of the
inputs to the failed process
● Cross-collection Consistency - provide consumers multiple synchronized
collections of data in one atomic, revertible action

24. Integrating Kafka with LakeFS
# Your lakeFS repository
s3.bucket.name=example-repo
# Your lakeFS S3 endpoint and credentials
store.url=https://s3.lakefs.example.com
aws.access.key.id=AKIAIOSFODNN7EXAMPLE
aws.secret.access.key=wJalrXUtnFEMIK7MDENGbPxRfiCYEXAMPLEKEY
# stream-1-branch being the branch we want to write to
topics.dir=stream-1-branch/topics
Lakectl branch revert lakefs://example-repo@stream-1-branch --commit dd8a60d5ef70809

25. Demo

26. Thanks!
Join the community eden.ohana@treeverse.io
GitHub
lakeFS.io
Check out the docs.lakefs.io
Contribute and star on Github:
Treeverse/lakeFS