The document discusses the future of Extract, Transform, Load (ETL) processes. It outlines how ETL has evolved from monthly batch processes 15 years ago to more real-time integration enabled by streaming data platforms. Key trends driving changes include cloud computing, microservices, and software engineers taking on more responsibilities. The document argues the future of ETL is to model all data as a continuous stream and leverage streaming data platforms to enable real-time integration that moves fast without breaking compatibility. This allows for fully integrated applications and databases at scale.

1. 1
The Future of ETL
(Revised)
Gwen Shapira, Principal Data Architect
@gwenshap
https://www.confluent.io/blog/the-future-of-etl-isnt-what-it-used-to-be/

3. You extrapolate the trends that you see

5. 5
So, to think about future of ETL, we want to look at…
• The future we imagined in the past
• The trends we are seeing now
• The future these trends indicate
• Practical suggestions
• Final thoughts

6. 6
Me
• Moving data around for 20 years
• Works at Confluent.
• Apache Kafka Committer
• Wrote a book or two
• Tweets a lot

7. 7
What was it like 15 years ago?
• Very few sources
• One Data Warehouse to rule them all
• One big “friendly” ETL tool
• Significant effort modeling, cleaning and
converging.
• “Although it is a huge leap to move from a
monthly or weekly process to a nightly one…”

8. 10
Main
pain-points
of data
integration?

9. 9
And 5 years ago?
• Many diverse sources.
• One big target
• Complete confusion of ETL tools
• “Schema on Read”
• … and still loading data around once a
day.

10. 10
And since change is difficult, we also got this:

11. 11
What did we dream about then?
• Real-time.
We don’t want to wait even 5 minutes for insights.
• Governance.
Save us from the mess we created.
• Better tooling.
We imagined really fancy drag-and-drop.

12. But we missed some key ways
in which the world is changing

13. #1 Cloud
13

14. #2 Microservices

15. Front End
Middleware
Backend
ETL
Databases
Ops
Sysadmins
Software
Engineers
Software
Engineers
Software
Engineers
Software
Engineers
Software
Engineers
Cloud and Microservices drove more changes

16. #3 Software Engineers do Everything
16

17. Bonus Future: Serverless

18. What shall we build now?

19. #1 Model Everything as a Stream
19

20. 20
all your data as a stream
Imagine…

21. 21

22. Old New
Messages are added at the end of the log
Kafka is simple. You start with a log.

23. Sally
is here
George
is here
Fred
is here
Old New
Scan Scan
Scan
Consumers have a position of their own

24. Old New
Read to offset & scan
Only sequential access. Order is guaranteed.

25. Partitions add scalability
Cluster
of
Machines
Producer 1 Producer 2 Producer 3
Messages are sent to
different partitions
Partitions are distributed
to different machines

26. Replication adds Fault Tolerance
Event!
Event!
Leader Follower
Machine A
Machine B

27. Replica takes over on machine failure
Replication adds Fault Tolerance

28. Producers
Kafka
Consumers
Consumer groups complete the scalability story

29. 29
A stream fixes the madness

30. #2 Move Fast, Don’t Break Compatibility
30

32. 32
Push,
Merge,
Build
Test,
Staging
Dev Prod
MVN
Plugin
Test
Registry
Prod
Registry

33. #3 Integrate Databases and Applications at Scale
33

34. Confidential 34
Do you think that’s a table
you are querying?

35. 35
The Stream/Table Duality
Account ID Balance
12345 €50
Account ID Amount
12345 + €50
12345 + €25
12345 -€60
Account ID Balance
12345 €75Account ID Balance
12345 €15
Time
Stream Table
http://docs.confluent.io/current/streams/concepts.html#duality-of-streams-and-tables

36. 36
Full streaming platform with Connect + Streams

37. 39
Promotion!
We want to send Platinum members
Who are looking at beach properties
An email about
Discount package in new Florida hotel
Lets look at an Example. Our marketing team has a small request:

38. Pageview Events
Consumer
Enrich
Decide
Produce
Promotions Events

39. 39

40. Pageview Events
Consumer
Enrich
Decide
Produce
Promotions Events
DB

41. Pageview Events
Consumer
Enrich
Decide
Produce
Promotions Events
latency, throughput,
availability
DB

42. Pageview Events
Consumer
Enrich
Decide
Produce
Promotions Events
DB Cache

43. Pageview Events
Consumer
Enrich
Decide
Produce
Promotions Events
DB
DB Change Events: Insert, update, delete
Consumer

44. Kafka Connect
+ CDC Connector
DB
KSQL> CREATE STREAM promotions AS
SELECT customer_email, customer_name
FROM pageview_events
LEFT OUTER JOIN customers
on c_id=pageview_cust_id
WHERE customer_class = ‘Prime’;

45. What’s next?

46. 46

47. • Serverless
• Observability
• Operational Patterns
• Metadata for Engineers
• Stronger guarantees
• Standardization
• Metadata-driven, ML-driven ETL
• …
• …
• Organizational Enlightenment
Almost here?
Will this ever happen?

48. Is ETL dead?
48

49. Resources and Next Steps
https://github.com/confluentinc/ksql
https://github.com/confluentinc/cp-demo
https://www.confluent.io/download/
https://slackpass.io/confluentcommunity
https://www.confluent.io/blog

50. Questions?
@gwenshap
gwen@confluent.io