The document discusses Confluent Stream Governance, a solution for governing data in motion with metadata. It introduces tools for managing schemas, classifying metadata, tracking lineage, and monitoring data quality. This helps bring order to what would otherwise be a "giant mess" of ungoverned data by enforcing standards and providing visibility into data flows and definitions.

1. Navigating your data in motion
David Araujo
Data Governance Product Manager
Michael Agnich
Sr. Director, Product Management

2. The Giant Mess
LINE OF BUSINESS 02
LINE OF BUSINESS 01 PUBLIC CLOUD

3. Solution: The Data in Motion Platform

4. Viral Growth of Data in Motion

5. Data
Metadata
● Structure
● Context
● Understanding
● Provenance
● Quality
● Ownership
● Usage
● ...

7. Introducing Confluent Stream Governance
Data
in motion
Metadata
in motion

8. Stream Governance
Now
Schema Registry
Schemas management UI (New)
Schema linking (New)
Next
Schema Registry PrivateLink access
Stream Validation - business quality rules
Stream Profiling - quality monitoring
Stream Quality
Metadata definition and
enforcement
Now
Auto technical metadata collection
Schemas classification with tagging
Stream Catalog UI & API
Next
Stream Catalog PrivateLink access
Key-value pairs metadata enrichment
Integration with 3rd party data catalogs
Stream Catalog
Metadata search and
discovery
Now
Automated lineage tracking
Metadata inspection
Live metrics
Next
Point in time lineage visualization
Cluster linking lineage visualization
Stream Lineage API
Stream Lineage
Metadata tracking and
analysis

9. Stream
Schema Registry
Stream Quality – contracts for data in motion
SCHEMA METADATA SCHEMA METADATA
Producer App 3
...
Producer App 1
Producer App 2
Consumer App 3
...
Consumer App 1
Consumer App 2
UNIVERSAL LANGUAGE

10. Stream
Catalog
Stream Catalog – knowledge base for data in motion
Tags
Key value pairs
TECHNICAL METADATA BUSINESS METADATA
INDEX
ENTITY TYPES
Owner
...
Name
Creation date
Integration with 3rd parties

11. Stream Lineage – maps for data in motion
Owner: David
Creation: 08/20/21
Retention: 30 days
Type: JSON
Tags: PII
Stream
Lineage

12. Demo
Sky In motion

14. Schema Linking
Schema Registry (Source) Schema Registry (Destination)
Staging Context
Test Context
Default Context
Destination Context
Exporter
Default Context
Schema

15. Confluent
Governance

16. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.

17. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Ungoverned Growth
Loose data definitions
No proper data
ownership
Siloed data
knowledge
No data visibility
Questionable
data quality
Inefficient data
usage
Slow expansion
and low ROI

18. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Governed Growth With Metadata
Explicit data
definitions and
enforcement
Safe data
evolvability
Data
classifications
and
organization
Democratized
data knowledge
Data flows
mapping
Self-service data
discovery and
access

19. Data
in motion
Metadata
in motion

20. Introducing Confluent Metadata in Motion

21. Connecting people & places
The Incredible History of Commercial Air Travel
Data from World Bank
Air transport, passengers carried World
(2019)
4.397
Billion

22. Before flying
Are we there yet?
A B

23. The Dawn of Commercial Flight
Air Mail Act of 1925

24. International Air Travel
You have arrived

25. Trust

26. Navigating a world in motion

27. Navigating a world in motion

28. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Slide explaining What is the problem for Kafka?

29. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Animation with Kafka mess without governance

30. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.

31. Validate
Classify
Follow
Discover
Monitor

32. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Data
in motion
Metadata
in motion

33. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.

34. Trust
Connect
data in motion
Discover
data in motion
Validate
data in motion
Classify
data in motion

35. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.