Getting to grips with governance on realtime data: Data Discovery, Data Catalogues, Schema Management, Lineage

1. Governing Data in Motion

2. GOVERN PROCESS
CONNECT
STREAM
The Capabilities of a Data Streaming Platform

3. {
customerId: 3453,
flightNumber: “200”
}
Producer
App 1
App 2
App 3
Topic
SCHEMAS

4. {
customerId: “3453”,
flightNumber; “200”
}
Producer
App 1
App 2
App 3
Topic
SCHEMAS

5. {
customerId: “3453”,
flightNumber; “200”
}
Producer
Topic
Producer
Producer
Producer
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App App App
App App App
App
App
App
App
App
App
App
App
App
App
App
App
SCHEMAS
Producer

6. Producer
Producer
Producer
Producer
Producer
Producer
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App
App App App
App App App
App
App
App
App
App
App
App
App
App
App
App
App
SCHEMAS

7. 7
PRODUCER APPS
Event 1
Event 2
Event 4
Event 5
Confluent
Kafka
Schema Registry
Data Producer/Owner Data Steward
Confluent Schema Registry
Organize, Validate, Evolve
Enforce Data Contract
SERIALIZER
Event 3
CONSUMER APPS
Event 1
Event 2
Event 4
Event 5
Event 3
Enforce Data Contract
DESERIALIZER

8. {
“customerId”: 3453,
“flightNumber”: “200”
}
Producer
App 1
App 2
App 3
DATA QUALITY RULES
Topic
Confluent
Data streaming platform
Kafka
Schema Registry
Topic
Data contract
The correct format of a flight number is string starting with 2 letters, a space, and then numbers, like “CF 200”

9. DATA QUALITY RULES
Schema Registry
Data contract
"schemaType": "AVRO",
"schema": "{
"type":"record",
"name":"Booking",
"Fields":[
{"name":"customerId", "type":"int"},
{"name":"flightNumber", "type":"string"},
]}",
"ruleSet": {
"domainRules": [{
"name": "checkFlightNumber",
"kind": "CONDITION",
"type": "CEL",
"mode": "WRITE",
"expr":"flightNumber.startsWith("CF ")",
"onFailure": "DLQ"
}]
}

10. {
“customerId”: 3453,
“flightNumber”: “200”,
“creditCard”: “4111111111111111”
}
Producer
App 1
App 2
App 3
FIELD LEVEL ENCRYPTION
Topic
Confluent
Data streaming platform
Kafka
Schema Registry
Topic
Data contract
“creditCard”: “4111111111111111”
“creditCard”: “4111111111111111”
“creditCard”: “4111111111111111”

11. {
“customerId”: 3453,
“flightNumber”: “200”,
“creditCard”: “pt4535Xfg@rt%gg3”
}
Producer
App 1
App 2
App 3
FIELD LEVEL ENCRYPTION
Topic
Confluent
Data streaming platform
Kafka
Schema Registry
Topic
Data contract
“creditCard”: “4111111111111111”
“creditCard”: “pt4535Xfg@rt%gg3”
“creditCard”: “pt4535Xfg@rt%gg3”

12. Schema Registry
Data contract
"schemaType": "AVRO",
"schema": "{
"type":"record",
"name":"Booking",
"Fields":[
{"name":"customerId", "type":"int"},
{"name":"flightNumber", "type":"string"},
{"name":"creditCard", "type":"string","confluent:tags":["PCI"]}
]}",
"ruleSet": {
"domainRules": [{
"name": "encryptPCI",
"type": "ENCRYPT",
"tags": ["PCI"],
"params": {
"encrypt.kek.name":"ce581594-3115-486e-b391-5ea874371e73",
"encrypt.kms.type": "aws-kms",
"Encrypt.kms.key.id": "arn:aws:kms:us-east-1:58605107309..."
}
}]
}
FIELD LEVEL ENCRYPTION

13. Data contract
An explicit agreement between a data producer
and a data consumer formalizing the expected
structure, semantics and enforcement policies of
the data.

14. 1
4
DATA CONTRACT
Structure
Integrity constraints
Metadata
Policies
DATA CONTRACT
Evolution

15. CLUSTER 1
Topic 1
Topic 2
Topic 3
CLUSTER 2
Topic 1
Topic 2
Topic 3
? My App
CATALOG

16. Central Log Systems (CLS)
Alerting Systems
Fraud & SIEM Systems
ITSM Systems
AI/ML Engines
Visualization Apps
Distribution
Watch Lists
Profile Management
Threat Vector
Order Fulfillment
Notification Engine
Payroll Systems
Supply Chain Systems
Customer Profile
Data
ML Engine
Revenue Systems
Customer Application
Clickstreams
Customer Loyalty
Change Logs
Transactions
Payments
Inventory
Weather
Telemetry
Purchase
Mainframe Data
IoT Data
Payment Service
Heatmap Service
Customer Churn
Incident Management
Supply Chain Systems
Merchant Profile
Data
Recommendation Engine
Transactions
Payments
Inventory
Weather
Logs Data
Obs Service
Fraud & SIEM Systems
ITSM Systems
AI/ML Engines
Visualization Apps
Distribution
Watch Lists
Profile Management
Threat Vector
Customers
Customer Churn
Incident Management
CATALOG
Table Application
Fraud System
Fraud clicks
Purchases
Refunds

17. Topic AR
Topic AL
Topic CH
Topic CS
Topic UG
Topic GH
Topic KA
Topic NL
Topic ES
Topic GE
Topic VB
Topic JP
Topic JA
Topic NZ
Topic BR
Topic IT
Topic US
Topic IN
Topic FR
Topic D
Topic SA
Topic PT
Topic FO
Topic GR
Topic CV
Topic IN
Topic AU
Topic JA
Topic VF
Topic NQ
Topic PT
Topic XE
Topic TA
Topic IS
Topic FI
Topic OV
Topic MB
Topic SX
Topic RS
Topic CB

18. STREAM CATALOG
TECHNICAL METADATA
BUSINESS METADATA
CATALOG
LINEAGE METADATA
Topics
Queries
Schemas
Connectors
Clusters
Environments
Cluster links
Data Portal
REST API
Terraform
GraphQL API

19. UI REST API Terraform
GraphQL API
Pipelines

20. DATA PORTAL
Search and discover

21. DATA PORTAL
Self-serve request access

22. DATA PORTAL
Build

23. DATA PORTAL
Process with Flink