SlideShare a Scribd company logo
Evolving Data Governance for the
Real-time Streaming and AI Era
Andrew Foo
Customer Solutions @ Confluent
Would you blindly cross the street with
traffic information that is 5 minutes old?
Generative AI is a revolutionary tool… and it’s
only getting better.
/imagine prompt:Street style photo of a woman shot on Kodak
July 2022 July 2023
Source: https://twitter.com/nickfloats/status/1676279157620199424?s=46&t=plcKoQYXnokFvxs3ieVg3Q
Recency, quality,
trustworthiness and
instant applicability
of data is as
important as the
models themselves.
Source: https://au.pcmag.com/ai/103906/air-canada-must-honor-a-fake-refund-policy-created-by-its-chatbot-court-says
Source: https://au.pcmag.com/ai/103906/air-canada-must-honor-a-fake-refund-policy-created-by-its-chatbot-court-says
Without context,
trustworthiness
or real-time data
applicability,
LLMs can’t drive
meaningful value
What is the status of
my flight to New York?
It is currently delayed by 2 hours and
expected to depart at 5 pm GMT.
Is there another flight available
to the same city that will depart
and arrive sooner? What are the
seating options and cost?
The next available flight to New York
with United departs later but will
arrive faster than your current flight.
The only available seats in this flight
are first class window seats and
costs $1,500.
Can your GenAI
assistant remember
data from an earlier
conversation?
What is the source of
this information? Is this
trustworthy? Is it fresh
and accurate?
How do you securely augment
customer data with real-time
data and process them on the fly
to provide meaningful insights?
“Our latest research estimates
that generative AI could add
the equivalent of $2.6 trillion
to $4.4 trillion annually across
the 63 use cases we analyzed.”
Source: Economic Potential of Generative AI, McKinsey
What we’ll
talk about
● The data architecture challenge
● Unifying the operational and analytical worlds
● Connecting governed data streams to power AI
● Benefits of a modern data streaming platform
Traditional enterprise data architecture
is a GenAI innovation bottleneck
Historic Public Data
Generative
AI Model
Intelligent
Business-Specific
Co-Pilot
User Interaction
??
Enterprise data architecture
In-context learning &
prompt-time assembly
9
10
11
DATA MESS = DEVELOPER PAIN
DATA MESS DATA PRODUCTS
12
Point-to-Point
Data Extracted by Consumer
Multi-Subscriber
Producer Presented
So, what’s stopping us?
13
ANALYTICAL ESTATE
OPERATIONAL ESTATE
14
ANALYTICAL ESTATE
OPERATIONAL ESTATE
15
DATA PRODUCTS
What if we could unite them?
16
17
Serve the needs of
applications to transact with
customers in real-time
Support after-the-fact business
analysis and reporting for various
stakeholders
OPERATIONAL ESTATE ANALYTICAL ESTATE
18
OPERATIONAL ESTATE
19
Kafka is the Open Standard for the Operational Estate
OPERATIONAL ESTATE
20
ANALYTICAL ESTATE
21
S3 / GCS / ABS
ANALYTICAL ESTATE
22
Iceberg is the Open Standard for the Analytical Estate
ANALYTICAL ESTATE
23
ANALYTICAL ESTATE
OPERATIONAL ESTATE
STREAM
Analytical Product
24
Universal Data Product
25
Universal Data Product
Powered by a Streaming Platform
26
Universal Data Product
Kafka Topic + Schema + Owner
27
DEVELOPERS
SECURITY & COMPLIANCE
28
29
Shift Your Governance Thinking to the Left
30
The Cleanest Data is Always Bottled at the Source
31
POINT-IN-TIME LINEAGE
LINEAGE SEARCH
Stream Lineage
TECHNICAL METADATA
BUSINESS METADATA
Stream Catalog
`
TECHNICAL METADATA
BUSINESS METADATA
Stream Quality
32
Online Purchase
In Store Purchase
Customer Detail
Purchases
Click Stream
Customer 360 Analytical Reports
Gen AI
Online Apps
From Vicious Cycle to Virtuous Circle
Challenge: Judo Bank needed to replace a series of
point-to-point integrations and core lending platform with a
new, unified system, and re-architect the foundational IT
infrastructure to drive event-first thinking and event-driven
principals.
Solution: Judo banke leverages Confluent Cloud for an easy,
agile, holistic management of a suite of services and creation of
a new CRM system and loan originator capabilities.
Results:
● Consistent,unified system
● Better data availability and time to market
● Bettersupportformicroservices
“Confluent is a strategic platform for us. With every
project we look at, we now think about how we use
Confluent to move things around and join things
together” — Niko Bielovich, General Manager, Services
Management
Industry: Banking | Geo: APAC | Product: Confluent Cloud
Click here to learn more.
“Everything we do is in real time because batch processing is an old
way of thinking. The longer your data waits, the less value it has. So,
as data comes through, you need to be able to act on it, or enrich it
quickly. Confluent enables this for us.”
— Rajay Rai, Chief Information Officer at Trust Bank
Challenge: Building a secure, digital-only bank to power unique, secure,
and real-time experiences for customers.
Solution: Trust Bank leverages Confluent’s data streaming platform for its
event-driven architecture, enabling different teams to produce, share, and
consume self-service data products in the form of real-time streams, drive
innovation, improve agility, reduce the total cost of ownership, and ensure
the appropriate quality controls and security policies are applied across
the organization.
Results:
● A scalable and resilient platform to power new, real-time experiences
for banking customers
● Built-in governance to meet regulations, gain customer trust, and
break down data silos so teams have self-service access to find, browse,
create, share, and reuse data, wherever and whenever it’s needed
● Lower total cost of ownership (TCO)
● Unscheduled downtime for critical systems does not exceed four hours
within any 12-month period
“Everything we do is in real time because batch processing is
an old way of thinking. The longer your data waits, the less
value it has. So, as data comes through, you need to be able to
act on it, or enrich it quickly. Confluent enables this for us.” —
Rajay Rai, Chief Information Officer
Industry: Financial Services | Geo: APAC | Product: Confluent Cloud
Click here to learn more.
CONNECT
PROCESS
GOVERN
SHARE
Custom Apps &
Microservices
Data Systems
STREAM
AI/ML Modeling
Inventory Payments
Personalization
Fraud Supply Chain
Recommendations
From Data Mess To Data Products
To Instant Value
Everywhere
36
DATA MESS = DEVELOPER PAIN
DATA PRODUCTS = DEVELOPER GAIN

More Related Content

Similar to Evolving Data Governance for the Real-time Streaming and AI Era

Fast Data for Competitive Advantage: 4 Steps to Expand your Window of Opportu...
Fast Data for Competitive Advantage: 4 Steps to Expand your Window of Opportu...Fast Data for Competitive Advantage: 4 Steps to Expand your Window of Opportu...
Fast Data for Competitive Advantage: 4 Steps to Expand your Window of Opportu...
VoltDB
 
Gartner CIO & IT Executive Summit -- Event Mesh: The Architecture Layer That ...
Gartner CIO & IT Executive Summit -- Event Mesh: The Architecture Layer That ...Gartner CIO & IT Executive Summit -- Event Mesh: The Architecture Layer That ...
Gartner CIO & IT Executive Summit -- Event Mesh: The Architecture Layer That ...
Solace
 
EY + Neo4j: Why graph technology makes sense for fraud detection and customer...
EY + Neo4j: Why graph technology makes sense for fraud detection and customer...EY + Neo4j: Why graph technology makes sense for fraud detection and customer...
EY + Neo4j: Why graph technology makes sense for fraud detection and customer...
Neo4j
 
Confluent Partner Tech Talk with BearingPoint
Confluent Partner Tech Talk with BearingPointConfluent Partner Tech Talk with BearingPoint
Confluent Partner Tech Talk with BearingPoint
confluent
 
In-Memory Computing Webcast. Market Predictions 2017
In-Memory Computing Webcast. Market Predictions 2017In-Memory Computing Webcast. Market Predictions 2017
In-Memory Computing Webcast. Market Predictions 2017
SingleStore
 
Flink Forward Berlin 2017 Keynote: Ferd Scheepers - Taking away customer fric...
Flink Forward Berlin 2017 Keynote: Ferd Scheepers - Taking away customer fric...Flink Forward Berlin 2017 Keynote: Ferd Scheepers - Taking away customer fric...
Flink Forward Berlin 2017 Keynote: Ferd Scheepers - Taking away customer fric...
Flink Forward
 
Agile Mumbai 2022 - Balvinder Kaur & Sushant Joshi | Real-Time Insights and A...
Agile Mumbai 2022 - Balvinder Kaur & Sushant Joshi | Real-Time Insights and A...Agile Mumbai 2022 - Balvinder Kaur & Sushant Joshi | Real-Time Insights and A...
Agile Mumbai 2022 - Balvinder Kaur & Sushant Joshi | Real-Time Insights and A...
AgileNetwork
 
Cloud Reshaping Banking
Cloud Reshaping BankingCloud Reshaping Banking
Cloud Reshaping Banking
Happiest Minds Technologies
 
Harness the Power of Big Data with Oracle
Harness the Power of Big Data with OracleHarness the Power of Big Data with Oracle
Harness the Power of Big Data with Oracle
Sai Janakiram Penumuru
 
Event Mesh: the Architecture Layer That Will Make Your Business Event-Driven
Event Mesh: the Architecture Layer That Will Make Your Business Event-DrivenEvent Mesh: the Architecture Layer That Will Make Your Business Event-Driven
Event Mesh: the Architecture Layer That Will Make Your Business Event-Driven
Solace
 
Gartner event mesh solace - phil scanlon - gold coast
Gartner event mesh   solace - phil scanlon - gold coastGartner event mesh   solace - phil scanlon - gold coast
Gartner event mesh solace - phil scanlon - gold coast
Phil Scanlon
 
CWIN17 New-York / real time customer experience for todays right now economy
CWIN17 New-York / real time customer experience for todays right now economyCWIN17 New-York / real time customer experience for todays right now economy
CWIN17 New-York / real time customer experience for todays right now economy
Capgemini
 
In-Memory Computing Driving Edge Computing and Blockchain Technologies
In-Memory Computing Driving Edge Computing and Blockchain TechnologiesIn-Memory Computing Driving Edge Computing and Blockchain Technologies
In-Memory Computing Driving Edge Computing and Blockchain Technologies
dsapps
 
Data and Analytics In The Digital Age
Data and Analytics In The Digital AgeData and Analytics In The Digital Age
Data and Analytics In The Digital Age
Nigel Wright Group
 
Di in the age of digital disruptions v1.0
Di in the age of digital disruptions v1.0Di in the age of digital disruptions v1.0
Di in the age of digital disruptions v1.0
Amar Roy
 
10 Keynotes in STRATA and HADOOP World Conference
10 Keynotes in STRATA and HADOOP World Conference10 Keynotes in STRATA and HADOOP World Conference
10 Keynotes in STRATA and HADOOP World Conference
KCC Software Ltd. & Easylearning.guru
 
Flink Forward Berlin 2017: Bas Geerdink, Martijn Visser - Fast Data at ING - ...
Flink Forward Berlin 2017: Bas Geerdink, Martijn Visser - Fast Data at ING - ...Flink Forward Berlin 2017: Bas Geerdink, Martijn Visser - Fast Data at ING - ...
Flink Forward Berlin 2017: Bas Geerdink, Martijn Visser - Fast Data at ING - ...
Flink Forward
 
Goodenough 110424192114-phpapp02
Goodenough 110424192114-phpapp02Goodenough 110424192114-phpapp02
Goodenough 110424192114-phpapp02
Gerson Orlando Jr
 
Modern Applications Demand Network Analytics
Modern Applications Demand Network AnalyticsModern Applications Demand Network Analytics
Modern Applications Demand Network Analytics
Pluribus Networks
 
Greetings david cutler inform and connect
Greetings   david cutler inform and connectGreetings   david cutler inform and connect
Greetings david cutler inform and connect
Sales Strategy and Innovation Delivery
 

Similar to Evolving Data Governance for the Real-time Streaming and AI Era (20)

Fast Data for Competitive Advantage: 4 Steps to Expand your Window of Opportu...
Fast Data for Competitive Advantage: 4 Steps to Expand your Window of Opportu...Fast Data for Competitive Advantage: 4 Steps to Expand your Window of Opportu...
Fast Data for Competitive Advantage: 4 Steps to Expand your Window of Opportu...
 
Gartner CIO & IT Executive Summit -- Event Mesh: The Architecture Layer That ...
Gartner CIO & IT Executive Summit -- Event Mesh: The Architecture Layer That ...Gartner CIO & IT Executive Summit -- Event Mesh: The Architecture Layer That ...
Gartner CIO & IT Executive Summit -- Event Mesh: The Architecture Layer That ...
 
EY + Neo4j: Why graph technology makes sense for fraud detection and customer...
EY + Neo4j: Why graph technology makes sense for fraud detection and customer...EY + Neo4j: Why graph technology makes sense for fraud detection and customer...
EY + Neo4j: Why graph technology makes sense for fraud detection and customer...
 
Confluent Partner Tech Talk with BearingPoint
Confluent Partner Tech Talk with BearingPointConfluent Partner Tech Talk with BearingPoint
Confluent Partner Tech Talk with BearingPoint
 
In-Memory Computing Webcast. Market Predictions 2017
In-Memory Computing Webcast. Market Predictions 2017In-Memory Computing Webcast. Market Predictions 2017
In-Memory Computing Webcast. Market Predictions 2017
 
Flink Forward Berlin 2017 Keynote: Ferd Scheepers - Taking away customer fric...
Flink Forward Berlin 2017 Keynote: Ferd Scheepers - Taking away customer fric...Flink Forward Berlin 2017 Keynote: Ferd Scheepers - Taking away customer fric...
Flink Forward Berlin 2017 Keynote: Ferd Scheepers - Taking away customer fric...
 
Agile Mumbai 2022 - Balvinder Kaur & Sushant Joshi | Real-Time Insights and A...
Agile Mumbai 2022 - Balvinder Kaur & Sushant Joshi | Real-Time Insights and A...Agile Mumbai 2022 - Balvinder Kaur & Sushant Joshi | Real-Time Insights and A...
Agile Mumbai 2022 - Balvinder Kaur & Sushant Joshi | Real-Time Insights and A...
 
Cloud Reshaping Banking
Cloud Reshaping BankingCloud Reshaping Banking
Cloud Reshaping Banking
 
Harness the Power of Big Data with Oracle
Harness the Power of Big Data with OracleHarness the Power of Big Data with Oracle
Harness the Power of Big Data with Oracle
 
Event Mesh: the Architecture Layer That Will Make Your Business Event-Driven
Event Mesh: the Architecture Layer That Will Make Your Business Event-DrivenEvent Mesh: the Architecture Layer That Will Make Your Business Event-Driven
Event Mesh: the Architecture Layer That Will Make Your Business Event-Driven
 
Gartner event mesh solace - phil scanlon - gold coast
Gartner event mesh   solace - phil scanlon - gold coastGartner event mesh   solace - phil scanlon - gold coast
Gartner event mesh solace - phil scanlon - gold coast
 
CWIN17 New-York / real time customer experience for todays right now economy
CWIN17 New-York / real time customer experience for todays right now economyCWIN17 New-York / real time customer experience for todays right now economy
CWIN17 New-York / real time customer experience for todays right now economy
 
In-Memory Computing Driving Edge Computing and Blockchain Technologies
In-Memory Computing Driving Edge Computing and Blockchain TechnologiesIn-Memory Computing Driving Edge Computing and Blockchain Technologies
In-Memory Computing Driving Edge Computing and Blockchain Technologies
 
Data and Analytics In The Digital Age
Data and Analytics In The Digital AgeData and Analytics In The Digital Age
Data and Analytics In The Digital Age
 
Di in the age of digital disruptions v1.0
Di in the age of digital disruptions v1.0Di in the age of digital disruptions v1.0
Di in the age of digital disruptions v1.0
 
10 Keynotes in STRATA and HADOOP World Conference
10 Keynotes in STRATA and HADOOP World Conference10 Keynotes in STRATA and HADOOP World Conference
10 Keynotes in STRATA and HADOOP World Conference
 
Flink Forward Berlin 2017: Bas Geerdink, Martijn Visser - Fast Data at ING - ...
Flink Forward Berlin 2017: Bas Geerdink, Martijn Visser - Fast Data at ING - ...Flink Forward Berlin 2017: Bas Geerdink, Martijn Visser - Fast Data at ING - ...
Flink Forward Berlin 2017: Bas Geerdink, Martijn Visser - Fast Data at ING - ...
 
Goodenough 110424192114-phpapp02
Goodenough 110424192114-phpapp02Goodenough 110424192114-phpapp02
Goodenough 110424192114-phpapp02
 
Modern Applications Demand Network Analytics
Modern Applications Demand Network AnalyticsModern Applications Demand Network Analytics
Modern Applications Demand Network Analytics
 
Greetings david cutler inform and connect
Greetings   david cutler inform and connectGreetings   david cutler inform and connect
Greetings david cutler inform and connect
 

More from confluent

Speed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in MinutesSpeed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in Minutes
confluent
 
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
confluent
 
Santander Stream Processing with Apache Flink
Santander Stream Processing with Apache FlinkSantander Stream Processing with Apache Flink
Santander Stream Processing with Apache Flink
confluent
 
Unlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insightsUnlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insights
confluent
 
Workshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con FlinkWorkshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con Flink
confluent
 
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
confluent
 
AWS Immersion Day Mapfre - Confluent
AWS Immersion Day Mapfre   -   ConfluentAWS Immersion Day Mapfre   -   Confluent
AWS Immersion Day Mapfre - Confluent
confluent
 
Eventos y Microservicios - Santander TechTalk
Eventos y Microservicios - Santander TechTalkEventos y Microservicios - Santander TechTalk
Eventos y Microservicios - Santander TechTalk
confluent
 
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
Q&A with Confluent Experts: Navigating Networking in Confluent CloudQ&A with Confluent Experts: Navigating Networking in Confluent Cloud
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
confluent
 
Citi TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep DiveCiti TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep Dive
confluent
 
Build real-time streaming data pipelines to AWS with Confluent
Build real-time streaming data pipelines to AWS with ConfluentBuild real-time streaming data pipelines to AWS with Confluent
Build real-time streaming data pipelines to AWS with Confluent
confluent
 
Q&A with Confluent Professional Services: Confluent Service Mesh
Q&A with Confluent Professional Services: Confluent Service MeshQ&A with Confluent Professional Services: Confluent Service Mesh
Q&A with Confluent Professional Services: Confluent Service Mesh
confluent
 
Citi Tech Talk: Event Driven Kafka Microservices
Citi Tech Talk: Event Driven Kafka MicroservicesCiti Tech Talk: Event Driven Kafka Microservices
Citi Tech Talk: Event Driven Kafka Microservices
confluent
 
Confluent & GSI Webinars series - Session 3
Confluent & GSI Webinars series - Session 3Confluent & GSI Webinars series - Session 3
Confluent & GSI Webinars series - Session 3
confluent
 
Citi Tech Talk: Messaging Modernization
Citi Tech Talk: Messaging ModernizationCiti Tech Talk: Messaging Modernization
Citi Tech Talk: Messaging Modernization
confluent
 
Citi Tech Talk: Data Governance for streaming and real time data
Citi Tech Talk: Data Governance for streaming and real time dataCiti Tech Talk: Data Governance for streaming and real time data
Citi Tech Talk: Data Governance for streaming and real time data
confluent
 
Confluent & GSI Webinars series: Session 2
Confluent & GSI Webinars series: Session 2Confluent & GSI Webinars series: Session 2
Confluent & GSI Webinars series: Session 2
confluent
 
Data In Motion Paris 2023
Data In Motion Paris 2023Data In Motion Paris 2023
Data In Motion Paris 2023
confluent
 
Confluent Partner Tech Talk with Synthesis
Confluent Partner Tech Talk with SynthesisConfluent Partner Tech Talk with Synthesis
Confluent Partner Tech Talk with Synthesis
confluent
 
The Future of Application Development - API Days - Melbourne 2023
The Future of Application Development - API Days - Melbourne 2023The Future of Application Development - API Days - Melbourne 2023
The Future of Application Development - API Days - Melbourne 2023
confluent
 

More from confluent (20)

Speed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in MinutesSpeed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in Minutes
 
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
 
Santander Stream Processing with Apache Flink
Santander Stream Processing with Apache FlinkSantander Stream Processing with Apache Flink
Santander Stream Processing with Apache Flink
 
Unlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insightsUnlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insights
 
Workshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con FlinkWorkshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con Flink
 
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
 
AWS Immersion Day Mapfre - Confluent
AWS Immersion Day Mapfre   -   ConfluentAWS Immersion Day Mapfre   -   Confluent
AWS Immersion Day Mapfre - Confluent
 
Eventos y Microservicios - Santander TechTalk
Eventos y Microservicios - Santander TechTalkEventos y Microservicios - Santander TechTalk
Eventos y Microservicios - Santander TechTalk
 
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
Q&A with Confluent Experts: Navigating Networking in Confluent CloudQ&A with Confluent Experts: Navigating Networking in Confluent Cloud
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
 
Citi TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep DiveCiti TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep Dive
 
Build real-time streaming data pipelines to AWS with Confluent
Build real-time streaming data pipelines to AWS with ConfluentBuild real-time streaming data pipelines to AWS with Confluent
Build real-time streaming data pipelines to AWS with Confluent
 
Q&A with Confluent Professional Services: Confluent Service Mesh
Q&A with Confluent Professional Services: Confluent Service MeshQ&A with Confluent Professional Services: Confluent Service Mesh
Q&A with Confluent Professional Services: Confluent Service Mesh
 
Citi Tech Talk: Event Driven Kafka Microservices
Citi Tech Talk: Event Driven Kafka MicroservicesCiti Tech Talk: Event Driven Kafka Microservices
Citi Tech Talk: Event Driven Kafka Microservices
 
Confluent & GSI Webinars series - Session 3
Confluent & GSI Webinars series - Session 3Confluent & GSI Webinars series - Session 3
Confluent & GSI Webinars series - Session 3
 
Citi Tech Talk: Messaging Modernization
Citi Tech Talk: Messaging ModernizationCiti Tech Talk: Messaging Modernization
Citi Tech Talk: Messaging Modernization
 
Citi Tech Talk: Data Governance for streaming and real time data
Citi Tech Talk: Data Governance for streaming and real time dataCiti Tech Talk: Data Governance for streaming and real time data
Citi Tech Talk: Data Governance for streaming and real time data
 
Confluent & GSI Webinars series: Session 2
Confluent & GSI Webinars series: Session 2Confluent & GSI Webinars series: Session 2
Confluent & GSI Webinars series: Session 2
 
Data In Motion Paris 2023
Data In Motion Paris 2023Data In Motion Paris 2023
Data In Motion Paris 2023
 
Confluent Partner Tech Talk with Synthesis
Confluent Partner Tech Talk with SynthesisConfluent Partner Tech Talk with Synthesis
Confluent Partner Tech Talk with Synthesis
 
The Future of Application Development - API Days - Melbourne 2023
The Future of Application Development - API Days - Melbourne 2023The Future of Application Development - API Days - Melbourne 2023
The Future of Application Development - API Days - Melbourne 2023
 

Recently uploaded

Liberarsi dai framework con i Web Component.pptx
Liberarsi dai framework con i Web Component.pptxLiberarsi dai framework con i Web Component.pptx
Liberarsi dai framework con i Web Component.pptx
Massimo Artizzu
 
What next after learning python programming basics
What next after learning python programming basicsWhat next after learning python programming basics
What next after learning python programming basics
Rakesh Kumar R
 
Malibou Pitch Deck For Its €3M Seed Round
Malibou Pitch Deck For Its €3M Seed RoundMalibou Pitch Deck For Its €3M Seed Round
Malibou Pitch Deck For Its €3M Seed Round
sjcobrien
 
在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样
在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样
在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样
mz5nrf0n
 
Oracle 23c New Features For DBAs and Developers.pptx
Oracle 23c New Features For DBAs and Developers.pptxOracle 23c New Features For DBAs and Developers.pptx
Oracle 23c New Features For DBAs and Developers.pptx
Remote DBA Services
 
Unveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdfUnveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdf
brainerhub1
 
Microservice Teams - How the cloud changes the way we work
Microservice Teams - How the cloud changes the way we workMicroservice Teams - How the cloud changes the way we work
Microservice Teams - How the cloud changes the way we work
Sven Peters
 
8 Best Automated Android App Testing Tool and Framework in 2024.pdf
8 Best Automated Android App Testing Tool and Framework in 2024.pdf8 Best Automated Android App Testing Tool and Framework in 2024.pdf
8 Best Automated Android App Testing Tool and Framework in 2024.pdf
kalichargn70th171
 
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
Bert Jan Schrijver
 
fiscal year variant fiscal year variant.
fiscal year variant fiscal year variant.fiscal year variant fiscal year variant.
fiscal year variant fiscal year variant.
AnkitaPandya11
 
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
gapen1
 
Modelling Up - DDDEurope 2024 - Amsterdam
Modelling Up - DDDEurope 2024 - AmsterdamModelling Up - DDDEurope 2024 - Amsterdam
Modelling Up - DDDEurope 2024 - Amsterdam
Alberto Brandolini
 
14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision
ShulagnaSarkar2
 
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CDKuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
rodomar2
 
How to write a program in any programming language
How to write a program in any programming languageHow to write a program in any programming language
How to write a program in any programming language
Rakesh Kumar R
 
The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...
The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...
The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...
kalichargn70th171
 
All you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVMAll you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVM
Alina Yurenko
 
Oracle Database 19c New Features for DBAs and Developers.pptx
Oracle Database 19c New Features for DBAs and Developers.pptxOracle Database 19c New Features for DBAs and Developers.pptx
Oracle Database 19c New Features for DBAs and Developers.pptx
Remote DBA Services
 
zOS Mainframe JES2-JES3 JCL-JECL Differences
zOS Mainframe JES2-JES3 JCL-JECL DifferenceszOS Mainframe JES2-JES3 JCL-JECL Differences
zOS Mainframe JES2-JES3 JCL-JECL Differences
YousufSait3
 
UI5con 2024 - Bring Your Own Design System
UI5con 2024 - Bring Your Own Design SystemUI5con 2024 - Bring Your Own Design System
UI5con 2024 - Bring Your Own Design System
Peter Muessig
 

Recently uploaded (20)

Liberarsi dai framework con i Web Component.pptx
Liberarsi dai framework con i Web Component.pptxLiberarsi dai framework con i Web Component.pptx
Liberarsi dai framework con i Web Component.pptx
 
What next after learning python programming basics
What next after learning python programming basicsWhat next after learning python programming basics
What next after learning python programming basics
 
Malibou Pitch Deck For Its €3M Seed Round
Malibou Pitch Deck For Its €3M Seed RoundMalibou Pitch Deck For Its €3M Seed Round
Malibou Pitch Deck For Its €3M Seed Round
 
在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样
在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样
在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样
 
Oracle 23c New Features For DBAs and Developers.pptx
Oracle 23c New Features For DBAs and Developers.pptxOracle 23c New Features For DBAs and Developers.pptx
Oracle 23c New Features For DBAs and Developers.pptx
 
Unveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdfUnveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdf
 
Microservice Teams - How the cloud changes the way we work
Microservice Teams - How the cloud changes the way we workMicroservice Teams - How the cloud changes the way we work
Microservice Teams - How the cloud changes the way we work
 
8 Best Automated Android App Testing Tool and Framework in 2024.pdf
8 Best Automated Android App Testing Tool and Framework in 2024.pdf8 Best Automated Android App Testing Tool and Framework in 2024.pdf
8 Best Automated Android App Testing Tool and Framework in 2024.pdf
 
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
 
fiscal year variant fiscal year variant.
fiscal year variant fiscal year variant.fiscal year variant fiscal year variant.
fiscal year variant fiscal year variant.
 
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
 
Modelling Up - DDDEurope 2024 - Amsterdam
Modelling Up - DDDEurope 2024 - AmsterdamModelling Up - DDDEurope 2024 - Amsterdam
Modelling Up - DDDEurope 2024 - Amsterdam
 
14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision14 th Edition of International conference on computer vision
14 th Edition of International conference on computer vision
 
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CDKuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
 
How to write a program in any programming language
How to write a program in any programming languageHow to write a program in any programming language
How to write a program in any programming language
 
The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...
The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...
The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...
 
All you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVMAll you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVM
 
Oracle Database 19c New Features for DBAs and Developers.pptx
Oracle Database 19c New Features for DBAs and Developers.pptxOracle Database 19c New Features for DBAs and Developers.pptx
Oracle Database 19c New Features for DBAs and Developers.pptx
 
zOS Mainframe JES2-JES3 JCL-JECL Differences
zOS Mainframe JES2-JES3 JCL-JECL DifferenceszOS Mainframe JES2-JES3 JCL-JECL Differences
zOS Mainframe JES2-JES3 JCL-JECL Differences
 
UI5con 2024 - Bring Your Own Design System
UI5con 2024 - Bring Your Own Design SystemUI5con 2024 - Bring Your Own Design System
UI5con 2024 - Bring Your Own Design System
 

Evolving Data Governance for the Real-time Streaming and AI Era

  • 1. Evolving Data Governance for the Real-time Streaming and AI Era Andrew Foo Customer Solutions @ Confluent
  • 2. Would you blindly cross the street with traffic information that is 5 minutes old?
  • 3. Generative AI is a revolutionary tool… and it’s only getting better. /imagine prompt:Street style photo of a woman shot on Kodak July 2022 July 2023 Source: https://twitter.com/nickfloats/status/1676279157620199424?s=46&t=plcKoQYXnokFvxs3ieVg3Q
  • 4. Recency, quality, trustworthiness and instant applicability of data is as important as the models themselves. Source: https://au.pcmag.com/ai/103906/air-canada-must-honor-a-fake-refund-policy-created-by-its-chatbot-court-says Source: https://au.pcmag.com/ai/103906/air-canada-must-honor-a-fake-refund-policy-created-by-its-chatbot-court-says
  • 5. Without context, trustworthiness or real-time data applicability, LLMs can’t drive meaningful value What is the status of my flight to New York? It is currently delayed by 2 hours and expected to depart at 5 pm GMT. Is there another flight available to the same city that will depart and arrive sooner? What are the seating options and cost? The next available flight to New York with United departs later but will arrive faster than your current flight. The only available seats in this flight are first class window seats and costs $1,500. Can your GenAI assistant remember data from an earlier conversation? What is the source of this information? Is this trustworthy? Is it fresh and accurate? How do you securely augment customer data with real-time data and process them on the fly to provide meaningful insights?
  • 6. “Our latest research estimates that generative AI could add the equivalent of $2.6 trillion to $4.4 trillion annually across the 63 use cases we analyzed.” Source: Economic Potential of Generative AI, McKinsey
  • 7. What we’ll talk about ● The data architecture challenge ● Unifying the operational and analytical worlds ● Connecting governed data streams to power AI ● Benefits of a modern data streaming platform
  • 8. Traditional enterprise data architecture is a GenAI innovation bottleneck Historic Public Data Generative AI Model Intelligent Business-Specific Co-Pilot User Interaction ?? Enterprise data architecture In-context learning & prompt-time assembly
  • 9. 9
  • 10. 10
  • 11. 11 DATA MESS = DEVELOPER PAIN
  • 12. DATA MESS DATA PRODUCTS 12 Point-to-Point Data Extracted by Consumer Multi-Subscriber Producer Presented
  • 16. What if we could unite them? 16
  • 17. 17 Serve the needs of applications to transact with customers in real-time Support after-the-fact business analysis and reporting for various stakeholders OPERATIONAL ESTATE ANALYTICAL ESTATE
  • 19. 19 Kafka is the Open Standard for the Operational Estate OPERATIONAL ESTATE
  • 21. 21 S3 / GCS / ABS ANALYTICAL ESTATE
  • 22. 22 Iceberg is the Open Standard for the Analytical Estate ANALYTICAL ESTATE
  • 25. 25 Universal Data Product Powered by a Streaming Platform
  • 26. 26 Universal Data Product Kafka Topic + Schema + Owner
  • 28. 28
  • 29. 29 Shift Your Governance Thinking to the Left
  • 30. 30 The Cleanest Data is Always Bottled at the Source
  • 31. 31 POINT-IN-TIME LINEAGE LINEAGE SEARCH Stream Lineage TECHNICAL METADATA BUSINESS METADATA Stream Catalog ` TECHNICAL METADATA BUSINESS METADATA Stream Quality
  • 32. 32 Online Purchase In Store Purchase Customer Detail Purchases Click Stream Customer 360 Analytical Reports Gen AI Online Apps From Vicious Cycle to Virtuous Circle
  • 33. Challenge: Judo Bank needed to replace a series of point-to-point integrations and core lending platform with a new, unified system, and re-architect the foundational IT infrastructure to drive event-first thinking and event-driven principals. Solution: Judo banke leverages Confluent Cloud for an easy, agile, holistic management of a suite of services and creation of a new CRM system and loan originator capabilities. Results: ● Consistent,unified system ● Better data availability and time to market ● Bettersupportformicroservices “Confluent is a strategic platform for us. With every project we look at, we now think about how we use Confluent to move things around and join things together” — Niko Bielovich, General Manager, Services Management Industry: Banking | Geo: APAC | Product: Confluent Cloud Click here to learn more.
  • 34. “Everything we do is in real time because batch processing is an old way of thinking. The longer your data waits, the less value it has. So, as data comes through, you need to be able to act on it, or enrich it quickly. Confluent enables this for us.” — Rajay Rai, Chief Information Officer at Trust Bank Challenge: Building a secure, digital-only bank to power unique, secure, and real-time experiences for customers. Solution: Trust Bank leverages Confluent’s data streaming platform for its event-driven architecture, enabling different teams to produce, share, and consume self-service data products in the form of real-time streams, drive innovation, improve agility, reduce the total cost of ownership, and ensure the appropriate quality controls and security policies are applied across the organization. Results: ● A scalable and resilient platform to power new, real-time experiences for banking customers ● Built-in governance to meet regulations, gain customer trust, and break down data silos so teams have self-service access to find, browse, create, share, and reuse data, wherever and whenever it’s needed ● Lower total cost of ownership (TCO) ● Unscheduled downtime for critical systems does not exceed four hours within any 12-month period “Everything we do is in real time because batch processing is an old way of thinking. The longer your data waits, the less value it has. So, as data comes through, you need to be able to act on it, or enrich it quickly. Confluent enables this for us.” — Rajay Rai, Chief Information Officer Industry: Financial Services | Geo: APAC | Product: Confluent Cloud Click here to learn more.
  • 35. CONNECT PROCESS GOVERN SHARE Custom Apps & Microservices Data Systems STREAM AI/ML Modeling Inventory Payments Personalization Fraud Supply Chain Recommendations From Data Mess To Data Products To Instant Value Everywhere
  • 36. 36 DATA MESS = DEVELOPER PAIN DATA PRODUCTS = DEVELOPER GAIN