Data in motion – Imperative for agile enterprise
•Download as PPTX, PDF•
1 like•38 views
This document discusses how data streaming platforms enable an agile enterprise by allowing data to be in constant motion rather than at rest. It outlines how traditional relational databases are slow and rigid, limiting an organization's ability to change and innovate. Data streaming platforms use event-driven architectures and real-time event streams to power applications with up-to-date data. This delivers benefits like increased revenue, lower costs, and reduced risks. The document advocates migrating from traditional monolithic systems to a microservices architecture built on top of an event streaming platform.
Recommended
Recommended
More Related Content
Similar to Data in motion – Imperative for agile enterprise
Similar to Data in motion – Imperative for agile enterprise (20)
More from confluent
More from confluent (20)
Recently uploaded
Recently uploaded (20)
Data in motion – Imperative for agile enterprise
- 1. DATA IN MOTION – IMPERATIVE FOR AGILE ENTERPRISE Zahoor ul Islam 1
- 2. Quest towards agility and removing dependencies
- 3. Year 1998
- 4. Paradigm for Data at Rest: Relational Databases Databases Slow, daily batch processing Simple, static real-time queries
- 5. 5 Application Data Structure Behavior User Interface • Integrated user interface • Limited Data Structure • Vertical impact for data change • Data always considered stable when in rest. • Data must be in normalized form. • Dependencies don’t allow to change the structure. • High cost for change. • Rescue to have multiple layers of application and data in the enterprises.
- 6. 6
- 7. 7 Be fast and agile in a highly competitive market Gain real time visibility into all aspects of the business Reduce the cost and burden of IT operations Personalize experiences to delight customers and increase loyalty The legacy architecture is monolithic and tightly coupled, slowing innovation Silos of data make it difficult to provide a consistent real-time experience Applications built around legacy infrastructure are batch based and slow On-premises infrastructure increases costs and escalates complexity business wants to... It’s hard to deliver because...
- 8. LINE OF BUSINESS 01 LINE OF BUSINESS 02 PUBLIC CLOUD Data architecture is rigid, complicated, and expensive - making it too hard, time consuming and cost-prohibitive to digitally transform 8
- 9. 10 Data Structure / Model Behavior User Interface • Not all data matters. • Data value diminishes over the period. • New data points replaces old. • Data captures become continuous process. • Rate of change in new type of data.
- 12. 13
- 13. Steaming Platform 14 Paradigm for Data in Motion: Data / Event Streams Rich customer experiences Real-time events Real-time Event Streams A Sale A shipment A Trade A Customer Experience Data driven operations
- 14. ... Device Logs ... ... ... Data Stores Logs 3rd Party Apps Custom Apps / Microservices Data Stream / Event Real-time Customer 360 Financial Fraud Detection Real-time Risk Analytics Real-time Payments Machine Learning Models ... Real-time Applications Universal Event Pipeline Amazon S3 SaaS apps
- 15. Streaming Platform Downstream datastores Ecosystem integrations Customer segmentation CUSTOMER BEHAVIOR CORE SYSTEMS Social ATM Call Center Website Mobile Payments Wealth Management Retail Events Wholesale Events Market Data Trade Data Customer Com Streaming Platform
- 16. Data Stream / Event Event Broker Integration Intermediaries ESP / CEP Event capable data stores Key Drivers • Delivery of Contextual Data • Event Driven architecture • Stream Data Integration • Segregation of event from consumer Data as state State is sequence of events Stream Analytics Event Enrichment / Layering
- 17. 18 Application A Application B • Raw Data / Event • Analytics • Complex Event • Event correlation • Enrichment
- 18. 19 19 Increase Revenue → Customer Experience, Loyalty Decrease Costs → Increase Operational Efficiency Mitigate Risks → Regulatory Compliance Regulatory Reporting & Compliance Risk Simulations Legacy IT Replacement (e.g. Middleware replacement) Cyber Security (incl. SIEM) Fraud Prevention (incl. Anti-Money-Laundering - AML) Legacy IT Modernization (e.g. Mainframe off-load / augmentation) Customer 360 (Call Center - Know Your Customer - KYC) Real-time App Updates (Digital mobile / online) Applications (e.g. Account Opening / Loans / Mortgages / Next Best Action / Targeted Offers) Real-time Payments Platform Migration to the Cloud (Hybrid on-prem / Cloud. Also Hybrid Public Cloud vendors) Microservices Architecture Data Infrastructure layer Business Application layer - the use cases Data Pipelines Messaging Microservice/ Event Sourcing Stream Processing Data Integration Streaming ETL Log Aggregation
- 19. Fast track your migration to the cloud Speed up integration to other cloud services Recover fast and mitigate risk of downtime DB2 DB1 DWH App2 App3 App4 KV2 KV3 DB3 App2-v2 App5 App7 App1-v2 App8 DWH App1 KV DB2 DB1 KV DWH App2 App4 KV2 KV3 App5 App7 App1-v2 App8 DWH App1 Data Stream Data Stream App3 DB3 App2-v2 VS
- 20. 21 Fraud service Trading service Loan service Reporting service Payment service UI service GUI Fraud service Trading service Loan service Reporting service Payment service UI service GUI Traditional REST based microservices Event-driven microservices Event Stream Platform
- 21. 22 Decentralized Data View Event Thinking Event Governance Event Standardization Organization Maturity Capabilities and Skills
- 22. THANK YOU!
Editor's Notes
- And this truly is a new way of thinking about data. The existing paradigm for data is one where data is at rest, with the relational database being the center of your data infrastructure. This worked remarkably well for many years, as it solved the enterprise need to query data and build transactional applications. But today, this is no longer enough. The challenge today is not just querying a database, The challenge is connecting across multiple systems and siloed lines of business.
- This results in most enterprises having an architecture that looks like this. In fact, your architecture very likely looks something like this. Here you see an untenable amount of point to point connections, and a data architecture that is rigid, complicated and expensive --- making it too difficult, time consuming and cost-prohibitive to innovate. This is the exact visual representation of what it means to be restricted by your infrastructure.
- We live in amazing times of change. The world is very different than when I was a kid, and a lot of this is due to technology. We see it all around us in everyday life. My son/daughter doesn’t understand, for example, how this black device there could be called a phone, since it doesn’t seem to do any of the things a phone does. It doesn’t take photos or videos, you can’t play video games, and you can’t message your friends. He/she similarly tries to swipe or pinch-zoom a television screen. And similar to this, the business world has changed dramatically.
- The new paradigm, for data-in-motion, is centered around event streams. Here, data is not static or passive, but continuously evolving and continuously being processed in real-time. Here, the fundamental notion is of data moving, not sitting at rest.
- And so what we’ve built is a new type of data platform --- centered around event streams. Here you see how all those solutions that were connected in that complex web --- are now connected in a seamless manner through a universal event pipeline. And what our customers tell us is essential about event streaming is that this pipeline successfully scales across their business. On the top you’ll see something entirely new, which is the ability to create a new category of applications --- event streaming applications. These include things like real-time customer interactions, real-time fraud detection, real-time machine learning models, and more. As our customers mature in fully embracing this architecture, this becomes the central nervous system that enables them to run their business in real-time.
- Single Source of Truth platform • Consolidate systems • Support data delivery to core banking systems • All data centrally persisted Deliver 360 understanding of customers • Support omni-channel banking initiatives • Enable real time customer experience • Leverage new sources of data
- Legacy IT infrastructure is difficult to manage or scale and prevents customers from efficiently building modern applications, limiting their long-term competitiveness. Traditional app architecture: On-prem, dedicated infrastructure Sprawls out of legacy IT Difficult to manage / scale High cost (capex) infra Challenges / downsides: Slow to update Does not have modern app features/capabilities Lagging competition Poor user experiences Challenges: Each team/department must execute their own cloud migration Maybe moving the same data multiple times Everything requires development, testing, deployment, monitoring and maintenance And this kind of architecture created untenable amount of point to point connections Benefits Continuous low-latency synchronization Centralized manageability and monitoring Track at event level data produced in all data centers Security and governance Track and control where data comes from and who is accessing it Cost Savings Move Data Once
- Messages are ordered chronologically and delivery is guaranteed. We wanted all services to have an accurate picture of the system at large. Durability, resilience and performance are all incredibly strong. We’re responsible for an enterprise system, it’s paramount we avoid outages, data loss or any form of service degradation.