SlideShare a Scribd company logo
What does a modern data
platform for analytics in the
company look like?
München, 2021-02-10, Arne Roßmann
Arne Roßmann
Education:
- Computer Science (BSc) with Focus on Business
Intelligence
- International Management (MBA)
Key Competences:
- Enterprise Architecture
- DevOps / DataOps
- Big Data / Cloud
Projects / Roles:
- Chief Architect
- DevOps Coach
- Solution Architect
- (before: Developer, Scrum Master, Project
Manager, …)
§ What are the characteristics of an Enterprise Data & AI Platform
§ How does a Reference Architecture for an Enterprise Data & AI Platform look like?
§ Why Kafka is a perfect key component in the modern platform?
§ A real world example
§ Q & A
Agenda
4
© Capgemini 2021. All rights reserved |
Modern Data Platform| Arne Roßmann | 2021-02-10
Enterprise Data &
AI Platform
Characteristics
Discoverable
Addressable
Trustworthy
(defined & monitored SLO)
Self Service
Inter Operable
(governed by open standards)
Secure
Reference Architecture for
an Enterprise Data & AI
Platform
5
Concept_Templates_April_2018.pptx
System of Insights Approach
6
NEXT-GEN ENTERPRISE DATA & AI PLATFORM
DATA CENTRIC MODELS
BUSINESS DATA HUB
DATA PROCESSING
DATA CURATION
STREAMING
SOURCES
BATCH
SOURCES
AI & ANALYTICS
FOUNDATION
SEARCH, AI & ANALYTICS
OLAP
PLATFORM MANAGEMENT & OPERATIONS
PLATFORM SECURITY
ACTIVE
DIRECTORY
ORCHESTRATION
KEY
VAULT
MFA SECURITY
CENTER
OMS
MONITOR
STREAM PROCESSING STREAM SERVING STORE
UNIVERSAL DATA LAKE
DATA TRANSFORMATION
DATA TRUST
AZURE DEVOPS
DATA INGESTION
LANDING
RAW DATA
INGESTION
TRANSFORMED DATA INGESTION
STREAM INGESTION
AI & ANALYTICS
EXECUTION
INTELLIGENT
APPs
SELF SERVICE &
ENTERPRISE BI
REAL-TIME
APPs
DATA
SERVICES
APP
DATA
SERVICES
CUSTOM AI
APPs
AI
&
ANALYTICS
SERVICES
Multiple sources converge to a common Data & AI platform using Microsoft Azure centric services
Goals: Consistent, repeatable methods and processes that can scale
Govern locally where it matters, manage centrally
AZURE DATA FACTORY
AZURE DATA FACTORY
AZURE DATABRICKS
AZURE DATABRICKS
AZURE DATABRICKS
EVENT HUB KAFKA
AZURE DATA LAKE
AZURE DATA LAKE
AZURE DATA LAKE
COSMOS DB
DATA LAKE SEARCH AZURE ML
AAS
ADLS
POWER BI
BOT
FRAMEWORK
COGNITIVE
SERVICES
APP SERVICE
LOGIC
APP
FUNCTION
SYNAPSE ANALYTICS
MGMT & SECURITY
AZURE INFORMATION PROTECTION AXON
SECURE@SOURCE ENTERPRISE DATA CATALOG
POLYBASE
AZURE SYNAPSE
ANALYTICS
API APP
Kafka as a key component
in the architecture
7
8
© Capgemini 2021. All rights reserved |
Modern Data Platform| Arne Roßmann | 2021-02-10
Event Driven Architecture – an approach to solve the Big
Ball of Mud problem
https://www.confluent.io/blog/using-apache-kafka-drive-cutting-edge-machine-learning/
https://www.confluent.io/blog/changing-face-etl/
9
© Capgemini 2021. All rights reserved |
Modern Data Platform| Arne Roßmann | 2021-02-10
Event Driven Architecture – “clean” architecture and AI
integration
https://www.confluent.io/blog/using-apache-kafka-drive-cutting-edge-machine-learning/
https://www.confluent.io/blog/changing-face-etl/
https://www.confluent.io/blog/importance-of-distributed-tracing-for-apache-kafka-based-applications/
A real world example
10
11
© Capgemini 2021. All rights reserved |
Modern Data Platform| Arne Roßmann | 2021-02-10
A real world example on Car Distribution
https://github.com/arossmann/metrics_exporter
12
© Capgemini 2021. All rights reserved |
Modern Data Platform| Arne Roßmann | 2021-02-10
Summary &
Q&A
13
© Capgemini 2021. All rights reserved |
Modern Data Platform| Arne Roßmann | 2021-02-10
A Modern
Enterprise Data &
AI Platform
The AI Engineering Data Centricity
Platform Reference Architecture
Discoverable
Addressable
Trustworthy
(defined & monitored SLO)
Self Service
Inter Operable
(governed by open standards)
Secure
AI & ANALYTICS EXECUTION
DATA CENTRICITY FOUNDATION
AI & ANALYTICS FOUNDATION
SOURCE DATA
PLATFORM FOUNDATION
DATA TRUST
Architecture & Advisory
Modern Enterprise Data & AI Platform
AI & ANALYTICS EXECUTION
AI & Analytics Execution
AI & ANALYTICS FOUNDATIONS
DATA CENTRICITY FOUNDATION
DATA
TRUST
Data
Engine
Virtual
Semantic
Layer
&
Caching
Data
Ingestion
Data
Landing
Zone
Raw
Data
Files
Batch
Ingestion
Streaming
Ingestion
Data Governance Strategy and Process
Reference &
Master Data
Data Lifecycle
Management
Data Privacy
Data Quality &
Profiling
Data Discovery
& Lineage
Data
Catalogue
Entity
Relationship
Management
Business
Glossary
Data Lake, Data
Hub, EDW
Business
Data
Hub
Modeled
data
store
Universal
Data
Lake
Curated
Data
Store
Data
Transformation
Data
Enrichment
Data
Curation
Custom AI Solutions
Intelligent Apps
AI
&
Analytics
-
Data
Services
Internal Data
External Data
CRM
ERP
ANALYTICAL
OTHER
THIRD PARTY
SOCIAL DATA
OTHER
SENSOR
IOT
AI Monitoring
Data Science Workbench
Model Catalogue
Analytics Orchestration
KPI Catalogue
Application
-
Data
Services
Traditional BI Reporting
Data
Centricity
-
Data
Services
PLATFORM FOUNDATION
Cloud Foundations I&D Platform Foundations & Run
Platform Security
& Governance
Network
Connectivity
Cyber
Security
Cloud
Infrastructure
Platform Management
& Operations
DevOps/DataOps
Automation
Hybrid Cloud
Services
Edge
Integration
Data Modeling
Aggregated Fact
APIs
Visualization
Data Exploration & Search
Analytics & Dashboards
Self-Serve BI
Reporting, BI & Visualization
14
© Capgemini 2021. All rights reserved |
Modern Data Platform| Arne Roßmann | 2021-02-10
SEC1
© 2020 Capgemini. All rights reserved.
14
Abendveranstaltung mit
Capgemini
Save the Date und sei bei unserem virtuellen
Abendprogramm am 11.02.2021 von 18:00 Uhr bis 20:00
Uhr dabei.
Es erwartet dich unter anderem eine Keynote „Coding at
Capgemini – Revolution or Evolution for Clients” von unserem
Head of Innovation Thilo Hermann.
Erhalte spannende Einblicke und Beispiele, wie Capgemini
seine Kunden revolutionär aber auch evolutionär mit Software
unterstützt.
Schnapp dir außerdem einen Drink für ein ausgelassenes
Get-Together: lerne dabei unsere Bereiche kennen und
vernetze dich mit Capgemini-Mitarbeitern.
Du hast Interesse?
Dann melde dich bis zum 11.02.2021 unter https://capgemini-
events.de/abendveranstaltung_mit_capgemini/ an.
Wir freuen uns auf dich!
15
© Capgemini 2021. All rights reserved |
Modern Data Platform| Arne Roßmann | 2021-02-10
SEC1
© 2020 Capgemini. All rights reserved.
15
Let‘s get in contact and visit us now at
our virtual booth/258
here on the OOP!
Architecting Our Future
16
© Capgemini 2021. All rights reserved |
Modern Data Platform| Arne Roßmann | 2021-02-10
Contact
Roßmann, Arne
Head of AI & Data Engineering Germany, Insights & Data
Chief Architect
Capgemini Nuremberg
Bahnhofstr. 30
90402 Nuremberg, Germany
arne.rossmann@capgemini.com
https://www.linkedin.com/in/arnerossmann/
Capgemini is a global leader in consulting, digital transformation, technology, and
engineering services. The Group is at the forefront of innovation to address the
entire breadth of clients’ opportunities in the evolving world of cloud, digital and
platforms. Building on its strong 50-year heritage and deep industry-specific
expertise, Capgemini enables organizations to realize their business ambitions
through an array of services from strategy to operations. A responsible and
multicultural company of 265,000 people in nearly 50 countries, Capgemini’s
purpose is to unleash human energy through technology for an inclusive and
sustainable future. With Altran, the Group reported 2019 combined global
revenues of €17 billion.
About Capgemini
Learn more about us at
www.capgemini.com
This presentation contains information that may be privileged or confidential
and is the property of the Capgemini Group.
Copyright © 2020 Capgemini. All rights reserved.

More Related Content

What's hot

The Data Trifecta – Privacy, Security & Governance Race from Reactivity to Re...
The Data Trifecta – Privacy, Security & Governance Race from Reactivity to Re...The Data Trifecta – Privacy, Security & Governance Race from Reactivity to Re...
The Data Trifecta – Privacy, Security & Governance Race from Reactivity to Re...
DATAVERSITY
 
Enabling a Data Mesh Architecture with Data Virtualization
Enabling a Data Mesh Architecture with Data VirtualizationEnabling a Data Mesh Architecture with Data Virtualization
Enabling a Data Mesh Architecture with Data Virtualization
Denodo
 

What's hot (20)

Snowflake: The most cost-effective agile and scalable data warehouse ever!
Snowflake: The most cost-effective agile and scalable data warehouse ever!Snowflake: The most cost-effective agile and scalable data warehouse ever!
Snowflake: The most cost-effective agile and scalable data warehouse ever!
 
Snowflake: Your Data. No Limits (Session sponsored by Snowflake) - AWS Summit...
Snowflake: Your Data. No Limits (Session sponsored by Snowflake) - AWS Summit...Snowflake: Your Data. No Limits (Session sponsored by Snowflake) - AWS Summit...
Snowflake: Your Data. No Limits (Session sponsored by Snowflake) - AWS Summit...
 
Data Warehouse - Incremental Migration to the Cloud
Data Warehouse - Incremental Migration to the CloudData Warehouse - Incremental Migration to the Cloud
Data Warehouse - Incremental Migration to the Cloud
 
Azure Synapse Analytics Overview (r1)
Azure Synapse Analytics Overview (r1)Azure Synapse Analytics Overview (r1)
Azure Synapse Analytics Overview (r1)
 
Architect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh ArchitectureArchitect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh Architecture
 
Databricks Fundamentals
Databricks FundamentalsDatabricks Fundamentals
Databricks Fundamentals
 
Data In Motion Paris 2023
Data In Motion Paris 2023Data In Motion Paris 2023
Data In Motion Paris 2023
 
Introduction to Azure Databricks
Introduction to Azure DatabricksIntroduction to Azure Databricks
Introduction to Azure Databricks
 
The Data Trifecta – Privacy, Security & Governance Race from Reactivity to Re...
The Data Trifecta – Privacy, Security & Governance Race from Reactivity to Re...The Data Trifecta – Privacy, Security & Governance Race from Reactivity to Re...
The Data Trifecta – Privacy, Security & Governance Race from Reactivity to Re...
 
Enabling a Data Mesh Architecture with Data Virtualization
Enabling a Data Mesh Architecture with Data VirtualizationEnabling a Data Mesh Architecture with Data Virtualization
Enabling a Data Mesh Architecture with Data Virtualization
 
Moving to Databricks & Delta
Moving to Databricks & DeltaMoving to Databricks & Delta
Moving to Databricks & Delta
 
Modern Data Architecture for a Data Lake with Informatica and Hortonworks Dat...
Modern Data Architecture for a Data Lake with Informatica and Hortonworks Dat...Modern Data Architecture for a Data Lake with Informatica and Hortonworks Dat...
Modern Data Architecture for a Data Lake with Informatica and Hortonworks Dat...
 
Data Quality Best Practices
Data Quality Best PracticesData Quality Best Practices
Data Quality Best Practices
 
Building a modern data warehouse
Building a modern data warehouseBuilding a modern data warehouse
Building a modern data warehouse
 
Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019)
Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019)Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019)
Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019)
 
Data Mesh
Data MeshData Mesh
Data Mesh
 
Data Lakehouse, Data Mesh, and Data Fabric (r2)
Data Lakehouse, Data Mesh, and Data Fabric (r2)Data Lakehouse, Data Mesh, and Data Fabric (r2)
Data Lakehouse, Data Mesh, and Data Fabric (r2)
 
Introduction to Azure Data Lake
Introduction to Azure Data LakeIntroduction to Azure Data Lake
Introduction to Azure Data Lake
 
Data mesh
Data meshData mesh
Data mesh
 
Azure+Databricks+Course+Slide+Deck+V4.pdf
Azure+Databricks+Course+Slide+Deck+V4.pdfAzure+Databricks+Course+Slide+Deck+V4.pdf
Azure+Databricks+Course+Slide+Deck+V4.pdf
 

Similar to Modern Data Platforms

A Journey to a Serverless Business Intelligence, Machine Learning and Big Dat...
A Journey to a Serverless Business Intelligence, Machine Learning and Big Dat...A Journey to a Serverless Business Intelligence, Machine Learning and Big Dat...
A Journey to a Serverless Business Intelligence, Machine Learning and Big Dat...
DataWorks Summit
 
Customer Presentation - IBM Cloud Pak for Data Overview (Level 100).PPTX
Customer Presentation - IBM Cloud Pak for Data Overview (Level 100).PPTXCustomer Presentation - IBM Cloud Pak for Data Overview (Level 100).PPTX
Customer Presentation - IBM Cloud Pak for Data Overview (Level 100).PPTX
tsigitnist02
 

Similar to Modern Data Platforms (20)

Meetup Spark UDF performance
Meetup Spark UDF performanceMeetup Spark UDF performance
Meetup Spark UDF performance
 
Compliant by Default - Digitaler Wandel - 14.08.2019 - Schlomo Schapiro
Compliant by Default - Digitaler Wandel - 14.08.2019 - Schlomo SchapiroCompliant by Default - Digitaler Wandel - 14.08.2019 - Schlomo Schapiro
Compliant by Default - Digitaler Wandel - 14.08.2019 - Schlomo Schapiro
 
Automating Data Lakes, Data Warehouses and Data Stores
Automating Data Lakes, Data Warehouses and Data StoresAutomating Data Lakes, Data Warehouses and Data Stores
Automating Data Lakes, Data Warehouses and Data Stores
 
The Future of Infrastructure: Key Trends to consider
The Future of Infrastructure: Key Trends to considerThe Future of Infrastructure: Key Trends to consider
The Future of Infrastructure: Key Trends to consider
 
Profile-FvanSteenveldt
Profile-FvanSteenveldtProfile-FvanSteenveldt
Profile-FvanSteenveldt
 
SharePoint Saturday Bremen - Unite your modern workplace with Microsoft's AI ...
SharePoint Saturday Bremen - Unite your modern workplace with Microsoft's AI ...SharePoint Saturday Bremen - Unite your modern workplace with Microsoft's AI ...
SharePoint Saturday Bremen - Unite your modern workplace with Microsoft's AI ...
 
A Journey to a Serverless Business Intelligence, Machine Learning and Big Dat...
A Journey to a Serverless Business Intelligence, Machine Learning and Big Dat...A Journey to a Serverless Business Intelligence, Machine Learning and Big Dat...
A Journey to a Serverless Business Intelligence, Machine Learning and Big Dat...
 
Cwin16 tls-datalab for scientists
Cwin16 tls-datalab for scientistsCwin16 tls-datalab for scientists
Cwin16 tls-datalab for scientists
 
Compliant by Default - Continuous Delivery at DB Systel - 16.10.2018 - Schlom...
Compliant by Default - Continuous Delivery at DB Systel - 16.10.2018 - Schlom...Compliant by Default - Continuous Delivery at DB Systel - 16.10.2018 - Schlom...
Compliant by Default - Continuous Delivery at DB Systel - 16.10.2018 - Schlom...
 
FAISAL SULEMAN_CV
FAISAL SULEMAN_CVFAISAL SULEMAN_CV
FAISAL SULEMAN_CV
 
CWIN17 New-York / demanding markets digital business dynamic outcomes
CWIN17 New-York / demanding markets digital business dynamic outcomesCWIN17 New-York / demanding markets digital business dynamic outcomes
CWIN17 New-York / demanding markets digital business dynamic outcomes
 
CIDEON SAP Engineering Control Center
CIDEON SAP Engineering Control CenterCIDEON SAP Engineering Control Center
CIDEON SAP Engineering Control Center
 
Platform Strategy to Deliver Digital Experiences on Azure
Platform Strategy to Deliver Digital Experiences on AzurePlatform Strategy to Deliver Digital Experiences on Azure
Platform Strategy to Deliver Digital Experiences on Azure
 
Jeff's Journey gets cloudy
Jeff's Journey gets cloudyJeff's Journey gets cloudy
Jeff's Journey gets cloudy
 
Dell boomi vs sap cpi
Dell boomi vs sap cpiDell boomi vs sap cpi
Dell boomi vs sap cpi
 
Customer Presentation - IBM Cloud Pak for Data Overview (Level 100).PPTX
Customer Presentation - IBM Cloud Pak for Data Overview (Level 100).PPTXCustomer Presentation - IBM Cloud Pak for Data Overview (Level 100).PPTX
Customer Presentation - IBM Cloud Pak for Data Overview (Level 100).PPTX
 
How To Convert Your SAP BusinessObjects Unused Licenses To SAP Analytics Cloud
How To Convert Your SAP BusinessObjects Unused Licenses To SAP Analytics CloudHow To Convert Your SAP BusinessObjects Unused Licenses To SAP Analytics Cloud
How To Convert Your SAP BusinessObjects Unused Licenses To SAP Analytics Cloud
 
Shrebo Case Study
Shrebo Case StudyShrebo Case Study
Shrebo Case Study
 
Integration architectures based on Microservices, APIs and events
Integration architectures based on Microservices,  APIs and eventsIntegration architectures based on Microservices,  APIs and events
Integration architectures based on Microservices, APIs and events
 
Top 6 Benefits of SAP Analytics Cloud – Central Hub of BI, Analytics & Planning
Top 6 Benefits of SAP Analytics Cloud – Central Hub of BI, Analytics & PlanningTop 6 Benefits of SAP Analytics Cloud – Central Hub of BI, Analytics & Planning
Top 6 Benefits of SAP Analytics Cloud – Central Hub of BI, Analytics & Planning
 

Recently uploaded

Search and Society: Reimagining Information Access for Radical Futures
Search and Society: Reimagining Information Access for Radical FuturesSearch and Society: Reimagining Information Access for Radical Futures
Search and Society: Reimagining Information Access for Radical Futures
Bhaskar Mitra
 

Recently uploaded (20)

Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........
 
"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi"Impact of front-end architecture on development cost", Viktor Turskyi
"Impact of front-end architecture on development cost", Viktor Turskyi
 
Free and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
Free and Effective: Making Flows Publicly Accessible, Yumi IbrahimzadeFree and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
Free and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
 
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya HalderCustom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder
 
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
 
Connector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonConnector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a button
 
IoT Analytics Company Presentation May 2024
IoT Analytics Company Presentation May 2024IoT Analytics Company Presentation May 2024
IoT Analytics Company Presentation May 2024
 
Introduction to Open Source RAG and RAG Evaluation
Introduction to Open Source RAG and RAG EvaluationIntroduction to Open Source RAG and RAG Evaluation
Introduction to Open Source RAG and RAG Evaluation
 
Demystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John StaveleyDemystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John Staveley
 
How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...
 
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdf
 
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
 
Search and Society: Reimagining Information Access for Radical Futures
Search and Society: Reimagining Information Access for Radical FuturesSearch and Society: Reimagining Information Access for Radical Futures
Search and Society: Reimagining Information Access for Radical Futures
 
UiPath Test Automation using UiPath Test Suite series, part 2
UiPath Test Automation using UiPath Test Suite series, part 2UiPath Test Automation using UiPath Test Suite series, part 2
UiPath Test Automation using UiPath Test Suite series, part 2
 
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
 
Optimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through ObservabilityOptimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through Observability
 
ODC, Data Fabric and Architecture User Group
ODC, Data Fabric and Architecture User GroupODC, Data Fabric and Architecture User Group
ODC, Data Fabric and Architecture User Group
 
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...
 
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
 

Modern Data Platforms

  • 1. What does a modern data platform for analytics in the company look like? München, 2021-02-10, Arne Roßmann
  • 2. Arne Roßmann Education: - Computer Science (BSc) with Focus on Business Intelligence - International Management (MBA) Key Competences: - Enterprise Architecture - DevOps / DataOps - Big Data / Cloud Projects / Roles: - Chief Architect - DevOps Coach - Solution Architect - (before: Developer, Scrum Master, Project Manager, …)
  • 3. § What are the characteristics of an Enterprise Data & AI Platform § How does a Reference Architecture for an Enterprise Data & AI Platform look like? § Why Kafka is a perfect key component in the modern platform? § A real world example § Q & A Agenda
  • 4. 4 © Capgemini 2021. All rights reserved | Modern Data Platform| Arne Roßmann | 2021-02-10 Enterprise Data & AI Platform Characteristics Discoverable Addressable Trustworthy (defined & monitored SLO) Self Service Inter Operable (governed by open standards) Secure
  • 5. Reference Architecture for an Enterprise Data & AI Platform 5
  • 6. Concept_Templates_April_2018.pptx System of Insights Approach 6 NEXT-GEN ENTERPRISE DATA & AI PLATFORM DATA CENTRIC MODELS BUSINESS DATA HUB DATA PROCESSING DATA CURATION STREAMING SOURCES BATCH SOURCES AI & ANALYTICS FOUNDATION SEARCH, AI & ANALYTICS OLAP PLATFORM MANAGEMENT & OPERATIONS PLATFORM SECURITY ACTIVE DIRECTORY ORCHESTRATION KEY VAULT MFA SECURITY CENTER OMS MONITOR STREAM PROCESSING STREAM SERVING STORE UNIVERSAL DATA LAKE DATA TRANSFORMATION DATA TRUST AZURE DEVOPS DATA INGESTION LANDING RAW DATA INGESTION TRANSFORMED DATA INGESTION STREAM INGESTION AI & ANALYTICS EXECUTION INTELLIGENT APPs SELF SERVICE & ENTERPRISE BI REAL-TIME APPs DATA SERVICES APP DATA SERVICES CUSTOM AI APPs AI & ANALYTICS SERVICES Multiple sources converge to a common Data & AI platform using Microsoft Azure centric services Goals: Consistent, repeatable methods and processes that can scale Govern locally where it matters, manage centrally AZURE DATA FACTORY AZURE DATA FACTORY AZURE DATABRICKS AZURE DATABRICKS AZURE DATABRICKS EVENT HUB KAFKA AZURE DATA LAKE AZURE DATA LAKE AZURE DATA LAKE COSMOS DB DATA LAKE SEARCH AZURE ML AAS ADLS POWER BI BOT FRAMEWORK COGNITIVE SERVICES APP SERVICE LOGIC APP FUNCTION SYNAPSE ANALYTICS MGMT & SECURITY AZURE INFORMATION PROTECTION AXON SECURE@SOURCE ENTERPRISE DATA CATALOG POLYBASE AZURE SYNAPSE ANALYTICS API APP
  • 7. Kafka as a key component in the architecture 7
  • 8. 8 © Capgemini 2021. All rights reserved | Modern Data Platform| Arne Roßmann | 2021-02-10 Event Driven Architecture – an approach to solve the Big Ball of Mud problem https://www.confluent.io/blog/using-apache-kafka-drive-cutting-edge-machine-learning/ https://www.confluent.io/blog/changing-face-etl/
  • 9. 9 © Capgemini 2021. All rights reserved | Modern Data Platform| Arne Roßmann | 2021-02-10 Event Driven Architecture – “clean” architecture and AI integration https://www.confluent.io/blog/using-apache-kafka-drive-cutting-edge-machine-learning/ https://www.confluent.io/blog/changing-face-etl/ https://www.confluent.io/blog/importance-of-distributed-tracing-for-apache-kafka-based-applications/
  • 10. A real world example 10
  • 11. 11 © Capgemini 2021. All rights reserved | Modern Data Platform| Arne Roßmann | 2021-02-10 A real world example on Car Distribution https://github.com/arossmann/metrics_exporter
  • 12. 12 © Capgemini 2021. All rights reserved | Modern Data Platform| Arne Roßmann | 2021-02-10 Summary & Q&A
  • 13. 13 © Capgemini 2021. All rights reserved | Modern Data Platform| Arne Roßmann | 2021-02-10 A Modern Enterprise Data & AI Platform The AI Engineering Data Centricity Platform Reference Architecture Discoverable Addressable Trustworthy (defined & monitored SLO) Self Service Inter Operable (governed by open standards) Secure AI & ANALYTICS EXECUTION DATA CENTRICITY FOUNDATION AI & ANALYTICS FOUNDATION SOURCE DATA PLATFORM FOUNDATION DATA TRUST Architecture & Advisory Modern Enterprise Data & AI Platform AI & ANALYTICS EXECUTION AI & Analytics Execution AI & ANALYTICS FOUNDATIONS DATA CENTRICITY FOUNDATION DATA TRUST Data Engine Virtual Semantic Layer & Caching Data Ingestion Data Landing Zone Raw Data Files Batch Ingestion Streaming Ingestion Data Governance Strategy and Process Reference & Master Data Data Lifecycle Management Data Privacy Data Quality & Profiling Data Discovery & Lineage Data Catalogue Entity Relationship Management Business Glossary Data Lake, Data Hub, EDW Business Data Hub Modeled data store Universal Data Lake Curated Data Store Data Transformation Data Enrichment Data Curation Custom AI Solutions Intelligent Apps AI & Analytics - Data Services Internal Data External Data CRM ERP ANALYTICAL OTHER THIRD PARTY SOCIAL DATA OTHER SENSOR IOT AI Monitoring Data Science Workbench Model Catalogue Analytics Orchestration KPI Catalogue Application - Data Services Traditional BI Reporting Data Centricity - Data Services PLATFORM FOUNDATION Cloud Foundations I&D Platform Foundations & Run Platform Security & Governance Network Connectivity Cyber Security Cloud Infrastructure Platform Management & Operations DevOps/DataOps Automation Hybrid Cloud Services Edge Integration Data Modeling Aggregated Fact APIs Visualization Data Exploration & Search Analytics & Dashboards Self-Serve BI Reporting, BI & Visualization
  • 14. 14 © Capgemini 2021. All rights reserved | Modern Data Platform| Arne Roßmann | 2021-02-10 SEC1 © 2020 Capgemini. All rights reserved. 14 Abendveranstaltung mit Capgemini Save the Date und sei bei unserem virtuellen Abendprogramm am 11.02.2021 von 18:00 Uhr bis 20:00 Uhr dabei. Es erwartet dich unter anderem eine Keynote „Coding at Capgemini – Revolution or Evolution for Clients” von unserem Head of Innovation Thilo Hermann. Erhalte spannende Einblicke und Beispiele, wie Capgemini seine Kunden revolutionär aber auch evolutionär mit Software unterstützt. Schnapp dir außerdem einen Drink für ein ausgelassenes Get-Together: lerne dabei unsere Bereiche kennen und vernetze dich mit Capgemini-Mitarbeitern. Du hast Interesse? Dann melde dich bis zum 11.02.2021 unter https://capgemini- events.de/abendveranstaltung_mit_capgemini/ an. Wir freuen uns auf dich!
  • 15. 15 © Capgemini 2021. All rights reserved | Modern Data Platform| Arne Roßmann | 2021-02-10 SEC1 © 2020 Capgemini. All rights reserved. 15 Let‘s get in contact and visit us now at our virtual booth/258 here on the OOP! Architecting Our Future
  • 16. 16 © Capgemini 2021. All rights reserved | Modern Data Platform| Arne Roßmann | 2021-02-10 Contact Roßmann, Arne Head of AI & Data Engineering Germany, Insights & Data Chief Architect Capgemini Nuremberg Bahnhofstr. 30 90402 Nuremberg, Germany arne.rossmann@capgemini.com https://www.linkedin.com/in/arnerossmann/
  • 17. Capgemini is a global leader in consulting, digital transformation, technology, and engineering services. The Group is at the forefront of innovation to address the entire breadth of clients’ opportunities in the evolving world of cloud, digital and platforms. Building on its strong 50-year heritage and deep industry-specific expertise, Capgemini enables organizations to realize their business ambitions through an array of services from strategy to operations. A responsible and multicultural company of 265,000 people in nearly 50 countries, Capgemini’s purpose is to unleash human energy through technology for an inclusive and sustainable future. With Altran, the Group reported 2019 combined global revenues of €17 billion. About Capgemini Learn more about us at www.capgemini.com This presentation contains information that may be privileged or confidential and is the property of the Capgemini Group. Copyright © 2020 Capgemini. All rights reserved.