SlideShare a Scribd company logo
BASEL | BERN | BRUGG | BUKAREST | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. | GENF
HAMBURG | KOPENHAGEN | LAUSANNE | MANNHEIM | MÜNCHEN | STUTTGART | WIEN | ZÜRICH
Blog.Trivadis.com@Trivadis
Provisioning of Data Platforms
Wie bringt man eine Data Analytics Plattform in die Cloud
Florian van Keulen
BASEL | BERN | BRUGG | BUKAREST | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. | GENF
HAMBURG | KOPENHAGEN | LAUSANNE | MANNHEIM | MÜNCHEN | STUTTGART | WIEN | ZÜRICH
Florian van Keulen
● Function at Trivadis:
Head of Product Design – Cloud & Security
Cloud Solution Architekt
● CV:
Studierte „Security in Distributed Systems“
bekämpfte Malware Weltweit
IT Security Officer & Cloud Architekt.
Identifiziere Chancen in der Cloud und nutze sie sicher!
● Hobbies:
Tauchen, BBQ, Woodwork…
Provisioning of Data Platforms
Provisioning of Data Platforms
Provisioning of Data Platforms - Cloud
Projekt
Projekt Details….
§ Zentralisierte, strategische Analytics Plattform
§ Sensor & Messdaten aus vielen Quellen
§ Auch für AdHoc Analytics
§ Automatisierte Daten Modelle
§ z.B. für Predictive Maintanance
§ Komplexe File / Daten Strukturen
§ Strikte und komplexe Zugriffssicherheit
Und alles „as a Service“…
NDA
Vorgehen
Daten Architekturen in Azure …
Big Data Workloads
Daten Architekturen in Azure …
Big Data Workloads
TVD Reference Architektur
TVD Reference Architektur
TVD Reference Architektur
TVD Reference Architektur
Data Ingestion
• M1 – Batch Data Ingestion • M2 – Streaming Data Ingestion
M1.1 - Automatic, Continuous Batch Data Ingestion M2.1 - Real-Time, Stream-based Data Ingestion
Data Processing
• M3 – Big Data Batch Processing • M4 – Event-/Stream-Processing
M3.1 - Spark-based Data Processing
M4.2 - Native Stream Processing
Analytics & Machine Learning
• M5 – Analytics & Machine Learning
M5.1 - Exploratory, ad-hoc Notebook-style Data Analytics M5.2 - Repetitive execution of Machine Learning algorithms
Accessing Data
• M6 – Accessing the Data Lake • M7 – Pushing Data to External Systems
M6.1 - Accessing Data through SQL M7.1 - Exporting Data into a Relational Database
Beispiel eines End-to-End Prozesses
Data Analytics
Data Processing
FTP Server
Data Ingestion
Format
Translation
Merge Data
& Perform
Analytics
Save to
redefined
data storage
Save data to
raw storage
Collect data
every
10mins
Enrich with
Metadata
Save data to
usage
optimized
data storage
Data Access
Access
through SQL
Visualize &
Reporting
Map to
Table &
Access
Control
Analytics & Machine Learning
(5.2)
Big Data Batch Processing
(M3.1)
FTP Server
Batch Data Ingestion
(M1.1)
Format
Translation
Merge Data
& Perform
Analytics
Save to
redefined
data storage
Save data to
raw storage
Collect data
every
10mins
Enrich with
Master Data
Save data to
usage
optimized
data storage
Accessing the Data Lake
(M6.1)
Access
through SQL
Visualize &
Reporting
Map to
Table &
Access
Control
Beispiel eines End-to-End Prozesses
Architektur nach Phase I
Integration
Bulk Data Flow
Create Blob
Disk Service
Analytical Platform Automation
Meta DataGeneratorTemplate
Create Blob
(Deployment)
Information Governance & Security
Event Catalog
Sync Data
Assets
Big Data Storage
Raw Zone
Trusted/Refined Zone
Usage-Optimized
Zone
Big Data Processing
Transform
Event Hub
SQL
REST / SOAP
Event Stream
Event Stream
API Call
Real-Time Big Data Processing
Stream
Analytics
Usage-Optimized
Data
Enterprise Apps
Big Data Analytics
Machine
Learning
Big Data Federation
Information
Consumer
Batch Data
Visualization
Self-Service
Analytics
EDWH
RDBMS
Data Flow
Data Science Lab
Service Bus
Business
Process
API Call
SQL
API Call
API Call
Read
Create / Delete
Read
SQL
APICall
API Call / SQL / Query
Load
Read
Create / Delete
Archival
API Call
Scheduler
API Call
Data Catalog
Containerized Apps
Microservice
SQL
API Call
Access Mgmt Encryption &
Protection
Multi-
Dimensional
ML Model
SQL / Query
Event Stream
Event Stream
Usage-
Optimized Data
Event Stream
Master Data
CRUD
Data Lineage
Master Data
Event
Handler
Event
Handler
Streaming Data
Visualization
Cleansing /
Validating
Enrichment Aggregation
Image/Video
Recognition
Timeseries
Analysis
Graph/Link
Analytics
Location
Analytics
Landing Zone
Sandbox Zone
App Marketplace
Query
Engine
API / Service
Master DataAPI / Service
Data Enterprise
App
API / Service
Archived Zone
API / Service
Azure Storage
Blob
Azure Storage
Blob
Azure Storage
Blob
Master Data
Services (MDS)
HDInsight Spark
Azure Data
Catalog
Trivadis biGENiUS
HDInsight
Kafka
Azure Functions
HDInsight
Interactive Query
StreamSets
Data Collector
Azure Cosmos
DB Azure SQL
Database
Azure Databricks
Azure Logic
Apps
Azure
Scheduler
Power BI
Tableau / SAP
BO
MATLAB
Data Catalog
Web UI
Azure Databricks
UI
Azure Data
Box
Azure
Import
StreamSets
Data Collector
Azure Time
Series Insights
Excel with
MDS Plugin
Azure Storage
Explorer
Azure Kubernetes
Service (AKS)
Azure SQL
Database
HDInsight
(Ranger)
Azure
StreamAnalytics
Spark
Streaming
Azure Event
Hub
Bulk Import
Event Stream
Edge
(Bulk) Data Flow
Stream
Analytics
Event
Stream
Bulk Data
Bulk Data
Event
Stream
Event Hub
API Call
API Call
Event
Handler
Data Sources
DB
Extract
File
Weather
DB
CDC
File
CDC
Mobile
Apps
Connected Car
Robot
Windpark
Air Traffic
Event
Message
Bulk
Stream
Service
Social
Media
Smart City
Sensor
Market Feed
Bulk Data Flow
Disk
Data Flow
Scheduler
API / Service
Control-M
FTP Server
Azure Event
Grid
Herausforderungen
Challenge: HD Insight
Hybrid Identities / Federation
Authorization in HD Insight
HD Insight Costs
Integration
Bulk Data Flow
Create Blob
Disk Service
Analytical Platform Automation
Meta DataGeneratorTemplate
Create Blob
(Deployment)
Information Governance & Security
Event Catalog
Sync Data
Assets
Big Data Storage
Raw Zone
Trusted/Refined Zone
Usage-Optimized
Zone
Big Data Processing
Transform
Event Hub
SQL
REST / SOAP
Event Stream
Event Stream
API Call
Real-Time Big Data Processing
Stream
Analytics
Usage-Optimized
Data
Enterprise Apps
Big Data Analytics
Machine
Learning
Big Data Federation
Information
Consumer
Batch Data
Visualization
Self-Service
Analytics
EDWH
RDBMS
Data Flow
Data Science Lab
Service Bus
Business
Process
API Call
SQL
API Call
API Call
Read
Create / Delete
Read
SQL
APICall
API Call / SQL / Query
Load
Read
Create / Delete
Archival
API Call
Scheduler
API Call
Data Catalog
Containerized Apps
Microservice
SQL
API Call
Access Mgmt Encryption &
Protection
Multi-
Dimensional
ML Model
SQL / Query
Event Stream
Event Stream
Usage-
Optimized Data
Event Stream
Master Data
CRUD
Data Lineage
Master Data
Event
Handler
Event
Handler
Streaming Data
Visualization
Cleansing /
Validating
Enrichment Aggregation
Image/Video
Recognition
Timeseries
Analysis
Graph/Link
Analytics
Location
Analytics
Landing Zone
Sandbox Zone
App Marketplace
Query
Engine
API / Service
Master DataAPI / Service
Data Enterprise
App
API / Service
Archived Zone
API / Service
Azure Storage
Blob
Azure Storage
Blob
Azure Storage
Blob
Master Data
Services (MDS)
HDInsight Spark
Azure Data
Catalog
Trivadis biGENiUS
HDInsight
Kafka
Azure Functions
HDInsight
Interactive Query
StreamSets
Data Collector
Azure Cosmos
DB Azure SQL
Database
Azure Databricks
Azure Logic
Apps
Azure
Scheduler
Power BI
Tableau / SAP
BO
MATLAB
Data Catalog
Web UI
Azure Databricks
UI
Azure Data
Box
Azure
Import
StreamSets
Data Collector
Azure Time
Series Insights
Excel with
MDS Plugin
Azure Storage
Explorer
Azure Kubernetes
Service (AKS)
Azure SQL
Database
HDInsight
(Ranger)
Azure
StreamAnalytics
Spark
Streaming
Azure Event
Hub
Bulk Import
Event Stream
Edge
(Bulk) Data Flow
Stream
Analytics
Event
Stream
Bulk Data
Bulk Data
Event
Stream
Event Hub
API Call
API Call
Event
Handler
Data Sources
DB
Extract
File
Weather
DB
CDC
File
CDC
Mobile
Apps
Connected Car
Robot
Windpark
Air Traffic
Event
Message
Bulk
Stream
Service
Social
Media
Smart City
Sensor
Market Feed
Bulk Data Flow
Disk
Data Flow
Scheduler
API / Service
Control-M
FTP Server
Azure Event
Grid
Challenge: HD Insight
Challenge: HD Insight - Authentication
HDInsight
Customer’s AzureAnalytics Platform Azure
Customer’s OnPrem
BigData
Storage
Azure AD
gateway
head nodeworker
node(s)
ranger
worker
node(s)
worker
node(s) Zeppelin
web services
SQL interface
other
Customer
Azure AD
Azure AD B2B
Federate
Sync/Federate
Authentication
OnPrem
Authentication/Authorization
(SAML,OAuth)
Benötigtes Setup für Azure HDInsight
Challenge: HD Insight - Authentication
HDInsight
Customer’s AzureAnalytics Platform Azure
Customer’s OnPrem
BigData
Storage
Azure AD
gateway
head nodeworker
node(s)
ranger
worker
node(s)
worker
node(s) Zeppelin
web services
SQL interface
other
Customer
Azure AD
Syncincl.Passwords
Authentication
OnPrem
Auth
Kerberos/LDAP
Azure Active
Directory
Domain
Services
Domain Join
Domain Join& Authentication
Empfohlenes Deployment von Microsoft
Challenge: HD Insight - Authentication
HDInsight
Customer’s AzureAnalytics Platform Azure
Customer’s OnPrem
BigData
Storage
Azure AD
gateway
head nodeworker
node(s)
ranger
worker
node(s)
worker
node(s) Zeppelin
web services
SQL interface
other
Customer
Azure AD
Auth
Kerberos/LDAP
Azure Active
Directory
Domain
Services
Domain Join
Synchronizing same identity
in 2 Azure ADs not possible
Syncincl.Passwords
Authentication
OnPrem
Sync
X
Möglicher Workaround 1
Challenge: HD Insight - Authentication
HDInsight
Ørsted AzureSMM Platform Azure
Ørsted OnPrem
BigData
Storage
Azure AD
Apache KNOX Gateway
head nodeworker
node(s)
ranger
worker
node(s)
worker
node(s) Zeppelin
web services
SQL interface
other
Ørsted
Azure AD
Azure AD B2B
Federate
Sync/Federate
Authentication
OnPrem
Authentication/Authorization
(SAML,OAuth)
Austausch einer
standard
HDInsight Komponente
Möglicher Workaround 2
Challenge: HD Insight
HDInsight
Customer’s AzureAnalytics Platform Azure
Customer’s OnPrem
BigData
Storage
Azure AD
gateway
head nodeworker
node(s)
ranger
worker
node(s)
worker
node(s) Zeppelin
web services
SQL interface
other
Customer
Azure AD
Provision/DeprovisionIdentities
Auth
Kerberos/LDAP
Azure Active
Directory
Domain
Services
Domain Join
Azure AD managing Identities
All Azure IAM features available
Self Service IAM additional possible
(e.g. Password reset)
provide initial credentials
Customers
Identity Manager
Script execution
Möglicher Workaround 3
Challenge: HD Insight - Kosten
HDInsight Spark
Head Node Worker
Min. 2 Nodes
HDInsight Kafka
Head Node Worker
Min. 2 Nodes
HDInsight Interactive Q
Head Node Worker
Min. 2 Nodes
3 Clusters, jeder min. 3 VMs
Kosten pro Stunde
Kein stoppen möglich, nur deprovisionieren
Challenge: HD Insight - Kosten
HDInsight Spark
Head Node Worker
Min. 2 Nodes
HDInsight Kafka
Head Node Worker
Min. 2 Nodes
HDInsight Interactive Q
Head Node Worker
Min. 2 Nodes
Challenge: Kostenmanagement
Challenge: Kostenmanagement
VM (8CPU) / hour Per GB +
Transactions
Per hour +
Outbound traffic
Instance / hour +
GB
Zone / Month +
Queries
Units / Hour +
Events
?
Challenge: Kostenmanagement
VM (8CPU) / hour Per GB +
Transactions
Per hour +
Outbound traffic
Instance / hour +
GB
Zone / Month +
Queries
Units / Hour +
Events
Tags !
Lessons Learned
Takeaways
§ HD Insight ist mächtig, aber nicht wirklich cloud aware…
§ Identity Management und Access Management für HD Insight eher traditionell
§ Kosten von HD Insight nicht unterschätzen…
§ Wo möglich automatisiert deprovisionieren und porvisionieren
§ Oder Databricks & Data Factory nutzen
§ Tagging ideal für Kosten Verteilungen
Azure Days 2019: Wie bringt man eine Data Analytics Plattform in die Cloud? (Florian van Keulen)

More Related Content

What's hot

Google Cloud Platform (GCP) At a Glance
Google Cloud Platform (GCP)  At a GlanceGoogle Cloud Platform (GCP)  At a Glance
Google Cloud Platform (GCP) At a Glance
Cloud Analogy
 
Big data in action
Big data in actionBig data in action
Big data in action
Tu Pham
 
IBM Cloud Pak for Integration with Confluent Platform powered by Apache Kafka
IBM Cloud Pak for Integration with Confluent Platform powered by Apache KafkaIBM Cloud Pak for Integration with Confluent Platform powered by Apache Kafka
IBM Cloud Pak for Integration with Confluent Platform powered by Apache Kafka
Kai Wähner
 
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
Cathrine Wilhelmsen
 
Customer Experience at Disney+ Through Data Perspective
Customer Experience at Disney+ Through Data PerspectiveCustomer Experience at Disney+ Through Data Perspective
Customer Experience at Disney+ Through Data Perspective
Databricks
 
Google Cloud Platform (GCP)
Google Cloud Platform (GCP)Google Cloud Platform (GCP)
Google Cloud Platform (GCP)
Chetan Sharma
 
Azure event hubs, Stream Analytics & Power BI (by Sam Vanhoutte)
Azure event hubs, Stream Analytics & Power BI (by Sam Vanhoutte)Azure event hubs, Stream Analytics & Power BI (by Sam Vanhoutte)
Azure event hubs, Stream Analytics & Power BI (by Sam Vanhoutte)
Codit
 
BDT201 AWS Data Pipeline - AWS re: Invent 2012
BDT201 AWS Data Pipeline - AWS re: Invent 2012BDT201 AWS Data Pipeline - AWS re: Invent 2012
BDT201 AWS Data Pipeline - AWS re: Invent 2012
Amazon Web Services
 
System Security on Cloud
System Security on CloudSystem Security on Cloud
System Security on Cloud
Tu Pham
 
Finding Meaning in the Noise: Understanding Big Data with AWS Analytics
Finding Meaning in the Noise: Understanding Big Data with AWS AnalyticsFinding Meaning in the Noise: Understanding Big Data with AWS Analytics
Finding Meaning in the Noise: Understanding Big Data with AWS Analytics
Amazon Web Services
 
A Complete Guide to the Google Cloud Platform
A Complete Guide to the Google Cloud PlatformA Complete Guide to the Google Cloud Platform
A Complete Guide to the Google Cloud Platform
BitMin Infosystems Pvt. Ltd
 
Introduction to Google Cloud Platform for Big Data - Trusted Conf
Introduction to Google Cloud Platform for Big Data - Trusted ConfIntroduction to Google Cloud Platform for Big Data - Trusted Conf
Introduction to Google Cloud Platform for Big Data - Trusted Conf
In Marketing We Trust
 
The Journey to the Hybrid Multi Cloud
The Journey to the Hybrid Multi CloudThe Journey to the Hybrid Multi Cloud
The Journey to the Hybrid Multi Cloud
Idan Tohami
 
cloud computing
cloud computingcloud computing
cloud computing
Shruti Gupta
 
Critical Breakthroughs and Challenges in Big Data and Analytics
Critical Breakthroughs and Challenges in Big Data and AnalyticsCritical Breakthroughs and Challenges in Big Data and Analytics
Critical Breakthroughs and Challenges in Big Data and Analytics
Data Driven Innovation
 
AWS re:Invent 2016: Predictive Security: Using Big Data to Fortify Your Defen...
AWS re:Invent 2016: Predictive Security: Using Big Data to Fortify Your Defen...AWS re:Invent 2016: Predictive Security: Using Big Data to Fortify Your Defen...
AWS re:Invent 2016: Predictive Security: Using Big Data to Fortify Your Defen...
Amazon Web Services
 
Journey Through the AWS Cloud - Big Data Analysis
Journey Through the AWS Cloud - Big Data AnalysisJourney Through the AWS Cloud - Big Data Analysis
Journey Through the AWS Cloud - Big Data Analysis
Amazon Web Services
 
(BDT306) Mission-Critical Stream Processing with Amazon EMR and Amazon Kinesi...
(BDT306) Mission-Critical Stream Processing with Amazon EMR and Amazon Kinesi...(BDT306) Mission-Critical Stream Processing with Amazon EMR and Amazon Kinesi...
(BDT306) Mission-Critical Stream Processing with Amazon EMR and Amazon Kinesi...
Amazon Web Services
 
Integrating Web and Business Data
Integrating Web and Business DataIntegrating Web and Business Data
Integrating Web and Business Data
Safe Software
 
Biml Tips and Tricks: Not Just for SSIS Packages! (SQLBits 2019)
Biml Tips and Tricks: Not Just for SSIS Packages! (SQLBits 2019)Biml Tips and Tricks: Not Just for SSIS Packages! (SQLBits 2019)
Biml Tips and Tricks: Not Just for SSIS Packages! (SQLBits 2019)
Cathrine Wilhelmsen
 

What's hot (20)

Google Cloud Platform (GCP) At a Glance
Google Cloud Platform (GCP)  At a GlanceGoogle Cloud Platform (GCP)  At a Glance
Google Cloud Platform (GCP) At a Glance
 
Big data in action
Big data in actionBig data in action
Big data in action
 
IBM Cloud Pak for Integration with Confluent Platform powered by Apache Kafka
IBM Cloud Pak for Integration with Confluent Platform powered by Apache KafkaIBM Cloud Pak for Integration with Confluent Platform powered by Apache Kafka
IBM Cloud Pak for Integration with Confluent Platform powered by Apache Kafka
 
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
 
Customer Experience at Disney+ Through Data Perspective
Customer Experience at Disney+ Through Data PerspectiveCustomer Experience at Disney+ Through Data Perspective
Customer Experience at Disney+ Through Data Perspective
 
Google Cloud Platform (GCP)
Google Cloud Platform (GCP)Google Cloud Platform (GCP)
Google Cloud Platform (GCP)
 
Azure event hubs, Stream Analytics & Power BI (by Sam Vanhoutte)
Azure event hubs, Stream Analytics & Power BI (by Sam Vanhoutte)Azure event hubs, Stream Analytics & Power BI (by Sam Vanhoutte)
Azure event hubs, Stream Analytics & Power BI (by Sam Vanhoutte)
 
BDT201 AWS Data Pipeline - AWS re: Invent 2012
BDT201 AWS Data Pipeline - AWS re: Invent 2012BDT201 AWS Data Pipeline - AWS re: Invent 2012
BDT201 AWS Data Pipeline - AWS re: Invent 2012
 
System Security on Cloud
System Security on CloudSystem Security on Cloud
System Security on Cloud
 
Finding Meaning in the Noise: Understanding Big Data with AWS Analytics
Finding Meaning in the Noise: Understanding Big Data with AWS AnalyticsFinding Meaning in the Noise: Understanding Big Data with AWS Analytics
Finding Meaning in the Noise: Understanding Big Data with AWS Analytics
 
A Complete Guide to the Google Cloud Platform
A Complete Guide to the Google Cloud PlatformA Complete Guide to the Google Cloud Platform
A Complete Guide to the Google Cloud Platform
 
Introduction to Google Cloud Platform for Big Data - Trusted Conf
Introduction to Google Cloud Platform for Big Data - Trusted ConfIntroduction to Google Cloud Platform for Big Data - Trusted Conf
Introduction to Google Cloud Platform for Big Data - Trusted Conf
 
The Journey to the Hybrid Multi Cloud
The Journey to the Hybrid Multi CloudThe Journey to the Hybrid Multi Cloud
The Journey to the Hybrid Multi Cloud
 
cloud computing
cloud computingcloud computing
cloud computing
 
Critical Breakthroughs and Challenges in Big Data and Analytics
Critical Breakthroughs and Challenges in Big Data and AnalyticsCritical Breakthroughs and Challenges in Big Data and Analytics
Critical Breakthroughs and Challenges in Big Data and Analytics
 
AWS re:Invent 2016: Predictive Security: Using Big Data to Fortify Your Defen...
AWS re:Invent 2016: Predictive Security: Using Big Data to Fortify Your Defen...AWS re:Invent 2016: Predictive Security: Using Big Data to Fortify Your Defen...
AWS re:Invent 2016: Predictive Security: Using Big Data to Fortify Your Defen...
 
Journey Through the AWS Cloud - Big Data Analysis
Journey Through the AWS Cloud - Big Data AnalysisJourney Through the AWS Cloud - Big Data Analysis
Journey Through the AWS Cloud - Big Data Analysis
 
(BDT306) Mission-Critical Stream Processing with Amazon EMR and Amazon Kinesi...
(BDT306) Mission-Critical Stream Processing with Amazon EMR and Amazon Kinesi...(BDT306) Mission-Critical Stream Processing with Amazon EMR and Amazon Kinesi...
(BDT306) Mission-Critical Stream Processing with Amazon EMR and Amazon Kinesi...
 
Integrating Web and Business Data
Integrating Web and Business DataIntegrating Web and Business Data
Integrating Web and Business Data
 
Biml Tips and Tricks: Not Just for SSIS Packages! (SQLBits 2019)
Biml Tips and Tricks: Not Just for SSIS Packages! (SQLBits 2019)Biml Tips and Tricks: Not Just for SSIS Packages! (SQLBits 2019)
Biml Tips and Tricks: Not Just for SSIS Packages! (SQLBits 2019)
 

Similar to Azure Days 2019: Wie bringt man eine Data Analytics Plattform in die Cloud? (Florian van Keulen)

Data Ingestion in Big Data and IoT platforms
Data Ingestion in Big Data and IoT platformsData Ingestion in Big Data and IoT platforms
Data Ingestion in Big Data and IoT platforms
Guido Schmutz
 
Bring Your Data Model Alive with Automation - Data Modeling Zone Europe 2018
Bring Your Data Model Alive with Automation - Data Modeling Zone Europe 2018 Bring Your Data Model Alive with Automation - Data Modeling Zone Europe 2018
Bring Your Data Model Alive with Automation - Data Modeling Zone Europe 2018
biGENiUS | Big Data & Data Warehouse Automation
 
Architecture of Big Data Solutions
Architecture of Big Data SolutionsArchitecture of Big Data Solutions
Architecture of Big Data Solutions
Guido Schmutz
 
Big Data: It’s all about the Use Cases
Big Data: It’s all about the Use CasesBig Data: It’s all about the Use Cases
Big Data: It’s all about the Use Cases
James Serra
 
Fundamentals Big Data and AI Architecture
Fundamentals Big Data and AI ArchitectureFundamentals Big Data and AI Architecture
Fundamentals Big Data and AI Architecture
Guido Schmutz
 
AzureML Welcome to the future of Predictive Analytics
AzureML Welcome to the future of Predictive Analytics AzureML Welcome to the future of Predictive Analytics
AzureML Welcome to the future of Predictive Analytics
Ruben Pertusa Lopez
 
Big Data Architectures @ JAX / BigDataCon 2016
Big Data Architectures @ JAX / BigDataCon 2016Big Data Architectures @ JAX / BigDataCon 2016
Big Data Architectures @ JAX / BigDataCon 2016
Guido Schmutz
 
Introduction to Stream Processing
Introduction to Stream ProcessingIntroduction to Stream Processing
Introduction to Stream Processing
Guido Schmutz
 
Big Data - in the cloud or rather on-premises?
Big Data - in the cloud or rather on-premises?Big Data - in the cloud or rather on-premises?
Big Data - in the cloud or rather on-premises?
Guido Schmutz
 
Introduction to Stream Processing
Introduction to Stream ProcessingIntroduction to Stream Processing
Introduction to Stream Processing
Guido Schmutz
 
Analytics in a Day Ft. Synapse Virtual Workshop
Analytics in a Day Ft. Synapse Virtual WorkshopAnalytics in a Day Ft. Synapse Virtual Workshop
Analytics in a Day Ft. Synapse Virtual Workshop
CCG
 
Secrets of Enterprise Data Mining 201310
Secrets of Enterprise Data Mining 201310Secrets of Enterprise Data Mining 201310
Secrets of Enterprise Data Mining 201310
Mark Tabladillo
 
Stream Processing – Concepts and Frameworks
Stream Processing – Concepts and FrameworksStream Processing – Concepts and Frameworks
Stream Processing – Concepts and Frameworks
Guido Schmutz
 
Windows Azure Platform - Jonathan Wong
Windows Azure Platform - Jonathan WongWindows Azure Platform - Jonathan Wong
Windows Azure Platform - Jonathan Wong
Spiffy
 
Continuous Delivery to Azure with VSTS
Continuous Delivery to Azure with VSTSContinuous Delivery to Azure with VSTS
Continuous Delivery to Azure with VSTS
Solidify
 
Introduction Big Data
Introduction Big DataIntroduction Big Data
Introduction Big Data
Frank Kienle
 
Spca2014 navigating clouds sp_con14_mackie
Spca2014 navigating clouds sp_con14_mackieSpca2014 navigating clouds sp_con14_mackie
Spca2014 navigating clouds sp_con14_mackie
NCCOMMS
 
MLOps Virtual Event | Building Machine Learning Platforms for the Full Lifecycle
MLOps Virtual Event | Building Machine Learning Platforms for the Full LifecycleMLOps Virtual Event | Building Machine Learning Platforms for the Full Lifecycle
MLOps Virtual Event | Building Machine Learning Platforms for the Full Lifecycle
Databricks
 
Azure Data Explorer deep dive - review 04.2020
Azure Data Explorer deep dive - review 04.2020Azure Data Explorer deep dive - review 04.2020
Azure Data Explorer deep dive - review 04.2020
Riccardo Zamana
 
Microservices+Approach+with+IBM+Cloud+Pak+for+Data+-+BACon+2019.pdf
Microservices+Approach+with+IBM+Cloud+Pak+for+Data+-+BACon+2019.pdfMicroservices+Approach+with+IBM+Cloud+Pak+for+Data+-+BACon+2019.pdf
Microservices+Approach+with+IBM+Cloud+Pak+for+Data+-+BACon+2019.pdf
AnandSivan7
 

Similar to Azure Days 2019: Wie bringt man eine Data Analytics Plattform in die Cloud? (Florian van Keulen) (20)

Data Ingestion in Big Data and IoT platforms
Data Ingestion in Big Data and IoT platformsData Ingestion in Big Data and IoT platforms
Data Ingestion in Big Data and IoT platforms
 
Bring Your Data Model Alive with Automation - Data Modeling Zone Europe 2018
Bring Your Data Model Alive with Automation - Data Modeling Zone Europe 2018 Bring Your Data Model Alive with Automation - Data Modeling Zone Europe 2018
Bring Your Data Model Alive with Automation - Data Modeling Zone Europe 2018
 
Architecture of Big Data Solutions
Architecture of Big Data SolutionsArchitecture of Big Data Solutions
Architecture of Big Data Solutions
 
Big Data: It’s all about the Use Cases
Big Data: It’s all about the Use CasesBig Data: It’s all about the Use Cases
Big Data: It’s all about the Use Cases
 
Fundamentals Big Data and AI Architecture
Fundamentals Big Data and AI ArchitectureFundamentals Big Data and AI Architecture
Fundamentals Big Data and AI Architecture
 
AzureML Welcome to the future of Predictive Analytics
AzureML Welcome to the future of Predictive Analytics AzureML Welcome to the future of Predictive Analytics
AzureML Welcome to the future of Predictive Analytics
 
Big Data Architectures @ JAX / BigDataCon 2016
Big Data Architectures @ JAX / BigDataCon 2016Big Data Architectures @ JAX / BigDataCon 2016
Big Data Architectures @ JAX / BigDataCon 2016
 
Introduction to Stream Processing
Introduction to Stream ProcessingIntroduction to Stream Processing
Introduction to Stream Processing
 
Big Data - in the cloud or rather on-premises?
Big Data - in the cloud or rather on-premises?Big Data - in the cloud or rather on-premises?
Big Data - in the cloud or rather on-premises?
 
Introduction to Stream Processing
Introduction to Stream ProcessingIntroduction to Stream Processing
Introduction to Stream Processing
 
Analytics in a Day Ft. Synapse Virtual Workshop
Analytics in a Day Ft. Synapse Virtual WorkshopAnalytics in a Day Ft. Synapse Virtual Workshop
Analytics in a Day Ft. Synapse Virtual Workshop
 
Secrets of Enterprise Data Mining 201310
Secrets of Enterprise Data Mining 201310Secrets of Enterprise Data Mining 201310
Secrets of Enterprise Data Mining 201310
 
Stream Processing – Concepts and Frameworks
Stream Processing – Concepts and FrameworksStream Processing – Concepts and Frameworks
Stream Processing – Concepts and Frameworks
 
Windows Azure Platform - Jonathan Wong
Windows Azure Platform - Jonathan WongWindows Azure Platform - Jonathan Wong
Windows Azure Platform - Jonathan Wong
 
Continuous Delivery to Azure with VSTS
Continuous Delivery to Azure with VSTSContinuous Delivery to Azure with VSTS
Continuous Delivery to Azure with VSTS
 
Introduction Big Data
Introduction Big DataIntroduction Big Data
Introduction Big Data
 
Spca2014 navigating clouds sp_con14_mackie
Spca2014 navigating clouds sp_con14_mackieSpca2014 navigating clouds sp_con14_mackie
Spca2014 navigating clouds sp_con14_mackie
 
MLOps Virtual Event | Building Machine Learning Platforms for the Full Lifecycle
MLOps Virtual Event | Building Machine Learning Platforms for the Full LifecycleMLOps Virtual Event | Building Machine Learning Platforms for the Full Lifecycle
MLOps Virtual Event | Building Machine Learning Platforms for the Full Lifecycle
 
Azure Data Explorer deep dive - review 04.2020
Azure Data Explorer deep dive - review 04.2020Azure Data Explorer deep dive - review 04.2020
Azure Data Explorer deep dive - review 04.2020
 
Microservices+Approach+with+IBM+Cloud+Pak+for+Data+-+BACon+2019.pdf
Microservices+Approach+with+IBM+Cloud+Pak+for+Data+-+BACon+2019.pdfMicroservices+Approach+with+IBM+Cloud+Pak+for+Data+-+BACon+2019.pdf
Microservices+Approach+with+IBM+Cloud+Pak+for+Data+-+BACon+2019.pdf
 

More from Trivadis

Azure Days 2019: Azure Chatbot Development for Airline Irregularities (Remco ...
Azure Days 2019: Azure Chatbot Development for Airline Irregularities (Remco ...Azure Days 2019: Azure Chatbot Development for Airline Irregularities (Remco ...
Azure Days 2019: Azure Chatbot Development for Airline Irregularities (Remco ...
Trivadis
 
Azure Days 2019: Trivadis Azure Foundation – Das Fundament für den ... (Nisan...
Azure Days 2019: Trivadis Azure Foundation – Das Fundament für den ... (Nisan...Azure Days 2019: Trivadis Azure Foundation – Das Fundament für den ... (Nisan...
Azure Days 2019: Trivadis Azure Foundation – Das Fundament für den ... (Nisan...
Trivadis
 
Azure Days 2019: Business Intelligence auf Azure (Marco Amhof & Yves Mauron)
Azure Days 2019: Business Intelligence auf Azure (Marco Amhof & Yves Mauron)Azure Days 2019: Business Intelligence auf Azure (Marco Amhof & Yves Mauron)
Azure Days 2019: Business Intelligence auf Azure (Marco Amhof & Yves Mauron)
Trivadis
 
Azure Days 2019: Master the Move to Azure (Konrad Brunner)
Azure Days 2019: Master the Move to Azure (Konrad Brunner)Azure Days 2019: Master the Move to Azure (Konrad Brunner)
Azure Days 2019: Master the Move to Azure (Konrad Brunner)
Trivadis
 
Azure Days 2019: Keynote Azure Switzerland – Status Quo und Ausblick (Primo A...
Azure Days 2019: Keynote Azure Switzerland – Status Quo und Ausblick (Primo A...Azure Days 2019: Keynote Azure Switzerland – Status Quo und Ausblick (Primo A...
Azure Days 2019: Keynote Azure Switzerland – Status Quo und Ausblick (Primo A...
Trivadis
 
Azure Days 2019: Grösser und Komplexer ist nicht immer besser (Meinrad Weiss)
Azure Days 2019: Grösser und Komplexer ist nicht immer besser (Meinrad Weiss)Azure Days 2019: Grösser und Komplexer ist nicht immer besser (Meinrad Weiss)
Azure Days 2019: Grösser und Komplexer ist nicht immer besser (Meinrad Weiss)
Trivadis
 
Azure Days 2019: Get Connected with Azure API Management (Gerry Keune & Stefa...
Azure Days 2019: Get Connected with Azure API Management (Gerry Keune & Stefa...Azure Days 2019: Get Connected with Azure API Management (Gerry Keune & Stefa...
Azure Days 2019: Get Connected with Azure API Management (Gerry Keune & Stefa...
Trivadis
 
Azure Days 2019: Infrastructure as Code auf Azure (Jonas Wanninger & Daniel H...
Azure Days 2019: Infrastructure as Code auf Azure (Jonas Wanninger & Daniel H...Azure Days 2019: Infrastructure as Code auf Azure (Jonas Wanninger & Daniel H...
Azure Days 2019: Infrastructure as Code auf Azure (Jonas Wanninger & Daniel H...
Trivadis
 
Azure Days 2019: Azure@Helsana: Die Erweiterung von Dynamics CRM mit Azure Po...
Azure Days 2019: Azure@Helsana: Die Erweiterung von Dynamics CRM mit Azure Po...Azure Days 2019: Azure@Helsana: Die Erweiterung von Dynamics CRM mit Azure Po...
Azure Days 2019: Azure@Helsana: Die Erweiterung von Dynamics CRM mit Azure Po...
Trivadis
 
TechEvent 2019: Kundenstory - Kein Angebot, kein Auftrag – Wie Du ein individ...
TechEvent 2019: Kundenstory - Kein Angebot, kein Auftrag – Wie Du ein individ...TechEvent 2019: Kundenstory - Kein Angebot, kein Auftrag – Wie Du ein individ...
TechEvent 2019: Kundenstory - Kein Angebot, kein Auftrag – Wie Du ein individ...
Trivadis
 
TechEvent 2019: Oracle Database Appliance M/L - Erfahrungen und Erfolgsmethod...
TechEvent 2019: Oracle Database Appliance M/L - Erfahrungen und Erfolgsmethod...TechEvent 2019: Oracle Database Appliance M/L - Erfahrungen und Erfolgsmethod...
TechEvent 2019: Oracle Database Appliance M/L - Erfahrungen und Erfolgsmethod...
Trivadis
 
TechEvent 2019: Security 101 für Web Entwickler; Roland Krüger - Trivadis
TechEvent 2019: Security 101 für Web Entwickler; Roland Krüger - TrivadisTechEvent 2019: Security 101 für Web Entwickler; Roland Krüger - Trivadis
TechEvent 2019: Security 101 für Web Entwickler; Roland Krüger - Trivadis
Trivadis
 
TechEvent 2019: Trivadis & Swisscom Partner Angebote; Konrad Häfeli, Markus O...
TechEvent 2019: Trivadis & Swisscom Partner Angebote; Konrad Häfeli, Markus O...TechEvent 2019: Trivadis & Swisscom Partner Angebote; Konrad Häfeli, Markus O...
TechEvent 2019: Trivadis & Swisscom Partner Angebote; Konrad Häfeli, Markus O...
Trivadis
 
TechEvent 2019: DBaaS from Swisscom Cloud powered by Trivadis; Konrad Häfeli ...
TechEvent 2019: DBaaS from Swisscom Cloud powered by Trivadis; Konrad Häfeli ...TechEvent 2019: DBaaS from Swisscom Cloud powered by Trivadis; Konrad Häfeli ...
TechEvent 2019: DBaaS from Swisscom Cloud powered by Trivadis; Konrad Häfeli ...
Trivadis
 
TechEvent 2019: Status of the partnership Trivadis and EDB - Comparing Postgr...
TechEvent 2019: Status of the partnership Trivadis and EDB - Comparing Postgr...TechEvent 2019: Status of the partnership Trivadis and EDB - Comparing Postgr...
TechEvent 2019: Status of the partnership Trivadis and EDB - Comparing Postgr...
Trivadis
 
TechEvent 2019: More Agile, More AI, More Cloud! Less Work?!; Oliver Dörr - T...
TechEvent 2019: More Agile, More AI, More Cloud! Less Work?!; Oliver Dörr - T...TechEvent 2019: More Agile, More AI, More Cloud! Less Work?!; Oliver Dörr - T...
TechEvent 2019: More Agile, More AI, More Cloud! Less Work?!; Oliver Dörr - T...
Trivadis
 
TechEvent 2019: Kundenstory - Vom Hauptmann zu Köpenick zum Polizisten 2020 -...
TechEvent 2019: Kundenstory - Vom Hauptmann zu Köpenick zum Polizisten 2020 -...TechEvent 2019: Kundenstory - Vom Hauptmann zu Köpenick zum Polizisten 2020 -...
TechEvent 2019: Kundenstory - Vom Hauptmann zu Köpenick zum Polizisten 2020 -...
Trivadis
 
TechEvent 2019: Vom Rechenzentrum in die Oracle Cloud - Übertragungsmethoden;...
TechEvent 2019: Vom Rechenzentrum in die Oracle Cloud - Übertragungsmethoden;...TechEvent 2019: Vom Rechenzentrum in die Oracle Cloud - Übertragungsmethoden;...
TechEvent 2019: Vom Rechenzentrum in die Oracle Cloud - Übertragungsmethoden;...
Trivadis
 
TechEvent 2019: The sleeping Power of Data; Eberhard Lösch - Trivadis
TechEvent 2019: The sleeping Power of Data; Eberhard Lösch - TrivadisTechEvent 2019: The sleeping Power of Data; Eberhard Lösch - Trivadis
TechEvent 2019: The sleeping Power of Data; Eberhard Lösch - Trivadis
Trivadis
 
TechEvent 2019: Tales from a Scrum Master; Ernst Jakob - Trivadis
TechEvent 2019: Tales from a Scrum Master; Ernst Jakob - TrivadisTechEvent 2019: Tales from a Scrum Master; Ernst Jakob - Trivadis
TechEvent 2019: Tales from a Scrum Master; Ernst Jakob - Trivadis
Trivadis
 

More from Trivadis (20)

Azure Days 2019: Azure Chatbot Development for Airline Irregularities (Remco ...
Azure Days 2019: Azure Chatbot Development for Airline Irregularities (Remco ...Azure Days 2019: Azure Chatbot Development for Airline Irregularities (Remco ...
Azure Days 2019: Azure Chatbot Development for Airline Irregularities (Remco ...
 
Azure Days 2019: Trivadis Azure Foundation – Das Fundament für den ... (Nisan...
Azure Days 2019: Trivadis Azure Foundation – Das Fundament für den ... (Nisan...Azure Days 2019: Trivadis Azure Foundation – Das Fundament für den ... (Nisan...
Azure Days 2019: Trivadis Azure Foundation – Das Fundament für den ... (Nisan...
 
Azure Days 2019: Business Intelligence auf Azure (Marco Amhof & Yves Mauron)
Azure Days 2019: Business Intelligence auf Azure (Marco Amhof & Yves Mauron)Azure Days 2019: Business Intelligence auf Azure (Marco Amhof & Yves Mauron)
Azure Days 2019: Business Intelligence auf Azure (Marco Amhof & Yves Mauron)
 
Azure Days 2019: Master the Move to Azure (Konrad Brunner)
Azure Days 2019: Master the Move to Azure (Konrad Brunner)Azure Days 2019: Master the Move to Azure (Konrad Brunner)
Azure Days 2019: Master the Move to Azure (Konrad Brunner)
 
Azure Days 2019: Keynote Azure Switzerland – Status Quo und Ausblick (Primo A...
Azure Days 2019: Keynote Azure Switzerland – Status Quo und Ausblick (Primo A...Azure Days 2019: Keynote Azure Switzerland – Status Quo und Ausblick (Primo A...
Azure Days 2019: Keynote Azure Switzerland – Status Quo und Ausblick (Primo A...
 
Azure Days 2019: Grösser und Komplexer ist nicht immer besser (Meinrad Weiss)
Azure Days 2019: Grösser und Komplexer ist nicht immer besser (Meinrad Weiss)Azure Days 2019: Grösser und Komplexer ist nicht immer besser (Meinrad Weiss)
Azure Days 2019: Grösser und Komplexer ist nicht immer besser (Meinrad Weiss)
 
Azure Days 2019: Get Connected with Azure API Management (Gerry Keune & Stefa...
Azure Days 2019: Get Connected with Azure API Management (Gerry Keune & Stefa...Azure Days 2019: Get Connected with Azure API Management (Gerry Keune & Stefa...
Azure Days 2019: Get Connected with Azure API Management (Gerry Keune & Stefa...
 
Azure Days 2019: Infrastructure as Code auf Azure (Jonas Wanninger & Daniel H...
Azure Days 2019: Infrastructure as Code auf Azure (Jonas Wanninger & Daniel H...Azure Days 2019: Infrastructure as Code auf Azure (Jonas Wanninger & Daniel H...
Azure Days 2019: Infrastructure as Code auf Azure (Jonas Wanninger & Daniel H...
 
Azure Days 2019: Azure@Helsana: Die Erweiterung von Dynamics CRM mit Azure Po...
Azure Days 2019: Azure@Helsana: Die Erweiterung von Dynamics CRM mit Azure Po...Azure Days 2019: Azure@Helsana: Die Erweiterung von Dynamics CRM mit Azure Po...
Azure Days 2019: Azure@Helsana: Die Erweiterung von Dynamics CRM mit Azure Po...
 
TechEvent 2019: Kundenstory - Kein Angebot, kein Auftrag – Wie Du ein individ...
TechEvent 2019: Kundenstory - Kein Angebot, kein Auftrag – Wie Du ein individ...TechEvent 2019: Kundenstory - Kein Angebot, kein Auftrag – Wie Du ein individ...
TechEvent 2019: Kundenstory - Kein Angebot, kein Auftrag – Wie Du ein individ...
 
TechEvent 2019: Oracle Database Appliance M/L - Erfahrungen und Erfolgsmethod...
TechEvent 2019: Oracle Database Appliance M/L - Erfahrungen und Erfolgsmethod...TechEvent 2019: Oracle Database Appliance M/L - Erfahrungen und Erfolgsmethod...
TechEvent 2019: Oracle Database Appliance M/L - Erfahrungen und Erfolgsmethod...
 
TechEvent 2019: Security 101 für Web Entwickler; Roland Krüger - Trivadis
TechEvent 2019: Security 101 für Web Entwickler; Roland Krüger - TrivadisTechEvent 2019: Security 101 für Web Entwickler; Roland Krüger - Trivadis
TechEvent 2019: Security 101 für Web Entwickler; Roland Krüger - Trivadis
 
TechEvent 2019: Trivadis & Swisscom Partner Angebote; Konrad Häfeli, Markus O...
TechEvent 2019: Trivadis & Swisscom Partner Angebote; Konrad Häfeli, Markus O...TechEvent 2019: Trivadis & Swisscom Partner Angebote; Konrad Häfeli, Markus O...
TechEvent 2019: Trivadis & Swisscom Partner Angebote; Konrad Häfeli, Markus O...
 
TechEvent 2019: DBaaS from Swisscom Cloud powered by Trivadis; Konrad Häfeli ...
TechEvent 2019: DBaaS from Swisscom Cloud powered by Trivadis; Konrad Häfeli ...TechEvent 2019: DBaaS from Swisscom Cloud powered by Trivadis; Konrad Häfeli ...
TechEvent 2019: DBaaS from Swisscom Cloud powered by Trivadis; Konrad Häfeli ...
 
TechEvent 2019: Status of the partnership Trivadis and EDB - Comparing Postgr...
TechEvent 2019: Status of the partnership Trivadis and EDB - Comparing Postgr...TechEvent 2019: Status of the partnership Trivadis and EDB - Comparing Postgr...
TechEvent 2019: Status of the partnership Trivadis and EDB - Comparing Postgr...
 
TechEvent 2019: More Agile, More AI, More Cloud! Less Work?!; Oliver Dörr - T...
TechEvent 2019: More Agile, More AI, More Cloud! Less Work?!; Oliver Dörr - T...TechEvent 2019: More Agile, More AI, More Cloud! Less Work?!; Oliver Dörr - T...
TechEvent 2019: More Agile, More AI, More Cloud! Less Work?!; Oliver Dörr - T...
 
TechEvent 2019: Kundenstory - Vom Hauptmann zu Köpenick zum Polizisten 2020 -...
TechEvent 2019: Kundenstory - Vom Hauptmann zu Köpenick zum Polizisten 2020 -...TechEvent 2019: Kundenstory - Vom Hauptmann zu Köpenick zum Polizisten 2020 -...
TechEvent 2019: Kundenstory - Vom Hauptmann zu Köpenick zum Polizisten 2020 -...
 
TechEvent 2019: Vom Rechenzentrum in die Oracle Cloud - Übertragungsmethoden;...
TechEvent 2019: Vom Rechenzentrum in die Oracle Cloud - Übertragungsmethoden;...TechEvent 2019: Vom Rechenzentrum in die Oracle Cloud - Übertragungsmethoden;...
TechEvent 2019: Vom Rechenzentrum in die Oracle Cloud - Übertragungsmethoden;...
 
TechEvent 2019: The sleeping Power of Data; Eberhard Lösch - Trivadis
TechEvent 2019: The sleeping Power of Data; Eberhard Lösch - TrivadisTechEvent 2019: The sleeping Power of Data; Eberhard Lösch - Trivadis
TechEvent 2019: The sleeping Power of Data; Eberhard Lösch - Trivadis
 
TechEvent 2019: Tales from a Scrum Master; Ernst Jakob - Trivadis
TechEvent 2019: Tales from a Scrum Master; Ernst Jakob - TrivadisTechEvent 2019: Tales from a Scrum Master; Ernst Jakob - Trivadis
TechEvent 2019: Tales from a Scrum Master; Ernst Jakob - Trivadis
 

Recently uploaded

Uncharted Together- Navigating AI's New Frontiers in Libraries
Uncharted Together- Navigating AI's New Frontiers in LibrariesUncharted Together- Navigating AI's New Frontiers in Libraries
Uncharted Together- Navigating AI's New Frontiers in Libraries
Brian Pichman
 
Mastering Board Best Practices: Essential Skills for Effective Non-profit Lea...
Mastering Board Best Practices: Essential Skills for Effective Non-profit Lea...Mastering Board Best Practices: Essential Skills for Effective Non-profit Lea...
Mastering Board Best Practices: Essential Skills for Effective Non-profit Lea...
OnBoard
 
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
alexjohnson7307
 
Finetuning GenAI For Hacking and Defending
Finetuning GenAI For Hacking and DefendingFinetuning GenAI For Hacking and Defending
Finetuning GenAI For Hacking and Defending
Priyanka Aash
 
Tailored CRM Software Development for Enhanced Customer Insights
Tailored CRM Software Development for Enhanced Customer InsightsTailored CRM Software Development for Enhanced Customer Insights
Tailored CRM Software Development for Enhanced Customer Insights
SynapseIndia
 
Semantic-Aware Code Model: Elevating the Future of Software Development
Semantic-Aware Code Model: Elevating the Future of Software DevelopmentSemantic-Aware Code Model: Elevating the Future of Software Development
Semantic-Aware Code Model: Elevating the Future of Software Development
Baishakhi Ray
 
UX Webinar Series: Aligning Authentication Experiences with Business Goals
UX Webinar Series: Aligning Authentication Experiences with Business GoalsUX Webinar Series: Aligning Authentication Experiences with Business Goals
UX Webinar Series: Aligning Authentication Experiences with Business Goals
FIDO Alliance
 
Integrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecaseIntegrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecase
shyamraj55
 
Intel Unveils Core Ultra 200V Lunar chip .pdf
Intel Unveils Core Ultra 200V Lunar chip .pdfIntel Unveils Core Ultra 200V Lunar chip .pdf
Intel Unveils Core Ultra 200V Lunar chip .pdf
Tech Guru
 
Redefining Cybersecurity with AI Capabilities
Redefining Cybersecurity with AI CapabilitiesRedefining Cybersecurity with AI Capabilities
Redefining Cybersecurity with AI Capabilities
Priyanka Aash
 
Zaitechno Handheld Raman Spectrometer.pdf
Zaitechno Handheld Raman Spectrometer.pdfZaitechno Handheld Raman Spectrometer.pdf
Zaitechno Handheld Raman Spectrometer.pdf
AmandaCheung15
 
Mastering OnlyFans Clone App Development: Key Strategies for Success
Mastering OnlyFans Clone App Development: Key Strategies for SuccessMastering OnlyFans Clone App Development: Key Strategies for Success
Mastering OnlyFans Clone App Development: Key Strategies for Success
David Wilson
 
Vulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive OverviewVulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive Overview
Steven Carlson
 
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
FIDO Alliance
 
Perth MuleSoft Meetup July 2024
Perth MuleSoft Meetup July 2024Perth MuleSoft Meetup July 2024
Perth MuleSoft Meetup July 2024
Michael Price
 
Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024
Nicolás Lopéz
 
Keynote : AI & Future Of Offensive Security
Keynote : AI & Future Of Offensive SecurityKeynote : AI & Future Of Offensive Security
Keynote : AI & Future Of Offensive Security
Priyanka Aash
 
Types of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technologyTypes of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technology
ldtexsolbl
 
NVIDIA at Breakthrough Discuss for Space Exploration
NVIDIA at Breakthrough Discuss for Space ExplorationNVIDIA at Breakthrough Discuss for Space Exploration
NVIDIA at Breakthrough Discuss for Space Exploration
Alison B. Lowndes
 
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
alexjohnson7307
 

Recently uploaded (20)

Uncharted Together- Navigating AI's New Frontiers in Libraries
Uncharted Together- Navigating AI's New Frontiers in LibrariesUncharted Together- Navigating AI's New Frontiers in Libraries
Uncharted Together- Navigating AI's New Frontiers in Libraries
 
Mastering Board Best Practices: Essential Skills for Effective Non-profit Lea...
Mastering Board Best Practices: Essential Skills for Effective Non-profit Lea...Mastering Board Best Practices: Essential Skills for Effective Non-profit Lea...
Mastering Board Best Practices: Essential Skills for Effective Non-profit Lea...
 
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
 
Finetuning GenAI For Hacking and Defending
Finetuning GenAI For Hacking and DefendingFinetuning GenAI For Hacking and Defending
Finetuning GenAI For Hacking and Defending
 
Tailored CRM Software Development for Enhanced Customer Insights
Tailored CRM Software Development for Enhanced Customer InsightsTailored CRM Software Development for Enhanced Customer Insights
Tailored CRM Software Development for Enhanced Customer Insights
 
Semantic-Aware Code Model: Elevating the Future of Software Development
Semantic-Aware Code Model: Elevating the Future of Software DevelopmentSemantic-Aware Code Model: Elevating the Future of Software Development
Semantic-Aware Code Model: Elevating the Future of Software Development
 
UX Webinar Series: Aligning Authentication Experiences with Business Goals
UX Webinar Series: Aligning Authentication Experiences with Business GoalsUX Webinar Series: Aligning Authentication Experiences with Business Goals
UX Webinar Series: Aligning Authentication Experiences with Business Goals
 
Integrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecaseIntegrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecase
 
Intel Unveils Core Ultra 200V Lunar chip .pdf
Intel Unveils Core Ultra 200V Lunar chip .pdfIntel Unveils Core Ultra 200V Lunar chip .pdf
Intel Unveils Core Ultra 200V Lunar chip .pdf
 
Redefining Cybersecurity with AI Capabilities
Redefining Cybersecurity with AI CapabilitiesRedefining Cybersecurity with AI Capabilities
Redefining Cybersecurity with AI Capabilities
 
Zaitechno Handheld Raman Spectrometer.pdf
Zaitechno Handheld Raman Spectrometer.pdfZaitechno Handheld Raman Spectrometer.pdf
Zaitechno Handheld Raman Spectrometer.pdf
 
Mastering OnlyFans Clone App Development: Key Strategies for Success
Mastering OnlyFans Clone App Development: Key Strategies for SuccessMastering OnlyFans Clone App Development: Key Strategies for Success
Mastering OnlyFans Clone App Development: Key Strategies for Success
 
Vulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive OverviewVulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive Overview
 
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
 
Perth MuleSoft Meetup July 2024
Perth MuleSoft Meetup July 2024Perth MuleSoft Meetup July 2024
Perth MuleSoft Meetup July 2024
 
Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024
 
Keynote : AI & Future Of Offensive Security
Keynote : AI & Future Of Offensive SecurityKeynote : AI & Future Of Offensive Security
Keynote : AI & Future Of Offensive Security
 
Types of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technologyTypes of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technology
 
NVIDIA at Breakthrough Discuss for Space Exploration
NVIDIA at Breakthrough Discuss for Space ExplorationNVIDIA at Breakthrough Discuss for Space Exploration
NVIDIA at Breakthrough Discuss for Space Exploration
 
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
 

Azure Days 2019: Wie bringt man eine Data Analytics Plattform in die Cloud? (Florian van Keulen)

  • 1. BASEL | BERN | BRUGG | BUKAREST | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. | GENF HAMBURG | KOPENHAGEN | LAUSANNE | MANNHEIM | MÜNCHEN | STUTTGART | WIEN | ZÜRICH Blog.Trivadis.com@Trivadis Provisioning of Data Platforms Wie bringt man eine Data Analytics Plattform in die Cloud Florian van Keulen
  • 2. BASEL | BERN | BRUGG | BUKAREST | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. | GENF HAMBURG | KOPENHAGEN | LAUSANNE | MANNHEIM | MÜNCHEN | STUTTGART | WIEN | ZÜRICH Florian van Keulen ● Function at Trivadis: Head of Product Design – Cloud & Security Cloud Solution Architekt ● CV: Studierte „Security in Distributed Systems“ bekämpfte Malware Weltweit IT Security Officer & Cloud Architekt. Identifiziere Chancen in der Cloud und nutze sie sicher! ● Hobbies: Tauchen, BBQ, Woodwork…
  • 5. Provisioning of Data Platforms - Cloud
  • 7. Projekt Details…. § Zentralisierte, strategische Analytics Plattform § Sensor & Messdaten aus vielen Quellen § Auch für AdHoc Analytics § Automatisierte Daten Modelle § z.B. für Predictive Maintanance § Komplexe File / Daten Strukturen § Strikte und komplexe Zugriffssicherheit Und alles „as a Service“… NDA
  • 9. Daten Architekturen in Azure … Big Data Workloads
  • 10. Daten Architekturen in Azure … Big Data Workloads
  • 15. Data Ingestion • M1 – Batch Data Ingestion • M2 – Streaming Data Ingestion M1.1 - Automatic, Continuous Batch Data Ingestion M2.1 - Real-Time, Stream-based Data Ingestion
  • 16. Data Processing • M3 – Big Data Batch Processing • M4 – Event-/Stream-Processing M3.1 - Spark-based Data Processing M4.2 - Native Stream Processing
  • 17. Analytics & Machine Learning • M5 – Analytics & Machine Learning M5.1 - Exploratory, ad-hoc Notebook-style Data Analytics M5.2 - Repetitive execution of Machine Learning algorithms
  • 18. Accessing Data • M6 – Accessing the Data Lake • M7 – Pushing Data to External Systems M6.1 - Accessing Data through SQL M7.1 - Exporting Data into a Relational Database
  • 19. Beispiel eines End-to-End Prozesses Data Analytics Data Processing FTP Server Data Ingestion Format Translation Merge Data & Perform Analytics Save to redefined data storage Save data to raw storage Collect data every 10mins Enrich with Metadata Save data to usage optimized data storage Data Access Access through SQL Visualize & Reporting Map to Table & Access Control
  • 20. Analytics & Machine Learning (5.2) Big Data Batch Processing (M3.1) FTP Server Batch Data Ingestion (M1.1) Format Translation Merge Data & Perform Analytics Save to redefined data storage Save data to raw storage Collect data every 10mins Enrich with Master Data Save data to usage optimized data storage Accessing the Data Lake (M6.1) Access through SQL Visualize & Reporting Map to Table & Access Control Beispiel eines End-to-End Prozesses
  • 21. Architektur nach Phase I Integration Bulk Data Flow Create Blob Disk Service Analytical Platform Automation Meta DataGeneratorTemplate Create Blob (Deployment) Information Governance & Security Event Catalog Sync Data Assets Big Data Storage Raw Zone Trusted/Refined Zone Usage-Optimized Zone Big Data Processing Transform Event Hub SQL REST / SOAP Event Stream Event Stream API Call Real-Time Big Data Processing Stream Analytics Usage-Optimized Data Enterprise Apps Big Data Analytics Machine Learning Big Data Federation Information Consumer Batch Data Visualization Self-Service Analytics EDWH RDBMS Data Flow Data Science Lab Service Bus Business Process API Call SQL API Call API Call Read Create / Delete Read SQL APICall API Call / SQL / Query Load Read Create / Delete Archival API Call Scheduler API Call Data Catalog Containerized Apps Microservice SQL API Call Access Mgmt Encryption & Protection Multi- Dimensional ML Model SQL / Query Event Stream Event Stream Usage- Optimized Data Event Stream Master Data CRUD Data Lineage Master Data Event Handler Event Handler Streaming Data Visualization Cleansing / Validating Enrichment Aggregation Image/Video Recognition Timeseries Analysis Graph/Link Analytics Location Analytics Landing Zone Sandbox Zone App Marketplace Query Engine API / Service Master DataAPI / Service Data Enterprise App API / Service Archived Zone API / Service Azure Storage Blob Azure Storage Blob Azure Storage Blob Master Data Services (MDS) HDInsight Spark Azure Data Catalog Trivadis biGENiUS HDInsight Kafka Azure Functions HDInsight Interactive Query StreamSets Data Collector Azure Cosmos DB Azure SQL Database Azure Databricks Azure Logic Apps Azure Scheduler Power BI Tableau / SAP BO MATLAB Data Catalog Web UI Azure Databricks UI Azure Data Box Azure Import StreamSets Data Collector Azure Time Series Insights Excel with MDS Plugin Azure Storage Explorer Azure Kubernetes Service (AKS) Azure SQL Database HDInsight (Ranger) Azure StreamAnalytics Spark Streaming Azure Event Hub Bulk Import Event Stream Edge (Bulk) Data Flow Stream Analytics Event Stream Bulk Data Bulk Data Event Stream Event Hub API Call API Call Event Handler Data Sources DB Extract File Weather DB CDC File CDC Mobile Apps Connected Car Robot Windpark Air Traffic Event Message Bulk Stream Service Social Media Smart City Sensor Market Feed Bulk Data Flow Disk Data Flow Scheduler API / Service Control-M FTP Server Azure Event Grid
  • 23. Challenge: HD Insight Hybrid Identities / Federation Authorization in HD Insight HD Insight Costs
  • 24. Integration Bulk Data Flow Create Blob Disk Service Analytical Platform Automation Meta DataGeneratorTemplate Create Blob (Deployment) Information Governance & Security Event Catalog Sync Data Assets Big Data Storage Raw Zone Trusted/Refined Zone Usage-Optimized Zone Big Data Processing Transform Event Hub SQL REST / SOAP Event Stream Event Stream API Call Real-Time Big Data Processing Stream Analytics Usage-Optimized Data Enterprise Apps Big Data Analytics Machine Learning Big Data Federation Information Consumer Batch Data Visualization Self-Service Analytics EDWH RDBMS Data Flow Data Science Lab Service Bus Business Process API Call SQL API Call API Call Read Create / Delete Read SQL APICall API Call / SQL / Query Load Read Create / Delete Archival API Call Scheduler API Call Data Catalog Containerized Apps Microservice SQL API Call Access Mgmt Encryption & Protection Multi- Dimensional ML Model SQL / Query Event Stream Event Stream Usage- Optimized Data Event Stream Master Data CRUD Data Lineage Master Data Event Handler Event Handler Streaming Data Visualization Cleansing / Validating Enrichment Aggregation Image/Video Recognition Timeseries Analysis Graph/Link Analytics Location Analytics Landing Zone Sandbox Zone App Marketplace Query Engine API / Service Master DataAPI / Service Data Enterprise App API / Service Archived Zone API / Service Azure Storage Blob Azure Storage Blob Azure Storage Blob Master Data Services (MDS) HDInsight Spark Azure Data Catalog Trivadis biGENiUS HDInsight Kafka Azure Functions HDInsight Interactive Query StreamSets Data Collector Azure Cosmos DB Azure SQL Database Azure Databricks Azure Logic Apps Azure Scheduler Power BI Tableau / SAP BO MATLAB Data Catalog Web UI Azure Databricks UI Azure Data Box Azure Import StreamSets Data Collector Azure Time Series Insights Excel with MDS Plugin Azure Storage Explorer Azure Kubernetes Service (AKS) Azure SQL Database HDInsight (Ranger) Azure StreamAnalytics Spark Streaming Azure Event Hub Bulk Import Event Stream Edge (Bulk) Data Flow Stream Analytics Event Stream Bulk Data Bulk Data Event Stream Event Hub API Call API Call Event Handler Data Sources DB Extract File Weather DB CDC File CDC Mobile Apps Connected Car Robot Windpark Air Traffic Event Message Bulk Stream Service Social Media Smart City Sensor Market Feed Bulk Data Flow Disk Data Flow Scheduler API / Service Control-M FTP Server Azure Event Grid Challenge: HD Insight
  • 25. Challenge: HD Insight - Authentication HDInsight Customer’s AzureAnalytics Platform Azure Customer’s OnPrem BigData Storage Azure AD gateway head nodeworker node(s) ranger worker node(s) worker node(s) Zeppelin web services SQL interface other Customer Azure AD Azure AD B2B Federate Sync/Federate Authentication OnPrem Authentication/Authorization (SAML,OAuth) Benötigtes Setup für Azure HDInsight
  • 26. Challenge: HD Insight - Authentication HDInsight Customer’s AzureAnalytics Platform Azure Customer’s OnPrem BigData Storage Azure AD gateway head nodeworker node(s) ranger worker node(s) worker node(s) Zeppelin web services SQL interface other Customer Azure AD Syncincl.Passwords Authentication OnPrem Auth Kerberos/LDAP Azure Active Directory Domain Services Domain Join Domain Join& Authentication Empfohlenes Deployment von Microsoft
  • 27. Challenge: HD Insight - Authentication HDInsight Customer’s AzureAnalytics Platform Azure Customer’s OnPrem BigData Storage Azure AD gateway head nodeworker node(s) ranger worker node(s) worker node(s) Zeppelin web services SQL interface other Customer Azure AD Auth Kerberos/LDAP Azure Active Directory Domain Services Domain Join Synchronizing same identity in 2 Azure ADs not possible Syncincl.Passwords Authentication OnPrem Sync X Möglicher Workaround 1
  • 28. Challenge: HD Insight - Authentication HDInsight Ørsted AzureSMM Platform Azure Ørsted OnPrem BigData Storage Azure AD Apache KNOX Gateway head nodeworker node(s) ranger worker node(s) worker node(s) Zeppelin web services SQL interface other Ørsted Azure AD Azure AD B2B Federate Sync/Federate Authentication OnPrem Authentication/Authorization (SAML,OAuth) Austausch einer standard HDInsight Komponente Möglicher Workaround 2
  • 29. Challenge: HD Insight HDInsight Customer’s AzureAnalytics Platform Azure Customer’s OnPrem BigData Storage Azure AD gateway head nodeworker node(s) ranger worker node(s) worker node(s) Zeppelin web services SQL interface other Customer Azure AD Provision/DeprovisionIdentities Auth Kerberos/LDAP Azure Active Directory Domain Services Domain Join Azure AD managing Identities All Azure IAM features available Self Service IAM additional possible (e.g. Password reset) provide initial credentials Customers Identity Manager Script execution Möglicher Workaround 3
  • 30. Challenge: HD Insight - Kosten HDInsight Spark Head Node Worker Min. 2 Nodes HDInsight Kafka Head Node Worker Min. 2 Nodes HDInsight Interactive Q Head Node Worker Min. 2 Nodes 3 Clusters, jeder min. 3 VMs Kosten pro Stunde Kein stoppen möglich, nur deprovisionieren
  • 31. Challenge: HD Insight - Kosten HDInsight Spark Head Node Worker Min. 2 Nodes HDInsight Kafka Head Node Worker Min. 2 Nodes HDInsight Interactive Q Head Node Worker Min. 2 Nodes
  • 33. Challenge: Kostenmanagement VM (8CPU) / hour Per GB + Transactions Per hour + Outbound traffic Instance / hour + GB Zone / Month + Queries Units / Hour + Events ?
  • 34. Challenge: Kostenmanagement VM (8CPU) / hour Per GB + Transactions Per hour + Outbound traffic Instance / hour + GB Zone / Month + Queries Units / Hour + Events Tags !
  • 36. Takeaways § HD Insight ist mächtig, aber nicht wirklich cloud aware… § Identity Management und Access Management für HD Insight eher traditionell § Kosten von HD Insight nicht unterschätzen… § Wo möglich automatisiert deprovisionieren und porvisionieren § Oder Databricks & Data Factory nutzen § Tagging ideal für Kosten Verteilungen