SlideShare a Scribd company logo
news.trivadis.com/blog@trivadis
Data Warehouse – (high) added value in Azure
Cloud
yves.mauron@trivadis.com
marco.amhof@trivadis.com
Agenda
• The modern data warehouse pattern
• Azure Cloud architecture pillars
• Ingest data (Azure Data Factory, ADF SSIS Runtime Integration, others)
• Store data (Azure SQL, Azure Data Lake, Azure SQL DWH)
• Process data (Azure Databricks)
• Azure SSAS – Advantages and possibilities of a BI semantic layer
• Power BI
• Trivadis Customer Cases
• Case 1 (Lift and Shift on-premise DWH to Azure Cloud)
• Case 2 (BI Solution in Azure Cloud with Data Lake and Databricks Technology)
• Case 3 (Green field a BI Solution from scratch in Azure Cloud)
© Microsoft Corporation
Modern data warehousing pattern
Advanced Analytics
Social
LOB
Graph
IoT
Image
CRM
INGEST STORE PREP MODEL & SERVE
(& store)
Data orchestration
and monitoring
Big data store Transform & Clean Data warehouse
AI
BI + Reporting
AI built-in | Most secure | Lowest TCO
Data warehouses
Data lakes
Operational databases
Data warehouses
Data lakes
Operational databasesIndustry leader 4 years in a row
#1 TPC-H performance
T-SQL query over any data
70% faster
2x the global reach
99.9% SLA
Easiest lift and shift
with no code changes
The Microsoft offering
SQL Server
Hybrid
Azure Data Services
Security and performanceFlexibility of choiceReason over any data, anywhere
SocialLOB Graph IoTImageCRM
The Azure data landscape
Azure
Data
Factory
Azure Import/Export
service
Azure SDKAzure
CLI
Cognitive servicesBot service
Azure Search Azure Data Catalog
Azure ExpressRoute Azure network
security groups
Azure Functions Visual StudioOperations
Management Suite
Azure Active Directory Azure key
management service
Azure Blob Storage Azure Data
Lake Store
Azure IoT Hub Azure event
hubs
Kafka on Azure HDInsight
Azure SQL data warehouseAzure SQL DB Azure Cosmos DB Azure Analysis Services Power BI
Azure
HDInsight
Azure
Databricks
Azure
HDInsight
Azure
Databricks
Azure Stream
Analytics
Azure
ML
Azure
Databricks
ML Server
10
01
SQL
NSG
>_
INGEST STORE PREP MODEL & SERVE
Ingest
ETL vs. ELT
vs.
Azure Data Factory
• Data Integration Service: Serverless, Scalable, Hybrid
Hybrid Pipeline Model
Seamlessly span: on prem, Azure, other clouds & SaaS
Run on-demand, scheduled, data-availability or on event
Data Movement @Scale
Cloud & Hybrid w/ 80+ connectors provided
Up to 1 GB/s
SSIS Package Execution
Lift existing SQL Server ETL to Azure
Use existing tools (SSMS, SSDT)
Author & Monitor
Programmability w/ multi-language SDK
Visual Tools
Azure
Pipeline foreach (…)Trigger
Linked
Service
© Microsoft Corporation
No-code data transformation @ scale
Mapping Data Flow
Data cleansing, transformation, aggregation, conversion, etc.
Cloud scale via Spark execution
Easily build resilient data flows
Store
Azure SQL Database resource types
Azure SQL Database
Database-scoped
deployment option with
predictable workload
performance
Shared resource model optimized
for greater efficiency of multi-
tenant applications
Best for apps that require resource
guarantee at database level
Best for SaaS apps with multiple
databases that can share resources
at database level, achieving better
cost efficiency
Best for modernization at scale
with low friction and effort
Elastic PoolSingle Managed Instance
Instance-scoped deployment option
with high compatibility with SQL Server
and full PaaS benefits
Azure SQL Data Warehouse
Best in class
price-performance
Up to 14X times faster
and 94% less expensive
than cloud competitors
Industry-leading
security
Defense-in-depth
security and 99.9%
financially backed
availability SLA
Intelligent workload
management
Separation of compute
and storage
Prioritize resources for
the most valuable
workloads
Developer productivityData flexibility
Azure SQL Data Warehouse MPP Architecture
© Microsoft Corporation
Data ingestion using external data sources
Polybase
-- Create Azure DataLake Gen2 Storage reference
CREATE EXTERNAL DATA SOURCE AzureStorage with
(
TYPE = HADOOP,
LOCATION='abfss://<container>@<storageaccnt>.blob.core.windows.net'
,
CREDENTIAL = AzureStorageCredential –- not required if using
managed identity
);
-- Type of format in Hadoop (CSV, RCFILE , ORC, PARQUET).
CREATE EXTERNAL FILE FORMAT TextFileFormat WITH
(
FORMAT_TYPE = DELIMITEDTEXT,
FORMAT_OPTIONS (FIELD_TERMINATOR ='|', USE_TYPE_DEFAULT =
TRUE)
)
-- LOCATION: path to file or directory that contains data
CREATE EXTERNAL TABLE [dbo].[CarSensor_Data]
(
[SensorKey] int NOT NULL,
[Speed] float NOT NULL,
[YearMeasured] int NOT NULL
)
WITH (LOCATION='/Demo/’, DATA_SOURCE = AzureStorage,
FILE_FORMAT = TextFileFormat
);
Overview
Polybase supports querying files (Parquet,
Delimited Text) stored in a Hadoop File System
(HDFS), Azure Blob storage, or Azure Data Lake
Store.
To query files, users create three objects:
External data source, external file format,
external table.
Starting in SQL Server 2019, you can now use
PolyBase to access external data in SQL
Server, Oracle, Teradata, and MongoDB.
DWU’s (CPU, memory, and IO) in $$$
SQL Azure Data Warehouse (a preview)
© Microsoft Corporation
It is a central storage repository that holds data coming from many sources in a raw, granular format. It can store structured,
semi-structured, or unstructured data, which means data ingested quickly and can be kept in a more flexible format for
future use cases.
What is a Data Lake?
Characteristics
• Schema-on-read
(ELT)
• Collection of data,
not a platform
• Perfect place for
evolving data
Benefits
• Quickly ingest high
volumes of diverse
data structures
• Enable advanced
analytics and data
exploration
• Scalability and
storage cost
reduction
BestPractices
• Data Governance
needed to avoid
Data Swamp
• Security
considerations
• Design your Data
Lake
• Metadata
management
© Microsoft Corporation
Data Lake Design Considerations
Data Lake Zones
Transient Landing Zone
Temporary storage of data to meet regulatory and quality control
requirements. Limited access. May not be required depending on
requirements.
Raw Zone
Original source of data ready for consumption. Metadata publicly
available but access to data still limited.
Trusted Zone
Standardized and enriched datasets ready for consumption to
those with appropriate role-based access. Metadata available to
all.
Curated/Refined Zone
Data transformed from Trusted Zone to meet specific business
requirements.
Sandbox Zone
Playground for Data Scientists for ad hoc exploratory use cases.
Data Governance Considerations
Security and Compliance
Access Control
Encryption
Row-Level Security
Metadata Management
Data Quality
Metadata Management
Lifecycle Management
Transform & Clean
© Microsoft Corporation
A fast, easy and collaborative Apache® Spark™ based analytics platform optimized for Azure
Azure Databricks
Best of Databricks Best of Microsoft
Designed in collaboration with the founders of Apache Spark
One-click set up; streamlined workflows
Interactive workspace that enables collaboration between data scientists, data engineers, and business
analysts.
Native integration with Azure services (Power BI, SQL DW, Cosmos DB, ADLS, Azure Storage, Azure Data
Factory, Azure AD, Event Hub, IoT Hub, HDInsight Kafka, SQL DB)
Enterprise-grade Azure security (Active Directory integration, compliance, enterprise-grade SLAs)
Azure Databricks
Optimized Databricks Runtime Engine
DATABRICKS I/O SERVERLESS
Collaborative Workspace
Cloud storage
Data warehouses
Hadoop storage
IoT / streaming data
Rest APIs
Machine learning models
BI tools
Data exports
Data warehouses
Azure Databricks
Enhance Productivity
Deploy Production Jobs & Workflows
APACHE SPARK
MULTI-STAGE
PIPELINES
DATA
ENGINEER
JOB SCHEDULER NOTIFICATION & LOGS
DATA
SCIENTIST
BUSINESS ANALYST
Build on secure & trusted cloud Scale without limits
Azure Databricks Deployment
Azure Resource
Manager APIs
Azure Portal
Azure
Databricks
Workspace
Managed
Resource
Group
Attached Azure
BLOB (DBFS)
Workspace
VNET
Workspace
NSG rulesCluster Node(s)
Notebooks
Clusters
Jobs
Run on
Interact using UI or Azure Databricks REST API
Integrate with other Azure Services
Azure BLOBs Data Lake
Event Hub IOT Hub Kafka
Cosmos DB SQL DW
Data Factory
Model & Serve
Why do I also need a cube if I have a data warehouse?
• Semantic layer
• Handle many concurrent users
• Implement complex business logic (DAX)
• Aggregating data for performance
• multidimensional analysis
• No joins or relationships
• Hierarchies, KPI’s
• Row-level Security
• Advanced time-calculations
• Slowly Changing Dimensions (SCD)
• Required for some reporting tools
What is Azure Analysis Services?
• Azure Analysis Services is a fully managed platform as a service (PaaS) that provides enterprise-
grade data models in the cloud.
• Use advanced mashup and modeling features to combine data from multiple data sources, define
metrics, and secure your data in a single, trusted tabular semantic data model.
• The data model provides an easier and faster way for users to browse massive amounts of data for
ad hoc data analysis.
Business / custom apps
(Structured)
Logs, files and media
(unstructured)
Azure Storage
Polybase
Azure SQL Data Warehouse
Data Factory
Data Factory
Azure Databricks
(Spark)
Analytical dashboards
(PowerBI)
Model & ServePrep & TrainStoreIngest Intelligence
Modern Data Analytics Landscape
AZURE DATA FACTORY ORCHESTRATES DATA PIPELINE ACTIVITY WORKFLOW & SCHEDULING
Azure Analysis ServicesOn Prem, Cloud
Apps & Data
BI & Reporting
Power BI
Power BI Report Server (on-premise)
Customer success stories
Case 1 – «on-premise» to Azure Cloud
Customer
Case 1 - Approach
Case 1 – Timeline
Case 2 – Modern Data Warehouse – (Delta) Lake
Case 3 – Management Reporting
PowerBI
Excel
…
Sub1
Sub2
Sub n
Structures
Messaging
Model & ServePrep / TrainStoreIngest
SQL DatabaseBLOB Storage
Logic App Function App
Analysis Services
Azure AD
DEMO
Questions?
Azure Days 2019: Business Intelligence auf Azure (Marco Amhof & Yves Mauron)

More Related Content

What's hot

Windowsazureplatform Overviewlatest
Windowsazureplatform OverviewlatestWindowsazureplatform Overviewlatest
Windowsazureplatform Overviewlatest
rajramab
 
Moving Your Data to The Cloud
Moving Your Data to The CloudMoving Your Data to The Cloud
Moving Your Data to The Cloud
Adwait Ullal
 

What's hot (20)

Azure Fundamentals || AZ-900
Azure Fundamentals || AZ-900Azure Fundamentals || AZ-900
Azure Fundamentals || AZ-900
 
Modern Business Intelligence and Advanced Analytics
Modern Business Intelligence and Advanced AnalyticsModern Business Intelligence and Advanced Analytics
Modern Business Intelligence and Advanced Analytics
 
Benefits of the Azure cloud
Benefits of the Azure cloudBenefits of the Azure cloud
Benefits of the Azure cloud
 
Azure intelligent edge solutions overview
Azure intelligent edge solutions overviewAzure intelligent edge solutions overview
Azure intelligent edge solutions overview
 
Azure data platform overview
Azure data platform overviewAzure data platform overview
Azure data platform overview
 
Windowsazureplatform Overviewlatest
Windowsazureplatform OverviewlatestWindowsazureplatform Overviewlatest
Windowsazureplatform Overviewlatest
 
What are the Business Benefits of Microsoft Azure
What are the Business Benefits of Microsoft AzureWhat are the Business Benefits of Microsoft Azure
What are the Business Benefits of Microsoft Azure
 
Moving Your Data to The Cloud
Moving Your Data to The CloudMoving Your Data to The Cloud
Moving Your Data to The Cloud
 
Simplify and Accelerate SQL Server Migration to Azure
Simplify and Accelerate SQL Server Migration to AzureSimplify and Accelerate SQL Server Migration to Azure
Simplify and Accelerate SQL Server Migration to Azure
 
Introduction to Azure
Introduction to AzureIntroduction to Azure
Introduction to Azure
 
Cloud computing 2
Cloud computing 2Cloud computing 2
Cloud computing 2
 
Migrate to Microsoft Azure with Confidence
Migrate to Microsoft Azure with ConfidenceMigrate to Microsoft Azure with Confidence
Migrate to Microsoft Azure with Confidence
 
Transforming ISV's to Azure
Transforming ISV's to AzureTransforming ISV's to Azure
Transforming ISV's to Azure
 
Cloud Computing and Microsoft Azure
Cloud Computing and Microsoft AzureCloud Computing and Microsoft Azure
Cloud Computing and Microsoft Azure
 
The Basics of Getting Started With Microsoft Azure
The Basics of Getting Started With Microsoft AzureThe Basics of Getting Started With Microsoft Azure
The Basics of Getting Started With Microsoft Azure
 
Designing a modern data warehouse in azure
Designing a modern data warehouse in azure   Designing a modern data warehouse in azure
Designing a modern data warehouse in azure
 
Cloud computing Azure
Cloud computing AzureCloud computing Azure
Cloud computing Azure
 
Cloud Scale Analytics Pitch Deck
Cloud Scale Analytics Pitch DeckCloud Scale Analytics Pitch Deck
Cloud Scale Analytics Pitch Deck
 
Azure SQL Database & Azure SQL Data Warehouse
Azure SQL Database & Azure SQL Data WarehouseAzure SQL Database & Azure SQL Data Warehouse
Azure SQL Database & Azure SQL Data Warehouse
 
04 Azure IAAS 101
04 Azure IAAS 10104 Azure IAAS 101
04 Azure IAAS 101
 

Similar to Azure Days 2019: Business Intelligence auf Azure (Marco Amhof & Yves Mauron)

Modern Analytics Academy - Data Modeling (1).pptx
Modern Analytics Academy - Data Modeling (1).pptxModern Analytics Academy - Data Modeling (1).pptx
Modern Analytics Academy - Data Modeling (1).pptx
ssuser290967
 
Azure Data Engineering.pdf
Azure Data Engineering.pdfAzure Data Engineering.pdf
Azure Data Engineering.pdf
akhilamadupativibhin
 

Similar to Azure Days 2019: Business Intelligence auf Azure (Marco Amhof & Yves Mauron) (20)

Azure Data.pptx
Azure Data.pptxAzure Data.pptx
Azure Data.pptx
 
Azure Data Factory ETL Patterns in the Cloud
Azure Data Factory ETL Patterns in the CloudAzure Data Factory ETL Patterns in the Cloud
Azure Data Factory ETL Patterns in the Cloud
 
Modern Analytics Academy - Data Modeling (1).pptx
Modern Analytics Academy - Data Modeling (1).pptxModern Analytics Academy - Data Modeling (1).pptx
Modern Analytics Academy - Data Modeling (1).pptx
 
SQL Saturday Redmond 2019 ETL Patterns in the Cloud
SQL Saturday Redmond 2019 ETL Patterns in the CloudSQL Saturday Redmond 2019 ETL Patterns in the Cloud
SQL Saturday Redmond 2019 ETL Patterns in the Cloud
 
Azure Data Engineering.pdf
Azure Data Engineering.pdfAzure Data Engineering.pdf
Azure Data Engineering.pdf
 
Azure databricks c sharp corner toronto feb 2019 heather grandy
Azure databricks c sharp corner toronto feb 2019 heather grandyAzure databricks c sharp corner toronto feb 2019 heather grandy
Azure databricks c sharp corner toronto feb 2019 heather grandy
 
Differentiate Big Data vs Data Warehouse use cases for a cloud solution
Differentiate Big Data vs Data Warehouse use cases for a cloud solutionDifferentiate Big Data vs Data Warehouse use cases for a cloud solution
Differentiate Big Data vs Data Warehouse use cases for a cloud solution
 
Azure Data Engineering.pptx
Azure Data Engineering.pptxAzure Data Engineering.pptx
Azure Data Engineering.pptx
 
Prague data management meetup 2018-03-27
Prague data management meetup 2018-03-27Prague data management meetup 2018-03-27
Prague data management meetup 2018-03-27
 
Cepta The Future of Data with Power BI
Cepta The Future of Data with Power BICepta The Future of Data with Power BI
Cepta The Future of Data with Power BI
 
Afternoons with Azure - Azure Data Services
Afternoons with Azure - Azure Data ServicesAfternoons with Azure - Azure Data Services
Afternoons with Azure - Azure Data Services
 
Introducing Azure SQL Data Warehouse
Introducing Azure SQL Data WarehouseIntroducing Azure SQL Data Warehouse
Introducing Azure SQL Data Warehouse
 
Azure fundamental -Introduction
Azure fundamental -IntroductionAzure fundamental -Introduction
Azure fundamental -Introduction
 
Azure Data Engineering course in hyderabad.pptx
Azure Data Engineering course in hyderabad.pptxAzure Data Engineering course in hyderabad.pptx
Azure Data Engineering course in hyderabad.pptx
 
Azure Data Engineering Course in Hyderabad
Azure Data Engineering  Course in HyderabadAzure Data Engineering  Course in Hyderabad
Azure Data Engineering Course in Hyderabad
 
"Azure Data Engineering Course in Hyderabad "
"Azure Data Engineering Course in Hyderabad ""Azure Data Engineering Course in Hyderabad "
"Azure Data Engineering Course in Hyderabad "
 
Data Modernization_Harinath Susairaj.pptx
Data Modernization_Harinath Susairaj.pptxData Modernization_Harinath Susairaj.pptx
Data Modernization_Harinath Susairaj.pptx
 
QuerySurge Slide Deck for Big Data Testing Webinar
QuerySurge Slide Deck for Big Data Testing WebinarQuerySurge Slide Deck for Big Data Testing Webinar
QuerySurge Slide Deck for Big Data Testing Webinar
 
USQL Trivadis Azure Data Lake Event
USQL Trivadis Azure Data Lake EventUSQL Trivadis Azure Data Lake Event
USQL Trivadis Azure Data Lake Event
 
Azure Data Factory for Redmond SQL PASS UG Sept 2018
Azure Data Factory for Redmond SQL PASS UG Sept 2018Azure Data Factory for Redmond SQL PASS UG Sept 2018
Azure Data Factory for Redmond SQL PASS UG Sept 2018
 

More from Trivadis

More from Trivadis (20)

Azure Days 2019: Azure Chatbot Development for Airline Irregularities (Remco ...
Azure Days 2019: Azure Chatbot Development for Airline Irregularities (Remco ...Azure Days 2019: Azure Chatbot Development for Airline Irregularities (Remco ...
Azure Days 2019: Azure Chatbot Development for Airline Irregularities (Remco ...
 
Azure Days 2019: Master the Move to Azure (Konrad Brunner)
Azure Days 2019: Master the Move to Azure (Konrad Brunner)Azure Days 2019: Master the Move to Azure (Konrad Brunner)
Azure Days 2019: Master the Move to Azure (Konrad Brunner)
 
Azure Days 2019: Infrastructure as Code auf Azure (Jonas Wanninger & Daniel H...
Azure Days 2019: Infrastructure as Code auf Azure (Jonas Wanninger & Daniel H...Azure Days 2019: Infrastructure as Code auf Azure (Jonas Wanninger & Daniel H...
Azure Days 2019: Infrastructure as Code auf Azure (Jonas Wanninger & Daniel H...
 
Azure Days 2019: Wie bringt man eine Data Analytics Plattform in die Cloud? (...
Azure Days 2019: Wie bringt man eine Data Analytics Plattform in die Cloud? (...Azure Days 2019: Wie bringt man eine Data Analytics Plattform in die Cloud? (...
Azure Days 2019: Wie bringt man eine Data Analytics Plattform in die Cloud? (...
 
Azure Days 2019: Azure@Helsana: Die Erweiterung von Dynamics CRM mit Azure Po...
Azure Days 2019: Azure@Helsana: Die Erweiterung von Dynamics CRM mit Azure Po...Azure Days 2019: Azure@Helsana: Die Erweiterung von Dynamics CRM mit Azure Po...
Azure Days 2019: Azure@Helsana: Die Erweiterung von Dynamics CRM mit Azure Po...
 
TechEvent 2019: Kundenstory - Kein Angebot, kein Auftrag – Wie Du ein individ...
TechEvent 2019: Kundenstory - Kein Angebot, kein Auftrag – Wie Du ein individ...TechEvent 2019: Kundenstory - Kein Angebot, kein Auftrag – Wie Du ein individ...
TechEvent 2019: Kundenstory - Kein Angebot, kein Auftrag – Wie Du ein individ...
 
TechEvent 2019: Oracle Database Appliance M/L - Erfahrungen und Erfolgsmethod...
TechEvent 2019: Oracle Database Appliance M/L - Erfahrungen und Erfolgsmethod...TechEvent 2019: Oracle Database Appliance M/L - Erfahrungen und Erfolgsmethod...
TechEvent 2019: Oracle Database Appliance M/L - Erfahrungen und Erfolgsmethod...
 
TechEvent 2019: Security 101 für Web Entwickler; Roland Krüger - Trivadis
TechEvent 2019: Security 101 für Web Entwickler; Roland Krüger - TrivadisTechEvent 2019: Security 101 für Web Entwickler; Roland Krüger - Trivadis
TechEvent 2019: Security 101 für Web Entwickler; Roland Krüger - Trivadis
 
TechEvent 2019: Trivadis & Swisscom Partner Angebote; Konrad Häfeli, Markus O...
TechEvent 2019: Trivadis & Swisscom Partner Angebote; Konrad Häfeli, Markus O...TechEvent 2019: Trivadis & Swisscom Partner Angebote; Konrad Häfeli, Markus O...
TechEvent 2019: Trivadis & Swisscom Partner Angebote; Konrad Häfeli, Markus O...
 
TechEvent 2019: DBaaS from Swisscom Cloud powered by Trivadis; Konrad Häfeli ...
TechEvent 2019: DBaaS from Swisscom Cloud powered by Trivadis; Konrad Häfeli ...TechEvent 2019: DBaaS from Swisscom Cloud powered by Trivadis; Konrad Häfeli ...
TechEvent 2019: DBaaS from Swisscom Cloud powered by Trivadis; Konrad Häfeli ...
 
TechEvent 2019: Status of the partnership Trivadis and EDB - Comparing Postgr...
TechEvent 2019: Status of the partnership Trivadis and EDB - Comparing Postgr...TechEvent 2019: Status of the partnership Trivadis and EDB - Comparing Postgr...
TechEvent 2019: Status of the partnership Trivadis and EDB - Comparing Postgr...
 
TechEvent 2019: More Agile, More AI, More Cloud! Less Work?!; Oliver Dörr - T...
TechEvent 2019: More Agile, More AI, More Cloud! Less Work?!; Oliver Dörr - T...TechEvent 2019: More Agile, More AI, More Cloud! Less Work?!; Oliver Dörr - T...
TechEvent 2019: More Agile, More AI, More Cloud! Less Work?!; Oliver Dörr - T...
 
TechEvent 2019: Kundenstory - Vom Hauptmann zu Köpenick zum Polizisten 2020 -...
TechEvent 2019: Kundenstory - Vom Hauptmann zu Köpenick zum Polizisten 2020 -...TechEvent 2019: Kundenstory - Vom Hauptmann zu Köpenick zum Polizisten 2020 -...
TechEvent 2019: Kundenstory - Vom Hauptmann zu Köpenick zum Polizisten 2020 -...
 
TechEvent 2019: Vom Rechenzentrum in die Oracle Cloud - Übertragungsmethoden;...
TechEvent 2019: Vom Rechenzentrum in die Oracle Cloud - Übertragungsmethoden;...TechEvent 2019: Vom Rechenzentrum in die Oracle Cloud - Übertragungsmethoden;...
TechEvent 2019: Vom Rechenzentrum in die Oracle Cloud - Übertragungsmethoden;...
 
TechEvent 2019: The sleeping Power of Data; Eberhard Lösch - Trivadis
TechEvent 2019: The sleeping Power of Data; Eberhard Lösch - TrivadisTechEvent 2019: The sleeping Power of Data; Eberhard Lösch - Trivadis
TechEvent 2019: The sleeping Power of Data; Eberhard Lösch - Trivadis
 
TechEvent 2019: Tales from a Scrum Master; Ernst Jakob - Trivadis
TechEvent 2019: Tales from a Scrum Master; Ernst Jakob - TrivadisTechEvent 2019: Tales from a Scrum Master; Ernst Jakob - Trivadis
TechEvent 2019: Tales from a Scrum Master; Ernst Jakob - Trivadis
 
TechEvent 2019: Serverless - Ist das was für mich?; Thorsten Maier - Trivadis
TechEvent 2019: Serverless - Ist das was für mich?; Thorsten Maier - TrivadisTechEvent 2019: Serverless - Ist das was für mich?; Thorsten Maier - Trivadis
TechEvent 2019: Serverless - Ist das was für mich?; Thorsten Maier - Trivadis
 
TechEvent 2019: Alexa, Netatmo, ZeptrionAir and Co, Home Automation with fun;...
TechEvent 2019: Alexa, Netatmo, ZeptrionAir and Co, Home Automation with fun;...TechEvent 2019: Alexa, Netatmo, ZeptrionAir and Co, Home Automation with fun;...
TechEvent 2019: Alexa, Netatmo, ZeptrionAir and Co, Home Automation with fun;...
 
TechEvent 2019: Ai and IoT in a real life scenario - how to operate a complex...
TechEvent 2019: Ai and IoT in a real life scenario - how to operate a complex...TechEvent 2019: Ai and IoT in a real life scenario - how to operate a complex...
TechEvent 2019: Ai and IoT in a real life scenario - how to operate a complex...
 
TechEvent 2019: Wie sichere ich eigentlich Kafka ab?; Markus Bente - Trivadis
TechEvent 2019: Wie sichere ich eigentlich Kafka ab?; Markus Bente - TrivadisTechEvent 2019: Wie sichere ich eigentlich Kafka ab?; Markus Bente - Trivadis
TechEvent 2019: Wie sichere ich eigentlich Kafka ab?; Markus Bente - Trivadis
 

Recently uploaded

Recently uploaded (20)

Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
 
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
 
ECS 2024 Teams Premium - Pretty Secure
ECS 2024   Teams Premium - Pretty SecureECS 2024   Teams Premium - Pretty Secure
ECS 2024 Teams Premium - Pretty Secure
 
Syngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdfSyngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdf
 
Agentic RAG What it is its types applications and implementation.pdf
Agentic RAG What it is its types applications and implementation.pdfAgentic RAG What it is its types applications and implementation.pdf
Agentic RAG What it is its types applications and implementation.pdf
 
Speed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in MinutesSpeed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in Minutes
 
PLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. StartupsPLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. Startups
 
UiPath Test Automation using UiPath Test Suite series, part 2
UiPath Test Automation using UiPath Test Suite series, part 2UiPath Test Automation using UiPath Test Suite series, part 2
UiPath Test Automation using UiPath Test Suite series, part 2
 
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
 
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya HalderCustom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder
 
SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...
SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...
SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...
 
Salesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
Salesforce Adoption – Metrics, Methods, and Motivation, Antone KomSalesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
Salesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
 
Top 10 Symfony Development Companies 2024
Top 10 Symfony Development Companies 2024Top 10 Symfony Development Companies 2024
Top 10 Symfony Development Companies 2024
 
Optimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through ObservabilityOptimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through Observability
 
Introduction to Open Source RAG and RAG Evaluation
Introduction to Open Source RAG and RAG EvaluationIntroduction to Open Source RAG and RAG Evaluation
Introduction to Open Source RAG and RAG Evaluation
 
AI revolution and Salesforce, Jiří Karpíšek
AI revolution and Salesforce, Jiří KarpíšekAI revolution and Salesforce, Jiří Karpíšek
AI revolution and Salesforce, Jiří Karpíšek
 
Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024
 
AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101
 
Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджера
 
A Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System StrategyA Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System Strategy
 

Azure Days 2019: Business Intelligence auf Azure (Marco Amhof & Yves Mauron)

  • 1. news.trivadis.com/blog@trivadis Data Warehouse – (high) added value in Azure Cloud yves.mauron@trivadis.com marco.amhof@trivadis.com
  • 2. Agenda • The modern data warehouse pattern • Azure Cloud architecture pillars • Ingest data (Azure Data Factory, ADF SSIS Runtime Integration, others) • Store data (Azure SQL, Azure Data Lake, Azure SQL DWH) • Process data (Azure Databricks) • Azure SSAS – Advantages and possibilities of a BI semantic layer • Power BI • Trivadis Customer Cases • Case 1 (Lift and Shift on-premise DWH to Azure Cloud) • Case 2 (BI Solution in Azure Cloud with Data Lake and Databricks Technology) • Case 3 (Green field a BI Solution from scratch in Azure Cloud)
  • 3. © Microsoft Corporation Modern data warehousing pattern Advanced Analytics Social LOB Graph IoT Image CRM INGEST STORE PREP MODEL & SERVE (& store) Data orchestration and monitoring Big data store Transform & Clean Data warehouse AI BI + Reporting
  • 4. AI built-in | Most secure | Lowest TCO Data warehouses Data lakes Operational databases Data warehouses Data lakes Operational databasesIndustry leader 4 years in a row #1 TPC-H performance T-SQL query over any data 70% faster 2x the global reach 99.9% SLA Easiest lift and shift with no code changes The Microsoft offering SQL Server Hybrid Azure Data Services Security and performanceFlexibility of choiceReason over any data, anywhere SocialLOB Graph IoTImageCRM
  • 5. The Azure data landscape Azure Data Factory Azure Import/Export service Azure SDKAzure CLI Cognitive servicesBot service Azure Search Azure Data Catalog Azure ExpressRoute Azure network security groups Azure Functions Visual StudioOperations Management Suite Azure Active Directory Azure key management service Azure Blob Storage Azure Data Lake Store Azure IoT Hub Azure event hubs Kafka on Azure HDInsight Azure SQL data warehouseAzure SQL DB Azure Cosmos DB Azure Analysis Services Power BI Azure HDInsight Azure Databricks Azure HDInsight Azure Databricks Azure Stream Analytics Azure ML Azure Databricks ML Server 10 01 SQL NSG >_ INGEST STORE PREP MODEL & SERVE
  • 8. Azure Data Factory • Data Integration Service: Serverless, Scalable, Hybrid Hybrid Pipeline Model Seamlessly span: on prem, Azure, other clouds & SaaS Run on-demand, scheduled, data-availability or on event Data Movement @Scale Cloud & Hybrid w/ 80+ connectors provided Up to 1 GB/s SSIS Package Execution Lift existing SQL Server ETL to Azure Use existing tools (SSMS, SSDT) Author & Monitor Programmability w/ multi-language SDK Visual Tools Azure
  • 10. © Microsoft Corporation No-code data transformation @ scale Mapping Data Flow Data cleansing, transformation, aggregation, conversion, etc. Cloud scale via Spark execution Easily build resilient data flows
  • 11. Store
  • 12. Azure SQL Database resource types Azure SQL Database Database-scoped deployment option with predictable workload performance Shared resource model optimized for greater efficiency of multi- tenant applications Best for apps that require resource guarantee at database level Best for SaaS apps with multiple databases that can share resources at database level, achieving better cost efficiency Best for modernization at scale with low friction and effort Elastic PoolSingle Managed Instance Instance-scoped deployment option with high compatibility with SQL Server and full PaaS benefits
  • 13. Azure SQL Data Warehouse Best in class price-performance Up to 14X times faster and 94% less expensive than cloud competitors Industry-leading security Defense-in-depth security and 99.9% financially backed availability SLA Intelligent workload management Separation of compute and storage Prioritize resources for the most valuable workloads Developer productivityData flexibility
  • 14. Azure SQL Data Warehouse MPP Architecture
  • 15. © Microsoft Corporation Data ingestion using external data sources Polybase -- Create Azure DataLake Gen2 Storage reference CREATE EXTERNAL DATA SOURCE AzureStorage with ( TYPE = HADOOP, LOCATION='abfss://<container>@<storageaccnt>.blob.core.windows.net' , CREDENTIAL = AzureStorageCredential –- not required if using managed identity ); -- Type of format in Hadoop (CSV, RCFILE , ORC, PARQUET). CREATE EXTERNAL FILE FORMAT TextFileFormat WITH ( FORMAT_TYPE = DELIMITEDTEXT, FORMAT_OPTIONS (FIELD_TERMINATOR ='|', USE_TYPE_DEFAULT = TRUE) ) -- LOCATION: path to file or directory that contains data CREATE EXTERNAL TABLE [dbo].[CarSensor_Data] ( [SensorKey] int NOT NULL, [Speed] float NOT NULL, [YearMeasured] int NOT NULL ) WITH (LOCATION='/Demo/’, DATA_SOURCE = AzureStorage, FILE_FORMAT = TextFileFormat ); Overview Polybase supports querying files (Parquet, Delimited Text) stored in a Hadoop File System (HDFS), Azure Blob storage, or Azure Data Lake Store. To query files, users create three objects: External data source, external file format, external table. Starting in SQL Server 2019, you can now use PolyBase to access external data in SQL Server, Oracle, Teradata, and MongoDB.
  • 16. DWU’s (CPU, memory, and IO) in $$$ SQL Azure Data Warehouse (a preview)
  • 17. © Microsoft Corporation It is a central storage repository that holds data coming from many sources in a raw, granular format. It can store structured, semi-structured, or unstructured data, which means data ingested quickly and can be kept in a more flexible format for future use cases. What is a Data Lake? Characteristics • Schema-on-read (ELT) • Collection of data, not a platform • Perfect place for evolving data Benefits • Quickly ingest high volumes of diverse data structures • Enable advanced analytics and data exploration • Scalability and storage cost reduction BestPractices • Data Governance needed to avoid Data Swamp • Security considerations • Design your Data Lake • Metadata management
  • 18. © Microsoft Corporation Data Lake Design Considerations Data Lake Zones Transient Landing Zone Temporary storage of data to meet regulatory and quality control requirements. Limited access. May not be required depending on requirements. Raw Zone Original source of data ready for consumption. Metadata publicly available but access to data still limited. Trusted Zone Standardized and enriched datasets ready for consumption to those with appropriate role-based access. Metadata available to all. Curated/Refined Zone Data transformed from Trusted Zone to meet specific business requirements. Sandbox Zone Playground for Data Scientists for ad hoc exploratory use cases. Data Governance Considerations Security and Compliance Access Control Encryption Row-Level Security Metadata Management Data Quality Metadata Management Lifecycle Management
  • 20. © Microsoft Corporation A fast, easy and collaborative Apache® Spark™ based analytics platform optimized for Azure Azure Databricks Best of Databricks Best of Microsoft Designed in collaboration with the founders of Apache Spark One-click set up; streamlined workflows Interactive workspace that enables collaboration between data scientists, data engineers, and business analysts. Native integration with Azure services (Power BI, SQL DW, Cosmos DB, ADLS, Azure Storage, Azure Data Factory, Azure AD, Event Hub, IoT Hub, HDInsight Kafka, SQL DB) Enterprise-grade Azure security (Active Directory integration, compliance, enterprise-grade SLAs)
  • 21. Azure Databricks Optimized Databricks Runtime Engine DATABRICKS I/O SERVERLESS Collaborative Workspace Cloud storage Data warehouses Hadoop storage IoT / streaming data Rest APIs Machine learning models BI tools Data exports Data warehouses Azure Databricks Enhance Productivity Deploy Production Jobs & Workflows APACHE SPARK MULTI-STAGE PIPELINES DATA ENGINEER JOB SCHEDULER NOTIFICATION & LOGS DATA SCIENTIST BUSINESS ANALYST Build on secure & trusted cloud Scale without limits
  • 22. Azure Databricks Deployment Azure Resource Manager APIs Azure Portal Azure Databricks Workspace Managed Resource Group Attached Azure BLOB (DBFS) Workspace VNET Workspace NSG rulesCluster Node(s) Notebooks Clusters Jobs Run on Interact using UI or Azure Databricks REST API Integrate with other Azure Services Azure BLOBs Data Lake Event Hub IOT Hub Kafka Cosmos DB SQL DW Data Factory
  • 24. Why do I also need a cube if I have a data warehouse? • Semantic layer • Handle many concurrent users • Implement complex business logic (DAX) • Aggregating data for performance • multidimensional analysis • No joins or relationships • Hierarchies, KPI’s • Row-level Security • Advanced time-calculations • Slowly Changing Dimensions (SCD) • Required for some reporting tools
  • 25. What is Azure Analysis Services? • Azure Analysis Services is a fully managed platform as a service (PaaS) that provides enterprise- grade data models in the cloud. • Use advanced mashup and modeling features to combine data from multiple data sources, define metrics, and secure your data in a single, trusted tabular semantic data model. • The data model provides an easier and faster way for users to browse massive amounts of data for ad hoc data analysis.
  • 26. Business / custom apps (Structured) Logs, files and media (unstructured) Azure Storage Polybase Azure SQL Data Warehouse Data Factory Data Factory Azure Databricks (Spark) Analytical dashboards (PowerBI) Model & ServePrep & TrainStoreIngest Intelligence Modern Data Analytics Landscape AZURE DATA FACTORY ORCHESTRATES DATA PIPELINE ACTIVITY WORKFLOW & SCHEDULING Azure Analysis ServicesOn Prem, Cloud Apps & Data
  • 28.
  • 29. Power BI Power BI Report Server (on-premise)
  • 31. Case 1 – «on-premise» to Azure Cloud Customer
  • 32. Case 1 - Approach
  • 33. Case 1 – Timeline
  • 34. Case 2 – Modern Data Warehouse – (Delta) Lake
  • 35. Case 3 – Management Reporting PowerBI Excel … Sub1 Sub2 Sub n Structures Messaging Model & ServePrep / TrainStoreIngest SQL DatabaseBLOB Storage Logic App Function App Analysis Services Azure AD
  • 36. DEMO