SlideShare a Scribd company logo
Building Modern Cloud
Analytics Solution
Dmitry Anoshin
Outline
• About Me
• Role of Analytics
• History of Cloud
• Analytics powered by Microsoft Azure
• DW modernization Project
• Use cases and Challenges
• Alternative Solution with Azure
About Myself
About Myself
• Work with Business Intelligence
since 2007
#dimaworkplace
Technical Skills Matrix
2015
2010
2007
Data
Warehouse
ETL/ELT
Business
Intelligence
Big Data
Cloud
Analytics
(AWS,
Azure,
GCP)
Machine
Learning
2019
Other Activities
Jumpstart Sno
wflake: A Step-
by-Step Guide
to Modern
Cloud Analytics.
• Victoria Power BI andVictoria SQL Server meetup
• Victoria andVancouverTableau User Group
• Conferences (EDW 2018, 2019, Data Architecture Summit)
• Amazon internal conferences
Role of Analytics
BusinessValue
Stakeholders Employees Customers
Value
”The goal of any organization is to generateValue”
The Future of Competition.
https://www.amazon.com/Future-Competition-Co-Creating-Unique-Customers/dp/1578519535
BIValue Chain
Stakeholders Employees Customers
Value
Decisions
Data
Value creation based on effective decisions
Effective decisions based on accurate
information
For Data to be a differentiator, customers
need to be able to…
• Capture and store new non-relational data at
PB-EB scale in real time
• Discover value in a new type of analytics that
go beyond batch reporting to incorporate
real-time, predictive, voice, and image
recognition
• Democratize access to data in a secure and
governed way
New types of analytics
Dashboards Predictive Image
Recognition
VoiceReal-time
New types of data
Cloud Analytics
Introduction
Cloud Early History
1970
Time Sharing Concept by
GE
1977
Cloud symbol
used in ARPANET
1990
VPN by telecom
1993
Cloud refer to
Distributed
Computing
1994 Cloud
metaphor for
virtualized
services
Cloud Recent History
2002
AWS
2006
AWS Elastic
Compute Cloud
2006
Google Docs
2008
Google App
Engine
2008
Microsoft
Announced Azure
2010
Microsoft Azure
Why moving to the Cloud?
• Elasticity
• Pay for what
you need
• Fail fast
• Fast time to
market
• Secure
• Reliable
• Business SLA
Downsides of on-premise solution
Scale
Constrained
Up-front cost Maintenance
Resources
Tuning and
Deployment
Cloud Restrictions -> Hybrid Clouds
Sensitive Data Data Moving
Cost
Public/Private
Cloud
Cloud Service Models
Cloud Service Models – friendly version
Cloud Analytics
with Microsoft
Azure
Microsoft Azure for Analytics
Data Analytics with Azure
• Data Factory
• Integration
Service
• Kafka
• Event Hub
• Data Lake Gen 1
• Data Lake Gen 2
• Blob Storage
• HD Insight
• Data Lake Analytics
• Streaming Analytics
• PolyBase
• CosmosDB
• SQL DW
• Analysis Service
• SQL Database
• SQL Server in
VM
• Cosmos DB
Data Integration
and
Transformation
Data Warehouse
and Data bases
Big Data
• Analysis Service
• ML Analytics
• Business Intelligence
Analytics
DW Modernization
Use Case
BI/DW (before)
Storage LayerSource Layer
Ad-hoc SQL
SFTP
Data Warehouse
ETL (PL/SQL)Files
Inventory
Sales
Access Layer
Cloud Migration Strategy
Lift & Shift
• Typical Approach
• Move all-at-once
• Target platform then evolve
• Approach gets you to the cloud quickly
• Relatively small barrier to learning new technology
since it tends to be a close fit
Split & Flip
• Split application into logical functional data layers
• Match the data functionality with the right
technology
• Leverage the wide selection of tools onAWS to
best fit the need
• Move data in phases — prototype, learn and
perfect
Migration Approach
Useful tools:
• Total Cost Ownership (TCO) Calculator
• Azure Database Migration Service
• Azure Migration Assistant
Cloud Data Warehouse
What is Azure DW?
• Decouple Storage
and Compute
• MPP
• Distribution Styles:
Hash/Robin/Replicat
e
MPP?
SQL Database vs SQL Data Warehouse
What is Azure Data Factory?
Azure Data Factory (ADF) is Microsoft’s fully managed ELT service
in the cloud that’s delivered as a Platform as a Service (PaaS)
Lack of Notification
Problem: Users are missing emails or they jump to spam.
Solution: Leverage Messenger with Webhooks. (Slack, Chime or so on).
Lack of Logging
Problem: We didn’t have any detail logs about our ETL performance and we didn’t
have any insights.
Solution: Collecting logs and events. In addition, we are able to collect logs on any
level of jobs and transformation.
Self-Service BI
Problem: Business Users wants Interactive and Self-Service tool. Fast time to Market
and less dependency on IT.
Solution: Implement modern Visual Analytics Platform
Marketing Automation
Problem: Marketing team wants “Move Fast and Break Things”.
Solution: Using ADF the gave Marketing template jobs and they doing their jobs
themselves.
Affiliates
Insights
Integration with BI
Problem: Having best BI tool doesn’t guaranty good SLA.
Solution: Build Integration between Matillion ETL and Tableau based on Trigger. Add
data quality checks.
Evolving to Cloud
Data Analytics
Platform
Streaming Data
Problem: Organization is using NoSQL database and mobile application. It is
critical to deliver near real time analytics
Solution: Using Apache Kaffka, we are able to stream data into the Data lake
and query this data in near real time
Data Lake Dashboard
Kafka
CosmoDB
Mobile App
Clickstream Analytics
Problem: Business wants to analyze Bots traffics and discover broken URLs.
Access logs are ~50GB per day, 5600 log files per day.
Solution: Leveraging Databricks in order to produce Parquet file and store in
Azure Data Lake Gen2. User are able query it with T-SQL and BI Tools.
Databricks ParquetBlob Storage
Access Logs
Load Balancer Data Lake Data Factory SQL DW
Query with SQL or Databricks
DevOps onboarding
Problem: Solution isn’t reliable and could easy break. As a result end users will
experience bad experience and it will affect business decisions.
Solution: Onboarding Continuous Integration methodology for Cloud Data
Platform
• Agile and Kanban board
• Code branching (Git)
• Gated check-ins
• Automated Tests
• Build
• Release
Evolving to Cloud Data Analytics Platform
Alternative Implementation
What is Matillion ETL?
What is Snowflake?

More Related Content

What's hot

Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...Cathrine Wilhelmsen
 
Azure Synapse 101 Webinar Presentation
Azure Synapse 101 Webinar PresentationAzure Synapse 101 Webinar Presentation
Azure Synapse 101 Webinar PresentationMatthew W. Bowers
 
Building the Data Lake with Azure Data Factory and Data Lake Analytics
Building the Data Lake with Azure Data Factory and Data Lake AnalyticsBuilding the Data Lake with Azure Data Factory and Data Lake Analytics
Building the Data Lake with Azure Data Factory and Data Lake AnalyticsKhalid Salama
 
Microsoft Data Platform - What's included
Microsoft Data Platform - What's includedMicrosoft Data Platform - What's included
Microsoft Data Platform - What's includedJames Serra
 
Data Lakehouse, Data Mesh, and Data Fabric (r2)
Data Lakehouse, Data Mesh, and Data Fabric (r2)Data Lakehouse, Data Mesh, and Data Fabric (r2)
Data Lakehouse, Data Mesh, and Data Fabric (r2)James Serra
 
Databricks Fundamentals
Databricks FundamentalsDatabricks Fundamentals
Databricks FundamentalsDalibor Wijas
 
Let’s get to know Snowflake
Let’s get to know SnowflakeLet’s get to know Snowflake
Let’s get to know SnowflakeKnoldus Inc.
 
Data Lakehouse Symposium | Day 4
Data Lakehouse Symposium | Day 4Data Lakehouse Symposium | Day 4
Data Lakehouse Symposium | Day 4Databricks
 
Building a modern data warehouse
Building a modern data warehouseBuilding a modern data warehouse
Building a modern data warehouseJames Serra
 
5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop
5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop
5 Critical Steps to Clean Your Data Swamp When Migrating Off of HadoopDatabricks
 
Time to Talk about Data Mesh
Time to Talk about Data MeshTime to Talk about Data Mesh
Time to Talk about Data MeshLibbySchulze
 
Data Lake Overview
Data Lake OverviewData Lake Overview
Data Lake OverviewJames Serra
 
Building an Effective Data Warehouse Architecture
Building an Effective Data Warehouse ArchitectureBuilding an Effective Data Warehouse Architecture
Building an Effective Data Warehouse ArchitectureJames Serra
 
Architect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh ArchitectureArchitect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh ArchitectureDatabricks
 
Azure Data Factory ETL Patterns in the Cloud
Azure Data Factory ETL Patterns in the CloudAzure Data Factory ETL Patterns in the Cloud
Azure Data Factory ETL Patterns in the CloudMark Kromer
 
Designing An Enterprise Data Fabric
Designing An Enterprise Data FabricDesigning An Enterprise Data Fabric
Designing An Enterprise Data FabricAlan McSweeney
 
Data Mesh for Dinner
Data Mesh for DinnerData Mesh for Dinner
Data Mesh for DinnerKent Graziano
 
Azure SQL Database Managed Instance
Azure SQL Database Managed InstanceAzure SQL Database Managed Instance
Azure SQL Database Managed InstanceJames Serra
 
Modernize & Automate Analytics Data Pipelines
Modernize & Automate Analytics Data PipelinesModernize & Automate Analytics Data Pipelines
Modernize & Automate Analytics Data PipelinesCarole Gunst
 

What's hot (20)

Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...
 
Azure Synapse 101 Webinar Presentation
Azure Synapse 101 Webinar PresentationAzure Synapse 101 Webinar Presentation
Azure Synapse 101 Webinar Presentation
 
Building the Data Lake with Azure Data Factory and Data Lake Analytics
Building the Data Lake with Azure Data Factory and Data Lake AnalyticsBuilding the Data Lake with Azure Data Factory and Data Lake Analytics
Building the Data Lake with Azure Data Factory and Data Lake Analytics
 
Microsoft Data Platform - What's included
Microsoft Data Platform - What's includedMicrosoft Data Platform - What's included
Microsoft Data Platform - What's included
 
Data Lakehouse, Data Mesh, and Data Fabric (r2)
Data Lakehouse, Data Mesh, and Data Fabric (r2)Data Lakehouse, Data Mesh, and Data Fabric (r2)
Data Lakehouse, Data Mesh, and Data Fabric (r2)
 
Databricks Fundamentals
Databricks FundamentalsDatabricks Fundamentals
Databricks Fundamentals
 
Let’s get to know Snowflake
Let’s get to know SnowflakeLet’s get to know Snowflake
Let’s get to know Snowflake
 
Data Mesh
Data MeshData Mesh
Data Mesh
 
Data Lakehouse Symposium | Day 4
Data Lakehouse Symposium | Day 4Data Lakehouse Symposium | Day 4
Data Lakehouse Symposium | Day 4
 
Building a modern data warehouse
Building a modern data warehouseBuilding a modern data warehouse
Building a modern data warehouse
 
5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop
5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop
5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop
 
Time to Talk about Data Mesh
Time to Talk about Data MeshTime to Talk about Data Mesh
Time to Talk about Data Mesh
 
Data Lake Overview
Data Lake OverviewData Lake Overview
Data Lake Overview
 
Building an Effective Data Warehouse Architecture
Building an Effective Data Warehouse ArchitectureBuilding an Effective Data Warehouse Architecture
Building an Effective Data Warehouse Architecture
 
Architect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh ArchitectureArchitect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh Architecture
 
Azure Data Factory ETL Patterns in the Cloud
Azure Data Factory ETL Patterns in the CloudAzure Data Factory ETL Patterns in the Cloud
Azure Data Factory ETL Patterns in the Cloud
 
Designing An Enterprise Data Fabric
Designing An Enterprise Data FabricDesigning An Enterprise Data Fabric
Designing An Enterprise Data Fabric
 
Data Mesh for Dinner
Data Mesh for DinnerData Mesh for Dinner
Data Mesh for Dinner
 
Azure SQL Database Managed Instance
Azure SQL Database Managed InstanceAzure SQL Database Managed Instance
Azure SQL Database Managed Instance
 
Modernize & Automate Analytics Data Pipelines
Modernize & Automate Analytics Data PipelinesModernize & Automate Analytics Data Pipelines
Modernize & Automate Analytics Data Pipelines
 

Similar to Building Modern Data Platform with Microsoft Azure

Develop a Custom Data Solution Architecture with NorthBay
Develop a Custom Data Solution Architecture with NorthBayDevelop a Custom Data Solution Architecture with NorthBay
Develop a Custom Data Solution Architecture with NorthBayAmazon Web Services
 
SPS Vancouver 2018 - What is CDM and CDS
SPS Vancouver 2018 - What is CDM and CDSSPS Vancouver 2018 - What is CDM and CDS
SPS Vancouver 2018 - What is CDM and CDSNicolas Georgeault
 
Building your first Analysis Services Tabular BI Semantic model with SQL Serv...
Building your first Analysis Services Tabular BI Semantic model with SQL Serv...Building your first Analysis Services Tabular BI Semantic model with SQL Serv...
Building your first Analysis Services Tabular BI Semantic model with SQL Serv...Microsoft TechNet - Belgium and Luxembourg
 
Feature Store as a Data Foundation for Machine Learning
Feature Store as a Data Foundation for Machine LearningFeature Store as a Data Foundation for Machine Learning
Feature Store as a Data Foundation for Machine LearningProvectus
 
How does Microsoft solve Big Data?
How does Microsoft solve Big Data?How does Microsoft solve Big Data?
How does Microsoft solve Big Data?James Serra
 
Enable Better Decision Making with Power BI Visualizations & Modern Data Estate
Enable Better Decision Making with Power BI Visualizations & Modern Data EstateEnable Better Decision Making with Power BI Visualizations & Modern Data Estate
Enable Better Decision Making with Power BI Visualizations & Modern Data EstateCCG
 
Microsoft Azure BI Solutions in the Cloud
Microsoft Azure BI Solutions in the CloudMicrosoft Azure BI Solutions in the Cloud
Microsoft Azure BI Solutions in the CloudMark Kromer
 
Best practices to deliver data analytics to the business with power bi
Best practices to deliver data analytics to the business with power biBest practices to deliver data analytics to the business with power bi
Best practices to deliver data analytics to the business with power biSatya Shyam K Jayanty
 
Preparing for BI in the Cloud with Windows Azure
Preparing for BI in the Cloud with Windows AzurePreparing for BI in the Cloud with Windows Azure
Preparing for BI in the Cloud with Windows AzurePerficient, Inc.
 
Creating a Modern Data Architecture for Digital Transformation
Creating a Modern Data Architecture for Digital TransformationCreating a Modern Data Architecture for Digital Transformation
Creating a Modern Data Architecture for Digital TransformationMongoDB
 
CDS Overview (May 2015)
CDS Overview (May 2015)CDS Overview (May 2015)
CDS Overview (May 2015)Karim Lalji
 
ADV Slides: Building and Growing Organizational Analytics with Data Lakes
ADV Slides: Building and Growing Organizational Analytics with Data LakesADV Slides: Building and Growing Organizational Analytics with Data Lakes
ADV Slides: Building and Growing Organizational Analytics with Data LakesDATAVERSITY
 
Unlocking the Value of Your Data Lake
Unlocking the Value of Your Data LakeUnlocking the Value of Your Data Lake
Unlocking the Value of Your Data LakeDATAVERSITY
 
Skillwise Big Data part 2
Skillwise Big Data part 2Skillwise Big Data part 2
Skillwise Big Data part 2Skillwise Group
 
Next Gen Analytics Going Beyond Data Warehouse
Next Gen Analytics Going Beyond Data WarehouseNext Gen Analytics Going Beyond Data Warehouse
Next Gen Analytics Going Beyond Data WarehouseDenodo
 
When and How Data Lakes Fit into a Modern Data Architecture
When and How Data Lakes Fit into a Modern Data ArchitectureWhen and How Data Lakes Fit into a Modern Data Architecture
When and How Data Lakes Fit into a Modern Data ArchitectureDATAVERSITY
 
MongoDB Breakfast Milan - Mainframe Offloading Strategies
MongoDB Breakfast Milan -  Mainframe Offloading StrategiesMongoDB Breakfast Milan -  Mainframe Offloading Strategies
MongoDB Breakfast Milan - Mainframe Offloading StrategiesMongoDB
 

Similar to Building Modern Data Platform with Microsoft Azure (20)

Develop a Custom Data Solution Architecture with NorthBay
Develop a Custom Data Solution Architecture with NorthBayDevelop a Custom Data Solution Architecture with NorthBay
Develop a Custom Data Solution Architecture with NorthBay
 
SPS Vancouver 2018 - What is CDM and CDS
SPS Vancouver 2018 - What is CDM and CDSSPS Vancouver 2018 - What is CDM and CDS
SPS Vancouver 2018 - What is CDM and CDS
 
Building your first Analysis Services Tabular BI Semantic model with SQL Serv...
Building your first Analysis Services Tabular BI Semantic model with SQL Serv...Building your first Analysis Services Tabular BI Semantic model with SQL Serv...
Building your first Analysis Services Tabular BI Semantic model with SQL Serv...
 
Feature Store as a Data Foundation for Machine Learning
Feature Store as a Data Foundation for Machine LearningFeature Store as a Data Foundation for Machine Learning
Feature Store as a Data Foundation for Machine Learning
 
Accelerating Data Warehouse Modernization
Accelerating Data Warehouse ModernizationAccelerating Data Warehouse Modernization
Accelerating Data Warehouse Modernization
 
How does Microsoft solve Big Data?
How does Microsoft solve Big Data?How does Microsoft solve Big Data?
How does Microsoft solve Big Data?
 
Serverless SQL
Serverless SQLServerless SQL
Serverless SQL
 
Enable Better Decision Making with Power BI Visualizations & Modern Data Estate
Enable Better Decision Making with Power BI Visualizations & Modern Data EstateEnable Better Decision Making with Power BI Visualizations & Modern Data Estate
Enable Better Decision Making with Power BI Visualizations & Modern Data Estate
 
Microsoft Azure BI Solutions in the Cloud
Microsoft Azure BI Solutions in the CloudMicrosoft Azure BI Solutions in the Cloud
Microsoft Azure BI Solutions in the Cloud
 
Best practices to deliver data analytics to the business with power bi
Best practices to deliver data analytics to the business with power biBest practices to deliver data analytics to the business with power bi
Best practices to deliver data analytics to the business with power bi
 
Preparing for BI in the Cloud with Windows Azure
Preparing for BI in the Cloud with Windows AzurePreparing for BI in the Cloud with Windows Azure
Preparing for BI in the Cloud with Windows Azure
 
Creating a Modern Data Architecture for Digital Transformation
Creating a Modern Data Architecture for Digital TransformationCreating a Modern Data Architecture for Digital Transformation
Creating a Modern Data Architecture for Digital Transformation
 
CDS Overview (May 2015)
CDS Overview (May 2015)CDS Overview (May 2015)
CDS Overview (May 2015)
 
Skilwise Big data
Skilwise Big dataSkilwise Big data
Skilwise Big data
 
ADV Slides: Building and Growing Organizational Analytics with Data Lakes
ADV Slides: Building and Growing Organizational Analytics with Data LakesADV Slides: Building and Growing Organizational Analytics with Data Lakes
ADV Slides: Building and Growing Organizational Analytics with Data Lakes
 
Unlocking the Value of Your Data Lake
Unlocking the Value of Your Data LakeUnlocking the Value of Your Data Lake
Unlocking the Value of Your Data Lake
 
Skillwise Big Data part 2
Skillwise Big Data part 2Skillwise Big Data part 2
Skillwise Big Data part 2
 
Next Gen Analytics Going Beyond Data Warehouse
Next Gen Analytics Going Beyond Data WarehouseNext Gen Analytics Going Beyond Data Warehouse
Next Gen Analytics Going Beyond Data Warehouse
 
When and How Data Lakes Fit into a Modern Data Architecture
When and How Data Lakes Fit into a Modern Data ArchitectureWhen and How Data Lakes Fit into a Modern Data Architecture
When and How Data Lakes Fit into a Modern Data Architecture
 
MongoDB Breakfast Milan - Mainframe Offloading Strategies
MongoDB Breakfast Milan -  Mainframe Offloading StrategiesMongoDB Breakfast Milan -  Mainframe Offloading Strategies
MongoDB Breakfast Milan - Mainframe Offloading Strategies
 

More from Dmitry Anoshin

Building Modern Data Platform with AWS
Building Modern Data Platform with AWSBuilding Modern Data Platform with AWS
Building Modern Data Platform with AWSDmitry Anoshin
 
Cloud Analytics Use Cases and Architecture, Math Marketing Conference, Russia...
Cloud Analytics Use Cases and Architecture, Math Marketing Conference, Russia...Cloud Analytics Use Cases and Architecture, Math Marketing Conference, Russia...
Cloud Analytics Use Cases and Architecture, Math Marketing Conference, Russia...Dmitry Anoshin
 
Victoria Tableau User Group - Getting started with Tableau
Victoria Tableau User Group - Getting started with TableauVictoria Tableau User Group - Getting started with Tableau
Victoria Tableau User Group - Getting started with TableauDmitry Anoshin
 
Hey, what is about data?
Hey, what is about data?Hey, what is about data?
Hey, what is about data?Dmitry Anoshin
 
Enterprise Data World 2018 - Building Cloud Self-Service Analytical Solution
Enterprise Data World 2018 - Building Cloud Self-Service Analytical SolutionEnterprise Data World 2018 - Building Cloud Self-Service Analytical Solution
Enterprise Data World 2018 - Building Cloud Self-Service Analytical SolutionDmitry Anoshin
 
AWS User Group: Building Cloud Analytics Solution with AWS
AWS User Group: Building Cloud Analytics Solution with AWSAWS User Group: Building Cloud Analytics Solution with AWS
AWS User Group: Building Cloud Analytics Solution with AWSDmitry Anoshin
 
My experience of writing technical books
My experience of writing technical booksMy experience of writing technical books
My experience of writing technical booksDmitry Anoshin
 
Business objects activities web intelligence
Business objects activities web intelligenceBusiness objects activities web intelligence
Business objects activities web intelligenceDmitry Anoshin
 
Splunk 6.2 new features
Splunk 6.2 new featuresSplunk 6.2 new features
Splunk 6.2 new featuresDmitry Anoshin
 
Business Analytics Paradigm Change
Business Analytics Paradigm ChangeBusiness Analytics Paradigm Change
Business Analytics Paradigm ChangeDmitry Anoshin
 
SAP BO and Teradata best practices
SAP BO and Teradata best practicesSAP BO and Teradata best practices
SAP BO and Teradata best practicesDmitry Anoshin
 
Splunk Digital Intelligence
Splunk Digital IntelligenceSplunk Digital Intelligence
Splunk Digital IntelligenceDmitry Anoshin
 
Role of Tableau on the Data Discovery Market
Role of Tableau on the Data Discovery MarketRole of Tableau on the Data Discovery Market
Role of Tableau on the Data Discovery MarketDmitry Anoshin
 
SAP Lumira - Building visualizations
SAP Lumira - Building visualizationsSAP Lumira - Building visualizations
SAP Lumira - Building visualizationsDmitry Anoshin
 
SAP Lumira - Acquiring data
SAP Lumira - Acquiring dataSAP Lumira - Acquiring data
SAP Lumira - Acquiring dataDmitry Anoshin
 
SAP Lumira - Enriching data
SAP Lumira - Enriching dataSAP Lumira - Enriching data
SAP Lumira - Enriching dataDmitry Anoshin
 
Microstrategy for Retailer Company
Microstrategy for Retailer CompanyMicrostrategy for Retailer Company
Microstrategy for Retailer CompanyDmitry Anoshin
 
SAP BusinessObjects 4.1 Web Intelligence Report Development
SAP BusinessObjects 4.1 Web Intelligence Report DevelopmentSAP BusinessObjects 4.1 Web Intelligence Report Development
SAP BusinessObjects 4.1 Web Intelligence Report DevelopmentDmitry Anoshin
 

More from Dmitry Anoshin (20)

Building Modern Data Platform with AWS
Building Modern Data Platform with AWSBuilding Modern Data Platform with AWS
Building Modern Data Platform with AWS
 
Cloud Analytics Use Cases and Architecture, Math Marketing Conference, Russia...
Cloud Analytics Use Cases and Architecture, Math Marketing Conference, Russia...Cloud Analytics Use Cases and Architecture, Math Marketing Conference, Russia...
Cloud Analytics Use Cases and Architecture, Math Marketing Conference, Russia...
 
Victoria Tableau User Group - Getting started with Tableau
Victoria Tableau User Group - Getting started with TableauVictoria Tableau User Group - Getting started with Tableau
Victoria Tableau User Group - Getting started with Tableau
 
Hey, what is about data?
Hey, what is about data?Hey, what is about data?
Hey, what is about data?
 
Enterprise Data World 2018 - Building Cloud Self-Service Analytical Solution
Enterprise Data World 2018 - Building Cloud Self-Service Analytical SolutionEnterprise Data World 2018 - Building Cloud Self-Service Analytical Solution
Enterprise Data World 2018 - Building Cloud Self-Service Analytical Solution
 
AWS User Group: Building Cloud Analytics Solution with AWS
AWS User Group: Building Cloud Analytics Solution with AWSAWS User Group: Building Cloud Analytics Solution with AWS
AWS User Group: Building Cloud Analytics Solution with AWS
 
Tableau API
Tableau APITableau API
Tableau API
 
My experience of writing technical books
My experience of writing technical booksMy experience of writing technical books
My experience of writing technical books
 
Business objects activities web intelligence
Business objects activities web intelligenceBusiness objects activities web intelligence
Business objects activities web intelligence
 
Splunk 6.2 new features
Splunk 6.2 new featuresSplunk 6.2 new features
Splunk 6.2 new features
 
Business Analytics Paradigm Change
Business Analytics Paradigm ChangeBusiness Analytics Paradigm Change
Business Analytics Paradigm Change
 
SAP BO and Teradata best practices
SAP BO and Teradata best practicesSAP BO and Teradata best practices
SAP BO and Teradata best practices
 
Exploring Splunk
Exploring SplunkExploring Splunk
Exploring Splunk
 
Splunk Digital Intelligence
Splunk Digital IntelligenceSplunk Digital Intelligence
Splunk Digital Intelligence
 
Role of Tableau on the Data Discovery Market
Role of Tableau on the Data Discovery MarketRole of Tableau on the Data Discovery Market
Role of Tableau on the Data Discovery Market
 
SAP Lumira - Building visualizations
SAP Lumira - Building visualizationsSAP Lumira - Building visualizations
SAP Lumira - Building visualizations
 
SAP Lumira - Acquiring data
SAP Lumira - Acquiring dataSAP Lumira - Acquiring data
SAP Lumira - Acquiring data
 
SAP Lumira - Enriching data
SAP Lumira - Enriching dataSAP Lumira - Enriching data
SAP Lumira - Enriching data
 
Microstrategy for Retailer Company
Microstrategy for Retailer CompanyMicrostrategy for Retailer Company
Microstrategy for Retailer Company
 
SAP BusinessObjects 4.1 Web Intelligence Report Development
SAP BusinessObjects 4.1 Web Intelligence Report DevelopmentSAP BusinessObjects 4.1 Web Intelligence Report Development
SAP BusinessObjects 4.1 Web Intelligence Report Development
 

Recently uploaded

Machine Learning For Career Growth..pptx
Machine Learning For Career Growth..pptxMachine Learning For Career Growth..pptx
Machine Learning For Career Growth..pptxbenishzehra469
 
Using PDB Relocation to Move a Single PDB to Another Existing CDB
Using PDB Relocation to Move a Single PDB to Another Existing CDBUsing PDB Relocation to Move a Single PDB to Another Existing CDB
Using PDB Relocation to Move a Single PDB to Another Existing CDBAlireza Kamrani
 
Jpolillo Amazon PPC - Bid Optimization Sample
Jpolillo Amazon PPC - Bid Optimization SampleJpolillo Amazon PPC - Bid Optimization Sample
Jpolillo Amazon PPC - Bid Optimization SampleJames Polillo
 
2024 Q1 Tableau User Group Leader Quarterly Call
2024 Q1 Tableau User Group Leader Quarterly Call2024 Q1 Tableau User Group Leader Quarterly Call
2024 Q1 Tableau User Group Leader Quarterly Calllward7
 
Artificial_General_Intelligence__storm_gen_article.pdf
Artificial_General_Intelligence__storm_gen_article.pdfArtificial_General_Intelligence__storm_gen_article.pdf
Artificial_General_Intelligence__storm_gen_article.pdfscitechtalktv
 
How can I successfully sell my pi coins in Philippines?
How can I successfully sell my pi coins in Philippines?How can I successfully sell my pi coins in Philippines?
How can I successfully sell my pi coins in Philippines?DOT TECH
 
Innovative Methods in Media and Communication Research by Sebastian Kubitschk...
Innovative Methods in Media and Communication Research by Sebastian Kubitschk...Innovative Methods in Media and Communication Research by Sebastian Kubitschk...
Innovative Methods in Media and Communication Research by Sebastian Kubitschk...correoyaya
 
Tabula.io Cheatsheet: automate your data workflows
Tabula.io Cheatsheet: automate your data workflowsTabula.io Cheatsheet: automate your data workflows
Tabula.io Cheatsheet: automate your data workflowsalex933524
 
basics of data science with application areas.pdf
basics of data science with application areas.pdfbasics of data science with application areas.pdf
basics of data science with application areas.pdfvyankatesh1
 
Exploratory Data Analysis - Dilip S.pptx
Exploratory Data Analysis - Dilip S.pptxExploratory Data Analysis - Dilip S.pptx
Exploratory Data Analysis - Dilip S.pptxDilipVasan
 
Pre-ProductionImproveddsfjgndflghtgg.pptx
Pre-ProductionImproveddsfjgndflghtgg.pptxPre-ProductionImproveddsfjgndflghtgg.pptx
Pre-ProductionImproveddsfjgndflghtgg.pptxStephen266013
 
Supply chain analytics to combat the effects of Ukraine-Russia-conflict
Supply chain analytics to combat the effects of Ukraine-Russia-conflictSupply chain analytics to combat the effects of Ukraine-Russia-conflict
Supply chain analytics to combat the effects of Ukraine-Russia-conflictJack Cole
 
how can i exchange pi coins for others currency like Bitcoin
how can i exchange pi coins for others currency like Bitcoinhow can i exchange pi coins for others currency like Bitcoin
how can i exchange pi coins for others currency like BitcoinDOT TECH
 
Webinar One View, Multiple Systems No-Code Integration of Salesforce and ERPs
Webinar One View, Multiple Systems No-Code Integration of Salesforce and ERPsWebinar One View, Multiple Systems No-Code Integration of Salesforce and ERPs
Webinar One View, Multiple Systems No-Code Integration of Salesforce and ERPsCEPTES Software Inc
 
Business update Q1 2024 Lar España Real Estate SOCIMI
Business update Q1 2024 Lar España Real Estate SOCIMIBusiness update Q1 2024 Lar España Real Estate SOCIMI
Business update Q1 2024 Lar España Real Estate SOCIMIAlejandraGmez176757
 
AI Imagen for data-storytelling Infographics.pdf
AI Imagen for data-storytelling Infographics.pdfAI Imagen for data-storytelling Infographics.pdf
AI Imagen for data-storytelling Infographics.pdfMichaelSenkow
 
社内勉強会資料_LLM Agents                              .
社内勉強会資料_LLM Agents                              .社内勉強会資料_LLM Agents                              .
社内勉強会資料_LLM Agents                              .NABLAS株式会社
 
2024-05-14 - Tableau User Group - TC24 Hot Topics - Tableau Pulse and Einstei...
2024-05-14 - Tableau User Group - TC24 Hot Topics - Tableau Pulse and Einstei...2024-05-14 - Tableau User Group - TC24 Hot Topics - Tableau Pulse and Einstei...
2024-05-14 - Tableau User Group - TC24 Hot Topics - Tableau Pulse and Einstei...elinavihriala
 
2024 Q2 Orange County (CA) Tableau User Group Meeting
2024 Q2 Orange County (CA) Tableau User Group Meeting2024 Q2 Orange County (CA) Tableau User Group Meeting
2024 Q2 Orange County (CA) Tableau User Group MeetingAlison Pitt
 

Recently uploaded (20)

Machine Learning For Career Growth..pptx
Machine Learning For Career Growth..pptxMachine Learning For Career Growth..pptx
Machine Learning For Career Growth..pptx
 
Using PDB Relocation to Move a Single PDB to Another Existing CDB
Using PDB Relocation to Move a Single PDB to Another Existing CDBUsing PDB Relocation to Move a Single PDB to Another Existing CDB
Using PDB Relocation to Move a Single PDB to Another Existing CDB
 
Jpolillo Amazon PPC - Bid Optimization Sample
Jpolillo Amazon PPC - Bid Optimization SampleJpolillo Amazon PPC - Bid Optimization Sample
Jpolillo Amazon PPC - Bid Optimization Sample
 
Slip-and-fall Injuries: Top Workers' Comp Claims
Slip-and-fall Injuries: Top Workers' Comp ClaimsSlip-and-fall Injuries: Top Workers' Comp Claims
Slip-and-fall Injuries: Top Workers' Comp Claims
 
2024 Q1 Tableau User Group Leader Quarterly Call
2024 Q1 Tableau User Group Leader Quarterly Call2024 Q1 Tableau User Group Leader Quarterly Call
2024 Q1 Tableau User Group Leader Quarterly Call
 
Artificial_General_Intelligence__storm_gen_article.pdf
Artificial_General_Intelligence__storm_gen_article.pdfArtificial_General_Intelligence__storm_gen_article.pdf
Artificial_General_Intelligence__storm_gen_article.pdf
 
How can I successfully sell my pi coins in Philippines?
How can I successfully sell my pi coins in Philippines?How can I successfully sell my pi coins in Philippines?
How can I successfully sell my pi coins in Philippines?
 
Innovative Methods in Media and Communication Research by Sebastian Kubitschk...
Innovative Methods in Media and Communication Research by Sebastian Kubitschk...Innovative Methods in Media and Communication Research by Sebastian Kubitschk...
Innovative Methods in Media and Communication Research by Sebastian Kubitschk...
 
Tabula.io Cheatsheet: automate your data workflows
Tabula.io Cheatsheet: automate your data workflowsTabula.io Cheatsheet: automate your data workflows
Tabula.io Cheatsheet: automate your data workflows
 
basics of data science with application areas.pdf
basics of data science with application areas.pdfbasics of data science with application areas.pdf
basics of data science with application areas.pdf
 
Exploratory Data Analysis - Dilip S.pptx
Exploratory Data Analysis - Dilip S.pptxExploratory Data Analysis - Dilip S.pptx
Exploratory Data Analysis - Dilip S.pptx
 
Pre-ProductionImproveddsfjgndflghtgg.pptx
Pre-ProductionImproveddsfjgndflghtgg.pptxPre-ProductionImproveddsfjgndflghtgg.pptx
Pre-ProductionImproveddsfjgndflghtgg.pptx
 
Supply chain analytics to combat the effects of Ukraine-Russia-conflict
Supply chain analytics to combat the effects of Ukraine-Russia-conflictSupply chain analytics to combat the effects of Ukraine-Russia-conflict
Supply chain analytics to combat the effects of Ukraine-Russia-conflict
 
how can i exchange pi coins for others currency like Bitcoin
how can i exchange pi coins for others currency like Bitcoinhow can i exchange pi coins for others currency like Bitcoin
how can i exchange pi coins for others currency like Bitcoin
 
Webinar One View, Multiple Systems No-Code Integration of Salesforce and ERPs
Webinar One View, Multiple Systems No-Code Integration of Salesforce and ERPsWebinar One View, Multiple Systems No-Code Integration of Salesforce and ERPs
Webinar One View, Multiple Systems No-Code Integration of Salesforce and ERPs
 
Business update Q1 2024 Lar España Real Estate SOCIMI
Business update Q1 2024 Lar España Real Estate SOCIMIBusiness update Q1 2024 Lar España Real Estate SOCIMI
Business update Q1 2024 Lar España Real Estate SOCIMI
 
AI Imagen for data-storytelling Infographics.pdf
AI Imagen for data-storytelling Infographics.pdfAI Imagen for data-storytelling Infographics.pdf
AI Imagen for data-storytelling Infographics.pdf
 
社内勉強会資料_LLM Agents                              .
社内勉強会資料_LLM Agents                              .社内勉強会資料_LLM Agents                              .
社内勉強会資料_LLM Agents                              .
 
2024-05-14 - Tableau User Group - TC24 Hot Topics - Tableau Pulse and Einstei...
2024-05-14 - Tableau User Group - TC24 Hot Topics - Tableau Pulse and Einstei...2024-05-14 - Tableau User Group - TC24 Hot Topics - Tableau Pulse and Einstei...
2024-05-14 - Tableau User Group - TC24 Hot Topics - Tableau Pulse and Einstei...
 
2024 Q2 Orange County (CA) Tableau User Group Meeting
2024 Q2 Orange County (CA) Tableau User Group Meeting2024 Q2 Orange County (CA) Tableau User Group Meeting
2024 Q2 Orange County (CA) Tableau User Group Meeting
 

Building Modern Data Platform with Microsoft Azure

  • 1. Building Modern Cloud Analytics Solution Dmitry Anoshin
  • 2. Outline • About Me • Role of Analytics • History of Cloud • Analytics powered by Microsoft Azure • DW modernization Project • Use cases and Challenges • Alternative Solution with Azure
  • 4. About Myself • Work with Business Intelligence since 2007
  • 6. Technical Skills Matrix 2015 2010 2007 Data Warehouse ETL/ELT Business Intelligence Big Data Cloud Analytics (AWS, Azure, GCP) Machine Learning 2019
  • 7. Other Activities Jumpstart Sno wflake: A Step- by-Step Guide to Modern Cloud Analytics. • Victoria Power BI andVictoria SQL Server meetup • Victoria andVancouverTableau User Group • Conferences (EDW 2018, 2019, Data Architecture Summit) • Amazon internal conferences
  • 9. BusinessValue Stakeholders Employees Customers Value ”The goal of any organization is to generateValue” The Future of Competition. https://www.amazon.com/Future-Competition-Co-Creating-Unique-Customers/dp/1578519535
  • 10. BIValue Chain Stakeholders Employees Customers Value Decisions Data Value creation based on effective decisions Effective decisions based on accurate information
  • 11. For Data to be a differentiator, customers need to be able to… • Capture and store new non-relational data at PB-EB scale in real time • Discover value in a new type of analytics that go beyond batch reporting to incorporate real-time, predictive, voice, and image recognition • Democratize access to data in a secure and governed way New types of analytics Dashboards Predictive Image Recognition VoiceReal-time New types of data
  • 13. Cloud Early History 1970 Time Sharing Concept by GE 1977 Cloud symbol used in ARPANET 1990 VPN by telecom 1993 Cloud refer to Distributed Computing 1994 Cloud metaphor for virtualized services
  • 14. Cloud Recent History 2002 AWS 2006 AWS Elastic Compute Cloud 2006 Google Docs 2008 Google App Engine 2008 Microsoft Announced Azure 2010 Microsoft Azure
  • 15. Why moving to the Cloud? • Elasticity • Pay for what you need • Fail fast • Fast time to market • Secure • Reliable • Business SLA
  • 16. Downsides of on-premise solution Scale Constrained Up-front cost Maintenance Resources Tuning and Deployment
  • 17. Cloud Restrictions -> Hybrid Clouds Sensitive Data Data Moving Cost Public/Private Cloud
  • 19. Cloud Service Models – friendly version
  • 21. Microsoft Azure for Analytics
  • 22. Data Analytics with Azure • Data Factory • Integration Service • Kafka • Event Hub • Data Lake Gen 1 • Data Lake Gen 2 • Blob Storage • HD Insight • Data Lake Analytics • Streaming Analytics • PolyBase • CosmosDB • SQL DW • Analysis Service • SQL Database • SQL Server in VM • Cosmos DB Data Integration and Transformation Data Warehouse and Data bases Big Data • Analysis Service • ML Analytics • Business Intelligence Analytics
  • 24. BI/DW (before) Storage LayerSource Layer Ad-hoc SQL SFTP Data Warehouse ETL (PL/SQL)Files Inventory Sales Access Layer
  • 25. Cloud Migration Strategy Lift & Shift • Typical Approach • Move all-at-once • Target platform then evolve • Approach gets you to the cloud quickly • Relatively small barrier to learning new technology since it tends to be a close fit Split & Flip • Split application into logical functional data layers • Match the data functionality with the right technology • Leverage the wide selection of tools onAWS to best fit the need • Move data in phases — prototype, learn and perfect
  • 26. Migration Approach Useful tools: • Total Cost Ownership (TCO) Calculator • Azure Database Migration Service • Azure Migration Assistant
  • 27.
  • 29. What is Azure DW? • Decouple Storage and Compute • MPP • Distribution Styles: Hash/Robin/Replicat e
  • 30. MPP?
  • 31. SQL Database vs SQL Data Warehouse
  • 32. What is Azure Data Factory? Azure Data Factory (ADF) is Microsoft’s fully managed ELT service in the cloud that’s delivered as a Platform as a Service (PaaS)
  • 33. Lack of Notification Problem: Users are missing emails or they jump to spam. Solution: Leverage Messenger with Webhooks. (Slack, Chime or so on).
  • 34. Lack of Logging Problem: We didn’t have any detail logs about our ETL performance and we didn’t have any insights. Solution: Collecting logs and events. In addition, we are able to collect logs on any level of jobs and transformation.
  • 35. Self-Service BI Problem: Business Users wants Interactive and Self-Service tool. Fast time to Market and less dependency on IT. Solution: Implement modern Visual Analytics Platform
  • 36. Marketing Automation Problem: Marketing team wants “Move Fast and Break Things”. Solution: Using ADF the gave Marketing template jobs and they doing their jobs themselves. Affiliates Insights
  • 37. Integration with BI Problem: Having best BI tool doesn’t guaranty good SLA. Solution: Build Integration between Matillion ETL and Tableau based on Trigger. Add data quality checks.
  • 38. Evolving to Cloud Data Analytics Platform
  • 39. Streaming Data Problem: Organization is using NoSQL database and mobile application. It is critical to deliver near real time analytics Solution: Using Apache Kaffka, we are able to stream data into the Data lake and query this data in near real time Data Lake Dashboard Kafka CosmoDB Mobile App
  • 40. Clickstream Analytics Problem: Business wants to analyze Bots traffics and discover broken URLs. Access logs are ~50GB per day, 5600 log files per day. Solution: Leveraging Databricks in order to produce Parquet file and store in Azure Data Lake Gen2. User are able query it with T-SQL and BI Tools. Databricks ParquetBlob Storage Access Logs Load Balancer Data Lake Data Factory SQL DW Query with SQL or Databricks
  • 41. DevOps onboarding Problem: Solution isn’t reliable and could easy break. As a result end users will experience bad experience and it will affect business decisions. Solution: Onboarding Continuous Integration methodology for Cloud Data Platform • Agile and Kanban board • Code branching (Git) • Gated check-ins • Automated Tests • Build • Release
  • 42. Evolving to Cloud Data Analytics Platform

Editor's Notes

  1. The cloud symbol was used to represent networks of computing equipment in the original ARPANET by as early as 1977 The term cloud was used to refer to platforms for distributed computing as early as 1993, when Apple spin-off General Magic and AT&T used it in describing their (paired) Telescript and PersonaLink technologies.
  2. The cloud symbol was used to represent networks of computing equipment in the original ARPANET by as early as 1977 The term cloud was used to refer to platforms for distributed computing as early as 1993, when Apple spin-off General Magic and AT&T used it in describing their (paired) Telescript and PersonaLink technologies.