Analytics in a day

Analytics in a day
Cloud analytics in the age of
self-service and data science

The heart of analytics
Section 1
Data businesses need
data warehouses
Section 2
Data warehouses &
data lakes come together
Section 3
BI & DW come together
Section 4
The cloud for modern analytics
Section 5
A new class of analytics

Section 1
Data businesses need
data warehouses

Is the data warehouse
still relevant?
What’s changed since 1988?
A 30-year-old architecture, still going strong
Commerce and technology
The data warehouse itself

Today, all businesses are
data businesses
Data is the lifeblood of modern work

All data businesses need
to be analytic businesses
Without analytics data is a cost center,
not a resource

Analytic businesses need
to evolve data science
Every business has opportunities to make
analytics faster, easier, and more insightful

Store
Data Ingestion Big Data Data Warehousing
The cloud data warehouse in the data-driven business

Store

Store
Cloud data
SaaS data
On-premises data
Devices data

Azure Synapse Analytics
Store

Store

Section 2
Data warehouses &
data lakes come together

80%
report struggling to
become mature users
of data*
55%
report data silos and
data management
difficulties as roadblocks*
* Harvard Business Review (2019), Understanding why analytics strategies fall short for some, but not for others
Analytics & AI is the #1 investment for business leaders,
however they struggle to maximize ROI

Big data
Experimentation
Fast exploration
Semi-structured
Data science
OR
Relational data
Proven security & privacy
Dependable performance
Structured
Business analytics
Data lake Data warehouse
Businesses are forced to maintain
two critical, yet independent analytics systems

©Microsoft Corporation
Azure
It’s a challenge to integrate these areas with security
Big data
Experimentation
Fast exploration
Semi-structured
Data science
OR
Relational data
Structured
Business analytics
Securing a business-critical
environment needs enterprise-
class features such as row and
column security
Relying on views only increases the
number of artifacts to be managed

Azure
Big data
Experimentation
Fast exploration
Semi-structured
Data science
OR
Relational data
Structured
Business analytics
column security
Securing a platform used for
both experimentation and
production demands the ability
to discover sensitive data
You can’t secure what you don’t know

Azure
Big data
Experimentation
Fast exploration
Semi-structured
Data science
OR
Relational data
Structured
Business analytics
column security
Securing a platform used for
both experimentation and
production demands the ability
to discover sensitive data
You can’t secure what you don’t know
User management in an
environment with diverse use
cases requires integrated
enterprise authentication

Azure
It’s a challenge to manage these diverse workloads
Big data
Experimentation
Fast exploration
Semi-structured
Data science
OR
Relational data
Structured
Business analytics
In this mixed
environment, not all
workloads have the
same priority
Mission-critical warehouse jobs are demanding and
predictable
Exploratory analytics and data science are important
but unpredictable
Executive queries are high-profile, although rare

Azure
It’s a challenge to manage these diverse workloads
Big data
Experimentation
Fast exploration
Semi-structured
Data science
OR
Relational data
Structured
Business analytics
In this mixed
environment, not all
workloads have the
same priority
Mission-critical warehouse jobs are demanding and
predictable
Exploratory analytics and data science are important
but unpredictable
Executive queries are high-profile, although rare
Throwing multiple
clusters at these
scenarios is easy,
especially in
lightweight scenarios,
however:
Each cluster has a cost
Bringing a cluster online has a lag, which can impact
important, if rare, workloads
It takes compute to maintain large caches

Azure
It’s a challenge to build integrated lifecycle management
Big data
Experimentation
Fast exploration
Semi-structured
Data science
OR
Relational data
Structured
Business analytics
Data Scientists
need a real
data lake
Query over open file formats such as Parquet and
ORC natively without loading the data to a
proprietary cluster
Enable data science platforms and the EDW to share a
common data set

Azure
It’s a challenge to build integrated lifecycle management
Big data
Experimentation
Fast exploration
Semi-structured
Data science
OR
Relational data
Structured
Business analytics
Developers need
real development
tools
Version control
Continuous integration and deployment
Unit testing, integration testing and load testing
Data Scientists
need a real
data lake
Query over open file formats such as Parquet and
ORC natively without loading the data to a
proprietary cluster
Enable data science platforms and the EDW to share a
common data set

Azure
Welcome to limitless
Ease of use
Fast exploration
Quick to start
Proven security
Airtight privacy
Data warehousing & big data analytics—all in one service
Azure meets these challenges,
with a single service to provide limitless analytics

Section 3
Azure meets these challenges,
with a single service to provide limitless analytics

Section 3

The new economy
thrives on data literacy
Communicating with data is a critical skill in
the new economy

Users and IT must come
together in the new enterprise
Get over the IT / business divide

Governance and self-service
enhance decision-making
Governance is not about making the right decisions,
it is about making decisions the right way

The importance of data models
BI models Power BI
• Built and maintained by business users or BI developers
• Use enterprise models, departmental data, and external sources
• Focused on a single subject area, but often widely shared
Machine Learning
models
Azure Synapse
Analytics
• Built and maintained by data scientists
• Mostly developed from raw sources in the data lake
• Often experimental, needing a data engineer for production use
Azure Synapse
AnalyticsEnterprise models
• Built and maintained by IT architects
• Consolidated data from many systems
• Centralized as an authoritative source for reporting and analysis

Enterprise models in the
self-service environment
If business users
are tech-smart and
data literate, why
do they need
enterprise models?
Consistency
Some business processes can be built once and shared as a
corporate standard
Governance
Certain data sets need complex security and privacy controls
Efficiency
No need to repeat design, preparing, and loading or securing
Line-of-business sources
Data ingestion &
transformation
Enterprise models
Azure Synapse
Analytics
Power BI

BI models in the enterprise
environment
If enterprise
models are so
important, why do
users need self-
service BI models?
Flexibility
Some data sets are temporary, external, or ad-hoc don’t need to
be consolidated
Efficiency
Tech-smart business users have fresh and innovative ideas they need to
explore with agility
Ad-hoc, departmental and
external sources
Line-of-business sources
Data ingestion &
transformation
Power BI
Enterprise models
Azure Synapse
Analytics
BI models

Section 4
Data science models in the
enterprise environment
What is the role of
the data
warehouse with
data science?
Integrating results with enterprise models
Making the results of data science easily available for business functions
Serving enterprise data for data scientists
Helps ensure consistency across diverse analyses
Power BI
Azure Synapse
Analytics
Azure
Databricks
Enterprise models
Azure Synapse
Analytics
Data science results

Section 4

Modern businesses
succeed in the cloud
The cloud is the default environment for
new technology initiatives

Cloud security offers a
new level of protection
Businesses benefit from built-in security
found only in the cloud

Price, performance, and agility
A cloud analytics platform
is an economic breakthrough

Structured, unstructured, and streaming data
integrated in a single, scalable, environment
A cloud analytics platform
is the hub for all data models

BI
Bring together the best of both worlds with the market-
leading BI service and the industry-leading analytics platform
Power BI can analyze and visualize
massive volumes of data
Azure Synapse Analytics provides a
scalable platform to enable real-time BI
Analytics

Section 5
Power BI can analyze and
visualize massive volumes of data
provides a scalable platform
to enable real-time BI
Azure Machine Learning natively
integrates with Azure Synapse &
Power BI to democratize AI across
your business
BI Analytics Machine learning

Section 5

Unified experience
Azure Synapse Studio
Integration Management Monitoring Security
Analytics runtimes
SQL
Azure Data Lake Storage
Azure Machine
Learning
On-premises data
Cloud data
SaaS data
Streaming data
Power BI
Azure Synapse lies at the heart of business, AI, and BI

Unified experienceAzure Synapse Studio
Integration Management Monitoring SecuritySQL
Azure Machine
Learning
On-premises
data
Cloud
data
SaaS data
Streaming
data
Cloud analytics has taken a leap forward
with a unified, unmatched platform
Power BI

Limitless analytics service with unmatched
time to insight

Introducing Azure
Synapse Analytics
A limitless analytics service with unmatched
time to insight, that delivers insights from all
your data, across data warehouses and big
data analytics systems, with blazing speed
Simply put, Azure Synapse is Azure SQL Data
Warehouse evolved
We have taken the same industry leading data
warehouse and elevated it to a whole new level of
performance and capabilities

Azure Synapse
Analytics
Snowflake
Standard
Amazon
Redshift
Google
BigQuery
per byte
$33
$103
$48
…$564
94% less
TPC-H benchmark comparison
Price-performance | Lower is better
* GigaOm TPC-H benchmark report, January 2019, “GigaOm report: Data Warehouse in the Cloud Benchmark
With the best price-performance
in the business
Up to 14x faster and costs 94%
less than other cloud providers
A breakthrough in the cost of enterprise analytics

Data consolidation using
Migration to the cloud for
efficient business operations
Using Azure Synapse Analytics
for predictive analytics
Organizations that fully harness their data outperform

At the core of all use cases is…Azure Synapse Analytics
Real-time
analytics
Modern data
warehousing
Advanced
analytics
"We want to analyze
data coming from
multiple sources and
in varied formats"
"We want to leverage
the analytics platform
for advanced fraud
detection"
“We’re trying to get
insights from our
devices in real-time”
Cloud-scale analytics

Store
Ingest Transform Model & serve Visualize
Modern Data Warehouse

Synapse SQL
Apache Spark
for Synapse
Synapse Pipelines Synapse Studio

Query and analyze data with
T-SQL using both provisioned
and serverless models
Quickly create notebooks with
your choice of Python, Scala,
SparkSQL, and .NET for
Apache Spark
Build end-to-end workflows
for your data movement and
data processing scenarios
Execute all data tasks with a
simple UI and unified
environment
Synapse SQL
Apache Spark
for Synapse
Synapse Pipelines Synapse Studio

Integrated analytics platform for AI, BI, and continuous intelligence
Platform
Azure
Data Lake Storage
Common Data Model
Enterprise Security
Optimized for Analytics
Data lake integrated and Common Data
Model aware
METASTORE
SECURITY
MANAGEMENT
MONITORING
Integrated platform services
for, management, security, monitoring,
and metastore
DATA INTEGRATION
Analytics Runtimes
Integrated analytics runtimes available
provisioned and serverless
Synapse SQL offering T-SQL for batch,
streaming, and interactive processing
Synapse Spark for big data processing
with Python, Scala, R and .NET
PROVISIONED (DW) SERVERLESS
Form Factors
SQL
Languages
Python .NET Java Scala R
Multiple languages suited to different
analytics workloads
Experience Synapse Studio
SaaS developer experiences for code
free and code first
Artificial Intelligence / Machine Learning / Internet of Things
Intelligent Apps / Business Intelligence
Designed for analytics workloads at any
scale

Integrated analytics platform for AI, BI, and continuous intelligence
Platform
Azure
Data Lake Storage
Common Data Model
Enterprise Security
Optimized for Analytics
METASTORE
SECURITY
MANAGEMENT
MONITORING
DATA INTEGRATION
Analytics Runtimes
PROVISIONED (DW) SERVERLESS
Form Factors
SQL
Languages
Python .NET Java Scala R
Experience Synapse Studio
Artificial Intelligence / Machine Learning / Internet of Things
Intelligent Apps / Business Intelligence
Connected Services
Azure Data Catalog
Azure Data Share
Azure Databricks
Azure HDInsight
Azure Machine Learning
Power BI
3rd Party Integration

Synapse Studio

Synapse Studio is divided into
Activity hubs
Hubs organize the tasks needed for
building analytics solutions
Synapse Studio
Overview Data
Monitor Manage
Quick-access to common
gestures, most-recently used
items, and links to tutorials
and documentation.
Explore structured and
unstructured data
Centralized view of all resource
usage and activities in the
workspace.
Configure the workspace,
pool, access to artifacts
Develop
Write code and the define
business logic of the pipeline
via notebooks, SQL scripts,
Data flows, etc.
Orchestrate
Design pipelines that that
move and transform data.

Start coding immediately
Begin with SQL scripts, notebook,
data flow and more
Overview hub

Explore data inside the workspace
and in linked storage accounts
Data Hub

Explore data inside the workspace
and in linked storage accounts
Data Hub
ADLS Gen2 Account
Container (filesystem)
Filepath

Preview a sample of your data
Data Hub –
Storage accounts

Manage access and configure
standard POSIX ACLs on files and
folders
Data Hub –
Storage accounts

Analyze SQL scripts or notebooks
with two simple actions
Autogenerate T-SQL or PySpark
Data Hub –
Storage accounts

SQL pool
SQL serverless
Apache Spark
Explore workspace databases
Databases

Author SQL Scripts
Execute SQL script on provisioned SQL
Pool or SQL Serverless
Publish individual SQL script or multiple
SQL scripts through Publish all feature
Support for languages and Intellisense
Develop hub -
SQL scripts

View results in table or chart form and
export results in several popular formats
Develop hub -
SQL scripts

Data flows are a visual way of
specifying how to transform data,
providing a code-free experience
Develop hub -
Data flows

Develop hub –
Power BI
Create Power BI reports in the workspace
Provide access to published reports in the
workspace
Update reports in real time from Synapse
workspace and show on Power BI service
Visually explore and analyze data

Synapse SQL

Best-in-class
Price-performance is calculated by GigaOm as the TPC-H metric of cost of ownership divided by composite query.
Results based on GigaOm’s TPC-H results, published in January 2019
Leader in price per performance

Amazon Redshift
$0
$10
$20
$30
$40
$50
$60
$550
$600
$40
$33
$47
$54
$48
$51
$564
Price-performance @ 30TB
Lower is Better
Google BigQueryAzure Synapse Analytics Snowflake
$103
$110
$152
$80
$100
$120
$140
Best-in-class

Price-performance @ 30TB
Lower is Better
Amazon
Redshift
Google BigQuery
Flat Rate
Azure Synapse
Analytics
Google BigQuery
Flat Rate
Snowflake
Standard
$1310
$570
$309
$206
$286
$153
$0
$100
$200
$300
$400
$500
$600
Snowflake
Standard
Best-in-class

--T-SQL syntax for scoring data in SQL DW
SELECT
d.*, p.Score
FROM PREDICT(MODEL = @onnx_model, DATA = dbo.mytable AS
d)
WITH (Score float) AS p;
Upload
models
Machine learning
enabled DW
Native PREDICT-ion
T-SQL based experience
(interactive/batch scoring)
Interoperability with other
models built elsewhere
Scoring executed where the
data lives
T-SQL Language
Data Warehouse
Data
+
Score models
Model Predictions
=
Synapse SQL
Create models

Event Hubs
IoT Hub
T-SQL language
Built-in streaming ingestion & analytics
Streaming Ingestion Data Warehouse
Synapse SQL
Heterogenous
data preparation
and ingestion
Native SQL streaming
High throughput ingestion
(up to 200MB/sec)
Delivery latencies in seconds
Ingestion throughput scales with
compute scale
Analytics capabilities

Empower more users
per data warehouse
Leverage up to 128 concurrent
slots, simultaneously, on a single
data warehouse
Number of simultaneous workloads
increases with data warehouse capacity
Utilize preset functions to allocate
resources that need them the most

Intra cluster workload isolation
(Scale in)
Marketing
CREATE WORKLOAD GROUP Sales
WITH
(
[ MIN_PERCENTAGE_RESOURCE = 60 ]
[ CAP_PERCENTAGE_RESOURCE = 100 ]
[ MAX_CONCURRENCY = 6 ] )
40%
Data
warehouse
Local In-Memory + SSD Cache
Compute
1000c DWU
60%
Sales
60%
100%
Workload aware
query execution
Workload isolation
Multiple workloads share
deployed resources
Reservation or shared resource
configuration
Online changes to workload policies

CREATE MATERIALZIED VIEW vw_ProductSales
WITH (DISTRIBUTION = HASH(ProductKey))
AS
SELECT
ProductName
ProductKey,
SUM(Amount) AS TotalSales
FROM
FactSales fs
INNER JOIN DimProduct dp ON fs.prodkey = dp.prodkey
GROUP BY
ProductName,
ProductKey
See more by scaling
to petabytes

ProductName ProductKey TotalSales
Product A 5453 784,943.00
Product B 763 48,723.00
… … …
FactSales
Table
10B Records
DimProduct
Table
1,000 Records
Materialized View
(1000 Records)
See more by scaling
to petabytes
FactInventory
Table
mvw_ProductSales
1,000 Records
CREATE MATERIALZIED VIEW
mvw_ProductSales
WITH (DISTRIBUTION = HASH(ProductKey))
AS
SELECT
ProductName
ProductKey,
FROM
FactSales fs
INNER JOIN DimProduct dp
ON fs.prodkey = dp.prodkey
GROUP BY
ProductName,
ProductKey
SELECT
<COLUMNS>
FROM FactSales fs
INNER JOIN
SELECT
ProductName
ProductKey,
FROM
FactSales fs
INNER JOIN DimProduct dp
GROUP BY
ProductName,
ProductKey ) ps
INNER JOIN FactInventory
GROUP BY …

Execution2
Cache Hit
~.2 seconds
Execution1
Cache Miss
Regular
Execution
SELECT
ProductName
ProductKey,
FROM
Fact Sales
INNER JOIN DimProduct
GROUP BY
ProductName,
ProductKey
Build confidence in your
data with result set cache
Data
Warehouse
Resultset
Cache

Most secure data
warehouse in the cloud
Multiple levels of security between the
user and the data warehouse
...at no additional cost
Threat Protection
Network Security
Authentication
Access Control
Data Protection
Customer Data

Comprehensive security
Category Feature
Data protection
Data in transit
Data encryption at rest
Data discovery and classification
Access control
Object level security (tables/views)
Row level security
Column level security
Dynamic data masking
SQL login
Authentication Azure active directory
Multi-factor authentication
Virtual networks
Network security Firewall
Azure ExpressRoute
Threat detection
Threat protection Auditing
Vulnerability assessment

Synapse SQL (serverless)

Discovery and
exploration
What’s in this file? How many rows are there? What’s the max value?
SQL serverless reduces data lake exploration to the right-click
Data
transformation
How to convert CSVs to Parquet quickly? How to transform the raw data?
Use the full power of T-SQL to transform the data in the data lake

Overview
An interactive query service that provides T-SQL
queries over high scale data in Azure Storage.
Benefits
Serverless
No infrastructure
Pay only for query execution
No ETL
Offers security
Data integration with Databricks, HDInsight
T-SQL syntax to query data
Supports data in various formats
(Parquet, CSV, JSON)
Support for BI ecosystem
Azure Storage
SQL
Serverless
Query
Power BI
Azure Data
Studio
SSMS
DW
Read and write
data files
Curate and
transform data
Sync table
definitions
Read and write
data files
Azure Synapse Analytics > SQL > SQL serverless

Apache Spark for Synapse

Allows multiple languages in one notebook
%%<Name of language>
Offers use of temporary tables across languages
Support for syntax highlight, syntax error, syntax code
completion, smart indent, and code folding
Export results
Quickly create &
configure notebooks

As notebook cells run, the underlying
Apache Spark application status is
shown, providing immediate feedback
and progress tracking.
Quickly create &
configure notebooks

Synapse Pipelines

Overview
Linked services defines the connection
information needed for pipelines to connect to
external resources
Benefits
Offers 90+ pre-built connectors
Allows easy cross platform data migration
Represents data store or compute resources

Prep and transform data
Mapping dataflow
Code free data transformation at scale
Wrangling dataflow
Code free data preparation at scale

Handle upserts,
updates, deletes
on sql sinks
Add new partition
methods
Add schema
drift support
Add file handling (move
files after read, write files
to file names described
in rows, etc.)
New inventory of
functions (e.g. Hash
functions for row
comparison)
Commonly used ETL
patterns (Sequence
generator/Lookup
transformation/SCD…)
Data lineage – Capturing
sink column lineage &
impact analysis
(invaluable if this is for
enterprise deployment)
Implement commonly
used ETL patterns as
templates (SCD type1,
type2, data vault)
Data flow
Capabilities

Insights for all with
Power BI + Azure
Power up your BI with Azure Synapse

Where do you find yourself on the curve?
Hindsight Insight Foresight
Value
Difficulty
What happened?
Descriptive Analysis
Why did it happen?
Diagnostic Analysis
What will happen?
Predictive Analysis
How can we make it happen?
Prescriptive Analysis

Where do you find yourself on the curve?
Value
Difficulty
What happened?
Why did it happen?
Diagnostic Analysis
What will happen?
Predictive Analysis
BI

BI + Analytics unlock the door to AI, machine learning, and
real-time insights
Value
Difficulty
What happened?
Why did it happen?
Diagnostic Analysis
What will happen?
Predictive Analysis
AnalyticsBI

Power BI can analyze and
visualize massive volumes of data
provides a scalable platform
to enable real-time BI
Azure Machine Learning natively
integrates with Azure Synapse &
Power BI to democratize AI across
your business
BI Analytics Machine learning

Accelerate business value with a powerful analytics platform
Business analysts IT professionals Data scientists
Frictionless
collaboration
Unified
analytics platform
Advanced analytics
and AI
Powerful visualization and
reporting
Unmatched
capabilities
Business value
Common Data Model on Azure Data Lake StorageUnified data
Azure Synapse AnalyticsPower BI
Powerful and
integrated
tooling
Azure Machine Learning

Visualize and
report
Power BI
Model &
serve
Azure Synapse
Analytics
CDM folders
Azure Data Lake
Storage
Respond instantly
Enable instant response times with
Power BI Aggregations on massive
datasets when querying at the
aggregated level
Get granular with your data
Queries at the granular level are
sent to Azure Synapse Analytics
with DirectQuery leveraging its
industry-leading performance
Save money with industry-
leading performance
Azure Synapse Analytics is up to
14x faster and 94% cheaper than
other cloud providers
View reports with a single pane
of glass
Skip the configuration when
connecting to Power BI with
integrated Power BI-authoring
directly in the Azure Synapse Studio
Accelerate business value with a powerful analytics platform

Customers using Azure Synapse & Power BI today
are transforming their business with purpose
27%
Faster time
to insights
271% Average ROI
26%
Lower total cost
of ownership
60%
Increased customer
satisfaction
* Forrester, October 2019, “The Total Economic Impact of Microsoft Azure Analytics with Power BI”

Build Power BI dashboards directly
from Azure Synapse
Azure Synapse + Power BI integration

View published reports in Power BI workspace
Azure Synapse + Power BI

Edit reports in Synapse workspace

Real-time publish on save

Hands-on lab
Build an end-to-end analytics solution
in the Azure Synapse Studio

Get Started Today
Create a free Azure account and get started with Azure Synapse Analytics:
https://azure.microsoft.com/en-us/free/synapse-analytics/
Get in touch with us:
https://info.microsoft.com/ww-landing-contact-me-azure-analytics.html
Attend a virtual Azure Analytics workshop with Informatica:
https://now.informatica.com/Microsoft_CDW_Workshops.html#fbid=uqwtl_SXNFV
Learn more:
https://aka.ms/synapse

Analytics in a day

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Analytics in a day

Similar to Analytics in a day (20)

Recently uploaded

Recently uploaded (20)

Analytics in a day