The document discusses cost-effective big data workload management using Azure Synapse and Azure Data Lake Storage. It highlights modernization strategies, cost optimization features, and the integration benefits of using these technologies, including shared metadata and reduced resource needs. Additionally, it introduces .NET support for Apache Spark, enabling .NET developers to engage in big data analytics without extensive retraining.

1. Running cost effective big data workloads with
Azure Synapse and Azure Data Lake Storage
James Baker, @BigJamesBigData
Michael Rys, @MikeDoesBigData

2. Agenda 1. Modernize your big data workloads
2. Cost improvements with Synapse and ADLS
3. Demo
4. .NET for Apache Spark

3. Traditional on-prem analytics pipeline
Operational
database
Business/custom apps
Operational
database
Operational
database
Enterprise data
warehouse
Data mart
Data mart
Data mart
ETL
ETL
ETL
ETL ETL
ETL
ETL
Reporting
Analytics
Data mining

4. Modern data warehouse
Logs (structured)
Media (unstructured)
Files (unstructured)
Business/custom apps
(structured)
Ingest Prep & train Model & serve
Store
Azure Data Lake Storage
Azure SQL
Data Warehouse
Azure DatabricksAzure Data Factory
Power BI

5. Modern data warehouse with Azure Synapse
Logs (structured)
Media (unstructured)
Files (unstructured)
Business/custom apps
(structured)
Azure
Synapse
Analytics Power BI
Store
Azure Data Lake Storage

6. Modern data warehouse with Azure Synapse
Logs (structured)
Media (unstructured)
Files (unstructured)
Business/custom apps
(structured)
Analytics runtimes
SQL
Common data estate
Shared meta data
Unified experience
Synapse Studio
Store
Azure Data Lake Storage
Power BI

7. Synapse component benefits
Existing tooling and skills
No requirement for retraining or new tooling to work with familiar T-SQL
environments
• .NET Support
Reducing training costs for big
data .NET developers
Provisioned model
 Workload Groups - maximize
resource utilization
 Materialized views and resultset
cache - smaller, cheaper clusters
Apache Spark Synapse SQL
• Hyperspace materialized
views
Requires smaller, cheaper
clusters to achieve the same
tasks
Serverless model
 Paying exactly for what you use - no
overprovisioning
 no clusters to monitor and manage -
lower maintenance costs

8. Cost optimization with Azure Data Lake Storage
Disaggregated compute
and storage with shared
metadata layer
Lifecycle management
for optimizing TCO
Lower compute resources
because of high performance

9. Azure Synapse and ADLS: Integration benefits
Integrated
workload
monitoring
Shared security Query
Acceleration
Shared
metadata
Single
management
portal
Data stewards
apply security
policies only once
Data owners
define data model
once only
Smaller, cheaper
clusters
Reduced
monitoring and
diagnosis costs
Lowers training costs

10. Synapse SQL - Serverless
Shared Metadata Experience
Spark Databases and Tables backed by Parquet
become automatically available in:
• Synapse SQL serverless
• Synapse SQL provisioned
• Synapse Pipelines
as external tables of the same name.
Spark Compute
X Auto-Expose Metadata Objects
CREATE DATABASE DBS1
CREATE SCHEMA $DBS1
CREATE DATABASE DBS1
CREATE TABLE DBS1.T1
CREATE EXTERNAL TABLE DBS1.dbo.T1
CREATE EXTERNAL TABLE $DBS1.T1
Auto-Expose Metadata Objects
Benefits:
• No need to run orchestration jobs to move data or meta
data between computes
• No duplication of data at the storage level
SELECT *
FROM DBS1.dbo.T1
SELECT *
FROM DBS1.T1 SELECT *
FROM $DBS1.T1
Synapse Hive Metastore Serverless SQL System Catalog
Provisioned Synapse SQL DB
Synapse SQL - Provisioned Instance

11. Synapse integration with ADLS Query Acceleration
Reduces total cost of ownership because analytics
frameworks don’t need to parse and load as much data
Delivers performance improvements due to less data
transferred over network
Optimize access to structured data by filtering data directly in
the storage service
Analytics queries typically require only ~20% of total data read
Deeply integrated into Azure Synapse Analytics for
improved performance and cost
AzureDataLakeStorage
Query
Acceleration
1
2
5
4
Data
3
Azure Synapse Analytics

12. Demo: .NET for Spark and shared
metadata experience in Azure
Synapse
Michael Rys, @MikeDoesBigData

13. Demo: .NET for Spark and shared metadata
experience in Azure Synapse
Analysis with
interactive .NET
for Spark
Notebook
Data prep with
Spark Scala
Twitter CSV files
Seamless analysis
with Synapse SQL
What has
Michael been
up?
Mentions
Topics
Who was
interacting
with Michael?
Michael
@MikeDoesBigData
Using Query
Acceleration
Synapse Shared
Meta Data

14. ANNOUNCING: .NET for Apache Spark v1.0 is released!
 First-class C# and F# bindings to Apache Spark,
bringing the power of big data analytics to .NET
developers
Apache Spark 2.4/3.0
Data Frames, Structured
Streaming, Delta Lake
.NET Standard 2.0
C# and F#
ML.NET
Performance optimized
with Apache Arrow and
HW Vectorization
First class integration in
Azure Synapse: Batch
Submission
Interactive .NET notebooks
Learn more at
http://dot.net/Spark

15. Call to action
 Check out sessions:
 OD217 - Real-time analytics and BI using Azure Synapse Link for Azure Cosmos DB by Euan Garden
 DB111 - Building real-time enterprise analytics solutions with Azure Synapse Analytics by Saveen Reddy
 Introduce to Azure Data Lake Storage – https://aka.ms/adls
 Get started with Azure Synapse – https://aka.ms/azuresynapse
 Learn more about .NET for Apache Spark - https://dot.net/Spark
 Leverage Informatica + Microsoft - DW migration
offer: https://aka.ms/SynapseInformaticaPOV
 Connect with us on Twitter:
Tweet us: @MikeDoesBigData @BigJamesBigData
Tag with: #OD220