The document summarizes the Databricks analytics platform, which provides a unified environment powered by Apache Spark. It integrates with Azure services and provides features like interactive collaboration, native security integration, and one-click setup. It also discusses capabilities like schema management, data consistency, elastic scalability, and automatic upgrades.

3. A fast, easy and collaborative Apache® Spark™ based analytics platform optimized for Azure
Best of Databricks Best of Microsoft
Designed in collaboration with the founders of Apache Spark
One-click set up; streamlined workflows
Interactive workspace that enables collaboration between data scientists, data engineers, and business analysts.
Native integration with Azure services (Power BI, SQL DW, Cosmos DB, Blob Storage)
Enterprise-grade Azure security (Active Directory integration, compliance, enterprise -grade SLAs)

5. 1. Various sources/formats
2. Schema mismatch
3. Different representation
4. Corrupted files and data
5. Scalability
5. Schema evolution
6. Continuous ETL
7. Monitoring & Auditing
8. Multi Activity integration

6. 6
Accelerate innovation by unifying data science,
engineering and business
• Founded by the creators of Apache Spark
• Contributes 75% of the open source code, 10x
more than any other company
• Trained 90k+ Spark users on the Databricks
platform
VISION
WHO WE
ARE
Unified Analytics Platform powered by Apache SparkPRODUCT

8. Building blocks for a big data ETL system
Big Data Schema Mgmt Data Consistency Fast Reads
Resource Estimation Failure Recovery Automatic Upgrades
Tooling & IntegrationElastic ScalabilityDistributed Framework
Infrastructure
Management
Platform
Management
Data
Management
Fast Data

9. Apache Spark - Distributed Framework
Multiple Use Cases
Multiple Languages
Multiple Data Sources

10. Efficient resource mgmt - Lower TCO

11. Automatic Recovery
• If cluster nodes fail, they get automatically replaced with new ones.
• If the whole cluster becomes unresponsive for some reason, then the
cluster will be automatically recovered.
Automatic Upgrades
• Automatic push of latest software updates to cluster every 2 weeks.
• Zero downtime of clusters during pushes.

12. Performance Optimizations
• Automatic caching for remote data
• Enhance vectorized parquet reader
• Dynamic query optimizations
• Coarse Grained Indexing (automatically cluster multi dimensional
data)
• Skewed Data Handling
• File Compaction
™

13. • Cloud Blob Storage
• Decouple Compute & Storage
• ACID Compliance & Data Upserts
• Schema Evolution
• Intelligent Partitioning & Compaction
• Data Skipping & Indexing
• Real-Time Streaming Ingest
• Structured Streaming
SCALE
PERFORMANCE
RELIABILITY
LOW LATENCY

15. An ETL Query in Apache Spark
spark.read.json("/source/path")
.filter(...)
.agg(...)
.write.mode("append")
.parquet("/output/path")
EXTRACT
TRANSFORM
LOAD

16. An ETL Query in Apache Spark
Extract
EXTRACT
TRANSFORM
LOAD
val csvTable = spark.read.csv("/source/path")
val jdbcTable = spark.read.format("jdbc")
.option("url", "jdbc:postgresql:...")
.option("dbtable", "TEST.PEOPLE")
.load()
csvTable
.join(jdbcTable, Seq("name"), "outer")
.filter("id <= 2999")
.write
.mode("overwrite")
.format("parquet")
.saveAsTable("outputTableName")

17. {"a":1, "b":2, "c":3}
{"a":{, b:3}
{"a":5, "b":6, "c":7}
spark.read
.option("mode", "PERMISSIVE")
.option("columnNameOfCorruptRecord", "_corrupt_record")
.json(corruptRecords)
.show() The default can be configured via
spark.sql.columnNameOfCorruptRecord
Json: Dealing with Corrupt Records

18. {"a":1, "b":2, "c":3}
{"a":{, b:3}
{"a":5, "b":6, "c":7}
spark.read
.option("mode", "DROPMALFORMED")
.json(corruptRecords)
.show()
Json: Dealing with Corrupt Records

19. {"a":1, "b":2, "c":3}
{"a":{, b:3}
{"a":5, "b":6, "c":7}
spark.read
.option("mode", "FAILFAST")
.json(corruptRecords)
.show()
org.apache.spark.sql.catalyst.json
.SparkSQLJsonProcessingException:
Malformed line in FAILFAST mode:
{"a":{, b:3}
Json: Dealing with Corrupt Records

20. spark.read
.option("mode", "FAILFAST")
.csv(corruptRecords)
.show()
java.lang.RuntimeException:
Malformed line in FAILFAST mode:
2015,Chevy,Volt
CSV: Dealing with Corrupt Records
year,make,model,comment,blank
"2012","Tesla","S","No comment",
1997,Ford,E350,"Go get one now they",
2015,Chevy,Volt

21. spark.read.
.option("mode", "PERMISSIVE")
.csv(corruptRecords)
.show()
CSV: Dealing with Corrupt Records
year,make,model,comment,blank
"2012","Tesla","S","No comment",
1997,Ford,E350,"Go get one now they",
2015,Chevy,Volt

22. year,make,model,comment,blank
"2012","Tesla","S","No comment",
1997,Ford,E350,"Go get one now they",
2015,Chevy,Volt
spark.read
.option("header", true)
.option("mode", "PERMISSIVE")
.csv(corruptRecords)
.show()
CSV: Dealing with Corrupt Records

23. val schema = "col1 INT, col2 STRING, col3 STRING, col4
STRING, " +
"col5 STRING, __corrupted_column_name STRING"
spark.read
.option("header", true)
.option("mode", "PERMISSIVE")
.csv(corruptRecords)
.show()
CSV: Dealing with Corrupt Records

24. year,make,model,comment,blank
"2012","Tesla","S","No comment",
1997,Ford,E350,"Go get one now they",
2015,Chevy,Volt
spark.read
.option("mode", ”DROPMALFORMED")
.csv(corruptRecords)
.show()
CSV: Dealing with Corrupt Records

25. Business / custom apps
(Structured)
Logs, files and media
(unstructured)
Azure storage
Polybase
Azure SQL Data Warehouse
Data factory
Data factory
Azure Databricks
(Spark)
Analytical dashboards
Model & ServePrep & TrainStoreIngest Intelligence

27. Any BI tool
Advanced
Analytics
Any language
Data Processing & MovementRelational data
Dashboards | Reporting
Mobile BI | Cubes
Machine Learning
Stream analytics Cognitive | AI
.NET | Java | R | Python
Ruby | PHP | Scala
Non-relational data
OLTP ERP CRM LOB
Social media DevicesWeb Media
CLOUD
ON-PREMISE
V-NET
AZURE DATA FACTORY ORCHESTRATES DATA PIPELINE ACTIVITY WORKFLOW & SCHEDULING

29. SSIS
Package
Pipeline
1
2
3 3
4 4
4 4
5

30. Call Log Files
Customer Table
Call Log Files
Customer Table
Customer
Churn Table
Data Sources Ingest Transform & Analyze Publish
Customer
Call Details
Customers
Likely to
Churn

31. Visual Drag-n-Drop Authoring
Visual Monitoring

32. Operationalize Databricks notebook using ADF

36. Compute
Control
Remote
Storage
Intelligent
Cache

37. ComputeRemotestorage
Cache TempDB
NVMe SSD
Cores Memory
Data
Log
Cache TempDB
NVMe SSD
Cores Memory
Cache TempDB
NVMe SSD
Cores Memory
Snapshot backups
Control
Cores Memory
SSD TempDB

38. Memory
Cache
Remote
Storage
∞

39. Parallel load via Azure Blob Storage
Compute Node
DMS Bridge
Compute Node
DMS Bridge
Compute Node
DMS Bridge
Control Node
DMS
AzureBlobStorage(WASB)

40. Microsoft Confidential
Import into Databricks from SQL DW
JDBC Connection
String
Azure Storage Blob Directory

41. Microsoft Confidential
Schema based on Table Definition

42. Microsoft Confidential
Display(DF)
• Auto Generated
Objects
• Column Projection
• Limit pushdown
• Query Label for easy
debugging

43. Microsoft Confidential
Export Databricks into SQL DW

44. Microsoft Confidential
Azure Storage Blob
What Happens
Master
Node
Worker
Node
Worker
Node
Worker
Node
Control
Node
Compute
Node
Compute
Node
Compute
Node
DF.save()
JDBC Connection
Parquet
Parquet
Parquet
tmpDir