Why is Azure Data Explorer fast in petabyte-scale analytics?

Sheik Uduman Ali
Sheik Uduman AliDirector, Industrial Digital Transformation
Why is Azure Data Explorer
fast in petabyte-scale
analytics?
www.linkedin.com/in/sheik-uduman-ali-m-54b5ab8
https://technicallysheik.com
Understand how its data storage architecture
makes this possible
sheikudumanali@gmail.com
Sheik (technicallysheik.com)
Azure Data Explorer (ADX)
• Managed large scale big data analytics platform
• Suitable for use cases that have high volume and variety of data ingestion at high velocity
• Internet of things – device telemetry data
• Timeseries data
• Log analytics
• Geo-spatial
• Big data analytics
• Variety of connectors available to ingest data from various sources including streaming data
• Simple query language even for complex data analytics
• Built-in data visualization and native support to Power BI and Grafana
Ingest Analyze (Query) Visualize
Outperforms all industry leading big data analytics services on performance and pricing
Sheik (technicallysheik.com)
"TableName": StormEvents,
"Schema": StartTime:datetime,EndTime:datetime,EpisodeId:int,EventId:int,
State:string,EventType:string,InjuriesDirect:int,InjuriesIndirect:int,
DeathsDirect:int,DeathsIndirect:int,DamageProperty:int,DamageCrops:int,
Source:string,BeginLocation:string,EndLocation:string,BeginLat:real,BeginLon:real,
EndLat:real,EndLon:real,EpisodeNarrative:string,EventNarrative:string,
StormSummary:dynamic,
"DatabaseName": Samples,
"Folder": Storm_Events,
"DocString": US storm events. Data source: https://www.ncdc.noaa.gov/stormevents
StormEvents - Sample table
let us take StormEvents table as a sample
this table contains 22 columns of information on US storm events
Sheik (technicallysheik.com)
"StartTime": 2007-09-18T20:00:00Z,
"EndTime": 2007-09-19T18:00:00Z,
"EpisodeId": 11074,
"EventId": 60904,
"State": FLORIDA,
"EventType": Heavy Rain,
"InjuriesDirect": 0,
"InjuriesIndirect": 0,
"DeathsDirect": 0,
"DeathsIndirect": 0,
"DamageProperty": 0,
"DamageCrops": 0,
"Source": Trained Spotter,
"BeginLocation": ORMOND BEACH,
"EndLocation": NEW SMYRNA BEACH,
"BeginLat": 29.28,
"BeginLon": -81.05,
"EndLat": 29.02,
"EndLon": -80.93,
"EpisodeNarrative": Thunderstorms lingered over Volusia County.,
"EventNarrative": As much as 9 inches of rain fell in a 24-hour period across parts of coastal Volusia County.,
"StormSummary": {
"TotalDamages": 0,
"StartTime": "2007-09-18T20:00:00.0000000Z",
"EndTime": "2007-09-19T18:00:00.0000000Z",
"Details": {
"Description": "As much as 9 inches of rain fell in a 24-hour period across parts of coastal Volusia County.",
"Location": "FLORIDA"
}
}
Sample record
Sheik (technicallysheik.com)
ADX
Storage
Columnar
Store
text
inverted
index
data shard
/ extent
Key tenets of ADX data store
Sheik (technicallysheik.com)
Columnar Store
stores the values from each column together
in separate files per column
instead of storing all the values from a row together
To return a row as a result of a query, it needs
to fetch corresponding position from each
column storage files
append only WRITE operation of ADX helps use
of this storage format
consider StormEvent table data
Sheik (technicallysheik.com)
Advantages of Columnar Store - 1
StormEvents
| take 5
| project StartTime, EndTime, EventType, State;
high query performance
among multiple columns, projection of few columns needs
less disk scans instead of searching all rows in the table
StormEvents
| summarize StormCount = count(),
TypeOfStorms = dcount(EventType) by State
| top 5 by StormCount desc
high performant
aggregation queries
as an immutable data nature, results can be cached
particularly aggregations.
Sheik (technicallysheik.com)
Advantages of Columnar Store - 2
Column compression compressed column storage on disk improves throughput.
by default ADX uses LZ4 compression
StormEvents
| where EventType =="Flood"
| summarize EventCount = count() by State
| where EventCount > 100
queries with WHERE predicate performs well
because the columns contain the rows in the same order
and compression improves disk I/O
vectorized processing
with the compressed columns, when a query needs to
fetch data from disk to apply projection or predicates may
fit into L1 cache itself that avoids unnecessary
memory and disk I/O
Memory
L1
Sheik (technicallysheik.com)
Extent or Shard
Shard 1 Shard 2 Shard 3
StartTime
EndTime
EpisodeId
EventId
State
EventType
StartTime Index
EndTime Index
EpisodeId Index
EventId Index
State Index
EventType Index
Table
An extent or shard holds a collection of records
that are physically arranged in columns
Shard 1 holds StartTime and EndTime
columns collection of records
A shard contains data, metadata and index
All columns are indexed
Sheik (technicallysheik.com)
Shard on both Ingestion and Queries
Shard 1
Shard 2
Shard 3
Table
Data
Ingestion
Cluster Node 1
Cluster Node 2
Distributed
Query
Engine
Query
Shards are evenly spread across the cluster nodes based on the partition key.
By default, ingestion time is the partition key
immutable nature, data
stored in both memory
and SSD
A query will be
distributed across
the nodes and run
concurrently
Distributed
Query
Plan
append only write with effective use of
free-text inverted indexing
A query result will
be fetched from
more than one
shards
ingest into Table
r1:= (c1, c2, c3, …, cn)
append c1, c2
append c3, c4, c5
append cn
query result
r1:= (c1, c8)
return c8
query
return c1
Sheik (technicallysheik.com)
Advantages of Shards
• Scale-out nature of sharding allows to effectively use computing on all nodes that
improves query performance
• Petabyte scale of ingestion and storage is very fast and reliable
Sheik (technicallysheik.com)
Closing Note
• The columnar store, column compression, inverted text index and data shard are the
key tenets of ADX to perform well on petabyte-scale analytics queries
• Immutable records with caching benefit makes your data analytics faster
• Materialized View and Query Result Cache are other ADX features that improves the
performance of data analytics
1 of 12

Recommended

Deploying your Data Warehouse on AWS by
Deploying your Data Warehouse on AWSDeploying your Data Warehouse on AWS
Deploying your Data Warehouse on AWSAmazon Web Services
5K views64 slides
Interactive Analytics on AWS - AWS Summit Tel Aviv 2017 by
Interactive Analytics on AWS - AWS Summit Tel Aviv 2017Interactive Analytics on AWS - AWS Summit Tel Aviv 2017
Interactive Analytics on AWS - AWS Summit Tel Aviv 2017Amazon Web Services
513 views30 slides
Masterclass - Redshift by
Masterclass - RedshiftMasterclass - Redshift
Masterclass - RedshiftAmazon Web Services
2.8K views82 slides
Amazon Athena Hands-On Workshop by
Amazon Athena Hands-On WorkshopAmazon Athena Hands-On Workshop
Amazon Athena Hands-On WorkshopDoiT International
2.8K views52 slides
InfluxDB IOx Tech Talks: Query Engine Design and the Rust-Based DataFusion in... by
InfluxDB IOx Tech Talks: Query Engine Design and the Rust-Based DataFusion in...InfluxDB IOx Tech Talks: Query Engine Design and the Rust-Based DataFusion in...
InfluxDB IOx Tech Talks: Query Engine Design and the Rust-Based DataFusion in...InfluxData
3.6K views55 slides
6° Sessione - Ambiti applicativi nella ricerca di tecnologie statistiche avan... by
6° Sessione - Ambiti applicativi nella ricerca di tecnologie statistiche avan...6° Sessione - Ambiti applicativi nella ricerca di tecnologie statistiche avan...
6° Sessione - Ambiti applicativi nella ricerca di tecnologie statistiche avan...Jürgen Ambrosi
538 views33 slides

More Related Content

Similar to Why is Azure Data Explorer fast in petabyte-scale analytics?

2021 04-20 apache arrow and its impact on the database industry.pptx by
2021 04-20  apache arrow and its impact on the database industry.pptx2021 04-20  apache arrow and its impact on the database industry.pptx
2021 04-20 apache arrow and its impact on the database industry.pptxAndrew Lamb
257 views37 slides
Making sense of your data jug by
Making sense of your data   jugMaking sense of your data   jug
Making sense of your data jugGerald Muecke
150 views58 slides
Azure Cosmos DB - The Swiss Army NoSQL Cloud Database by
Azure Cosmos DB - The Swiss Army NoSQL Cloud DatabaseAzure Cosmos DB - The Swiss Army NoSQL Cloud Database
Azure Cosmos DB - The Swiss Army NoSQL Cloud DatabaseBizTalk360
483 views61 slides
Introduction to Amazon Athena by
Introduction to Amazon AthenaIntroduction to Amazon Athena
Introduction to Amazon AthenaAmazon Web Services
3.7K views58 slides
IBM Cloud Native Day April 2021: Serverless Data Lake by
IBM Cloud Native Day April 2021: Serverless Data LakeIBM Cloud Native Day April 2021: Serverless Data Lake
IBM Cloud Native Day April 2021: Serverless Data LakeTorsten Steinbach
105 views27 slides
Amazon Athena Capabilities and Use Cases Overview by
Amazon Athena Capabilities and Use Cases Overview Amazon Athena Capabilities and Use Cases Overview
Amazon Athena Capabilities and Use Cases Overview Amazon Web Services
7.9K views67 slides

Similar to Why is Azure Data Explorer fast in petabyte-scale analytics?(20)

2021 04-20 apache arrow and its impact on the database industry.pptx by Andrew Lamb
2021 04-20  apache arrow and its impact on the database industry.pptx2021 04-20  apache arrow and its impact on the database industry.pptx
2021 04-20 apache arrow and its impact on the database industry.pptx
Andrew Lamb257 views
Making sense of your data jug by Gerald Muecke
Making sense of your data   jugMaking sense of your data   jug
Making sense of your data jug
Gerald Muecke150 views
Azure Cosmos DB - The Swiss Army NoSQL Cloud Database by BizTalk360
Azure Cosmos DB - The Swiss Army NoSQL Cloud DatabaseAzure Cosmos DB - The Swiss Army NoSQL Cloud Database
Azure Cosmos DB - The Swiss Army NoSQL Cloud Database
BizTalk360483 views
IBM Cloud Native Day April 2021: Serverless Data Lake by Torsten Steinbach
IBM Cloud Native Day April 2021: Serverless Data LakeIBM Cloud Native Day April 2021: Serverless Data Lake
IBM Cloud Native Day April 2021: Serverless Data Lake
Torsten Steinbach105 views
Amazon Athena Capabilities and Use Cases Overview by Amazon Web Services
Amazon Athena Capabilities and Use Cases Overview Amazon Athena Capabilities and Use Cases Overview
Amazon Athena Capabilities and Use Cases Overview
Amazon Web Services7.9K views
Think Like Spark: Some Spark Concepts and a Use Case by Rachel Warren
Think Like Spark: Some Spark Concepts and a Use CaseThink Like Spark: Some Spark Concepts and a Use Case
Think Like Spark: Some Spark Concepts and a Use Case
Rachel Warren600 views
Writing Continuous Applications with Structured Streaming PySpark API by Databricks
Writing Continuous Applications with Structured Streaming PySpark APIWriting Continuous Applications with Structured Streaming PySpark API
Writing Continuous Applications with Structured Streaming PySpark API
Databricks2.2K views
The life of a query (oracle edition) by maclean liu
The life of a query (oracle edition)The life of a query (oracle edition)
The life of a query (oracle edition)
maclean liu2.2K views
Apache IOTDB: a Time Series Database for Industrial IoT by jixuan1989
Apache IOTDB: a Time Series Database for Industrial IoTApache IOTDB: a Time Series Database for Industrial IoT
Apache IOTDB: a Time Series Database for Industrial IoT
jixuan19893.2K views
Microsoft Azure Big Data Analytics by Mark Kromer
Microsoft Azure Big Data AnalyticsMicrosoft Azure Big Data Analytics
Microsoft Azure Big Data Analytics
Mark Kromer4.2K views
A Rusty introduction to Apache Arrow and how it applies to a time series dat... by Andrew Lamb
A Rusty introduction to Apache Arrow and how it applies to a  time series dat...A Rusty introduction to Apache Arrow and how it applies to a  time series dat...
A Rusty introduction to Apache Arrow and how it applies to a time series dat...
Andrew Lamb343 views
Supercharging the Value of Your Data with Amazon S3 by Amazon Web Services
Supercharging the Value of Your Data with Amazon S3Supercharging the Value of Your Data with Amazon S3
Supercharging the Value of Your Data with Amazon S3
Amazon Web Services1.3K views
Writing Continuous Applications with Structured Streaming in PySpark by Databricks
Writing Continuous Applications with Structured Streaming in PySparkWriting Continuous Applications with Structured Streaming in PySpark
Writing Continuous Applications with Structured Streaming in PySpark
Databricks2.2K views
Interactively Querying Large-scale Datasets on Amazon S3 by Amazon Web Services
Interactively Querying Large-scale Datasets on Amazon S3Interactively Querying Large-scale Datasets on Amazon S3
Interactively Querying Large-scale Datasets on Amazon S3
Amazon Web Services3.7K views
Structuring Spark: DataFrames, Datasets, and Streaming by Michael Armbrust by Spark Summit
Structuring Spark: DataFrames, Datasets, and Streaming by Michael ArmbrustStructuring Spark: DataFrames, Datasets, and Streaming by Michael Armbrust
Structuring Spark: DataFrames, Datasets, and Streaming by Michael Armbrust
Spark Summit8.8K views

Recently uploaded

3196 The Case of The East River by
3196 The Case of The East River3196 The Case of The East River
3196 The Case of The East RiverErickANDRADE90
16 views4 slides
Data about the sector workshop by
Data about the sector workshopData about the sector workshop
Data about the sector workshopinfo828217
12 views27 slides
TGP 2.docx by
TGP 2.docxTGP 2.docx
TGP 2.docxsandi636490
10 views8 slides
[DSC Europe 23] Spela Poklukar & Tea Brasanac - Retrieval Augmented Generation by
[DSC Europe 23] Spela Poklukar & Tea Brasanac - Retrieval Augmented Generation[DSC Europe 23] Spela Poklukar & Tea Brasanac - Retrieval Augmented Generation
[DSC Europe 23] Spela Poklukar & Tea Brasanac - Retrieval Augmented GenerationDataScienceConferenc1
13 views29 slides
CRM stick or twist.pptx by
CRM stick or twist.pptxCRM stick or twist.pptx
CRM stick or twist.pptxinfo828217
10 views16 slides
Organic Shopping in Google Analytics 4.pdf by
Organic Shopping in Google Analytics 4.pdfOrganic Shopping in Google Analytics 4.pdf
Organic Shopping in Google Analytics 4.pdfGA4 Tutorials
14 views13 slides

Recently uploaded(20)

3196 The Case of The East River by ErickANDRADE90
3196 The Case of The East River3196 The Case of The East River
3196 The Case of The East River
ErickANDRADE9016 views
Data about the sector workshop by info828217
Data about the sector workshopData about the sector workshop
Data about the sector workshop
info82821712 views
[DSC Europe 23] Spela Poklukar & Tea Brasanac - Retrieval Augmented Generation by DataScienceConferenc1
[DSC Europe 23] Spela Poklukar & Tea Brasanac - Retrieval Augmented Generation[DSC Europe 23] Spela Poklukar & Tea Brasanac - Retrieval Augmented Generation
[DSC Europe 23] Spela Poklukar & Tea Brasanac - Retrieval Augmented Generation
CRM stick or twist.pptx by info828217
CRM stick or twist.pptxCRM stick or twist.pptx
CRM stick or twist.pptx
info82821710 views
Organic Shopping in Google Analytics 4.pdf by GA4 Tutorials
Organic Shopping in Google Analytics 4.pdfOrganic Shopping in Google Analytics 4.pdf
Organic Shopping in Google Analytics 4.pdf
GA4 Tutorials14 views
UNEP FI CRS Climate Risk Results.pptx by pekka28
UNEP FI CRS Climate Risk Results.pptxUNEP FI CRS Climate Risk Results.pptx
UNEP FI CRS Climate Risk Results.pptx
pekka2811 views
Survey on Factuality in LLM's.pptx by NeethaSherra1
Survey on Factuality in LLM's.pptxSurvey on Factuality in LLM's.pptx
Survey on Factuality in LLM's.pptx
NeethaSherra16 views
Advanced_Recommendation_Systems_Presentation.pptx by neeharikasingh29
Advanced_Recommendation_Systems_Presentation.pptxAdvanced_Recommendation_Systems_Presentation.pptx
Advanced_Recommendation_Systems_Presentation.pptx
CRM stick or twist workshop by info828217
CRM stick or twist workshopCRM stick or twist workshop
CRM stick or twist workshop
info8282179 views
CRIJ4385_Death Penalty_F23.pptx by yvettemm100
CRIJ4385_Death Penalty_F23.pptxCRIJ4385_Death Penalty_F23.pptx
CRIJ4385_Death Penalty_F23.pptx
yvettemm1006 views
[DSC Europe 23] Milos Grubjesic Empowering Business with Pepsico s Advanced M... by DataScienceConferenc1
[DSC Europe 23] Milos Grubjesic Empowering Business with Pepsico s Advanced M...[DSC Europe 23] Milos Grubjesic Empowering Business with Pepsico s Advanced M...
[DSC Europe 23] Milos Grubjesic Empowering Business with Pepsico s Advanced M...
Ukraine Infographic_22NOV2023_v2.pdf by AnastosiyaGurin
Ukraine Infographic_22NOV2023_v2.pdfUkraine Infographic_22NOV2023_v2.pdf
Ukraine Infographic_22NOV2023_v2.pdf
AnastosiyaGurin1.4K views
Data Journeys Hard Talk workshop final.pptx by info828217
Data Journeys Hard Talk workshop final.pptxData Journeys Hard Talk workshop final.pptx
Data Journeys Hard Talk workshop final.pptx
info82821710 views
[DSC Europe 23] Zsolt Feleki - Machine Translation should we trust it.pptx by DataScienceConferenc1
[DSC Europe 23] Zsolt Feleki - Machine Translation should we trust it.pptx[DSC Europe 23] Zsolt Feleki - Machine Translation should we trust it.pptx
[DSC Europe 23] Zsolt Feleki - Machine Translation should we trust it.pptx

Why is Azure Data Explorer fast in petabyte-scale analytics?

  • 1. Why is Azure Data Explorer fast in petabyte-scale analytics? www.linkedin.com/in/sheik-uduman-ali-m-54b5ab8 https://technicallysheik.com Understand how its data storage architecture makes this possible sheikudumanali@gmail.com
  • 2. Sheik (technicallysheik.com) Azure Data Explorer (ADX) • Managed large scale big data analytics platform • Suitable for use cases that have high volume and variety of data ingestion at high velocity • Internet of things – device telemetry data • Timeseries data • Log analytics • Geo-spatial • Big data analytics • Variety of connectors available to ingest data from various sources including streaming data • Simple query language even for complex data analytics • Built-in data visualization and native support to Power BI and Grafana Ingest Analyze (Query) Visualize Outperforms all industry leading big data analytics services on performance and pricing
  • 3. Sheik (technicallysheik.com) "TableName": StormEvents, "Schema": StartTime:datetime,EndTime:datetime,EpisodeId:int,EventId:int, State:string,EventType:string,InjuriesDirect:int,InjuriesIndirect:int, DeathsDirect:int,DeathsIndirect:int,DamageProperty:int,DamageCrops:int, Source:string,BeginLocation:string,EndLocation:string,BeginLat:real,BeginLon:real, EndLat:real,EndLon:real,EpisodeNarrative:string,EventNarrative:string, StormSummary:dynamic, "DatabaseName": Samples, "Folder": Storm_Events, "DocString": US storm events. Data source: https://www.ncdc.noaa.gov/stormevents StormEvents - Sample table let us take StormEvents table as a sample this table contains 22 columns of information on US storm events
  • 4. Sheik (technicallysheik.com) "StartTime": 2007-09-18T20:00:00Z, "EndTime": 2007-09-19T18:00:00Z, "EpisodeId": 11074, "EventId": 60904, "State": FLORIDA, "EventType": Heavy Rain, "InjuriesDirect": 0, "InjuriesIndirect": 0, "DeathsDirect": 0, "DeathsIndirect": 0, "DamageProperty": 0, "DamageCrops": 0, "Source": Trained Spotter, "BeginLocation": ORMOND BEACH, "EndLocation": NEW SMYRNA BEACH, "BeginLat": 29.28, "BeginLon": -81.05, "EndLat": 29.02, "EndLon": -80.93, "EpisodeNarrative": Thunderstorms lingered over Volusia County., "EventNarrative": As much as 9 inches of rain fell in a 24-hour period across parts of coastal Volusia County., "StormSummary": { "TotalDamages": 0, "StartTime": "2007-09-18T20:00:00.0000000Z", "EndTime": "2007-09-19T18:00:00.0000000Z", "Details": { "Description": "As much as 9 inches of rain fell in a 24-hour period across parts of coastal Volusia County.", "Location": "FLORIDA" } } Sample record
  • 6. Sheik (technicallysheik.com) Columnar Store stores the values from each column together in separate files per column instead of storing all the values from a row together To return a row as a result of a query, it needs to fetch corresponding position from each column storage files append only WRITE operation of ADX helps use of this storage format consider StormEvent table data
  • 7. Sheik (technicallysheik.com) Advantages of Columnar Store - 1 StormEvents | take 5 | project StartTime, EndTime, EventType, State; high query performance among multiple columns, projection of few columns needs less disk scans instead of searching all rows in the table StormEvents | summarize StormCount = count(), TypeOfStorms = dcount(EventType) by State | top 5 by StormCount desc high performant aggregation queries as an immutable data nature, results can be cached particularly aggregations.
  • 8. Sheik (technicallysheik.com) Advantages of Columnar Store - 2 Column compression compressed column storage on disk improves throughput. by default ADX uses LZ4 compression StormEvents | where EventType =="Flood" | summarize EventCount = count() by State | where EventCount > 100 queries with WHERE predicate performs well because the columns contain the rows in the same order and compression improves disk I/O vectorized processing with the compressed columns, when a query needs to fetch data from disk to apply projection or predicates may fit into L1 cache itself that avoids unnecessary memory and disk I/O Memory L1
  • 9. Sheik (technicallysheik.com) Extent or Shard Shard 1 Shard 2 Shard 3 StartTime EndTime EpisodeId EventId State EventType StartTime Index EndTime Index EpisodeId Index EventId Index State Index EventType Index Table An extent or shard holds a collection of records that are physically arranged in columns Shard 1 holds StartTime and EndTime columns collection of records A shard contains data, metadata and index All columns are indexed
  • 10. Sheik (technicallysheik.com) Shard on both Ingestion and Queries Shard 1 Shard 2 Shard 3 Table Data Ingestion Cluster Node 1 Cluster Node 2 Distributed Query Engine Query Shards are evenly spread across the cluster nodes based on the partition key. By default, ingestion time is the partition key immutable nature, data stored in both memory and SSD A query will be distributed across the nodes and run concurrently Distributed Query Plan append only write with effective use of free-text inverted indexing A query result will be fetched from more than one shards ingest into Table r1:= (c1, c2, c3, …, cn) append c1, c2 append c3, c4, c5 append cn query result r1:= (c1, c8) return c8 query return c1
  • 11. Sheik (technicallysheik.com) Advantages of Shards • Scale-out nature of sharding allows to effectively use computing on all nodes that improves query performance • Petabyte scale of ingestion and storage is very fast and reliable
  • 12. Sheik (technicallysheik.com) Closing Note • The columnar store, column compression, inverted text index and data shard are the key tenets of ADX to perform well on petabyte-scale analytics queries • Immutable records with caching benefit makes your data analytics faster • Materialized View and Query Result Cache are other ADX features that improves the performance of data analytics