Understand how its data storage architecture makes this possible

1. Why is Azure Data Explorer
fast in petabyte-scale
analytics?
www.linkedin.com/in/sheik-uduman-ali-m-54b5ab8
https://technicallysheik.com
Understand how its data storage architecture
makes this possible
sheikudumanali@gmail.com

2. Sheik (technicallysheik.com)
Azure Data Explorer (ADX)
• Managed large scale big data analytics platform
• Suitable for use cases that have high volume and variety of data ingestion at high velocity
• Internet of things – device telemetry data
• Timeseries data
• Log analytics
• Geo-spatial
• Big data analytics
• Variety of connectors available to ingest data from various sources including streaming data
• Simple query language even for complex data analytics
• Built-in data visualization and native support to Power BI and Grafana
Ingest Analyze (Query) Visualize
Outperforms all industry leading big data analytics services on performance and pricing

3. Sheik (technicallysheik.com)
"TableName": StormEvents,
"Schema": StartTime:datetime,EndTime:datetime,EpisodeId:int,EventId:int,
State:string,EventType:string,InjuriesDirect:int,InjuriesIndirect:int,
DeathsDirect:int,DeathsIndirect:int,DamageProperty:int,DamageCrops:int,
Source:string,BeginLocation:string,EndLocation:string,BeginLat:real,BeginLon:real,
EndLat:real,EndLon:real,EpisodeNarrative:string,EventNarrative:string,
StormSummary:dynamic,
"DatabaseName": Samples,
"Folder": Storm_Events,
"DocString": US storm events. Data source: https://www.ncdc.noaa.gov/stormevents
StormEvents - Sample table
let us take StormEvents table as a sample
this table contains 22 columns of information on US storm events

4. Sheik (technicallysheik.com)
"StartTime": 2007-09-18T20:00:00Z,
"EndTime": 2007-09-19T18:00:00Z,
"EpisodeId": 11074,
"EventId": 60904,
"State": FLORIDA,
"EventType": Heavy Rain,
"InjuriesDirect": 0,
"InjuriesIndirect": 0,
"DeathsDirect": 0,
"DeathsIndirect": 0,
"DamageProperty": 0,
"DamageCrops": 0,
"Source": Trained Spotter,
"BeginLocation": ORMOND BEACH,
"EndLocation": NEW SMYRNA BEACH,
"BeginLat": 29.28,
"BeginLon": -81.05,
"EndLat": 29.02,
"EndLon": -80.93,
"EpisodeNarrative": Thunderstorms lingered over Volusia County.,
"EventNarrative": As much as 9 inches of rain fell in a 24-hour period across parts of coastal Volusia County.,
"StormSummary": {
"TotalDamages": 0,
"StartTime": "2007-09-18T20:00:00.0000000Z",
"EndTime": "2007-09-19T18:00:00.0000000Z",
"Details": {
"Description": "As much as 9 inches of rain fell in a 24-hour period across parts of coastal Volusia County.",
"Location": "FLORIDA"
}
}
Sample record

5. Sheik (technicallysheik.com)
ADX
Storage
Columnar
Store
text
inverted
index
data shard
/ extent
Key tenets of ADX data store

6. Sheik (technicallysheik.com)
Columnar Store
stores the values from each column together
in separate files per column
instead of storing all the values from a row together
To return a row as a result of a query, it needs
to fetch corresponding position from each
column storage files
append only WRITE operation of ADX helps use
of this storage format
consider StormEvent table data

7. Sheik (technicallysheik.com)
Advantages of Columnar Store - 1
StormEvents
| take 5
| project StartTime, EndTime, EventType, State;
high query performance
among multiple columns, projection of few columns needs
less disk scans instead of searching all rows in the table
StormEvents
| summarize StormCount = count(),
TypeOfStorms = dcount(EventType) by State
| top 5 by StormCount desc
high performant
aggregation queries
as an immutable data nature, results can be cached
particularly aggregations.

8. Sheik (technicallysheik.com)
Advantages of Columnar Store - 2
Column compression compressed column storage on disk improves throughput.
by default ADX uses LZ4 compression
StormEvents
| where EventType =="Flood"
| summarize EventCount = count() by State
| where EventCount > 100
queries with WHERE predicate performs well
because the columns contain the rows in the same order
and compression improves disk I/O
vectorized processing
with the compressed columns, when a query needs to
fetch data from disk to apply projection or predicates may
fit into L1 cache itself that avoids unnecessary
memory and disk I/O
Memory
L1

9. Sheik (technicallysheik.com)
Extent or Shard
Shard 1 Shard 2 Shard 3
StartTime
EndTime
EpisodeId
EventId
State
EventType
StartTime Index
EndTime Index
EpisodeId Index
EventId Index
State Index
EventType Index
Table
An extent or shard holds a collection of records
that are physically arranged in columns
Shard 1 holds StartTime and EndTime
columns collection of records
A shard contains data, metadata and index
All columns are indexed

10. Sheik (technicallysheik.com)
Shard on both Ingestion and Queries
Shard 1
Shard 2
Shard 3
Table
Data
Ingestion
Cluster Node 1
Cluster Node 2
Distributed
Query
Engine
Query
Shards are evenly spread across the cluster nodes based on the partition key.
By default, ingestion time is the partition key
immutable nature, data
stored in both memory
and SSD
A query will be
distributed across
the nodes and run
concurrently
Distributed
Query
Plan
append only write with effective use of
free-text inverted indexing
A query result will
be fetched from
more than one
shards
ingest into Table
r1:= (c1, c2, c3, …, cn)
append c1, c2
append c3, c4, c5
append cn
query result
r1:= (c1, c8)
return c8
query
return c1

11. Sheik (technicallysheik.com)
Advantages of Shards
• Scale-out nature of sharding allows to effectively use computing on all nodes that
improves query performance
• Petabyte scale of ingestion and storage is very fast and reliable

12. Sheik (technicallysheik.com)
Closing Note
• The columnar store, column compression, inverted text index and data shard are the
key tenets of ADX to perform well on petabyte-scale analytics queries
• Immutable records with caching benefit makes your data analytics faster
• Materialized View and Query Result Cache are other ADX features that improves the
performance of data analytics