SlideShare a Scribd company logo
Improve Power BI
Performance
@xuanalytics
Who I am
• Dan (Annie) Xu
• Microsoft Cloud Solution Architect (OCP)
• Dax@Microsoft.com
Understand it ->
Improve it
Agenda
• Level setting
• The Vertipaq engine
• Query Architecture Essentials
• Optimization for Query Performance
• Thinking in a bigger picture
Level setting - Imported Mode vs. Direct Query mode
Data Source Power BI
Compare the difference
Direct Query Mode
Import Mode
Understand the Vertipaq engine
Column
Store vs.
Row Store
http://saphanatutorial.com/column-data-storage-and-row-data-storage-sap-hana/
Compression
http://saphanatutorial.com/column-data-storage-and-row-data-storage-sap-hana/
What happens when data is imported from source
to Power BI
• Reading of the source dataset, transformation into a columnar
data structure of VertiPaq, encoding and compressing each
column.
• Creation of dictionaries and indexes for each column.
• Creation of the data structures for relationships.
• Computation and compression of all the calculated columns.
https://www.microsoftpressstore.com/articles/article.aspx?p=2449192
Optimization
for Vertipaq
memory
usage
• Total table size, Number of records and columns
• Number of partition and segments, ~ records per
segments
Analyze biggest tables
• Total column size, dictionary size (Hash)
• Data Type, Cardinality
• Encoding type (value, hash)
Analyze biggest columns
Analyze largest relationships
Data Model
is a Key
Driver for
query
performance
• Even the best DAX in the world can struggle
against a poorly designed model
• Lots of performance issues are cause by model
designs which not follow best practices
• As model get larger, adherence to best practice in
design becomes more and more important
• Recommended reading: Modeling for AS Tabular
scalability whitepaper
Demo time!
Your Tool: Vertipaq Analyzer
Query Architecture Essential
Formula Engine vs. Storage Engine
Category Formula Engine Storage Engine
Thread Single - Threaded Multi – Threaded
Characteristics Very Smart! Very Fast!
Main function Build execution
plans
Handle complex
expressions against
datacaches
Handle simple arithmetic
calculations
Execute queries against
the compressed data in
VertiPaq storage
Cache utilization No Yes
Performance Tuning Check physical plan Check xmSQL queries (a
textual representation of
SE query)
Vertipaq Cache
• Goal: Improve the performance of multiple requests of the same data
cache within the same query or different queries requesting the same
datacache
• Only result of Storage Engine is Cached (FE does not have any)
• Vertipaq engine can reuses data in cache only when the cardinality is
the same and the columns are a subset of a previous query
CallbackDataID
• The SE only supports a limited set of operators and functions in
xmSQL. It cannot calculated complex logic such as conditional logic,
advanced math
• When a calculation is required within a VertiPaq iterator, the SE may
call FE using a special xmSQL function called CallbackDataID.
• If the CallbackDataID is generated in query plan of a iterator, the SE
calls the FE for every row, passing the DAX expression and the values
of its members as arguments.
• The result of the CallbackDataID is a data cache with only one row,
corresponding to the aggregated result
CallBackDataID Performance
• Slower than pure Vertipaq
• Faster than pure Formula Engine
• – Highly parallelized
• – Works on compressed data
• Not cached
Optimization for Query Performance
Consider
Query
Memory
Usage
• Simple queries require some memory
• Complex queries require more memory Ex:
Materialization of datasets
• Data Cache (SE query) also requires memory
Optimization
Standard
Process
1. Measure performance Power BI Performance
Analyzer
2. Analyze the query plan Dax Studio
3. Find a new way to rethink calculation & query
(Reduce FE usage and Increase SE usage)
4. Measure performance gain
5. Loop until the expected level of performance
Demo time!
Your Tool: Dax Studio, Power BI Performance Analyzer
Thinking in a bigger picture
More horse
power
• Power BI pro, Power BI premium, Power BI
embedded
• Use SSAS Tabular Server (Hardware
considerations)
• Direct Query: more powerful data pipeline
engine (SQL Server, Azure SQL Data
Warehouse edc.
Appendix List
• Useful Links
• Key Concepts
• Useful Tools for DAX and Tabular Modeling
Useful Links
• https://www.sqlbi.com/
• https://powerpivotpro.com
• http://anniexu1990.com
• Book: The Definitive Guide to DAX
Key Concepts
• Column store vs. row store technology
• Vertipaq compression
• Row context vs. filter context
• Formula Engine vs. Storage Engine
Useful Tools for
DAX and Tabular Modeling
• DAXStudio
• Vertipaq Analyzer
• DAX Editor
Performance Tuning for DAX
“The most important factor of DAX formula speed is data distribution”
• Vertipaq Analyzer
Formula Engine Bottlenecks
• Redundant logic steps
• Long iterations over datacaches
Storage Engine Bottlenecks
• Long Scan time
• Large cardinality
• High frequency of CallbackdataID (a function in Storage Engine communicating back
to formula engine for complicated calculations which disable the cache)
• Large materialization
Some Query
Performance
Tuning
Techniques
• User Variables when you can precompute some calculations
• When possible, replace cumulated IF conditions by CALCULATE with
separated conditions expression
• Avoid complex FILTERS in the came CALCULATE single conditions (use
separated conditions)
• Use IFERROR sparingly
• When possible replace DIVIDE by condition in the CALCULATE
• Reduce useless iterations (change the grain of the table inside the
iterator)
• Adjust the model design to remove complex “on the fly” computation
(reduce FE cpu time)
• Look at the number of xmSQL queries and rows returned by Storage
Engine
• Try to avoid CallBack (ex: complex filter)
• Try to avoid Materializations (ex: complex join or iterator)

More Related Content

What's hot

Introduction to Power BI to make smart decisions
Introduction to Power BI to make smart decisionsIntroduction to Power BI to make smart decisions
Introduction to Power BI to make smart decisions
VIVEK GURURANI
 
SSIS Presentation
SSIS PresentationSSIS Presentation
SSIS Presentation
BarbaraBederman
 
Best practices to deliver data analytics to the business with power bi
Best practices to deliver data analytics to the business with power biBest practices to deliver data analytics to the business with power bi
Best practices to deliver data analytics to the business with power bi
Satya Shyam K Jayanty
 
Building Modern Data Platform with Microsoft Azure
Building Modern Data Platform with Microsoft AzureBuilding Modern Data Platform with Microsoft Azure
Building Modern Data Platform with Microsoft Azure
Dmitry Anoshin
 
Power bi overview
Power bi overview Power bi overview
Power bi overview
Kiki Noviandi
 
Introduction to Azure Data Factory
Introduction to Azure Data FactoryIntroduction to Azure Data Factory
Introduction to Azure Data Factory
Slava Kokaev
 
Power BI Made Simple
Power BI Made SimplePower BI Made Simple
Power BI Made Simple
James Serra
 
Power BI Governance and Development Best Practices - Presentation at #MSBIFI ...
Power BI Governance and Development Best Practices - Presentation at #MSBIFI ...Power BI Governance and Development Best Practices - Presentation at #MSBIFI ...
Power BI Governance and Development Best Practices - Presentation at #MSBIFI ...
Jouko Nyholm
 
Power BI new workspace experience in power bi
Power BI  new workspace experience in power biPower BI  new workspace experience in power bi
Power BI new workspace experience in power bi
Amit Kumar ☁
 
NOVA SQL User Group - Azure Synapse Analytics Overview - May 2020
NOVA SQL User Group - Azure Synapse Analytics Overview -  May 2020NOVA SQL User Group - Azure Synapse Analytics Overview -  May 2020
NOVA SQL User Group - Azure Synapse Analytics Overview - May 2020
Timothy McAliley
 
Introduction SQL Analytics on Lakehouse Architecture
Introduction SQL Analytics on Lakehouse ArchitectureIntroduction SQL Analytics on Lakehouse Architecture
Introduction SQL Analytics on Lakehouse Architecture
Databricks
 
Data Lakehouse, Data Mesh, and Data Fabric (r2)
Data Lakehouse, Data Mesh, and Data Fabric (r2)Data Lakehouse, Data Mesh, and Data Fabric (r2)
Data Lakehouse, Data Mesh, and Data Fabric (r2)
James Serra
 
Building an Effective Data Warehouse Architecture
Building an Effective Data Warehouse ArchitectureBuilding an Effective Data Warehouse Architecture
Building an Effective Data Warehouse Architecture
James Serra
 
Power BI
Power BIPower BI
Modern Data Warehousing with the Microsoft Analytics Platform System
Modern Data Warehousing with the Microsoft Analytics Platform SystemModern Data Warehousing with the Microsoft Analytics Platform System
Modern Data Warehousing with the Microsoft Analytics Platform System
James Serra
 
Azure Synapse Analytics Overview (r1)
Azure Synapse Analytics Overview (r1)Azure Synapse Analytics Overview (r1)
Azure Synapse Analytics Overview (r1)
James Serra
 
Power BI Overview
Power BI OverviewPower BI Overview
Power BI Overview
Nikkia Carter
 
Microsoft Power BI Overview
Microsoft Power BI OverviewMicrosoft Power BI Overview
Microsoft Power BI Overview
Netwoven Inc.
 
Modernizing to a Cloud Data Architecture
Modernizing to a Cloud Data ArchitectureModernizing to a Cloud Data Architecture
Modernizing to a Cloud Data Architecture
Databricks
 
Big data on aws
Big data on awsBig data on aws
Big data on aws
Serkan Özal
 

What's hot (20)

Introduction to Power BI to make smart decisions
Introduction to Power BI to make smart decisionsIntroduction to Power BI to make smart decisions
Introduction to Power BI to make smart decisions
 
SSIS Presentation
SSIS PresentationSSIS Presentation
SSIS Presentation
 
Best practices to deliver data analytics to the business with power bi
Best practices to deliver data analytics to the business with power biBest practices to deliver data analytics to the business with power bi
Best practices to deliver data analytics to the business with power bi
 
Building Modern Data Platform with Microsoft Azure
Building Modern Data Platform with Microsoft AzureBuilding Modern Data Platform with Microsoft Azure
Building Modern Data Platform with Microsoft Azure
 
Power bi overview
Power bi overview Power bi overview
Power bi overview
 
Introduction to Azure Data Factory
Introduction to Azure Data FactoryIntroduction to Azure Data Factory
Introduction to Azure Data Factory
 
Power BI Made Simple
Power BI Made SimplePower BI Made Simple
Power BI Made Simple
 
Power BI Governance and Development Best Practices - Presentation at #MSBIFI ...
Power BI Governance and Development Best Practices - Presentation at #MSBIFI ...Power BI Governance and Development Best Practices - Presentation at #MSBIFI ...
Power BI Governance and Development Best Practices - Presentation at #MSBIFI ...
 
Power BI new workspace experience in power bi
Power BI  new workspace experience in power biPower BI  new workspace experience in power bi
Power BI new workspace experience in power bi
 
NOVA SQL User Group - Azure Synapse Analytics Overview - May 2020
NOVA SQL User Group - Azure Synapse Analytics Overview -  May 2020NOVA SQL User Group - Azure Synapse Analytics Overview -  May 2020
NOVA SQL User Group - Azure Synapse Analytics Overview - May 2020
 
Introduction SQL Analytics on Lakehouse Architecture
Introduction SQL Analytics on Lakehouse ArchitectureIntroduction SQL Analytics on Lakehouse Architecture
Introduction SQL Analytics on Lakehouse Architecture
 
Data Lakehouse, Data Mesh, and Data Fabric (r2)
Data Lakehouse, Data Mesh, and Data Fabric (r2)Data Lakehouse, Data Mesh, and Data Fabric (r2)
Data Lakehouse, Data Mesh, and Data Fabric (r2)
 
Building an Effective Data Warehouse Architecture
Building an Effective Data Warehouse ArchitectureBuilding an Effective Data Warehouse Architecture
Building an Effective Data Warehouse Architecture
 
Power BI
Power BIPower BI
Power BI
 
Modern Data Warehousing with the Microsoft Analytics Platform System
Modern Data Warehousing with the Microsoft Analytics Platform SystemModern Data Warehousing with the Microsoft Analytics Platform System
Modern Data Warehousing with the Microsoft Analytics Platform System
 
Azure Synapse Analytics Overview (r1)
Azure Synapse Analytics Overview (r1)Azure Synapse Analytics Overview (r1)
Azure Synapse Analytics Overview (r1)
 
Power BI Overview
Power BI OverviewPower BI Overview
Power BI Overview
 
Microsoft Power BI Overview
Microsoft Power BI OverviewMicrosoft Power BI Overview
Microsoft Power BI Overview
 
Modernizing to a Cloud Data Architecture
Modernizing to a Cloud Data ArchitectureModernizing to a Cloud Data Architecture
Modernizing to a Cloud Data Architecture
 
Big data on aws
Big data on awsBig data on aws
Big data on aws
 

Similar to Improve power bi performance

Виталий Бондаренко "Fast Data Platform for Real-Time Analytics. Architecture ...
Виталий Бондаренко "Fast Data Platform for Real-Time Analytics. Architecture ...Виталий Бондаренко "Fast Data Platform for Real-Time Analytics. Architecture ...
Виталий Бондаренко "Fast Data Platform for Real-Time Analytics. Architecture ...
Fwdays
 
Introduction to Amazon Athena
Introduction to Amazon AthenaIntroduction to Amazon Athena
Introduction to Amazon Athena
Amazon Web Services
 
Introduction to Amazon Athena
Introduction to Amazon AthenaIntroduction to Amazon Athena
Introduction to Amazon Athena
Amazon Web Services
 
Announcing Amazon Athena - Instantly Analyze Your Data in S3 Using SQL
Announcing Amazon Athena - Instantly Analyze Your Data in S3 Using SQLAnnouncing Amazon Athena - Instantly Analyze Your Data in S3 Using SQL
Announcing Amazon Athena - Instantly Analyze Your Data in S3 Using SQL
Amazon Web Services
 
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...
Databricks
 
In-memory ColumnStore Index
In-memory ColumnStore IndexIn-memory ColumnStore Index
In-memory ColumnStore Index
SolidQ
 
StackMate - CloudFormation for CloudStack
StackMate - CloudFormation for CloudStackStackMate - CloudFormation for CloudStack
StackMate - CloudFormation for CloudStack
Chiradeep Vittal
 
Enabling Key Business Advantage from Big Data through Advanced Ingest Process...
Enabling Key Business Advantage from Big Data through Advanced Ingest Process...Enabling Key Business Advantage from Big Data through Advanced Ingest Process...
Enabling Key Business Advantage from Big Data through Advanced Ingest Process...
StampedeCon
 
NEW LAUNCH! Intro to Amazon Athena. Easily analyze data in S3, using SQL.
NEW LAUNCH! Intro to Amazon Athena. Easily analyze data in S3, using SQL.NEW LAUNCH! Intro to Amazon Athena. Easily analyze data in S3, using SQL.
NEW LAUNCH! Intro to Amazon Athena. Easily analyze data in S3, using SQL.
Amazon Web Services
 
NEW LAUNCH! Intro to Amazon Athena. Easily analyze data in S3, using SQL.
NEW LAUNCH! Intro to Amazon Athena. Easily analyze data in S3, using SQL.NEW LAUNCH! Intro to Amazon Athena. Easily analyze data in S3, using SQL.
NEW LAUNCH! Intro to Amazon Athena. Easily analyze data in S3, using SQL.
Amazon Web Services
 
impalapresentation-130130105033-phpapp02 (1)_221220_235919.pdf
impalapresentation-130130105033-phpapp02 (1)_221220_235919.pdfimpalapresentation-130130105033-phpapp02 (1)_221220_235919.pdf
impalapresentation-130130105033-phpapp02 (1)_221220_235919.pdf
ssusere05ec21
 
SQL Analytics for Search Engineers - Timothy Potter, Lucidworksngineers
SQL Analytics for Search Engineers - Timothy Potter, LucidworksngineersSQL Analytics for Search Engineers - Timothy Potter, Lucidworksngineers
SQL Analytics for Search Engineers - Timothy Potter, Lucidworksngineers
Lucidworks
 
Webinar: SAP HANA - Features, Architecture and Advantages
Webinar: SAP HANA - Features, Architecture and AdvantagesWebinar: SAP HANA - Features, Architecture and Advantages
Webinar: SAP HANA - Features, Architecture and Advantages
APPSeCONNECT
 
Cheetah:Data Warehouse on Top of MapReduce
Cheetah:Data Warehouse on Top of MapReduceCheetah:Data Warehouse on Top of MapReduce
Cheetah:Data Warehouse on Top of MapReduce
Tilani Gunawardena PhD(UNIBAS), BSc(Pera), FHEA(UK), CEng, MIESL
 
Basic Application Performance Optimization Techniques (Backend)
Basic Application Performance Optimization Techniques (Backend)Basic Application Performance Optimization Techniques (Backend)
Basic Application Performance Optimization Techniques (Backend)
Klas Berlič Fras
 
Qubole - Big data in cloud
Qubole - Big data in cloudQubole - Big data in cloud
Qubole - Big data in cloud
Dmitry Tolpeko
 
2018 data warehouse features in spark
2018   data warehouse features in spark2018   data warehouse features in spark
2018 data warehouse features in spark
Chester Chen
 
Low Latency Polyglot Model Scoring using Apache Apex
Low Latency Polyglot Model Scoring using Apache ApexLow Latency Polyglot Model Scoring using Apache Apex
Low Latency Polyglot Model Scoring using Apache Apex
Apache Apex
 
Apache Kylin: OLAP Engine on Hadoop - Tech Deep Dive
Apache Kylin: OLAP Engine on Hadoop - Tech Deep DiveApache Kylin: OLAP Engine on Hadoop - Tech Deep Dive
Apache Kylin: OLAP Engine on Hadoop - Tech Deep Dive
Xu Jiang
 
Big Data Day LA 2016/ Big Data Track - How To Use Impala and Kudu To Optimize...
Big Data Day LA 2016/ Big Data Track - How To Use Impala and Kudu To Optimize...Big Data Day LA 2016/ Big Data Track - How To Use Impala and Kudu To Optimize...
Big Data Day LA 2016/ Big Data Track - How To Use Impala and Kudu To Optimize...
Data Con LA
 

Similar to Improve power bi performance (20)

Виталий Бондаренко "Fast Data Platform for Real-Time Analytics. Architecture ...
Виталий Бондаренко "Fast Data Platform for Real-Time Analytics. Architecture ...Виталий Бондаренко "Fast Data Platform for Real-Time Analytics. Architecture ...
Виталий Бондаренко "Fast Data Platform for Real-Time Analytics. Architecture ...
 
Introduction to Amazon Athena
Introduction to Amazon AthenaIntroduction to Amazon Athena
Introduction to Amazon Athena
 
Introduction to Amazon Athena
Introduction to Amazon AthenaIntroduction to Amazon Athena
Introduction to Amazon Athena
 
Announcing Amazon Athena - Instantly Analyze Your Data in S3 Using SQL
Announcing Amazon Athena - Instantly Analyze Your Data in S3 Using SQLAnnouncing Amazon Athena - Instantly Analyze Your Data in S3 Using SQL
Announcing Amazon Athena - Instantly Analyze Your Data in S3 Using SQL
 
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...
 
In-memory ColumnStore Index
In-memory ColumnStore IndexIn-memory ColumnStore Index
In-memory ColumnStore Index
 
StackMate - CloudFormation for CloudStack
StackMate - CloudFormation for CloudStackStackMate - CloudFormation for CloudStack
StackMate - CloudFormation for CloudStack
 
Enabling Key Business Advantage from Big Data through Advanced Ingest Process...
Enabling Key Business Advantage from Big Data through Advanced Ingest Process...Enabling Key Business Advantage from Big Data through Advanced Ingest Process...
Enabling Key Business Advantage from Big Data through Advanced Ingest Process...
 
NEW LAUNCH! Intro to Amazon Athena. Easily analyze data in S3, using SQL.
NEW LAUNCH! Intro to Amazon Athena. Easily analyze data in S3, using SQL.NEW LAUNCH! Intro to Amazon Athena. Easily analyze data in S3, using SQL.
NEW LAUNCH! Intro to Amazon Athena. Easily analyze data in S3, using SQL.
 
NEW LAUNCH! Intro to Amazon Athena. Easily analyze data in S3, using SQL.
NEW LAUNCH! Intro to Amazon Athena. Easily analyze data in S3, using SQL.NEW LAUNCH! Intro to Amazon Athena. Easily analyze data in S3, using SQL.
NEW LAUNCH! Intro to Amazon Athena. Easily analyze data in S3, using SQL.
 
impalapresentation-130130105033-phpapp02 (1)_221220_235919.pdf
impalapresentation-130130105033-phpapp02 (1)_221220_235919.pdfimpalapresentation-130130105033-phpapp02 (1)_221220_235919.pdf
impalapresentation-130130105033-phpapp02 (1)_221220_235919.pdf
 
SQL Analytics for Search Engineers - Timothy Potter, Lucidworksngineers
SQL Analytics for Search Engineers - Timothy Potter, LucidworksngineersSQL Analytics for Search Engineers - Timothy Potter, Lucidworksngineers
SQL Analytics for Search Engineers - Timothy Potter, Lucidworksngineers
 
Webinar: SAP HANA - Features, Architecture and Advantages
Webinar: SAP HANA - Features, Architecture and AdvantagesWebinar: SAP HANA - Features, Architecture and Advantages
Webinar: SAP HANA - Features, Architecture and Advantages
 
Cheetah:Data Warehouse on Top of MapReduce
Cheetah:Data Warehouse on Top of MapReduceCheetah:Data Warehouse on Top of MapReduce
Cheetah:Data Warehouse on Top of MapReduce
 
Basic Application Performance Optimization Techniques (Backend)
Basic Application Performance Optimization Techniques (Backend)Basic Application Performance Optimization Techniques (Backend)
Basic Application Performance Optimization Techniques (Backend)
 
Qubole - Big data in cloud
Qubole - Big data in cloudQubole - Big data in cloud
Qubole - Big data in cloud
 
2018 data warehouse features in spark
2018   data warehouse features in spark2018   data warehouse features in spark
2018 data warehouse features in spark
 
Low Latency Polyglot Model Scoring using Apache Apex
Low Latency Polyglot Model Scoring using Apache ApexLow Latency Polyglot Model Scoring using Apache Apex
Low Latency Polyglot Model Scoring using Apache Apex
 
Apache Kylin: OLAP Engine on Hadoop - Tech Deep Dive
Apache Kylin: OLAP Engine on Hadoop - Tech Deep DiveApache Kylin: OLAP Engine on Hadoop - Tech Deep Dive
Apache Kylin: OLAP Engine on Hadoop - Tech Deep Dive
 
Big Data Day LA 2016/ Big Data Track - How To Use Impala and Kudu To Optimize...
Big Data Day LA 2016/ Big Data Track - How To Use Impala and Kudu To Optimize...Big Data Day LA 2016/ Big Data Track - How To Use Impala and Kudu To Optimize...
Big Data Day LA 2016/ Big Data Track - How To Use Impala and Kudu To Optimize...
 

Recently uploaded

GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)
Javier Junquera
 
"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota
Fwdays
 
Getting the Most Out of ScyllaDB Monitoring: ShareChat's Tips
Getting the Most Out of ScyllaDB Monitoring: ShareChat's TipsGetting the Most Out of ScyllaDB Monitoring: ShareChat's Tips
Getting the Most Out of ScyllaDB Monitoring: ShareChat's Tips
ScyllaDB
 
Discover the Unseen: Tailored Recommendation of Unwatched Content
Discover the Unseen: Tailored Recommendation of Unwatched ContentDiscover the Unseen: Tailored Recommendation of Unwatched Content
Discover the Unseen: Tailored Recommendation of Unwatched Content
ScyllaDB
 
Y-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PPY-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PP
c5vrf27qcz
 
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance PanelsNorthern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving
 
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity serverDandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
Antonios Katsarakis
 
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green MasterplanJavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
Miro Wengner
 
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid ResearchHarnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Neo4j
 
The Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptxThe Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptx
operationspcvita
 
Apps Break Data
Apps Break DataApps Break Data
Apps Break Data
Ivo Velitchkov
 
Principle of conventional tomography-Bibash Shahi ppt..pptx
Principle of conventional tomography-Bibash Shahi ppt..pptxPrinciple of conventional tomography-Bibash Shahi ppt..pptx
Principle of conventional tomography-Bibash Shahi ppt..pptx
BibashShahi
 
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge GraphGraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
Neo4j
 
Demystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through StorytellingDemystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through Storytelling
Enterprise Knowledge
 
Christine's Product Research Presentation.pptx
Christine's Product Research Presentation.pptxChristine's Product Research Presentation.pptx
Christine's Product Research Presentation.pptx
christinelarrosa
 
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
Fwdays
 
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin..."$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
Fwdays
 
AppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSFAppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSF
Ajin Abraham
 
Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |
AstuteBusiness
 
Mutation Testing for Task-Oriented Chatbots
Mutation Testing for Task-Oriented ChatbotsMutation Testing for Task-Oriented Chatbots
Mutation Testing for Task-Oriented Chatbots
Pablo Gómez Abajo
 

Recently uploaded (20)

GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)
 
"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota
 
Getting the Most Out of ScyllaDB Monitoring: ShareChat's Tips
Getting the Most Out of ScyllaDB Monitoring: ShareChat's TipsGetting the Most Out of ScyllaDB Monitoring: ShareChat's Tips
Getting the Most Out of ScyllaDB Monitoring: ShareChat's Tips
 
Discover the Unseen: Tailored Recommendation of Unwatched Content
Discover the Unseen: Tailored Recommendation of Unwatched ContentDiscover the Unseen: Tailored Recommendation of Unwatched Content
Discover the Unseen: Tailored Recommendation of Unwatched Content
 
Y-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PPY-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PP
 
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance PanelsNorthern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
 
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity serverDandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
 
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green MasterplanJavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
 
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid ResearchHarnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
 
The Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptxThe Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptx
 
Apps Break Data
Apps Break DataApps Break Data
Apps Break Data
 
Principle of conventional tomography-Bibash Shahi ppt..pptx
Principle of conventional tomography-Bibash Shahi ppt..pptxPrinciple of conventional tomography-Bibash Shahi ppt..pptx
Principle of conventional tomography-Bibash Shahi ppt..pptx
 
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge GraphGraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
 
Demystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through StorytellingDemystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through Storytelling
 
Christine's Product Research Presentation.pptx
Christine's Product Research Presentation.pptxChristine's Product Research Presentation.pptx
Christine's Product Research Presentation.pptx
 
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
 
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin..."$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
 
AppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSFAppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSF
 
Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |
 
Mutation Testing for Task-Oriented Chatbots
Mutation Testing for Task-Oriented ChatbotsMutation Testing for Task-Oriented Chatbots
Mutation Testing for Task-Oriented Chatbots
 

Improve power bi performance

  • 2. Who I am • Dan (Annie) Xu • Microsoft Cloud Solution Architect (OCP) • Dax@Microsoft.com
  • 4. Agenda • Level setting • The Vertipaq engine • Query Architecture Essentials • Optimization for Query Performance • Thinking in a bigger picture
  • 5. Level setting - Imported Mode vs. Direct Query mode Data Source Power BI
  • 6. Compare the difference Direct Query Mode Import Mode
  • 10. What happens when data is imported from source to Power BI • Reading of the source dataset, transformation into a columnar data structure of VertiPaq, encoding and compressing each column. • Creation of dictionaries and indexes for each column. • Creation of the data structures for relationships. • Computation and compression of all the calculated columns. https://www.microsoftpressstore.com/articles/article.aspx?p=2449192
  • 11. Optimization for Vertipaq memory usage • Total table size, Number of records and columns • Number of partition and segments, ~ records per segments Analyze biggest tables • Total column size, dictionary size (Hash) • Data Type, Cardinality • Encoding type (value, hash) Analyze biggest columns Analyze largest relationships
  • 12. Data Model is a Key Driver for query performance • Even the best DAX in the world can struggle against a poorly designed model • Lots of performance issues are cause by model designs which not follow best practices • As model get larger, adherence to best practice in design becomes more and more important • Recommended reading: Modeling for AS Tabular scalability whitepaper
  • 13. Demo time! Your Tool: Vertipaq Analyzer
  • 15. Formula Engine vs. Storage Engine Category Formula Engine Storage Engine Thread Single - Threaded Multi – Threaded Characteristics Very Smart! Very Fast! Main function Build execution plans Handle complex expressions against datacaches Handle simple arithmetic calculations Execute queries against the compressed data in VertiPaq storage Cache utilization No Yes Performance Tuning Check physical plan Check xmSQL queries (a textual representation of SE query)
  • 16. Vertipaq Cache • Goal: Improve the performance of multiple requests of the same data cache within the same query or different queries requesting the same datacache • Only result of Storage Engine is Cached (FE does not have any) • Vertipaq engine can reuses data in cache only when the cardinality is the same and the columns are a subset of a previous query
  • 17. CallbackDataID • The SE only supports a limited set of operators and functions in xmSQL. It cannot calculated complex logic such as conditional logic, advanced math • When a calculation is required within a VertiPaq iterator, the SE may call FE using a special xmSQL function called CallbackDataID. • If the CallbackDataID is generated in query plan of a iterator, the SE calls the FE for every row, passing the DAX expression and the values of its members as arguments. • The result of the CallbackDataID is a data cache with only one row, corresponding to the aggregated result
  • 18. CallBackDataID Performance • Slower than pure Vertipaq • Faster than pure Formula Engine • – Highly parallelized • – Works on compressed data • Not cached
  • 19. Optimization for Query Performance
  • 20. Consider Query Memory Usage • Simple queries require some memory • Complex queries require more memory Ex: Materialization of datasets • Data Cache (SE query) also requires memory
  • 21. Optimization Standard Process 1. Measure performance Power BI Performance Analyzer 2. Analyze the query plan Dax Studio 3. Find a new way to rethink calculation & query (Reduce FE usage and Increase SE usage) 4. Measure performance gain 5. Loop until the expected level of performance
  • 22. Demo time! Your Tool: Dax Studio, Power BI Performance Analyzer
  • 23. Thinking in a bigger picture
  • 24. More horse power • Power BI pro, Power BI premium, Power BI embedded • Use SSAS Tabular Server (Hardware considerations) • Direct Query: more powerful data pipeline engine (SQL Server, Azure SQL Data Warehouse edc.
  • 25.
  • 26. Appendix List • Useful Links • Key Concepts • Useful Tools for DAX and Tabular Modeling
  • 27. Useful Links • https://www.sqlbi.com/ • https://powerpivotpro.com • http://anniexu1990.com • Book: The Definitive Guide to DAX
  • 28. Key Concepts • Column store vs. row store technology • Vertipaq compression • Row context vs. filter context • Formula Engine vs. Storage Engine
  • 29. Useful Tools for DAX and Tabular Modeling • DAXStudio • Vertipaq Analyzer • DAX Editor
  • 30. Performance Tuning for DAX “The most important factor of DAX formula speed is data distribution” • Vertipaq Analyzer Formula Engine Bottlenecks • Redundant logic steps • Long iterations over datacaches Storage Engine Bottlenecks • Long Scan time • Large cardinality • High frequency of CallbackdataID (a function in Storage Engine communicating back to formula engine for complicated calculations which disable the cache) • Large materialization
  • 31. Some Query Performance Tuning Techniques • User Variables when you can precompute some calculations • When possible, replace cumulated IF conditions by CALCULATE with separated conditions expression • Avoid complex FILTERS in the came CALCULATE single conditions (use separated conditions) • Use IFERROR sparingly • When possible replace DIVIDE by condition in the CALCULATE • Reduce useless iterations (change the grain of the table inside the iterator) • Adjust the model design to remove complex “on the fly” computation (reduce FE cpu time) • Look at the number of xmSQL queries and rows returned by Storage Engine • Try to avoid CallBack (ex: complex filter) • Try to avoid Materializations (ex: complex join or iterator)