Welcome To The 2016 Query Store!

Welcome To The 2016 Query Store!
Janis Griffin
Senior DBA / Performance Evangelist

2
• Senior DBA / Performance Evangelist for Solarwinds
• JanisGriffin@solarwinds.com
• Twitter® - @DoBoutAnything
• Current – 25+ Years in Oracle®, SQL Server®, ASE, and now MySQL®
• DBA and Developer
• Specialize in Performance Tuning
• Review Database Performance for Customers and Prospects
• Common Question – How do I tune it?
Who am I
© 2017 SolarWinds Worldwide, LLC. All rights reserved.

3
• What is Query Store?
• In the Beginning: Tracing, DMVs & Extended Events
• Overview of Query Store
• Why use Query Store?
• Benefits & Gotchas
• How Query Store Works
• Enabling, configuring & managing it
• Best practice considerations – configuration & storage
• Performance Impact
• What’s New In 2017?
• Reporting & Several Use Cases
• Demos - using both scripts & SSMS
Agenda

4
• User complains query is slow
• DBA says nothing has changed
• No new releases
• No additional load
• What changed?
• Changes in data causing out-of-date statistics
• Changes in execution parameters
• Parameter sniffing and/or data skew
• System changes
• Troubleshooting plan changes difficult in previous versions
• Had to trace
• Had to know xQuery to decipher the plan cache XML
In the Beginning…

5
• <2005 had tracing
• High overhead
• Difficult to use (had to start & stop)
• 2005+ DMVs: dm_exec_cached_plans & query_stats
• Real time only or since last restart or clear
• Can’t see history of plan changes or regression overtime
• 2012+ Extended Events
• Easier to use than trace
• System Health session on by default
• No history of plan changes / regression
In the Beginning…

6
• New feature in 2016, Improved in 2017
• Provides insight on Query Plans
• Useful in troubleshooting performance degradation
• Quickly find differences when query plan changes
• Can see history of plans and runtime statistics
• Data is persisted in database
• Not just stored in memory
• Shows workload and database usage patterns
• Makes upgrades easier
• Between versions of SQL Server
• Put in compatibility mode 100 (lowest) or keep at prior level
• Turn on query store
• Collect representative workload
• Force plans where needed
What is Query Store?

7
• Simplifies performance tuning
• Can see plan changes & quickly compare their differences
• Can aggregate runtime statistics over different time intervals
• Can enable at database level
• You get to control when, where and how you use it
• Tracks both runtime and compile time metrics
• Has graphical interface in SSMS
• Supported on all SQL Server editions
Benefits Of Using Query Store
2016 2017

8
• Allows forcing or pinning of a better plan for a query
• Can be a quick temporary fix
• However, can be easily forgotten
• When data and/or schema changes
• Easy T-SQL for scripting
• Can quickly find queries:
• Which frequently fail
• Which use literals
• Which are resource intensive and/or most regressed over time
• Many SPs, QSD Wait types, & Extended Events
• For management purposes
Benefits – cont.

9
How Query Store Works
https://msdn.microsoft.com/en-us/library/mt631173.aspx© 2017 SolarWinds Worldwide, LLC. All rights reserved.

10
How Query Store Works – cont.
https://msdn.microsoft.com/en-us/library/mt631173.aspx

11
• Query Store is local for each database
• Is disabled by default
• To enable in SSMS
Enabling the Query Store

12
• SSMS
• T-SQL
Changing Query Store Settings
ALTER DATABASE AdventureWorks
SET QUERY_STORE = ON
( OPERATION_MODE = READ_WRITE,
CLEANUP_POLICY = ( STALE_QUERY_THRESHOLD_DAYS = 30 ),
DATA_FLUSH_INTERVAL_SECONDS = 900,
MAX_STORAGE_SIZE_MB = 100,
INTERVAL_LENGTH_MINUTES = 10);

13
• https://msdn.microsoft.com/en-us/library/dn818146.aspx
View Query Store Settings
select * from sys.database_query_store_options;
Must watch size!
If full or state
changes, errors
aren’t written to log.
Monitor actual_state.

14
• Operation Mode (Requested) / operation_mode
• Default value: Off
• Valid values are Off, Read Write, Read Only
• T-SQL: off, read_write, read_only
• Actual operation mode can change to “read only” with full
• Consider monitoring – change not written to logs
• Data Flush Interval (Minutes)
• When data is written from memory to disk
• Default is 15 minutes
• T-SQL: DATA_FLUSH_INTERVAL_SECONDS
• 900 secs = 15 min
Query Store Settings

15
• Statistics Collection Interval
• Defaults to 1 hour
• Consider changing to 10 minutes to get granular aggregation
• Higher frequency will require more storage
• Lowest level – 1 minute
• T-SQL: INTERVAL_LENGTH_MINUTES
• Max Size (MB)
• Maximum size allocated to the Query Store
• Defaults to 100 MB
• Stored in primary file of database
• When full, changes to read-only
• T-SQL: MAX_STORGE_SIZE_MB
Query Store Settings – cont.
ALTER DATABASE AdventureWorks
SET QUERY_STORE CLEAR;

16
• Query Store Capture Mode
• ‘All’ collects all plans (default)
• ‘Auto’ captures queries based on resource consumption
• ‘None’ stops the capture process
• T-SQL: QUERY_CAPTURE_MODE
• Size Based Cleanup Mode
• Activates data cleanup when it reaches maximum size
• 2016 - defaults to ‘Off’, 2017 – defaults to ‘Auto’
• Consider changing to ‘Auto’
• Flushes older queries and trivial plans as new data comes in
• Will not change to ‘read only’
• T-SQL: SIZE_BASED_CLEANUP_MODE
Query Store Settings – cont.

17
• Stale Query Threshold (Days)
• Duration to retain query store statistics
• Default 30 days
• T-SQL: CLEANUP_POLICY (STALE_QUERY_THRESHOLD_DAYS)
• MAX_PLANS_PER_QUERY
• Default is 200 plans
• Have you ever seen 200 plans to 1 query?
• Consider changing to something less
• Must change via T-SQL not in SSMS
Query Store Settings– cont.

18
• Consider enabling in the MODEL database
• In SSMS - no Query Store property in MODEL db
• Need to turn on via T-SQL
• ALTER DATABASE MODEL SET QUERY_STORE=ON;
• Consider proper sizing of Query Store
• Default is 100mb
• May be too small for busy database
• By default, it converts to read-only when full
select max_storage_size_mb - current_storage_size_mb qs_free_space_mb
from sys.database_query_store_options;
Query Store Considerations

19
• Disabled by default
• Need to enable at each individual database level
• Can modify Model so it’s on by default
• Consider sizing / performance implications
• Can’t enable for master or tempdb
• Can’t enable for Read-Only databases
• E.g. AG replication for read-only
• No global way to control configuration
• Without manual scripting or setting up Model DB
• DBAs can easily change in SSMS so hard to track
• Changes to QS settings are logged in ERRORLOG
Gotchas Of Using Query Store

20
• Query Store (QS) requires care & feeding
• Default sizes may be too small for a busy database
• If QS storage is full, switches to READ-Only mode
• May miss important performance info
• Query Store data is stored in primary file
• May conflict or compete for resource with user data
• Can add 3-5% load to the database load
• May take longer database restores
• Plans may contain literal values (HIPAA)
• No additional context – Who ran it, etc…
Gotchas – cont.

21
• Don’t blindly turn it on / plan, plan, plan
• Consider what you will use it for
• Which databases are enabled / what’s the retention period
• How will you manage it?
• Must monitor QS status, performance & storage
• Create standard setup & configuration scripts
• Consider backups / restores
• Restore into development / test or NOT
• Replication – sensitive data, forced plans, etc…
Query Store Management

22
• Microsoft states 3% to 5% impact on average
• Depends on
• # of databases monitored
• Transaction / Compile rates
• Query Store Configurations
• Can monitor Query Store via
• Perfmon Counters
• QDS Wait Types
• Extended Events
Query Store Performance Impact

23
Query Store Performance Impact
QS Events
2016 – 58
2017 – 90+
Not very well
documented
- 17 in 2016
- 20 in 2017

24
• sp_query_store_flush_db;
• Flushes in-memory portion of data to disk
• sp_query_store_reset_exec_stats plan_id;
• Clears runtime stats for a specific query plan
• sp_query_store_force_plan query_id, plan_id;
• Enables forcing a particular plan, if forcing fails a Xevent is fired to optimize normally
• sp_query_store_unforce_plan query_id, plan_id;
• Removes forcing a particular plan for a particular query
• sp_query_store_remove_plan plan_id;
• Removes a single plan from the Query Store
• sp_query_store_remove_query query_id;
• Removes a single query, all associated plans and statistics from Query Store
Management – Stored Procedures

…
…
…
…
…
25
• query_store_query_text - text entered by user, includes whites space, hints, etc... msdn-dn818159.aspx
• query_context_settings - unique combinations of plan affecting settings under which queries run msdn-dn818148.aspx
• query_store_plan - execution plan information for queries msdn-dn818155.aspx
• query_store_query - query information and overall aggregated runtime statistics msdn-dn818156.aspx
• query_store_runtime_stats - runtime execution statistics for queries (avg,min,max,std deviation) msdn-dn818158.aspx
• query_store_runtime_stats_interval - start and end times intervals for statistics collected msdn-dn818147.aspx2
Query Store DMVs

26
• SSMS Reports
• Force Plans
• View Forced Plans
• Compare Plans
• Find queries with high variation
• See overall resource consumption
• T-SQL
• Useful Queries
• Find Multiple Plans
• Find Long Running Queries
• Find Queries Doubling in time
• Find AD-Hoc Queries
• Using literals
• Plan Guides versus Forced Plans
Query Store Reporting & Use Cases

• New Wait Stats Store
• Plus plan store
• And runtime stats store
• On = 1 (default)
• T-SQL: WAIT_STATS_CAPTURE_MODE
• OFF = 0
• New 23 Wait Categories
• Summarizes wait types
• Common resolution
• Using-the-query-store
What’s New In 2017 Query Store?
Join to query_store_plan and
query_store_runtime_stats_interval

2017 Query Store – Cont.
• Find Top 10 Queries Waiting The Most
Over 1 hour waiting on Parallelism
SELECT TOP 10
qt.query_sql_text,
q.query_id,
p.plan_id,
ws.wait_category_desc,
sum(total_query_wait_time_ms) AS sum_total_wait_ms
FROM sys.query_store_wait_stats ws
JOIN sys.query_store_plan p ON ws.plan_id = p.plan_id
JOIN sys.query_store_query q ON p.query_id = q.query_id
JOIN sys.query_store_query_text qt ON q.query_text_id = qt.query_text_id
WHERE ws.wait_category_desc != 'Idle'
GROUP BY qt.query_sql_text, q.query_id, p.plan_id, ws.wait_category_desc
ORDER BY sum_total_wait_ms DESC

2017 Query Store – Cont.
• Query Store and the New Adaptive Query Processing
• Captures and can force batch mode adaptive join plan
• Can force plans with interleaved execution (MSTVFs)

• Command-line Tools
• sqlcmd & bcp
• SSMS
• Connect remotely from Windows
• Includes PowerShell
• Visual Studio Code
• How-to-develop-use-vscode
• SQL Server Data Tools (SSDT)
• For Visual Studio
• Build project on Windows
• Deploy to Linux
• SQL Operations Studio
• Installs on Linux/Windows/MacOS
• GUI interface – sqlops
https://docs.microsoft.com/en-us/sql/sql-operations-studio/download
Client Connectivity & Tools for 2017

31
• Review Properties
• Database defaults
• Turn on QS in Model
• Database with QS on
• Show full QS
• Look at T-SQL Code
• Multiplan Queries
• Long running – doubled in duration (microseconds)
• Parameter Canidates
• Review SSMS Reports
• Force a Plan
• Show Forced Plan Failures
• Extended Events
• Q & A
Demo Time

32
SSMS - Regressed Queries Report

33
Overall Resource Consumption

34
Top Resource Consuming Queries
Canceled
Failed

35
Tracked Queries

36
Compare Query Plans

37
Forcing Plans
SELECT p.plan_id, p.query_id, q.object_id as containing_object_id, force_failure_count, last_force_failure_reason_desc
FROM sys.query_store_plan p
JOIN sys.query_store_query q on p.query_id = q.query_id
WHERE is_forced_plan = 1;
To remove force plan via TSQL:
sp_query_store_unforce_plan 615,665;

38
Failed Forced Plans
Reasons why plan forcing failed
0: no failure
8637: ONLINE_INDEX_BUILD
8683: INVALID_STARJOIN
8684: TIME_OUT
8689: NO_DB
8690: HINT_CONFLICT
8691: SETOPT_CONFLICT
8694: DQ_NO_FORCING_SUPPORTED
8698: NO_PLAN
8712: NO_INDEX
8713: VIEW_COMPILE_FAILED
<other value>: GENERAL_FAILURE

39
• https://msdn.microsoft.com/en-us/library/dn817826.aspx
;WITH Query_MultPlans
AS
(
SELECT COUNT(*) AS cnt, q.query_id
FROM sys.query_store_query_text AS qt
JOIN sys.query_store_query AS q
ON qt.query_text_id = q.query_text_id
JOIN sys.query_store_plan AS p
ON p.query_id = q.query_id
GROUP BY q.query_id
HAVING COUNT(distinct plan_id) > 1
)
Useful Query Store Queries
SELECT q.query_id, object_name(object_id) AS ContainingObject,
query_sql_text, plan_id, convert(xml,p.query_plan) AS plan_xml,
p.last_compile_start_time, p.last_execution_time
FROM Query_MultPlans AS qm
ON qm.query_id = q.query_id
JOIN sys.query_store_plan AS p
ON q.query_id = p.query_id
JOIN sys.query_store_query_text qt
ORDER BY query_id, plan_id;
GO

40
Multi-plan Queries

41
;WITH AggregatedDurationLastHour
AS
(
SELECT q.query_id, SUM(count_executions * avg_duration) AS total_duration,
COUNT (distinct p.plan_id) AS number_of_plans
FROM sys.query_store_query_text AS qt JOIN sys.query_store_query AS q
JOIN sys.query_store_plan AS p ON q.query_id = p.query_id
JOIN sys.query_store_runtime_stats AS rs ON rs.plan_id = p.plan_id
JOIN sys.query_store_runtime_stats_interval AS rsi
ON rsi.runtime_stats_interval_id = rs.runtime_stats_interval_id
WHERE rsi.start_time >= DATEADD(hour, -1, GETUTCDATE())
AND rs.execution_type_desc = 'Regular'
GROUP BY q.query_id
) ,OrderedDuration
AS
(
SELECT query_id, total_duration, number_of_plans,
ROW_NUMBER () OVER (ORDER BY total_duration DESC, query_id) AS RN
FROM AggregatedDurationLastHour
)
Long Running – Last Hour
SELECT qt.query_sql_text, object_name(q.object_id) AS
containing_object, q.query_id,
p.plan_id,rsi.start_time as interval_start, rs.avg_duration,
CONVERT(xml, p.query_plan) AS query_plan_xml
FROM OrderedDuration od JOIN sys.query_store_query
AS q ON q.query_id = od.query_id
JOIN sys.query_store_query_text AS qt ON
q.query_text_id = qt.query_text_id
JOIN sys.query_store_plan AS p ON q.query_id =
p.query_id
JOIN sys.query_store_runtime_stats AS rs ON rs.plan_id
= p.plan_id
JOIN sys.query_store_runtime_stats_interval AS rsi ON
rsi.runtime_stats_interval_id = rs.runtime_stats_interval_id
WHERE rsi.start_time >= DATEADD(hour, -1,
GETUTCDATE())
AND number_of_plans > 1
ORDER BY total_duration DESC, query_id,
rsi.runtime_stats_interval_id, p.plan_id

42
Long Running – Last Hour

© 2017 SolarWinds Worldwide, LLC. All rights reserved. 43
SELECT
qt.query_sql_text,
q.query_id,
qt.query_text_id,
rs1.runtime_stats_id AS runtime_stats_id_1,
rsi1.start_time AS interval_1,
p1.plan_id AS plan_1,
rs1.avg_duration AS avg_duration_1,
rs2.avg_duration AS avg_duration_2,
p2.plan_id AS plan_2,
rsi2.start_time AS interval_2,
rs2.runtime_stats_id AS runtime_stats_id_2
FROM sys.query_store_query_text AS qt
JOIN sys.query_store_plan AS p1
ON q.query_id = p1.query_id
JOIN sys.query_store_runtime_stats AS rs1
ON p1.plan_id = rs1.plan_id
Execution Time - Doubled in Last 48hrs
JOIN sys.query_store_runtime_stats_interval AS rsi1
ON rsi1.runtime_stats_interval_id = rs1.runtime_stats_interval_id
JOIN sys.query_store_plan AS p2
ON q.query_id = p2.query_id
JOIN sys.query_store_runtime_stats AS rs2
ON p2.plan_id = rs2.plan_id
JOIN sys.query_store_runtime_stats_interval AS rsi2
ON rsi2.runtime_stats_interval_id = rs2.runtime_stats_interval_id
WHERE rsi1.start_time > DATEADD(hour, -48, GETUTCDATE())
AND rsi2.start_time > rsi1.start_time
AND p1.plan_id <> p2.plan_id
AND rs2.avg_duration > 2*rs1.avg_duration
ORDER BY q.query_id, rsi1.start_time, rsi2.start_time;

44
Execution Time - Doubled in Last 48hrs
Plan_1 = 370ms Plan_2 = 3.7s

45
• Query Store helps you manage query performance
• See plan changes & performance regression
• Find bad parameter sniffing & stale statistics
• See resource consumption at query & database level
• Info is only kept in database where query runs
• If you run them out of master or tempdb, they won’t be kept
• Need to plan & maintain the Query Store
• Size appropriately for workload
• Consider how long to keep the data
• Consider backups & restores
Summary

The SolarWinds, SolarWinds & Design, Orion, and THWACK trademarks are the exclusive
property of SolarWinds Worldwide, LLC or its affiliates, are registered with the U.S.
Patent and Trademark Office, and may be registered or pending registration in other
countries. All other SolarWinds trademarks, service marks, and logos may be common
law marks or are registered or pending registration. All other trademarks mentioned
herein are used for identification purposes only and are trademarks of (and may be
registered trademarks) of their respective companies.
Thank You!

Welcome To The 2016 Query Store!

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Welcome To The 2016 Query Store!

Similar to Welcome To The 2016 Query Store! (20)

More from SolarWinds

More from SolarWinds (20)

Recently uploaded

Recently uploaded (20)

Welcome To The 2016 Query Store!