SlideShare a Scribd company logo

Advanced pg_stat_statements: Filtering, Regression Testing & more

Talk on how we use pg_stat_statements at https://pganalyze.com/. Lessons learned and introducing our pg_query library for parsing SQL queries.

1 of 71
Download to read offline
Advanced 
pg_stat_statements: 
Filtering, Regression Testing 
& more 
@LukasFittl
Skilled Developer 
Amateur Hacker 
@LukasFittl
pganalyze.com 
1.6 million unique queries tracked 
using pg_stat_statements
Intro 
pg_stat_statements 
userid | 10 
dbid | 1397527 
query | SELECT * FROM x WHERE y = ? 
calls | 5 
total_time | 15.249 
rows | 0 
shared_blks_hit | 451 
shared_blks_read | 41 
shared_blks_dirtied | 26 
shared_blks_written | 0 
local_blks_hit | 0 
local_blks_read | 0 
local_blks_dirtied | 0 
local_blks_written | 0 
temp_blks_read | 0 
temp_blks_written | 0 
blk_read_time | 0 
blk_write_time | 0
Intro 
query | SELECT * FROM x WHERE y = ? 
calls | 5 
total_time | 15.249 
Query + Avg Time + Timeframe
Intro
Improving 
Data Quality 
pg_query 
Filtering & 
Regression 
Testing
Improving 
Data Quality 
pg_query 
Filtering & 
Regression 
Testing
SELECT "postgres_settings".* FROM "postgres_settings" WHERE 
"postgres_settings"."database_id" = $1 AND 
"postgres_settings"."invalidated_at_snapshot_id" IS NULL AND (id 
not in 
(70288,70289,70290,70291,70292,70293,70294,70295,70296,70297,70298 
,70299,70300,70301,70302,70303,70304,70305,70306,70307,70308,70309 
,70310,70311,70312,70313,70314,70315,70316,70317,70318,70319,70320 
,70321,70322,70323,70324,70325,70326,70327,99059,99060,70330,70331 
,70332,70333,70334,70335,70336,70337,70338,99061,70340,70341,70342 
,70343,70344,70345,70346,70347,70348,70349,70350,70351,70352,70353 
,70354,70355,70356,70357,70358,70359,70360,99062,70362,70363,70364 
,70365,70366,70367,70368,70369,70370,70371,70372,70373,70374,70375 
,70376,70377,70378,70379,70380,70381,70382,70383,70384,70385,70386 
,99063,99064,99065,99066,99067,70392,70393,70394,70395,70396,70397 
,70398,70399,70400,70401,70402,70403,70404,70405,99068,70407,70408 
,70409,70410,70411,70412,70413,70414,70415,70416,70417,99069,70419 
,70420,70421,99070,70423,70424,70425,70426,70427,70428, 
Truncation 
Improving 
Data Quality
Improving 
Data Quality 
-[ RECORD 1 ]———+-------------------------------- 
query | SELECT * FROM x WHERE y = ? 
calls | 5 
total_time | 15.249 
-[ RECORD 2 ]———+-------------------------------- 
query | SELECT * FROM z WHERE a = 123 
calls | 50 
total_time | 104.19 
Race Condition during 
pg_stat_statements_reset()
Lesson Learned: 
Avoid frequent 
Improving 
Data Quality 
pg_stat_statements_reset()
Fingerprinting 
SELECT a AS b == SELECT a AS c 
Problematic: 
y IN (?, ?, ?) != y IN (?, ?) 
Improving 
Data Quality 
SELECT a, b FROM x != SELECT b, a FROM x 
DEALLOCATE p141 != DEALLOCATE p150
Limited Statistical Information 
! 
Histogram / MAX(runtime) 
would be super-useful 
Improving 
Data Quality
pg_stat_plans 
Improving 
Data Quality 
pg_stat_statements variant that 
differentiates between query plans. 
Slower + Don’t use it before this bug is fixed: 
https://github.com/2ndQuadrant/pg_stat_plans/issues/39
Improving 
Data Quality 
Filtering & 
Regression 
Testing 
pg_query
Storing & Cleaning 
pg_stat_statements data 
pg_query
pg_query 
Monitoring Setup 
Snapshot 
{“schema”: 
{“n_live_tup”: 75, 
"relpages": 1, 
"reltuples": 75.0,…}, 
“queries”: 
[{..}, {..}]} 
Production 
Database Collector 
Normalize 
{“schema”: 
{“n_live_tup”: 75, 
"relpages": 1, 
"reltuples": 75.0,…}, 
“queries”: 
[{..}, {..}]} 
Monitoring 
Database 
Parse 
Fingerprint 
Extract Tables
pg_query 
queries 
id | 7053479 
database_id | 1 
received_query | SELECT * FROM x WHERE y = ? 
normalized_query | SELECT * FROM x WHERE y = ? 
created_at | 2014-06-27 16:20:08.334705 
updated_at | 2014-06-27 16:20:08.334705 
parse_tree | [{"SELECT":{...}] 
parse_error | 
parse_warnings | 
statement_types | {SELECT} 
truncated | f 
fingerprint | 00704f1fd8442b7c17821cb8a61856c3d61b330e
pg_query 
query_snapshots 
id | 170661585 
query_id | 7053479 
calls | 29 
total_time | 94.38 
rows | 29 
snapshot_id | 3386118 
snapshots 
id | 3386118 
database_id | 408 
collected_at | 2014-09-09 20:10:01 
submitter | pganalyze-collector 0.6.1 
query_source | pg_stat_statements
pg_query 
Normalize 
Parse 
Fingerprint 
Extract Tables
pg_query 
Normalize 
Parse Parsing an SQL Query 
Fingerprint 
Extract Tables
EXPLAIN (PARSETREE TRUE) pg_query 
SELECT * FROM x WHERE y = 1 
({SELECT :distinctClause <> 
:intoClause <> 
:targetList ( 
{RESTARGET 
:name <> 
:indirection <> 
:val {COLUMNREF :fields ({A_STAR}) :location 7} 
:location 7}) 
:fromClause ( 
{RANGEVAR 
:schemaname <> 
:relname x 
:inhOpt 2 
:relpersistence p 
:alias <> 
:location 14}) 
:whereClause {AEXPR :name (“=") 
:lexpr {COLUMNREF :fields ("y") :location 22} 
:rexpr {PARAMREF :number 0 :location 26} 
:location 24} 
Unfortunately doesn’t exist.
pg_query 
Parse Statement 
raw_parse(..) 
pg_catalog 
Rewrite 
Query 
Query 
Planner Execute
tree = raw_parser(query_str); pg_query 
str = nodeToString(tree); 
printf(str); 
({SELECT :distinctClause <> 
:intoClause <> 
:targetList ( 
{RESTARGET 
:name <> 
:indirection <> 
:val {COLUMNREF :fields ({A_STAR}) :location 7} 
:location 7}) 
:fromClause ( 
{RANGEVAR 
:schemaname <> 
:relname x 
:inhOpt 2 
:relpersistence p 
:alias <> 
:location 14}) 
:whereClause {AEXPR :name (“=") 
:lexpr {COLUMNREF :fields ("y") :location 22} 
:rexpr {PARAMREF :number 0 :location 26} 
:location 24}
pg_query 
Parse Statement 
raw_parse(..) 
pg_catalog 
Rewrite 
Query 
Query 
Planner Execute
github.com/pganalyze/pg_query 
pg_query 
Extension 
Compiles a full copy of PostgreSQL 
when you do “gem install pg_query”
pg_query 
PgQuery._raw_parse( 
“SELECT * FROM x WHERE y = 1”) 
({SELECT :distinctClause <> 
:intoClause <> 
:targetList ( 
{RESTARGET 
:name <> 
:indirection <> 
:val {COLUMNREF :fields ({A_STAR}) :location 7} 
:location 7}) 
:fromClause ( 
{RANGEVAR 
:schemaname <> 
:relname x 
:inhOpt 2 
:relpersistence p 
:alias <> 
:location 14}) 
:whereClause {AEXPR :name (“=") 
:lexpr {COLUMNREF :fields ("y") :location 22} 
:rexpr {PARAMREF :number 0 :location 26} 
:location 24} 
:groupClause <> 
:havingClause <> 
:windowClause <> 
:valuesLists <> 
:sortClause <> 
:limitOffset <> 
:limitCount <> 
:lockingClause <> 
:withClause <>
pg_query 
nodeToString is incomplete :( 
PgQuery._raw_parse(“CREATE SCHEMA foo”) 
WARNING: 01000: could not dump 
unrecognized node type: 754
src/backend/nodes/outfuncs.c pg_query 
Patch: Generate automatically, JSON output
PgQuery._raw_parse( pg_query 
“SELECT * FROM x WHERE y = 1”) 
[{"SELECT": { 
"targetList": [{ 
"RESTARGET": { 
"val": { 
"COLUMNREF": { 
"fields": [{"A_STAR": {}}], 
"location": 7 
} 
}, 
"location": 7 
} 
} 
], 
"fromClause": [ 
{ 
"RANGEVAR": { 
"relname": "x", 
"inhOpt": 2, 
"relpersistence": "p", 
"location": 14 
} 
} 
], 
"whereClause": { 
"AEXPR": { 
"name": [ 
"=" 
], 
"lexpr": {
pg_query 
Parsing a normalized 
Normalize 
Parse SQL query 
Fingerprint 
Extract Tables
EXPLAIN SELECT * FROM x WHERE y = 1 
QUERY PLAN 
--------------------------------------------------------------------- 
Index Scan using idx_for_y on x (cost=0.15..8.17 rows=1 width=140) 
Index Cond: (id = 1) 
Parse Analyze Plan 
pg_query
EXPLAIN SELECT * FROM x WHERE y = ? 
ERROR: syntax error at or near ";" 
LINE 1: EXPLAIN SELECT * FROM x WHERE y = ?; 
Parse Analyze Plan 
pg_query
EXPLAIN SELECT * FROM x WHERE y = ? 
EXPLAIN SELECT * FROM x WHERE y = $1 
ERROR: there is no parameter $1 
LINE 1: EXPLAIN SELECT * FROM x WHERE y = $1; 
Parse Analyze Plan 
pg_query
pg_query 
Parser Patch to support parsing “?”
pg_query 
Downside: 
Breaks ? operator in some cases 
Real fix: Don’t use ? as 
a replacement character.
pg_query 
Fingerprinting 
Normalize 
Parse 
Fingerprint 
Extract Tables
pg_query 
> require ‘pg_query’ 
! 
> q1 = PgQuery.parse(‘SELECT a, b FROM x’) 
> q1.fingerprint 
[“c72f1bc9feda72c0b4ba030eea90b4fed3ac8e86”] 
! 
> q2 = PgQuery.parse(‘SELECT b, a FROM x’) 
> q2.fingerprint 
[“c72f1bc9feda72c0b4ba030eea90b4fed3ac8e86”]
pg_query 
40 lines of unit-tested Ruby code
pg_query 
Extracting Table References 
Normalize 
Parse 
Fingerprint 
Extract Tables
pg_query 
> require ‘pg_query’ 
> q = PgQuery.parse(‘SELECT * FROM x’) 
> q.tables 
[“x”]
pg_query 
~90 lines of unit-tested Ruby code
github.com/pganalyze/pg_query 
pg_query
Improving 
Data Quality 
pg_query 
Filtering & 
Regression 
Testing
Filtering 
Filtering & 
Regression 
Testing
monitor.rb 
Filtering & 
Regression 
Testing 
Simple top-like tool that shows 
pg_stat_statements data 
https://gist.github.com/lfittl/301542602607b738b23f
Filtering & 
Regression 
Testing 
monitor.rb -d testdb 
AVG | QUERY 
-------------------------------------------------------------------------------- 
10.7ms | SELECT oid, typname, typelem, typdelim, typinput FROM pg_type 
3.0ms | SET time zone 'UTC' 
0.4ms | SELECT a.attname, format_type(a.atttypid, a.atttypmod), pg_get_expr(d.adbin, d.adrelid), 
a.attnotnull, a.atttypid, a.atttypmod FROM pg_attribute a LEFT JOIN pg_attrdef d ON a.attrelid 
= d.adrelid AND a.attnum = d.adnum WHERE a.attrelid = ?::regclass AND a.attnum > ? AND NOT 
a.attisdropped ORDER BY a.attnum 
0.2ms | SELECT pg_stat_statements_reset() 
0.1ms | SELECT query, calls, total_time FROM pg_stat_statements 
0.1ms | SELECT attr.attname FROM pg_attribute attr INNER JOIN pg_constraint cons ON attr.attrelid 
= cons.conrelid AND attr.attnum = cons.conkey[?] WHERE cons.contype = ? AND cons.conrelid = ?: 
:regclass 
0.0ms | SELECT COUNT(*) FROM pg_class c LEFT JOIN pg_namespace n ON n.oid = c.relnamespace WHERE 
c.relkind in (?,?) AND c.relname = ? AND n.nspname = ANY (current_schemas(?)) 
0.0ms | SELECT * FROM posts JOIN users ON (posts.author_id = users.id) WHERE users.login = ?; 
0.0ms | SET client_min_messages TO 'panic' 
0.0ms | set client_encoding to 'UTF8' 
0.0ms | SHOW client_min_messages 
0.0ms | SELECT * FROM ad_reels WHERE id = ?; 
0.0ms | SELECT * FROM posts WHERE guid = ?; 
0.0ms | SELECT ? 
0.0ms | SET client_min_messages TO 'warning' 
0.0ms | SET standard_conforming_strings = on 
0.0ms | SELECT "posts".* FROM "posts" ORDER BY "posts"."id" DESC LIMIT ? 
0.0ms | SHOW TIME ZONE
Filtering & 
Regression 
Testing 
monitor.rb -d testdb -t posts 
AVG | QUERY 
-------------------------------------------------------------------------------- 
0.0ms | SELECT * FROM posts JOIN users ON (posts.author_id = users.id) WHERE users.login = ?; 
0.0ms | SELECT * FROM posts WHERE guid = ?; 
0.0ms | SELECT "posts".* FROM "posts" ORDER BY "posts"."id" DESC LIMIT ?
Filtering & 
Regression 
Testing 
if cli.config[:table] 
q = PgQuery.parse(query["query"]) 
next unless q.tables.include?(cli.config[:table]) 
end
Regression 
Testing 
Filtering & 
Regression 
Testing
Which query plans are affected by 
removal of an index? 
! 
How would execution plans be 
affected by an upgrade to 9.X? 
Filtering & 
Regression 
Testing
Regression Test based on 
pg_stat_statements 
+ table statistics. 
! 
(no actual data) 
Filtering & 
Regression 
Testing
Schema Dump + 
Table Level Statistics 
"n_live_tup": 75, 
"relpages": 1, 
"reltuples": 75.0, 
“stanumbers1": [..], 
"stavalues1": “{..}”, 
… 
Local Test 
Database 
Testing Setup 
Production 
Database 
EXPLAIN SELECT FROM x WHERE y = ? 
Filtering & 
Regression 
Testing
EXPLAIN SELECT * FROM x WHERE y = ? 
EXPLAIN SELECT * FROM x WHERE y = $1 
ERROR: there is no parameter $1 
LINE 1: EXPLAIN SELECT * FROM x WHERE y = $1; 
Parse Analyze Plan 
Filtering & 
Regression 
Testing
y = $1 
ERROR: there is no parameter $0 
LINE 1: EXPLAIN SELECT * FROM x WHERE y = $0; 
Filtering & 
Regression 
Testing
y = $1 
ERROR: there is no parameter $0 
LINE 1: EXPLAIN SELECT * FROM x WHERE y = $0; 
y = NULL 
QUERY PLAN 
---------------------------------------------------------------- 
Result (cost=0.00..21.60 rows=1 width=40) 
One-Time Filter: NULL::boolean 
-> Seq Scan on x (cost=0.00..21.60 rows=1 width=40) 
Filtering & 
Regression 
Testing
y = $1 
ERROR: there is no parameter $0 
LINE 1: EXPLAIN SELECT * FROM x WHERE y = $0; 
y = NULL 
QUERY PLAN 
---------------------------------------------------------------- 
Result (cost=0.00..21.60 rows=1 width=40) 
One-Time Filter: NULL::boolean 
-> Seq Scan on x (cost=0.00..21.60 rows=1 width=40) 
y = (SELECT null) 
ERROR: failed to find conversion function from unknown to integer 
Filtering & 
Regression 
Testing
y = $1 
ERROR: there is no parameter $0 
LINE 1: EXPLAIN SELECT * FROM x WHERE y = $0; 
y = NULL 
QUERY PLAN 
---------------------------------------------------------------- 
Result (cost=0.00..21.60 rows=1 width=40) 
One-Time Filter: NULL::boolean 
-> Seq Scan on x (cost=0.00..21.60 rows=1 width=40) 
y = (SELECT null) 
ERROR: failed to find conversion function from unknown to integer 
y = (SELECT null::integer) 
QUERY PLAN 
---------------------------------------------------------------------- 
Index Scan using idx_for_y on x (cost=0.16..8.18 rows=1 width=144) 
Index Cond: (y = $0) 
InitPlan 1 (returns $0) 
-> Result (cost=0.00..0.01 rows=1 width=0) 
Filtering & 
Regression 
Testing
Finding out the type 
y = $1 
ERROR: there is no parameter $1 
LINE 1: EXPLAIN SELECT * FROM x WHERE y = $1; 
pg_prepared_statements 
PREPARE tmp AS SELECT * FROM x WHERE y = $1; 
SELECT unnest(parameter_types) AS data_type 
FROM pg_prepared_statements WHERE name = ‘tmp’; 
DEALLOCATE tmp; 
data_type 
----------- 
integer 
Filtering & 
Regression 
Testing
EXPLAIN SELECT * FROM x WHERE y = ? 
EXPLAIN SELECT * FROM x WHERE y = $0 
EXPLAIN SELECT * FROM x WHERE y = 
((SELECT null::integer)::integer) 
QUERY PLAN 
--------------------------------------------------------------------- 
Index Scan using idx_for_y on x (cost=0.16..8.18 rows=1 width=144) 
Index Cond: (y = $0) 
InitPlan 1 (returns $0) 
-> Result (cost=0.00..0.01 rows=1 width=0) 
Parse Analyze Plan 
Filtering & 
Regression 
Testing
Open Issue: 
Planner reads actual 
physical size whilst planning 
Filtering & 
Regression 
Testing
github.com/pganalyze/pg_simulator 
Filtering & 
Regression 
Testing
Improving 
Data Quality 
pg_query 
Filtering & 
Regression 
Testing
9.5 proposal for pg_s_s: 
Closing 
! 
Instead of ? use $0 as replacement character - 
making the output parseable again.
9.5 proposal for outfuncs.c: 
! 
Generate automatically from struct definitions, 
cutting 3000 hand-written lines down to 1000. 
! 
Add JSON output support. 
Closing
9.X proposal: 
! 
Consider adding a way to get a parsetree 
more easily. 
! 
Via SQL / shared library / helper tool. 
Closing
Tools & libraries available at: 
Closing 
github.com/pganalyze
@LukasFittl 
Thank you! 
github.com/pganalyze 
pganalyze.com
Advanced pg_stat_statements: Filtering, Regression Testing & more
Backup Slides
Classifying queries 
Improving 
Data Quality 
Frequent/OLTP vs analytical query

Recommended

Deep dive into PostgreSQL statistics.
Deep dive into PostgreSQL statistics.Deep dive into PostgreSQL statistics.
Deep dive into PostgreSQL statistics.Alexey Lesovsky
 
Inside PostgreSQL Shared Memory
Inside PostgreSQL Shared MemoryInside PostgreSQL Shared Memory
Inside PostgreSQL Shared MemoryEDB
 
Apache Calcite: One planner fits all
Apache Calcite: One planner fits allApache Calcite: One planner fits all
Apache Calcite: One planner fits allJulian Hyde
 
PostgreSQL High_Performance_Cheatsheet
PostgreSQL High_Performance_CheatsheetPostgreSQL High_Performance_Cheatsheet
PostgreSQL High_Performance_CheatsheetLucian Oprea
 
Top 10 Mistakes When Migrating From Oracle to PostgreSQL
Top 10 Mistakes When Migrating From Oracle to PostgreSQLTop 10 Mistakes When Migrating From Oracle to PostgreSQL
Top 10 Mistakes When Migrating From Oracle to PostgreSQLJim Mlodgenski
 
Adding measures to Calcite SQL
Adding measures to Calcite SQLAdding measures to Calcite SQL
Adding measures to Calcite SQLJulian Hyde
 
What is new in PostgreSQL 14?
What is new in PostgreSQL 14?What is new in PostgreSQL 14?
What is new in PostgreSQL 14?Mydbops
 
PostgreSQL Deep Internal
PostgreSQL Deep InternalPostgreSQL Deep Internal
PostgreSQL Deep InternalEXEM
 

More Related Content

What's hot

Oracle statistics by example
Oracle statistics by exampleOracle statistics by example
Oracle statistics by exampleMauro Pagano
 
MySQL Query And Index Tuning
MySQL Query And Index TuningMySQL Query And Index Tuning
MySQL Query And Index TuningManikanda kumar
 
NoSQL @ CodeMash 2010
NoSQL @ CodeMash 2010NoSQL @ CodeMash 2010
NoSQL @ CodeMash 2010Ben Scofield
 
[Pgday.Seoul 2019] Citus를 이용한 분산 데이터베이스
[Pgday.Seoul 2019] Citus를 이용한 분산 데이터베이스[Pgday.Seoul 2019] Citus를 이용한 분산 데이터베이스
[Pgday.Seoul 2019] Citus를 이용한 분산 데이터베이스PgDay.Seoul
 
Auditing and Monitoring PostgreSQL/EPAS
Auditing and Monitoring PostgreSQL/EPASAuditing and Monitoring PostgreSQL/EPAS
Auditing and Monitoring PostgreSQL/EPASEDB
 
Troubleshooting Complex Oracle Performance Problems with Tanel Poder
Troubleshooting Complex Oracle Performance Problems with Tanel PoderTroubleshooting Complex Oracle Performance Problems with Tanel Poder
Troubleshooting Complex Oracle Performance Problems with Tanel PoderTanel Poder
 
Parquet performance tuning: the missing guide
Parquet performance tuning: the missing guideParquet performance tuning: the missing guide
Parquet performance tuning: the missing guideRyan Blue
 
Adapting and adopting spm v04
Adapting and adopting spm v04Adapting and adopting spm v04
Adapting and adopting spm v04Carlos Sierra
 
2021 04-20 apache arrow and its impact on the database industry.pptx
2021 04-20  apache arrow and its impact on the database industry.pptx2021 04-20  apache arrow and its impact on the database industry.pptx
2021 04-20 apache arrow and its impact on the database industry.pptxAndrew Lamb
 
Incremental View Maintenance with Coral, DBT, and Iceberg
Incremental View Maintenance with Coral, DBT, and IcebergIncremental View Maintenance with Coral, DBT, and Iceberg
Incremental View Maintenance with Coral, DBT, and IcebergWalaa Eldin Moustafa
 
Data Wrangling with Pandas
Data Wrangling with PandasData Wrangling with Pandas
Data Wrangling with PandasLuis Carrasco
 
Ash masters : advanced ash analytics on Oracle
Ash masters : advanced ash analytics on Oracle Ash masters : advanced ash analytics on Oracle
Ash masters : advanced ash analytics on Oracle Kyle Hailey
 
[pgday.Seoul 2022] POSTGRES 테스트코드로 기여하기 - 이동욱
[pgday.Seoul 2022] POSTGRES 테스트코드로 기여하기 - 이동욱[pgday.Seoul 2022] POSTGRES 테스트코드로 기여하기 - 이동욱
[pgday.Seoul 2022] POSTGRES 테스트코드로 기여하기 - 이동욱PgDay.Seoul
 
[pgday.Seoul 2022] PostgreSQL구조 - 윤성재
[pgday.Seoul 2022] PostgreSQL구조 - 윤성재[pgday.Seoul 2022] PostgreSQL구조 - 윤성재
[pgday.Seoul 2022] PostgreSQL구조 - 윤성재PgDay.Seoul
 
How to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better PerformanceHow to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better Performanceoysteing
 
How does PostgreSQL work with disks: a DBA's checklist in detail. PGConf.US 2015
How does PostgreSQL work with disks: a DBA's checklist in detail. PGConf.US 2015How does PostgreSQL work with disks: a DBA's checklist in detail. PGConf.US 2015
How does PostgreSQL work with disks: a DBA's checklist in detail. PGConf.US 2015PostgreSQL-Consulting
 
MySQL Indexing - Best practices for MySQL 5.6
MySQL Indexing - Best practices for MySQL 5.6MySQL Indexing - Best practices for MySQL 5.6
MySQL Indexing - Best practices for MySQL 5.6MYXPLAIN
 
Spark SQL Catalyst Code Optimization using Function Outlining with Kavana Bha...
Spark SQL Catalyst Code Optimization using Function Outlining with Kavana Bha...Spark SQL Catalyst Code Optimization using Function Outlining with Kavana Bha...
Spark SQL Catalyst Code Optimization using Function Outlining with Kavana Bha...Databricks
 

What's hot (20)

Oracle statistics by example
Oracle statistics by exampleOracle statistics by example
Oracle statistics by example
 
PostgreSQL and RAM usage
PostgreSQL and RAM usagePostgreSQL and RAM usage
PostgreSQL and RAM usage
 
MySQL Query And Index Tuning
MySQL Query And Index TuningMySQL Query And Index Tuning
MySQL Query And Index Tuning
 
NoSQL @ CodeMash 2010
NoSQL @ CodeMash 2010NoSQL @ CodeMash 2010
NoSQL @ CodeMash 2010
 
[Pgday.Seoul 2019] Citus를 이용한 분산 데이터베이스
[Pgday.Seoul 2019] Citus를 이용한 분산 데이터베이스[Pgday.Seoul 2019] Citus를 이용한 분산 데이터베이스
[Pgday.Seoul 2019] Citus를 이용한 분산 데이터베이스
 
Auditing and Monitoring PostgreSQL/EPAS
Auditing and Monitoring PostgreSQL/EPASAuditing and Monitoring PostgreSQL/EPAS
Auditing and Monitoring PostgreSQL/EPAS
 
Troubleshooting Complex Oracle Performance Problems with Tanel Poder
Troubleshooting Complex Oracle Performance Problems with Tanel PoderTroubleshooting Complex Oracle Performance Problems with Tanel Poder
Troubleshooting Complex Oracle Performance Problems with Tanel Poder
 
Parquet performance tuning: the missing guide
Parquet performance tuning: the missing guideParquet performance tuning: the missing guide
Parquet performance tuning: the missing guide
 
Adapting and adopting spm v04
Adapting and adopting spm v04Adapting and adopting spm v04
Adapting and adopting spm v04
 
2021 04-20 apache arrow and its impact on the database industry.pptx
2021 04-20  apache arrow and its impact on the database industry.pptx2021 04-20  apache arrow and its impact on the database industry.pptx
2021 04-20 apache arrow and its impact on the database industry.pptx
 
Incremental View Maintenance with Coral, DBT, and Iceberg
Incremental View Maintenance with Coral, DBT, and IcebergIncremental View Maintenance with Coral, DBT, and Iceberg
Incremental View Maintenance with Coral, DBT, and Iceberg
 
Data Wrangling with Pandas
Data Wrangling with PandasData Wrangling with Pandas
Data Wrangling with Pandas
 
Ash masters : advanced ash analytics on Oracle
Ash masters : advanced ash analytics on Oracle Ash masters : advanced ash analytics on Oracle
Ash masters : advanced ash analytics on Oracle
 
[pgday.Seoul 2022] POSTGRES 테스트코드로 기여하기 - 이동욱
[pgday.Seoul 2022] POSTGRES 테스트코드로 기여하기 - 이동욱[pgday.Seoul 2022] POSTGRES 테스트코드로 기여하기 - 이동욱
[pgday.Seoul 2022] POSTGRES 테스트코드로 기여하기 - 이동욱
 
[pgday.Seoul 2022] PostgreSQL구조 - 윤성재
[pgday.Seoul 2022] PostgreSQL구조 - 윤성재[pgday.Seoul 2022] PostgreSQL구조 - 윤성재
[pgday.Seoul 2022] PostgreSQL구조 - 윤성재
 
How to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better PerformanceHow to Analyze and Tune MySQL Queries for Better Performance
How to Analyze and Tune MySQL Queries for Better Performance
 
How does PostgreSQL work with disks: a DBA's checklist in detail. PGConf.US 2015
How does PostgreSQL work with disks: a DBA's checklist in detail. PGConf.US 2015How does PostgreSQL work with disks: a DBA's checklist in detail. PGConf.US 2015
How does PostgreSQL work with disks: a DBA's checklist in detail. PGConf.US 2015
 
MySQL Indexing - Best practices for MySQL 5.6
MySQL Indexing - Best practices for MySQL 5.6MySQL Indexing - Best practices for MySQL 5.6
MySQL Indexing - Best practices for MySQL 5.6
 
Spark SQL Catalyst Code Optimization using Function Outlining with Kavana Bha...
Spark SQL Catalyst Code Optimization using Function Outlining with Kavana Bha...Spark SQL Catalyst Code Optimization using Function Outlining with Kavana Bha...
Spark SQL Catalyst Code Optimization using Function Outlining with Kavana Bha...
 
How to Use JSON in MySQL Wrong
How to Use JSON in MySQL WrongHow to Use JSON in MySQL Wrong
How to Use JSON in MySQL Wrong
 

Viewers also liked

Researching postgresql
Researching postgresqlResearching postgresql
Researching postgresqlFernando Ike
 
5 Tips to Simplify the Management of Your Postgres Database
5 Tips to Simplify the Management of Your Postgres Database5 Tips to Simplify the Management of Your Postgres Database
5 Tips to Simplify the Management of Your Postgres DatabaseEDB
 
Four Levels of High Availability in Cloud Foundry (Cloud Foundry Summit 2014)
Four Levels of High Availability in Cloud Foundry (Cloud Foundry Summit 2014)Four Levels of High Availability in Cloud Foundry (Cloud Foundry Summit 2014)
Four Levels of High Availability in Cloud Foundry (Cloud Foundry Summit 2014)VMware Tanzu
 
Pandora FMS: PostgreSQL Plugin
Pandora FMS: PostgreSQL PluginPandora FMS: PostgreSQL Plugin
Pandora FMS: PostgreSQL PluginPandora FMS
 
Ajal filters
Ajal filtersAjal filters
Ajal filtersAJAL A J
 
Isolation amplifier
Isolation amplifierIsolation amplifier
Isolation amplifiervickeysv
 
Regression testing
Regression testingRegression testing
Regression testingHarsh verma
 
What will testing look like in year 2020
What will testing look like in year 2020What will testing look like in year 2020
What will testing look like in year 2020BugRaptors
 
Customer Satisfaction Measures
Customer Satisfaction MeasuresCustomer Satisfaction Measures
Customer Satisfaction MeasuresMichael Lieberman
 
All you need to know about regression testing | David Tzemach
All you need to know about regression testing | David TzemachAll you need to know about regression testing | David Tzemach
All you need to know about regression testing | David TzemachDavid Tzemach
 
Different Android Test Automation Frameworks - What Works You the Best?
Different Android Test Automation Frameworks - What Works You the Best?Different Android Test Automation Frameworks - What Works You the Best?
Different Android Test Automation Frameworks - What Works You the Best?Bitbar
 
Automated visual-regression-testing
Automated visual-regression-testingAutomated visual-regression-testing
Automated visual-regression-testingSriram Angajala
 
Automated Testing for Embedded Software in C or C++
Automated Testing for Embedded Software in C or C++Automated Testing for Embedded Software in C or C++
Automated Testing for Embedded Software in C or C++Lars Thorup
 
PostgreSQL Portland Performance Practice Project - Database Test 2 Filesystem...
PostgreSQL Portland Performance Practice Project - Database Test 2 Filesystem...PostgreSQL Portland Performance Practice Project - Database Test 2 Filesystem...
PostgreSQL Portland Performance Practice Project - Database Test 2 Filesystem...Mark Wong
 
24/7 Monitoring and Alerting of PostgreSQL
24/7 Monitoring and Alerting of PostgreSQL24/7 Monitoring and Alerting of PostgreSQL
24/7 Monitoring and Alerting of PostgreSQLInMobi Technology
 
Regression testing
Regression testingRegression testing
Regression testingMohua Amin
 
Automated Regression Testing for Embedded Systems in Action
Automated Regression Testing for Embedded Systems in ActionAutomated Regression Testing for Embedded Systems in Action
Automated Regression Testing for Embedded Systems in ActionAANDTech
 

Viewers also liked (20)

PGDay India 2016
PGDay India 2016PGDay India 2016
PGDay India 2016
 
Isolation amplifier
Isolation amplifier Isolation amplifier
Isolation amplifier
 
Researching postgresql
Researching postgresqlResearching postgresql
Researching postgresql
 
5 Tips to Simplify the Management of Your Postgres Database
5 Tips to Simplify the Management of Your Postgres Database5 Tips to Simplify the Management of Your Postgres Database
5 Tips to Simplify the Management of Your Postgres Database
 
Four Levels of High Availability in Cloud Foundry (Cloud Foundry Summit 2014)
Four Levels of High Availability in Cloud Foundry (Cloud Foundry Summit 2014)Four Levels of High Availability in Cloud Foundry (Cloud Foundry Summit 2014)
Four Levels of High Availability in Cloud Foundry (Cloud Foundry Summit 2014)
 
Pandora FMS: PostgreSQL Plugin
Pandora FMS: PostgreSQL PluginPandora FMS: PostgreSQL Plugin
Pandora FMS: PostgreSQL Plugin
 
Ajal filters
Ajal filtersAjal filters
Ajal filters
 
Isolation amplifier
Isolation amplifierIsolation amplifier
Isolation amplifier
 
Regression testing
Regression testingRegression testing
Regression testing
 
What will testing look like in year 2020
What will testing look like in year 2020What will testing look like in year 2020
What will testing look like in year 2020
 
Customer Satisfaction Measures
Customer Satisfaction MeasuresCustomer Satisfaction Measures
Customer Satisfaction Measures
 
All you need to know about regression testing | David Tzemach
All you need to know about regression testing | David TzemachAll you need to know about regression testing | David Tzemach
All you need to know about regression testing | David Tzemach
 
Different Android Test Automation Frameworks - What Works You the Best?
Different Android Test Automation Frameworks - What Works You the Best?Different Android Test Automation Frameworks - What Works You the Best?
Different Android Test Automation Frameworks - What Works You the Best?
 
Automated visual-regression-testing
Automated visual-regression-testingAutomated visual-regression-testing
Automated visual-regression-testing
 
Automated Testing for Embedded Software in C or C++
Automated Testing for Embedded Software in C or C++Automated Testing for Embedded Software in C or C++
Automated Testing for Embedded Software in C or C++
 
PostgreSQL Portland Performance Practice Project - Database Test 2 Filesystem...
PostgreSQL Portland Performance Practice Project - Database Test 2 Filesystem...PostgreSQL Portland Performance Practice Project - Database Test 2 Filesystem...
PostgreSQL Portland Performance Practice Project - Database Test 2 Filesystem...
 
24/7 Monitoring and Alerting of PostgreSQL
24/7 Monitoring and Alerting of PostgreSQL24/7 Monitoring and Alerting of PostgreSQL
24/7 Monitoring and Alerting of PostgreSQL
 
Adaptive filter
Adaptive filterAdaptive filter
Adaptive filter
 
Regression testing
Regression testingRegression testing
Regression testing
 
Automated Regression Testing for Embedded Systems in Action
Automated Regression Testing for Embedded Systems in ActionAutomated Regression Testing for Embedded Systems in Action
Automated Regression Testing for Embedded Systems in Action
 

Similar to Advanced pg_stat_statements: Filtering, Regression Testing & more

Using PostgreSQL statistics to optimize performance
Using PostgreSQL statistics to optimize performance Using PostgreSQL statistics to optimize performance
Using PostgreSQL statistics to optimize performance Alexey Ermakov
 
Explaining the Postgres Query Optimizer
Explaining the Postgres Query OptimizerExplaining the Postgres Query Optimizer
Explaining the Postgres Query OptimizerEDB
 
New Tuning Features in Oracle 11g - How to make your database as boring as po...
New Tuning Features in Oracle 11g - How to make your database as boring as po...New Tuning Features in Oracle 11g - How to make your database as boring as po...
New Tuning Features in Oracle 11g - How to make your database as boring as po...Sage Computing Services
 
Explaining the Postgres Query Optimizer (Bruce Momjian)
Explaining the Postgres Query Optimizer (Bruce Momjian)Explaining the Postgres Query Optimizer (Bruce Momjian)
Explaining the Postgres Query Optimizer (Bruce Momjian)Ontico
 
Postgres performance for humans
Postgres performance for humansPostgres performance for humans
Postgres performance for humansCraig Kerstiens
 
Advanced Postgres Monitoring
Advanced Postgres MonitoringAdvanced Postgres Monitoring
Advanced Postgres MonitoringDenish Patel
 
Basic Query Tuning Primer - Pg West 2009
Basic Query Tuning Primer - Pg West 2009Basic Query Tuning Primer - Pg West 2009
Basic Query Tuning Primer - Pg West 2009mattsmiley
 
Histograms : Pre-12c and Now
Histograms : Pre-12c and NowHistograms : Pre-12c and Now
Histograms : Pre-12c and NowAnju Garg
 
Extending Spark for Qbeast's SQL Data Source​ with Paola Pardo and Cesare Cug...
Extending Spark for Qbeast's SQL Data Source​ with Paola Pardo and Cesare Cug...Extending Spark for Qbeast's SQL Data Source​ with Paola Pardo and Cesare Cug...
Extending Spark for Qbeast's SQL Data Source​ with Paola Pardo and Cesare Cug...Qbeast
 
Nyc open-data-2015-andvanced-sklearn-expanded
Nyc open-data-2015-andvanced-sklearn-expandedNyc open-data-2015-andvanced-sklearn-expanded
Nyc open-data-2015-andvanced-sklearn-expandedVivian S. Zhang
 
PostgreSQL query planner's internals
PostgreSQL query planner's internalsPostgreSQL query planner's internals
PostgreSQL query planner's internalsAlexey Ermakov
 
PerlApp2Postgresql (2)
PerlApp2Postgresql (2)PerlApp2Postgresql (2)
PerlApp2Postgresql (2)Jerome Eteve
 
Explaining the Postgres Query Optimizer - PGCon 2014
Explaining the Postgres Query Optimizer - PGCon 2014Explaining the Postgres Query Optimizer - PGCon 2014
Explaining the Postgres Query Optimizer - PGCon 2014EDB
 
Postgres Performance for Humans
Postgres Performance for HumansPostgres Performance for Humans
Postgres Performance for HumansCitus Data
 
query-optimization-techniques_talk.pdf
query-optimization-techniques_talk.pdfquery-optimization-techniques_talk.pdf
query-optimization-techniques_talk.pdfgaros1
 
PostgreSQL 9.6 새 기능 소개
PostgreSQL 9.6 새 기능 소개PostgreSQL 9.6 새 기능 소개
PostgreSQL 9.6 새 기능 소개PgDay.Seoul
 
pg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQLpg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQLCommand Prompt., Inc
 
pg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQLpg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQLMark Wong
 

Similar to Advanced pg_stat_statements: Filtering, Regression Testing & more (20)

Using PostgreSQL statistics to optimize performance
Using PostgreSQL statistics to optimize performance Using PostgreSQL statistics to optimize performance
Using PostgreSQL statistics to optimize performance
 
Explaining the Postgres Query Optimizer
Explaining the Postgres Query OptimizerExplaining the Postgres Query Optimizer
Explaining the Postgres Query Optimizer
 
New Tuning Features in Oracle 11g - How to make your database as boring as po...
New Tuning Features in Oracle 11g - How to make your database as boring as po...New Tuning Features in Oracle 11g - How to make your database as boring as po...
New Tuning Features in Oracle 11g - How to make your database as boring as po...
 
Explaining the Postgres Query Optimizer (Bruce Momjian)
Explaining the Postgres Query Optimizer (Bruce Momjian)Explaining the Postgres Query Optimizer (Bruce Momjian)
Explaining the Postgres Query Optimizer (Bruce Momjian)
 
Postgres performance for humans
Postgres performance for humansPostgres performance for humans
Postgres performance for humans
 
Advanced Postgres Monitoring
Advanced Postgres MonitoringAdvanced Postgres Monitoring
Advanced Postgres Monitoring
 
query_tuning.pdf
query_tuning.pdfquery_tuning.pdf
query_tuning.pdf
 
Basic Query Tuning Primer - Pg West 2009
Basic Query Tuning Primer - Pg West 2009Basic Query Tuning Primer - Pg West 2009
Basic Query Tuning Primer - Pg West 2009
 
Basic Query Tuning Primer
Basic Query Tuning PrimerBasic Query Tuning Primer
Basic Query Tuning Primer
 
Histograms : Pre-12c and Now
Histograms : Pre-12c and NowHistograms : Pre-12c and Now
Histograms : Pre-12c and Now
 
Extending Spark for Qbeast's SQL Data Source​ with Paola Pardo and Cesare Cug...
Extending Spark for Qbeast's SQL Data Source​ with Paola Pardo and Cesare Cug...Extending Spark for Qbeast's SQL Data Source​ with Paola Pardo and Cesare Cug...
Extending Spark for Qbeast's SQL Data Source​ with Paola Pardo and Cesare Cug...
 
Nyc open-data-2015-andvanced-sklearn-expanded
Nyc open-data-2015-andvanced-sklearn-expandedNyc open-data-2015-andvanced-sklearn-expanded
Nyc open-data-2015-andvanced-sklearn-expanded
 
PostgreSQL query planner's internals
PostgreSQL query planner's internalsPostgreSQL query planner's internals
PostgreSQL query planner's internals
 
PerlApp2Postgresql (2)
PerlApp2Postgresql (2)PerlApp2Postgresql (2)
PerlApp2Postgresql (2)
 
Explaining the Postgres Query Optimizer - PGCon 2014
Explaining the Postgres Query Optimizer - PGCon 2014Explaining the Postgres Query Optimizer - PGCon 2014
Explaining the Postgres Query Optimizer - PGCon 2014
 
Postgres Performance for Humans
Postgres Performance for HumansPostgres Performance for Humans
Postgres Performance for Humans
 
query-optimization-techniques_talk.pdf
query-optimization-techniques_talk.pdfquery-optimization-techniques_talk.pdf
query-optimization-techniques_talk.pdf
 
PostgreSQL 9.6 새 기능 소개
PostgreSQL 9.6 새 기능 소개PostgreSQL 9.6 새 기능 소개
PostgreSQL 9.6 새 기능 소개
 
pg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQLpg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQL
 
pg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQLpg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQL
 

More from Lukas Fittl

Lean Startup & UX for Agile Teams
Lean Startup & UX for Agile TeamsLean Startup & UX for Agile Teams
Lean Startup & UX for Agile TeamsLukas Fittl
 
Lean Stack - A Story Of Continuous Improvement
Lean Stack - A Story Of Continuous ImprovementLean Stack - A Story Of Continuous Improvement
Lean Stack - A Story Of Continuous ImprovementLukas Fittl
 
Lean & Agile ♥ UX Design
Lean & Agile ♥ UX DesignLean & Agile ♥ UX Design
Lean & Agile ♥ UX DesignLukas Fittl
 
We Built It, And They Didn't Come!
We Built It, And They Didn't Come!We Built It, And They Didn't Come!
We Built It, And They Didn't Come!Lukas Fittl
 
Actionable Metrics @ Lean Startup Meetup Berlin
Actionable Metrics @ Lean Startup Meetup BerlinActionable Metrics @ Lean Startup Meetup Berlin
Actionable Metrics @ Lean Startup Meetup BerlinLukas Fittl
 
Enabling Innovation using Lean Startup
Enabling Innovation using Lean StartupEnabling Innovation using Lean Startup
Enabling Innovation using Lean StartupLukas Fittl
 
Lean Startup Essentials - March 2012
Lean Startup Essentials - March 2012Lean Startup Essentials - March 2012
Lean Startup Essentials - March 2012Lukas Fittl
 
Lean Startup Essentials - Le Camping Edition
Lean Startup Essentials - Le Camping EditionLean Startup Essentials - Le Camping Edition
Lean Startup Essentials - Le Camping EditionLukas Fittl
 
Lean Startup Essentials - SeedHack Edition
Lean Startup Essentials - SeedHack EditionLean Startup Essentials - SeedHack Edition
Lean Startup Essentials - SeedHack EditionLukas Fittl
 
Lean Startup Essentials - STARTup Live Hagenberg
Lean Startup Essentials - STARTup Live HagenbergLean Startup Essentials - STARTup Live Hagenberg
Lean Startup Essentials - STARTup Live HagenbergLukas Fittl
 
Lean Startup Essentials - STARTup Live Graz
Lean Startup Essentials - STARTup Live GrazLean Startup Essentials - STARTup Live Graz
Lean Startup Essentials - STARTup Live GrazLukas Fittl
 
Introduction to Git
Introduction to GitIntroduction to Git
Introduction to GitLukas Fittl
 

More from Lukas Fittl (12)

Lean Startup & UX for Agile Teams
Lean Startup & UX for Agile TeamsLean Startup & UX for Agile Teams
Lean Startup & UX for Agile Teams
 
Lean Stack - A Story Of Continuous Improvement
Lean Stack - A Story Of Continuous ImprovementLean Stack - A Story Of Continuous Improvement
Lean Stack - A Story Of Continuous Improvement
 
Lean & Agile ♥ UX Design
Lean & Agile ♥ UX DesignLean & Agile ♥ UX Design
Lean & Agile ♥ UX Design
 
We Built It, And They Didn't Come!
We Built It, And They Didn't Come!We Built It, And They Didn't Come!
We Built It, And They Didn't Come!
 
Actionable Metrics @ Lean Startup Meetup Berlin
Actionable Metrics @ Lean Startup Meetup BerlinActionable Metrics @ Lean Startup Meetup Berlin
Actionable Metrics @ Lean Startup Meetup Berlin
 
Enabling Innovation using Lean Startup
Enabling Innovation using Lean StartupEnabling Innovation using Lean Startup
Enabling Innovation using Lean Startup
 
Lean Startup Essentials - March 2012
Lean Startup Essentials - March 2012Lean Startup Essentials - March 2012
Lean Startup Essentials - March 2012
 
Lean Startup Essentials - Le Camping Edition
Lean Startup Essentials - Le Camping EditionLean Startup Essentials - Le Camping Edition
Lean Startup Essentials - Le Camping Edition
 
Lean Startup Essentials - SeedHack Edition
Lean Startup Essentials - SeedHack EditionLean Startup Essentials - SeedHack Edition
Lean Startup Essentials - SeedHack Edition
 
Lean Startup Essentials - STARTup Live Hagenberg
Lean Startup Essentials - STARTup Live HagenbergLean Startup Essentials - STARTup Live Hagenberg
Lean Startup Essentials - STARTup Live Hagenberg
 
Lean Startup Essentials - STARTup Live Graz
Lean Startup Essentials - STARTup Live GrazLean Startup Essentials - STARTup Live Graz
Lean Startup Essentials - STARTup Live Graz
 
Introduction to Git
Introduction to GitIntroduction to Git
Introduction to Git
 

Recently uploaded

Growing through Artificial Intelligence (AI)
Growing  through Artificial Intelligence (AI)Growing  through Artificial Intelligence (AI)
Growing through Artificial Intelligence (AI)OmChou6
 
Customer Satisfaction Data - Multiple Linear Regression Model.pdf
Customer Satisfaction Data -  Multiple Linear Regression Model.pdfCustomer Satisfaction Data -  Multiple Linear Regression Model.pdf
Customer Satisfaction Data - Multiple Linear Regression Model.pdfruwanp2000
 
presentation big data analytics on Apache spark
presentation big data analytics on Apache sparkpresentation big data analytics on Apache spark
presentation big data analytics on Apache sparkVarun Garg
 
EXCEL-VLOOKUP-AND-HLOOKUP LECTURE NOTES ALL EXCEL VLOOKUP NOTES PDF
EXCEL-VLOOKUP-AND-HLOOKUP LECTURE NOTES ALL EXCEL VLOOKUP NOTES PDFEXCEL-VLOOKUP-AND-HLOOKUP LECTURE NOTES ALL EXCEL VLOOKUP NOTES PDF
EXCEL-VLOOKUP-AND-HLOOKUP LECTURE NOTES ALL EXCEL VLOOKUP NOTES PDFProject Cubicle
 
Discover the Best Free Web Hosting Services with SSL in 2023
Discover the Best Free Web Hosting Services with SSL in 2023Discover the Best Free Web Hosting Services with SSL in 2023
Discover the Best Free Web Hosting Services with SSL in 2023maker Money
 
introduction-to-crimean-congo-haemorrhagic-fever.pdf
introduction-to-crimean-congo-haemorrhagic-fever.pdfintroduction-to-crimean-congo-haemorrhagic-fever.pdf
introduction-to-crimean-congo-haemorrhagic-fever.pdfSalamaAdel
 
EIS-Webinar-Info-Governance-Age-AI-2024-02-27-for-distr.pdf
EIS-Webinar-Info-Governance-Age-AI-2024-02-27-for-distr.pdfEIS-Webinar-Info-Governance-Age-AI-2024-02-27-for-distr.pdf
EIS-Webinar-Info-Governance-Age-AI-2024-02-27-for-distr.pdfEarley Information Science
 
Introduction to data science.pdf-Definition,types and application of Data Sci...
Introduction to data science.pdf-Definition,types and application of Data Sci...Introduction to data science.pdf-Definition,types and application of Data Sci...
Introduction to data science.pdf-Definition,types and application of Data Sci...DrSumathyV
 
HayleyDerby_Market_Research_Spotify.docx
HayleyDerby_Market_Research_Spotify.docxHayleyDerby_Market_Research_Spotify.docx
HayleyDerby_Market_Research_Spotify.docxHayleyDerby
 
Choose your perfect jacket.pdf
Choose your perfect jacket.pdfChoose your perfect jacket.pdf
Choose your perfect jacket.pdfAlexia Trejo
 
Basics of Creating Graphs / Charts using Microsoft Excel
Basics of Creating Graphs / Charts using Microsoft ExcelBasics of Creating Graphs / Charts using Microsoft Excel
Basics of Creating Graphs / Charts using Microsoft ExcelTope Osanyintuyi
 
Ratio analysis, Formulas, Advantage PPt.pptx
Ratio analysis, Formulas, Advantage PPt.pptxRatio analysis, Formulas, Advantage PPt.pptx
Ratio analysis, Formulas, Advantage PPt.pptxSugumarVenkai
 
Cousera Cap Course Datasets containing datasets from a Fictional Fitness Trac...
Cousera Cap Course Datasets containing datasets from a Fictional Fitness Trac...Cousera Cap Course Datasets containing datasets from a Fictional Fitness Trac...
Cousera Cap Course Datasets containing datasets from a Fictional Fitness Trac...Samuel Chukwuma
 
Pereira_EBM_Assign3.docx1234123412341234
Pereira_EBM_Assign3.docx1234123412341234Pereira_EBM_Assign3.docx1234123412341234
Pereira_EBM_Assign3.docx1234123412341234LR1709MUSIC
 
WOMEN IN TECH EVENT : Explore Salesforce Metadata.pptx
WOMEN IN TECH EVENT : Explore Salesforce Metadata.pptxWOMEN IN TECH EVENT : Explore Salesforce Metadata.pptx
WOMEN IN TECH EVENT : Explore Salesforce Metadata.pptxyosra Saidani
 
TainoMoreiraPereiraLuis_Assignment_4.docx
TainoMoreiraPereiraLuis_Assignment_4.docxTainoMoreiraPereiraLuis_Assignment_4.docx
TainoMoreiraPereiraLuis_Assignment_4.docxLR1709MUSIC
 
Artificial Intelligence for Vision: A walkthrough of recent breakthroughs
Artificial Intelligence for Vision:  A walkthrough of recent breakthroughsArtificial Intelligence for Vision:  A walkthrough of recent breakthroughs
Artificial Intelligence for Vision: A walkthrough of recent breakthroughsNikolas Markou
 
AWS_projects related AWS services such as feature store store and clarify
AWS_projects related AWS services such as feature store store and clarifyAWS_projects related AWS services such as feature store store and clarify
AWS_projects related AWS services such as feature store store and clarifyVarun Garg
 
Why Choose Canada for your next film project
Why Choose Canada for your next film projectWhy Choose Canada for your next film project
Why Choose Canada for your next film projectnubreed2019
 
Prometheus Grafana Dashboard for Cassandra 5
Prometheus Grafana Dashboard for Cassandra 5Prometheus Grafana Dashboard for Cassandra 5
Prometheus Grafana Dashboard for Cassandra 5Sarma Pydipally
 

Recently uploaded (20)

Growing through Artificial Intelligence (AI)
Growing  through Artificial Intelligence (AI)Growing  through Artificial Intelligence (AI)
Growing through Artificial Intelligence (AI)
 
Customer Satisfaction Data - Multiple Linear Regression Model.pdf
Customer Satisfaction Data -  Multiple Linear Regression Model.pdfCustomer Satisfaction Data -  Multiple Linear Regression Model.pdf
Customer Satisfaction Data - Multiple Linear Regression Model.pdf
 
presentation big data analytics on Apache spark
presentation big data analytics on Apache sparkpresentation big data analytics on Apache spark
presentation big data analytics on Apache spark
 
EXCEL-VLOOKUP-AND-HLOOKUP LECTURE NOTES ALL EXCEL VLOOKUP NOTES PDF
EXCEL-VLOOKUP-AND-HLOOKUP LECTURE NOTES ALL EXCEL VLOOKUP NOTES PDFEXCEL-VLOOKUP-AND-HLOOKUP LECTURE NOTES ALL EXCEL VLOOKUP NOTES PDF
EXCEL-VLOOKUP-AND-HLOOKUP LECTURE NOTES ALL EXCEL VLOOKUP NOTES PDF
 
Discover the Best Free Web Hosting Services with SSL in 2023
Discover the Best Free Web Hosting Services with SSL in 2023Discover the Best Free Web Hosting Services with SSL in 2023
Discover the Best Free Web Hosting Services with SSL in 2023
 
introduction-to-crimean-congo-haemorrhagic-fever.pdf
introduction-to-crimean-congo-haemorrhagic-fever.pdfintroduction-to-crimean-congo-haemorrhagic-fever.pdf
introduction-to-crimean-congo-haemorrhagic-fever.pdf
 
EIS-Webinar-Info-Governance-Age-AI-2024-02-27-for-distr.pdf
EIS-Webinar-Info-Governance-Age-AI-2024-02-27-for-distr.pdfEIS-Webinar-Info-Governance-Age-AI-2024-02-27-for-distr.pdf
EIS-Webinar-Info-Governance-Age-AI-2024-02-27-for-distr.pdf
 
Introduction to data science.pdf-Definition,types and application of Data Sci...
Introduction to data science.pdf-Definition,types and application of Data Sci...Introduction to data science.pdf-Definition,types and application of Data Sci...
Introduction to data science.pdf-Definition,types and application of Data Sci...
 
HayleyDerby_Market_Research_Spotify.docx
HayleyDerby_Market_Research_Spotify.docxHayleyDerby_Market_Research_Spotify.docx
HayleyDerby_Market_Research_Spotify.docx
 
Choose your perfect jacket.pdf
Choose your perfect jacket.pdfChoose your perfect jacket.pdf
Choose your perfect jacket.pdf
 
Basics of Creating Graphs / Charts using Microsoft Excel
Basics of Creating Graphs / Charts using Microsoft ExcelBasics of Creating Graphs / Charts using Microsoft Excel
Basics of Creating Graphs / Charts using Microsoft Excel
 
Ratio analysis, Formulas, Advantage PPt.pptx
Ratio analysis, Formulas, Advantage PPt.pptxRatio analysis, Formulas, Advantage PPt.pptx
Ratio analysis, Formulas, Advantage PPt.pptx
 
Cousera Cap Course Datasets containing datasets from a Fictional Fitness Trac...
Cousera Cap Course Datasets containing datasets from a Fictional Fitness Trac...Cousera Cap Course Datasets containing datasets from a Fictional Fitness Trac...
Cousera Cap Course Datasets containing datasets from a Fictional Fitness Trac...
 
Pereira_EBM_Assign3.docx1234123412341234
Pereira_EBM_Assign3.docx1234123412341234Pereira_EBM_Assign3.docx1234123412341234
Pereira_EBM_Assign3.docx1234123412341234
 
WOMEN IN TECH EVENT : Explore Salesforce Metadata.pptx
WOMEN IN TECH EVENT : Explore Salesforce Metadata.pptxWOMEN IN TECH EVENT : Explore Salesforce Metadata.pptx
WOMEN IN TECH EVENT : Explore Salesforce Metadata.pptx
 
TainoMoreiraPereiraLuis_Assignment_4.docx
TainoMoreiraPereiraLuis_Assignment_4.docxTainoMoreiraPereiraLuis_Assignment_4.docx
TainoMoreiraPereiraLuis_Assignment_4.docx
 
Artificial Intelligence for Vision: A walkthrough of recent breakthroughs
Artificial Intelligence for Vision:  A walkthrough of recent breakthroughsArtificial Intelligence for Vision:  A walkthrough of recent breakthroughs
Artificial Intelligence for Vision: A walkthrough of recent breakthroughs
 
AWS_projects related AWS services such as feature store store and clarify
AWS_projects related AWS services such as feature store store and clarifyAWS_projects related AWS services such as feature store store and clarify
AWS_projects related AWS services such as feature store store and clarify
 
Why Choose Canada for your next film project
Why Choose Canada for your next film projectWhy Choose Canada for your next film project
Why Choose Canada for your next film project
 
Prometheus Grafana Dashboard for Cassandra 5
Prometheus Grafana Dashboard for Cassandra 5Prometheus Grafana Dashboard for Cassandra 5
Prometheus Grafana Dashboard for Cassandra 5
 

Advanced pg_stat_statements: Filtering, Regression Testing & more

  • 1. Advanced pg_stat_statements: Filtering, Regression Testing & more @LukasFittl
  • 2. Skilled Developer Amateur Hacker @LukasFittl
  • 3. pganalyze.com 1.6 million unique queries tracked using pg_stat_statements
  • 4. Intro pg_stat_statements userid | 10 dbid | 1397527 query | SELECT * FROM x WHERE y = ? calls | 5 total_time | 15.249 rows | 0 shared_blks_hit | 451 shared_blks_read | 41 shared_blks_dirtied | 26 shared_blks_written | 0 local_blks_hit | 0 local_blks_read | 0 local_blks_dirtied | 0 local_blks_written | 0 temp_blks_read | 0 temp_blks_written | 0 blk_read_time | 0 blk_write_time | 0
  • 5. Intro query | SELECT * FROM x WHERE y = ? calls | 5 total_time | 15.249 Query + Avg Time + Timeframe
  • 7. Improving Data Quality pg_query Filtering & Regression Testing
  • 8. Improving Data Quality pg_query Filtering & Regression Testing
  • 9. SELECT "postgres_settings".* FROM "postgres_settings" WHERE "postgres_settings"."database_id" = $1 AND "postgres_settings"."invalidated_at_snapshot_id" IS NULL AND (id not in (70288,70289,70290,70291,70292,70293,70294,70295,70296,70297,70298 ,70299,70300,70301,70302,70303,70304,70305,70306,70307,70308,70309 ,70310,70311,70312,70313,70314,70315,70316,70317,70318,70319,70320 ,70321,70322,70323,70324,70325,70326,70327,99059,99060,70330,70331 ,70332,70333,70334,70335,70336,70337,70338,99061,70340,70341,70342 ,70343,70344,70345,70346,70347,70348,70349,70350,70351,70352,70353 ,70354,70355,70356,70357,70358,70359,70360,99062,70362,70363,70364 ,70365,70366,70367,70368,70369,70370,70371,70372,70373,70374,70375 ,70376,70377,70378,70379,70380,70381,70382,70383,70384,70385,70386 ,99063,99064,99065,99066,99067,70392,70393,70394,70395,70396,70397 ,70398,70399,70400,70401,70402,70403,70404,70405,99068,70407,70408 ,70409,70410,70411,70412,70413,70414,70415,70416,70417,99069,70419 ,70420,70421,99070,70423,70424,70425,70426,70427,70428, Truncation Improving Data Quality
  • 10. Improving Data Quality -[ RECORD 1 ]———+-------------------------------- query | SELECT * FROM x WHERE y = ? calls | 5 total_time | 15.249 -[ RECORD 2 ]———+-------------------------------- query | SELECT * FROM z WHERE a = 123 calls | 50 total_time | 104.19 Race Condition during pg_stat_statements_reset()
  • 11. Lesson Learned: Avoid frequent Improving Data Quality pg_stat_statements_reset()
  • 12. Fingerprinting SELECT a AS b == SELECT a AS c Problematic: y IN (?, ?, ?) != y IN (?, ?) Improving Data Quality SELECT a, b FROM x != SELECT b, a FROM x DEALLOCATE p141 != DEALLOCATE p150
  • 13. Limited Statistical Information ! Histogram / MAX(runtime) would be super-useful Improving Data Quality
  • 14. pg_stat_plans Improving Data Quality pg_stat_statements variant that differentiates between query plans. Slower + Don’t use it before this bug is fixed: https://github.com/2ndQuadrant/pg_stat_plans/issues/39
  • 15. Improving Data Quality Filtering & Regression Testing pg_query
  • 16. Storing & Cleaning pg_stat_statements data pg_query
  • 17. pg_query Monitoring Setup Snapshot {“schema”: {“n_live_tup”: 75, "relpages": 1, "reltuples": 75.0,…}, “queries”: [{..}, {..}]} Production Database Collector Normalize {“schema”: {“n_live_tup”: 75, "relpages": 1, "reltuples": 75.0,…}, “queries”: [{..}, {..}]} Monitoring Database Parse Fingerprint Extract Tables
  • 18. pg_query queries id | 7053479 database_id | 1 received_query | SELECT * FROM x WHERE y = ? normalized_query | SELECT * FROM x WHERE y = ? created_at | 2014-06-27 16:20:08.334705 updated_at | 2014-06-27 16:20:08.334705 parse_tree | [{"SELECT":{...}] parse_error | parse_warnings | statement_types | {SELECT} truncated | f fingerprint | 00704f1fd8442b7c17821cb8a61856c3d61b330e
  • 19. pg_query query_snapshots id | 170661585 query_id | 7053479 calls | 29 total_time | 94.38 rows | 29 snapshot_id | 3386118 snapshots id | 3386118 database_id | 408 collected_at | 2014-09-09 20:10:01 submitter | pganalyze-collector 0.6.1 query_source | pg_stat_statements
  • 20. pg_query Normalize Parse Fingerprint Extract Tables
  • 21. pg_query Normalize Parse Parsing an SQL Query Fingerprint Extract Tables
  • 22. EXPLAIN (PARSETREE TRUE) pg_query SELECT * FROM x WHERE y = 1 ({SELECT :distinctClause <> :intoClause <> :targetList ( {RESTARGET :name <> :indirection <> :val {COLUMNREF :fields ({A_STAR}) :location 7} :location 7}) :fromClause ( {RANGEVAR :schemaname <> :relname x :inhOpt 2 :relpersistence p :alias <> :location 14}) :whereClause {AEXPR :name (“=") :lexpr {COLUMNREF :fields ("y") :location 22} :rexpr {PARAMREF :number 0 :location 26} :location 24} Unfortunately doesn’t exist.
  • 23. pg_query Parse Statement raw_parse(..) pg_catalog Rewrite Query Query Planner Execute
  • 24. tree = raw_parser(query_str); pg_query str = nodeToString(tree); printf(str); ({SELECT :distinctClause <> :intoClause <> :targetList ( {RESTARGET :name <> :indirection <> :val {COLUMNREF :fields ({A_STAR}) :location 7} :location 7}) :fromClause ( {RANGEVAR :schemaname <> :relname x :inhOpt 2 :relpersistence p :alias <> :location 14}) :whereClause {AEXPR :name (“=") :lexpr {COLUMNREF :fields ("y") :location 22} :rexpr {PARAMREF :number 0 :location 26} :location 24}
  • 25. pg_query Parse Statement raw_parse(..) pg_catalog Rewrite Query Query Planner Execute
  • 26. github.com/pganalyze/pg_query pg_query Extension Compiles a full copy of PostgreSQL when you do “gem install pg_query”
  • 27. pg_query PgQuery._raw_parse( “SELECT * FROM x WHERE y = 1”) ({SELECT :distinctClause <> :intoClause <> :targetList ( {RESTARGET :name <> :indirection <> :val {COLUMNREF :fields ({A_STAR}) :location 7} :location 7}) :fromClause ( {RANGEVAR :schemaname <> :relname x :inhOpt 2 :relpersistence p :alias <> :location 14}) :whereClause {AEXPR :name (“=") :lexpr {COLUMNREF :fields ("y") :location 22} :rexpr {PARAMREF :number 0 :location 26} :location 24} :groupClause <> :havingClause <> :windowClause <> :valuesLists <> :sortClause <> :limitOffset <> :limitCount <> :lockingClause <> :withClause <>
  • 28. pg_query nodeToString is incomplete :( PgQuery._raw_parse(“CREATE SCHEMA foo”) WARNING: 01000: could not dump unrecognized node type: 754
  • 29. src/backend/nodes/outfuncs.c pg_query Patch: Generate automatically, JSON output
  • 30. PgQuery._raw_parse( pg_query “SELECT * FROM x WHERE y = 1”) [{"SELECT": { "targetList": [{ "RESTARGET": { "val": { "COLUMNREF": { "fields": [{"A_STAR": {}}], "location": 7 } }, "location": 7 } } ], "fromClause": [ { "RANGEVAR": { "relname": "x", "inhOpt": 2, "relpersistence": "p", "location": 14 } } ], "whereClause": { "AEXPR": { "name": [ "=" ], "lexpr": {
  • 31. pg_query Parsing a normalized Normalize Parse SQL query Fingerprint Extract Tables
  • 32. EXPLAIN SELECT * FROM x WHERE y = 1 QUERY PLAN --------------------------------------------------------------------- Index Scan using idx_for_y on x (cost=0.15..8.17 rows=1 width=140) Index Cond: (id = 1) Parse Analyze Plan pg_query
  • 33. EXPLAIN SELECT * FROM x WHERE y = ? ERROR: syntax error at or near ";" LINE 1: EXPLAIN SELECT * FROM x WHERE y = ?; Parse Analyze Plan pg_query
  • 34. EXPLAIN SELECT * FROM x WHERE y = ? EXPLAIN SELECT * FROM x WHERE y = $1 ERROR: there is no parameter $1 LINE 1: EXPLAIN SELECT * FROM x WHERE y = $1; Parse Analyze Plan pg_query
  • 35. pg_query Parser Patch to support parsing “?”
  • 36. pg_query Downside: Breaks ? operator in some cases Real fix: Don’t use ? as a replacement character.
  • 37. pg_query Fingerprinting Normalize Parse Fingerprint Extract Tables
  • 38. pg_query > require ‘pg_query’ ! > q1 = PgQuery.parse(‘SELECT a, b FROM x’) > q1.fingerprint [“c72f1bc9feda72c0b4ba030eea90b4fed3ac8e86”] ! > q2 = PgQuery.parse(‘SELECT b, a FROM x’) > q2.fingerprint [“c72f1bc9feda72c0b4ba030eea90b4fed3ac8e86”]
  • 39. pg_query 40 lines of unit-tested Ruby code
  • 40. pg_query Extracting Table References Normalize Parse Fingerprint Extract Tables
  • 41. pg_query > require ‘pg_query’ > q = PgQuery.parse(‘SELECT * FROM x’) > q.tables [“x”]
  • 42. pg_query ~90 lines of unit-tested Ruby code
  • 44. Improving Data Quality pg_query Filtering & Regression Testing
  • 45. Filtering Filtering & Regression Testing
  • 46. monitor.rb Filtering & Regression Testing Simple top-like tool that shows pg_stat_statements data https://gist.github.com/lfittl/301542602607b738b23f
  • 47. Filtering & Regression Testing monitor.rb -d testdb AVG | QUERY -------------------------------------------------------------------------------- 10.7ms | SELECT oid, typname, typelem, typdelim, typinput FROM pg_type 3.0ms | SET time zone 'UTC' 0.4ms | SELECT a.attname, format_type(a.atttypid, a.atttypmod), pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod FROM pg_attribute a LEFT JOIN pg_attrdef d ON a.attrelid = d.adrelid AND a.attnum = d.adnum WHERE a.attrelid = ?::regclass AND a.attnum > ? AND NOT a.attisdropped ORDER BY a.attnum 0.2ms | SELECT pg_stat_statements_reset() 0.1ms | SELECT query, calls, total_time FROM pg_stat_statements 0.1ms | SELECT attr.attname FROM pg_attribute attr INNER JOIN pg_constraint cons ON attr.attrelid = cons.conrelid AND attr.attnum = cons.conkey[?] WHERE cons.contype = ? AND cons.conrelid = ?: :regclass 0.0ms | SELECT COUNT(*) FROM pg_class c LEFT JOIN pg_namespace n ON n.oid = c.relnamespace WHERE c.relkind in (?,?) AND c.relname = ? AND n.nspname = ANY (current_schemas(?)) 0.0ms | SELECT * FROM posts JOIN users ON (posts.author_id = users.id) WHERE users.login = ?; 0.0ms | SET client_min_messages TO 'panic' 0.0ms | set client_encoding to 'UTF8' 0.0ms | SHOW client_min_messages 0.0ms | SELECT * FROM ad_reels WHERE id = ?; 0.0ms | SELECT * FROM posts WHERE guid = ?; 0.0ms | SELECT ? 0.0ms | SET client_min_messages TO 'warning' 0.0ms | SET standard_conforming_strings = on 0.0ms | SELECT "posts".* FROM "posts" ORDER BY "posts"."id" DESC LIMIT ? 0.0ms | SHOW TIME ZONE
  • 48. Filtering & Regression Testing monitor.rb -d testdb -t posts AVG | QUERY -------------------------------------------------------------------------------- 0.0ms | SELECT * FROM posts JOIN users ON (posts.author_id = users.id) WHERE users.login = ?; 0.0ms | SELECT * FROM posts WHERE guid = ?; 0.0ms | SELECT "posts".* FROM "posts" ORDER BY "posts"."id" DESC LIMIT ?
  • 49. Filtering & Regression Testing if cli.config[:table] q = PgQuery.parse(query["query"]) next unless q.tables.include?(cli.config[:table]) end
  • 50. Regression Testing Filtering & Regression Testing
  • 51. Which query plans are affected by removal of an index? ! How would execution plans be affected by an upgrade to 9.X? Filtering & Regression Testing
  • 52. Regression Test based on pg_stat_statements + table statistics. ! (no actual data) Filtering & Regression Testing
  • 53. Schema Dump + Table Level Statistics "n_live_tup": 75, "relpages": 1, "reltuples": 75.0, “stanumbers1": [..], "stavalues1": “{..}”, … Local Test Database Testing Setup Production Database EXPLAIN SELECT FROM x WHERE y = ? Filtering & Regression Testing
  • 54. EXPLAIN SELECT * FROM x WHERE y = ? EXPLAIN SELECT * FROM x WHERE y = $1 ERROR: there is no parameter $1 LINE 1: EXPLAIN SELECT * FROM x WHERE y = $1; Parse Analyze Plan Filtering & Regression Testing
  • 55. y = $1 ERROR: there is no parameter $0 LINE 1: EXPLAIN SELECT * FROM x WHERE y = $0; Filtering & Regression Testing
  • 56. y = $1 ERROR: there is no parameter $0 LINE 1: EXPLAIN SELECT * FROM x WHERE y = $0; y = NULL QUERY PLAN ---------------------------------------------------------------- Result (cost=0.00..21.60 rows=1 width=40) One-Time Filter: NULL::boolean -> Seq Scan on x (cost=0.00..21.60 rows=1 width=40) Filtering & Regression Testing
  • 57. y = $1 ERROR: there is no parameter $0 LINE 1: EXPLAIN SELECT * FROM x WHERE y = $0; y = NULL QUERY PLAN ---------------------------------------------------------------- Result (cost=0.00..21.60 rows=1 width=40) One-Time Filter: NULL::boolean -> Seq Scan on x (cost=0.00..21.60 rows=1 width=40) y = (SELECT null) ERROR: failed to find conversion function from unknown to integer Filtering & Regression Testing
  • 58. y = $1 ERROR: there is no parameter $0 LINE 1: EXPLAIN SELECT * FROM x WHERE y = $0; y = NULL QUERY PLAN ---------------------------------------------------------------- Result (cost=0.00..21.60 rows=1 width=40) One-Time Filter: NULL::boolean -> Seq Scan on x (cost=0.00..21.60 rows=1 width=40) y = (SELECT null) ERROR: failed to find conversion function from unknown to integer y = (SELECT null::integer) QUERY PLAN ---------------------------------------------------------------------- Index Scan using idx_for_y on x (cost=0.16..8.18 rows=1 width=144) Index Cond: (y = $0) InitPlan 1 (returns $0) -> Result (cost=0.00..0.01 rows=1 width=0) Filtering & Regression Testing
  • 59. Finding out the type y = $1 ERROR: there is no parameter $1 LINE 1: EXPLAIN SELECT * FROM x WHERE y = $1; pg_prepared_statements PREPARE tmp AS SELECT * FROM x WHERE y = $1; SELECT unnest(parameter_types) AS data_type FROM pg_prepared_statements WHERE name = ‘tmp’; DEALLOCATE tmp; data_type ----------- integer Filtering & Regression Testing
  • 60. EXPLAIN SELECT * FROM x WHERE y = ? EXPLAIN SELECT * FROM x WHERE y = $0 EXPLAIN SELECT * FROM x WHERE y = ((SELECT null::integer)::integer) QUERY PLAN --------------------------------------------------------------------- Index Scan using idx_for_y on x (cost=0.16..8.18 rows=1 width=144) Index Cond: (y = $0) InitPlan 1 (returns $0) -> Result (cost=0.00..0.01 rows=1 width=0) Parse Analyze Plan Filtering & Regression Testing
  • 61. Open Issue: Planner reads actual physical size whilst planning Filtering & Regression Testing
  • 63. Improving Data Quality pg_query Filtering & Regression Testing
  • 64. 9.5 proposal for pg_s_s: Closing ! Instead of ? use $0 as replacement character - making the output parseable again.
  • 65. 9.5 proposal for outfuncs.c: ! Generate automatically from struct definitions, cutting 3000 hand-written lines down to 1000. ! Add JSON output support. Closing
  • 66. 9.X proposal: ! Consider adding a way to get a parsetree more easily. ! Via SQL / shared library / helper tool. Closing
  • 67. Tools & libraries available at: Closing github.com/pganalyze
  • 68. @LukasFittl Thank you! github.com/pganalyze pganalyze.com
  • 71. Classifying queries Improving Data Quality Frequent/OLTP vs analytical query