P90 X Your Database!!

Your Database!!
Denish Patel
Database Architect
OmniTi Computer Consulting Inc.
denish@omniti.com

1
Friday, March 25, 2011

Who am I ?
• Denish Patel
• Database Architect
• 4+ Years with OmniTi
• Expertise
• Heterogeneous Databases
• PostgreSQL, Oracle, MySQL
• Very Large databases
• Performance tuning
• Application level
• SQL tuning
• Database and OS parameters
• Objects tuning 2

Who am I ?
• Denish Patel
• Database Architect
• 4+ Years with OmniTi
• Expertise
• Heterogeneous Databases
• PostgreSQL, Oracle, MySQL
• Very Large databases
• Performance tuning
• Application level OmniTi is hiring!
• SQL tuning
• Database and OS parameters
• Objects tuning 2

Agenda
•Why bloat ?
•Issues
•Identify bloat
•Tables
•Indexes
•Bloat removal toolkit
•Monitoring and trending
3

why bloat?

•MVCC
•ACID
•Updates & Deletes

4

MVCC method-1

5

MVCC method-1
!

34..-+56''
' !:723,'
7$89$"&'
!"#$%&'
,+-.$'
/+&+' (0/1,2'

()*+&$ ><??!,'
'
/2;2,2'

72;2>,''
,31:7'
;<='
5

MVCC method-2

6

MVCC method-2
!
& 3./,-!&
!"#$%&
/,0,4!&& '"("&

)*'+!,&

!-+./& ',0,!,&
012&

41553!&

6

Issues
•Unnecessary costly I/O operations
•Poor database tables access time
•Increase maintenance time
•Waste of resources
•RAM
•CPU
•Disk
7

Prevent Bloat
• autovacuum =on
• vacuum manually between batch updates
• Truncate table
• Free Space Map
• <= PostgreSQL 8.3
• max_fsm_pages
• max_fsm_tables
• >=PostgreSQL 8.4
• auto sizing free space map
8

BEWARE!
• Bug: long running auto-vacuum workers
• PostgreSQL Version Affected
• 8.3.10
• 8.3.11
• 8.3.12
• 8.4.4
• 8.4.5
9

BEWARE!
• Bug: long running auto-vacuum workers
• PostgreSQL Version Affected
• 8.3.10
• 8.3.11
• 8.3.12
• 8.4.4
• 8.4.5
Upgrade to latest release !!
9

Measure Bloat

•Bloat Report
•pg_bloat_report (OmniTI Labs)
•Nagios Monitoring
•Check_postgres_bloat

10

pg_bloat_report
Bloat Report Tables

1. omniti.hits 1969348 of 4255960 pages wasted (46.3%),
15 GB of 32 GB.
2. omniti.clients 279979 of 370509 pages wasted (75.6%),
2187 MB of 2895 MB.

Bloat Report Indexes

1. omniti.hits_pk 51275 of 71075 pages wasted (72.1%),
401 MB of 555 MB.
2. omniti.clients_tz 47550 of 71271 pages wasted (66.7%),
371 MB 11 557 MB.
of

check_postgres_bloat

Notification Type: PROBLEM
Host: omniti
State: CRITICAL
Date/Time: Wed Feb 16 11:24:56 UTC 2011
Additional Info: POSTGRES_BLOAT
CRITICAL: DB "omniti" (host:omniti)
(port=5432) table omniti.hits rows:23995164
pages:2858893 shouldbe:2278176 (1.3X)
wasted size:4757233664 (4 GB)

12

bloat removal
toolkit
• Offline - with locking
• Vacuum
• Vacuum Full
• Cluster
• Cluster vs Vacuum Full
• Online - with minimal locking
• compact_table
• pg_reorg
13

vacuum
• Auto vacuum
• VACUUM [verbose|analyze] table
• Reclaims storage occupied by dead
tuples

• Reclaimed space available to re-use
• Normally free space doesn’t return to
OS

• No exclusive table lock
15

vacuum full
• VACUUM FULL table
• Compact table based on dead rows
• Starting PostgreSQL 9.0 , it’s rewrite
entire table and indexes (like CLUSTER)

• Reclaimed space available to re-use
• Space returned to OS
• Table Exclusive lock
• Expensive operation
16

cluster
•CLUSTER [verbose] table_name
[USING index_name]

•Create reorganized copy of table
based on index

•Rebuild indexes too
•Requires Exclusive Lock on table
•Expensive Operation
18

cluster vs vacuum full

It’s just rewrite of table without
Order based on Index
any speciﬁc order

Table needs index Doesn’t need index

Bloat indexes (Pre PostgreSQL
Doesn’t bloat index
9.0)
Overall Operation Slower (Pre
Overall operation Faster
PostgreSQL 9.0)

Recommended Not recommended
19

compact_table
•Developed by OmniTi
•Reorganize rows to empty pages at the
end based on ctid

•ctid - (page number, tuple number)
•When we update a row, the row's ctid
changes, because the update creates a
new version of the row and leaves the
old version behind
21

compact_table
postgres=# select ctid,* from bar;
ctid | a | b
-------+---+---
(0,1) | 1 | 1
(0,2) | 2 | 1
(2 rows)

postgres=# update bar set a=3 where a=2;
UPDATE 1
postgres=# select ctid,* from bar;
ctid | a | b
-------+---+---
(0,1) | 1 | 1
(0,3) | 3 | 1
(2 rows)

postgres=# vacuum verbose bar;
.
.
INFO: "bar": found 1 removable, 2 nonremovable row versions in 1 out of 1 pages
DETAIL: 0 dead row versions cannot be removed yet.
VACUUM
22

compact_table

23

compact_table
postgres=# truncate table bar;
TRUNCATE TABLE
postgres=# insert into bar select generate_series(1,1000);
INSERT 0 1000
postgres=# delete from bar where a %2 = 0;
DELETE 500
postgres=# select max(ctid) from bar;
max
--------
(4,95)
(1 row)
INFO: "bar": found 0 removable, 500
nonremovable row versions in 5 out of 5 pages
VACUUM
23

compact_table
postgres=# begin;
BEGIN
postgres=# update bar set a=a where ctid>='(3,0)';
UPDATE 161
UPDATE 161
UPDATE 48
UPDATE 48
UPDATE 34
UPDATE 0
postgres=# commit;
COMMIT
.
.
INFO: "bar": truncated 5 to 3 pages
.
VACUUM
24

compact_table
•Pretty safe update process
•No additional changes required
•Slow process
•Generate Index bloat
•Under development
•Not tested in production
•Future consideration?
25

pg_reorg
• Developed by NTT OSS Center
• Reorganize table with very short lock
• Needs Primary Key
• Not Null Unique Key works!
• CLUSTER online
• ORDER BY Key
• VACUUM FULL Online
• Rewrite without ORDER BY
• Build and Install like other Contrib Module
27

!
3,4-$*(
8,$%"&-/(
9"#:(($-./*(
<%$=(
!"#$%&'(
;-#'*$( 8,*#-$%"&-/(
9"#:( )"'(
;-./*(
;-./*(

(
6*,/-7(
;*+,(
)"'((
;-./*(
)"'($"($*+,(
$-./*0( !<-,,%&'(
( 1&2*#$(
( 3,4-$*(
<%$=(>*#7(
( 5*/*$*( 2="#$()"?:(
28

!
pg_reorg
3,4-$*(
8,$%"&-/(
9"#:(($-./*(
<%$=(
!"#$%&'(
;-#'*$( 8,*#-$%"&-/(
9"#:( )"'(
;-./*(
;-./*(

(
6*,/-7(
;*+,(
)"'((
;-./*(
)"'($"($*+,(
$-./*0( !<-,,%&'(
( 1&2*#$(
( 3,4-$*(
<%$=(>*#7(
( 5*/*$*( 2="#$()"?:(
28

pg_reorg Options
pg_reorg --no-order -table test -d omniti
-n
-- no-order
Do online VACUUM FULL.

-o columns [,...]
--order-by=columns [,...]
Do online CLUSTER ordered by specified columns.

-t table
--table=table

-Z
--no-analyze

-T seconds
--wait-timeout=seconds
29

pg_reorg in action
$ /opt/pgsql/bin/pg_reorg -e -t public.test --order-by=test_id -U postgres -d test

LOG: (query) SET statement_timeout = 0
LOG: (query) SET search_path = pg_catalog, pg_temp, public
LOG: (query) SET client_min_messages = warning
LOG: (query) SELECT * FROM reorg.tables WHERE relid = $1::regclass
LOG: (param:0) = public.test
LOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED

LOG: (query) SET LOCAL statement_timeout = 100
LOG: (query) LOCK TABLE test IN ACCESS EXCLUSIVE MODE
LOG: (query) RESET statement_timeout
LOG: (query) SELECT reorg.conflicted_triggers($1)
LOG: (param:0) = 7441636
LOG: (query) CREATE TYPE reorg.pk_7441636 AS (test_id integer)
LOG: (query) CREATE TABLE reorg.log_7441636 (id bigserial PRIMARY KEY, pk
reorg.pk_7441636, row pulbic.test)
LOG: (query) CREATE TRIGGER z_reorg_trigger BEFORE INSERT OR DELETE OR
UPDATE ON pulbic.test FOR EACH ROW EXECUTE PROCEDURE
reorg.reorg_trigger('INSERT INTO reorg.log_7441636(pk, row) VALUES( CASE
WHEN $1 IS NULL THEN NULL ELSE (ROW($1.test_id)::reorg.pk_7441636) END,
$2)')
LOG: (query) SELECT reorg.disable_autovacuum('reorg.log_7441636')
LOG: (query) COMMIT
30

pg_reorg in action

LOG: (query) SET search_path = pg_catalog, pg_temp, public SET
variables

LOG: (param:0) = 7441636
$2)')
LOG: (query) COMMIT
30

pg_reorg in action

LOG: (query) SET search_path = pg_catalog, pg_temp, public SET
variables

Place Triggers
LOG: (param:0) = 7441636
$2)')
LOG: (query) COMMIT
30

pg_reorg in action
LOG: (query) BEGIN ISOLATION LEVEL SERIALIZABLE
LOG: (query) SELECT set_config('work_mem', current_setting('maintenance_work_mem'),
true)
LOG: (query) SELECT reorg.array_accum(virtualtransaction) FROM pg_locks WHERE
locktype = 'virtualxid' AND pid <> pg_backend_pid()
LOG: (query) DELETE FROM reorg.log_7441636
LOG: (query) CREATE TABLE reorg.table_7441636 WITH (oids=false) TABLESPACE
pg_default AS SELECT * FROM ONLY public.test ORDER BY test_id
LOG: (query) SELECT reorg.disable_autovacuum('reorg.table_7441636')
LOG: (query) COMMIT
LOG: (query) SELECT indexrelid, reorg.reorg_indexdef(indexrelid, indrelid) FROM
pg_index WHERE indrelid = $1
LOG: (param:0) = 7441636
LOG: (query) CREATE UNIQUE INDEX index_201337719 ON reorg.table_7441636 USING
btree (test_id)

LOG: (query) SELECT reorg.reorg_apply($1, $2, $3, $4, $5, $6)
LOG: (param:0) = SELECT * FROM reorg.log_7441636 ORDER BY id LIMIT $1
LOG: (param:1) = INSERT INTO reorg.table_7441636 VALUES ($1.*)
LOG: (param:2) = DELETE FROM reorg.table_7441636 WHERE (test_id) = ($1.test_id)
LOG: (param:3) = UPDATE reorg.table_7441636 SET (test_id, name, lat, lon, city, country,
state_abr) = ($2.test_id, $2.name, $2.lat, $2.lon, $2.city, $2.country, $2.state_abr) WHERE
(test_id) = ($1.test_id)
LOG: (param:4) = DELETE FROM reorg.log_7441636 WHERE id <= $1
LOG: (param:5) = 1000 31

pg_reorg in action
true)
locktype = 'virtualxid' AND pid <> pg_backend_pid() CTAS
LOG: (query) DELETE FROM reorg.log_7441636 Create
pg_default AS SELECT * FROM ONLY public.test ORDER BY test_id Indexes
LOG: (query) COMMIT
LOG: (param:0) = 7441636
btree (test_id)

LOG: (param:5) = 1000 31

pg_reorg in action
true)
locktype = 'virtualxid' AND pid <> pg_backend_pid() CTAS
LOG: (query) DELETE FROM reorg.log_7441636 Create
pg_default AS SELECT * FROM ONLY public.test ORDER BY test_id Indexes
LOG: (query) COMMIT
LOG: (param:0) = 7441636
btree (test_id)
Apply
LOG: (param:0) = SELECT * FROM reorg.log_7441636 ORDER BY id LIMIT $1 Operational
LOG: (param:1) = INSERT INTO reorg.table_7441636 VALUES ($1.*) logs
LOG: (param:5) = 1000 31

pg_reorg in action
LOG: (query) LOCK TABLE public.test IN ACCESS EXCLUSIVE MODE
LOG: (param:3) = UPDATE reorg.table_7441636 SET (test_id, name, lat, lon, city,
country, state_abr) = ($2.test_id, $2.name, $2.lat, $2.lon, $2.city, $2.country, $2.state_abr)
WHERE (test_id) = ($1.test_id)
LOG: (param:5) = 0

LOG: (query) SELECT reorg.reorg_swap($1)
LOG: (param:0) = 7441636
LOG: (query) COMMIT

LOG: (query) SELECT reorg.reorg_drop($1)
LOG: (param:0) = 7441636
LOG: (query) COMMIT
LOG: (query) ANALYZE publoc.test
32
LOG: (query) COMMIT

pg_reorg in action
LOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED Lock
LOG: (query) SET LOCAL statement_timeout = 100 table and
LOG: (query) LOCK TABLE public.test IN ACCESS EXCLUSIVE MODE apply
LOG: (query) SELECT reorg.reorg_apply($1, $2, $3, $4, $5, $6) operational logs
LOG: (param:0) = SELECT * FROM reorg.log_7441636 ORDER BY id LIMIT $1 last time
LOG: (param:5) = 0

LOG: (query) SELECT reorg.reorg_swap($1)
LOG: (param:0) = 7441636
LOG: (query) COMMIT

LOG: (param:0) = 7441636
LOG: (query) COMMIT
32
LOG: (query) COMMIT

pg_reorg in action
LOG: (param:5) = 0

LOG: (query) SELECT reorg.reorg_swap($1) Swapping
LOG: (param:0) = 7441636
LOG: (query) COMMIT

LOG: (param:0) = 7441636
LOG: (query) COMMIT
32
LOG: (query) COMMIT

pg_reorg in action
LOG: (param:5) = 0

LOG: (query) SELECT reorg.reorg_swap($1) Swapping
LOG: (param:0) = 7441636
LOG: (query) COMMIT

LOG: (query) SELECT reorg.reorg_drop($1) Drop work and
LOG: (param:0) = 7441636 temp tables
LOG: (query) COMMIT
Analyze new table
32
LOG: (query) COMMIT

pg_reorg in
Production
•Database Size 540GB
•OLTP database 2-3K transactions/sec
•Largest pre-rebuild table Size was
127GB

•Took 5.75 hours to rebuild
•Reclaimed 52GB
•No outage reported for website!
33

pg_reorg in
Production
Table Size Before Pg_Reorg
Bloat Removed
Reorg Execution time

35GB 24m 10sec 13GB

27GB 116m 8 Sec 6GB

31GB 64m 32sec 4GB

44GB 128m 44sec 3GB

34

Comparison

CLUSTER VACUUM FULL PG_REORG

Execution Time: Execution Time: Execution Time:
2 min 1 min 50 Sec 1 min 4 Sec
Table Size: 594MB Table Size: 594MB Table Size: 594MB
Index Size: 128 MB Index Size: 128 MB Index Size: 128 MB

ofﬂine ofﬂine online

Built - in Built-in External Tool

38

Other Tips

•Remove unused indexes
•Reindex indexes as needed
•Partitioned table
•Detach child table -> Cluster child
table -> Attach

•Avoid vacuum full (Pre PostgrSQL 9.0)
39

Monitoring

•Circonus
•Nagios monitoring
•Bloat report

40

trending

•Cacti
•Circonus

41

trending

42

thanks!

•OmniTi
•Robert Treat
•Depesz
•Joshua (End Point)
•PGEast Conference Committee

43

References
•http://www.postgresql.org/docs/9.0/
•http://reorg.projects.postgresql.org/
pg_reorg.html

•https://labs.omniti.com/pgtreats/trunk/
tools/compact_table

•http://blog.endpoint.com/2010/09/
reducing-bloat-without-locking.html

44

open discussion

•When PostgreSQL will have built in
features to rebuild table and index
online ?

•Is it worth to change MVCC handling
to avoid bloating?

45

Questions?

•Email : denish@omniti.com
•Twitter : http://twitter.com/#!/denishpatel

46

P90 X Your Database!!

More Related Content

Similar to P90 X Your Database!!

More from Denish Patel

P90 X Your Database!!