P90 X Your Database!!
Upcoming SlideShare
Loading in...5
×
 

P90 X Your Database!!

on

  • 2,879 views

 

Statistics

Views

Total Views
2,879
Views on SlideShare
1,686
Embed Views
1,193

Actions

Likes
2
Downloads
34
Comments
1

12 Embeds 1,193

http://denishjpatel.blogspot.com 1144
http://denishjpatel.blogspot.in 26
http://www.mybestcv2.co.il 6
http://denishjpatel.blogspot.de 3
https://www.linkedin.com 3
http://feeds.feedburner.com 2
https://planet.omniti.com 2
http://webcache.googleusercontent.com 2
http://denishjpatel.blogspot.com.au 2
http://denishjpatel.blogspot.com.es 1
http://static.slidesharecdn.com 1
http://denishjpatel.blogspot.gr 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

P90 X Your Database!! P90 X Your Database!! Presentation Transcript

  • 1Friday, March 25, 2011
  • Your Database!! Denish Patel Database Architect OmniTi Computer Consulting Inc. denish@omniti.com 1Friday, March 25, 2011
  • 2Friday, March 25, 2011
  • Who am I ? • Denish Patel • Database Architect • 4+ Years with OmniTi • Expertise • Heterogeneous Databases • PostgreSQL, Oracle, MySQL • Very Large databases • Performance tuning • Application level • SQL tuning • Database and OS parameters • Objects tuning 2Friday, March 25, 2011
  • Who am I ? • Denish Patel • Database Architect • 4+ Years with OmniTi • Expertise • Heterogeneous Databases • PostgreSQL, Oracle, MySQL • Very Large databases • Performance tuning • Application level OmniTi is hiring! • SQL tuning • Database and OS parameters • Objects tuning 2Friday, March 25, 2011
  • 3Friday, March 25, 2011
  • Agenda •Why bloat ? •Issues •Identify bloat •Tables •Indexes •Bloat removal toolkit •Monitoring and trending 3Friday, March 25, 2011
  • 4Friday, March 25, 2011
  • why bloat? •MVCC •ACID •Updates & Deletes 4Friday, March 25, 2011
  • 5Friday, March 25, 2011
  • MVCC method-1 5Friday, March 25, 2011
  • MVCC method-1 5Friday, March 25, 2011
  • MVCC method-1 ! 34..-+56 !:723, 7$89$"& !"#$%& ,+-.$ /+&+ (0/1,2 ()*+&$ ><??!, /2;2,2 72;2>, ,31:7 ;<= 5Friday, March 25, 2011
  • 6Friday, March 25, 2011
  • MVCC method-2 6Friday, March 25, 2011
  • MVCC method-2 6Friday, March 25, 2011
  • MVCC method-2 ! & 3./,-!& !"#$%& /,0,4!&& "("& )*+!,& !-+./& ,0,!,& 012& 41553!& 6Friday, March 25, 2011
  • 7Friday, March 25, 2011
  • Issues •Unnecessary costly I/O operations •Poor database tables access time •Increase maintenance time •Waste of resources •RAM •CPU •Disk 7Friday, March 25, 2011
  • 8Friday, March 25, 2011
  • Prevent Bloat • autovacuum =on • vacuum manually between batch updates • Truncate table • Free Space Map • <= PostgreSQL 8.3 • max_fsm_pages • max_fsm_tables • >=PostgreSQL 8.4 • auto sizing free space map 8Friday, March 25, 2011
  • 9Friday, March 25, 2011
  • BEWARE! • Bug: long running auto-vacuum workers • PostgreSQL Version Affected • 8.3.10 • 8.3.11 • 8.3.12 • 8.4.4 • 8.4.5 9Friday, March 25, 2011
  • BEWARE! • Bug: long running auto-vacuum workers • PostgreSQL Version Affected • 8.3.10 • 8.3.11 • 8.3.12 • 8.4.4 • 8.4.5 Upgrade to latest release !! 9Friday, March 25, 2011
  • 10Friday, March 25, 2011
  • Measure Bloat •Bloat Report •pg_bloat_report (OmniTI Labs) •Nagios Monitoring •Check_postgres_bloat 10Friday, March 25, 2011
  • 11Friday, March 25, 2011
  • pg_bloat_report Bloat Report Tables 1. omniti.hits 1969348 of 4255960 pages wasted (46.3%), 15 GB of 32 GB. 2. omniti.clients 279979 of 370509 pages wasted (75.6%), 2187 MB of 2895 MB. Bloat Report Indexes 1. omniti.hits_pk 51275 of 71075 pages wasted (72.1%), 401 MB of 555 MB. 2. omniti.clients_tz 47550 of 71271 pages wasted (66.7%), 371 MB 11 557 MB. ofFriday, March 25, 2011
  • 12Friday, March 25, 2011
  • check_postgres_bloat Notification Type: PROBLEM Host: omniti State: CRITICAL Date/Time: Wed Feb 16 11:24:56 UTC 2011 Additional Info: POSTGRES_BLOAT CRITICAL: DB "omniti" (host:omniti) (port=5432) table omniti.hits rows:23995164 pages:2858893 shouldbe:2278176 (1.3X) wasted size:4757233664 (4 GB) 12Friday, March 25, 2011
  • 13Friday, March 25, 2011
  • bloat removal toolkit • Offline - with locking • Vacuum • Vacuum Full • Cluster • Cluster vs Vacuum Full • Online - with minimal locking • compact_table • pg_reorg 13Friday, March 25, 2011
  • 14Friday, March 25, 2011
  • 14Friday, March 25, 2011
  • 15Friday, March 25, 2011
  • vacuum • Auto vacuum • VACUUM [verbose|analyze] table • Reclaims storage occupied by dead tuples • Reclaimed space available to re-use • Normally free space doesn’t return to OS • No exclusive table lock 15Friday, March 25, 2011
  • 16Friday, March 25, 2011
  • vacuum full • VACUUM FULL table • Compact table based on dead rows • Starting PostgreSQL 9.0 , it’s rewrite entire table and indexes (like CLUSTER) • Reclaimed space available to re-use • Space returned to OS • Table Exclusive lock • Expensive operation 16Friday, March 25, 2011
  • 17Friday, March 25, 2011
  • 17Friday, March 25, 2011
  • 18Friday, March 25, 2011
  • cluster •CLUSTER [verbose] table_name [USING index_name] •Create reorganized copy of table based on index •Rebuild indexes too •Requires Exclusive Lock on table •Expensive Operation 18Friday, March 25, 2011
  • 19Friday, March 25, 2011
  • cluster vs vacuum full It’s just rewrite of table without Order based on Index any specific order Table needs index Doesn’t need index Bloat indexes (Pre PostgreSQL Doesn’t bloat index 9.0) Overall Operation Slower (Pre Overall operation Faster PostgreSQL 9.0) Recommended Not recommended 19Friday, March 25, 2011
  • 20Friday, March 25, 2011
  • 20Friday, March 25, 2011
  • 21Friday, March 25, 2011
  • compact_table •Developed by OmniTi •Reorganize rows to empty pages at the end based on ctid •ctid - (page number, tuple number) •When we update a row, the rows ctid changes, because the update creates a new version of the row and leaves the old version behind 21Friday, March 25, 2011
  • 22Friday, March 25, 2011
  • compact_table postgres=# select ctid,* from bar; ctid | a | b -------+---+--- (0,1) | 1 | 1 (0,2) | 2 | 1 (2 rows) postgres=# update bar set a=3 where a=2; UPDATE 1 postgres=# select ctid,* from bar; ctid | a | b -------+---+--- (0,1) | 1 | 1 (0,3) | 3 | 1 (2 rows) postgres=# vacuum verbose bar; . . INFO: "bar": found 1 removable, 2 nonremovable row versions in 1 out of 1 pages DETAIL: 0 dead row versions cannot be removed yet. VACUUM 22Friday, March 25, 2011
  • 23Friday, March 25, 2011
  • compact_table 23Friday, March 25, 2011
  • compact_table postgres=# truncate table bar; TRUNCATE TABLE postgres=# insert into bar select generate_series(1,1000); INSERT 0 1000 postgres=# delete from bar where a %2 = 0; DELETE 500 postgres=# select max(ctid) from bar; max -------- (4,95) (1 row) postgres=# vacuum verbose bar; INFO: "bar": found 0 removable, 500 nonremovable row versions in 5 out of 5 pages VACUUM 23Friday, March 25, 2011
  • 24Friday, March 25, 2011
  • compact_table postgres=# begin; BEGIN postgres=# update bar set a=a where ctid>=(3,0); UPDATE 161 postgres=# update bar set a=a where ctid>=(3,0); UPDATE 161 postgres=# update bar set a=a where ctid>=(3,0); UPDATE 48 postgres=# update bar set a=a where ctid>=(3,0); UPDATE 48 postgres=# update bar set a=a where ctid>=(3,0); UPDATE 34 postgres=# update bar set a=a where ctid>=(3,0); UPDATE 0 postgres=# commit; COMMIT postgres=# vacuum verbose bar; . . INFO: "bar": truncated 5 to 3 pages . VACUUM 24Friday, March 25, 2011
  • 25Friday, March 25, 2011
  • compact_table •Pretty safe update process •No additional changes required •Slow process •Generate Index bloat •Under development •Not tested in production •Future consideration? 25Friday, March 25, 2011
  • 26Friday, March 25, 2011
  • 26Friday, March 25, 2011
  • 27Friday, March 25, 2011
  • pg_reorg • Developed by NTT OSS Center • Reorganize table with very short lock • Needs Primary Key • Not Null Unique Key works! • CLUSTER online • ORDER BY Key • VACUUM FULL Online • Rewrite without ORDER BY • Build and Install like other Contrib Module 27Friday, March 25, 2011
  • ! 3,4-$*( 8,$%"&-/( 9"#:(($-./*( <%$=( !"#$%&( ;-#*$( 8,*#-$%"&-/( 9"#:( )"( ;-./*( ;-./*( ( 6*,/-7( ;*+,( )"(( ;-./*( )"($"($*+,( $-./*0( !<-,,%&( ( 1&2*#$( ( 3,4-$*( <%$=(>*#7( ( 5*/*$*( 2="#$()"?:( 28Friday, March 25, 2011
  • ! pg_reorg 3,4-$*( 8,$%"&-/( 9"#:(($-./*( <%$=( !"#$%&( ;-#*$( 8,*#-$%"&-/( 9"#:( )"( ;-./*( ;-./*( ( 6*,/-7( ;*+,( )"(( ;-./*( )"($"($*+,( $-./*0( !<-,,%&( ( 1&2*#$( ( 3,4-$*( <%$=(>*#7( ( 5*/*$*( 2="#$()"?:( 28Friday, March 25, 2011
  • 29Friday, March 25, 2011
  • pg_reorg Options pg_reorg --no-order -table test -d omniti -n -- no-order Do online VACUUM FULL. -o columns [,...] --order-by=columns [,...] Do online CLUSTER ordered by specified columns. -t table --table=table -Z --no-analyze -T seconds --wait-timeout=seconds 29Friday, March 25, 2011
  • 30Friday, March 25, 2011
  • pg_reorg in action $ /opt/pgsql/bin/pg_reorg -e -t public.test --order-by=test_id -U postgres -d test LOG: (query) SET statement_timeout = 0 LOG: (query) SET search_path = pg_catalog, pg_temp, public LOG: (query) SET client_min_messages = warning LOG: (query) SELECT * FROM reorg.tables WHERE relid = $1::regclass LOG: (param:0) = public.test LOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED LOG: (query) SET LOCAL statement_timeout = 100 LOG: (query) LOCK TABLE test IN ACCESS EXCLUSIVE MODE LOG: (query) RESET statement_timeout LOG: (query) SELECT reorg.conflicted_triggers($1) LOG: (param:0) = 7441636 LOG: (query) CREATE TYPE reorg.pk_7441636 AS (test_id integer) LOG: (query) CREATE TABLE reorg.log_7441636 (id bigserial PRIMARY KEY, pk reorg.pk_7441636, row pulbic.test) LOG: (query) CREATE TRIGGER z_reorg_trigger BEFORE INSERT OR DELETE OR UPDATE ON pulbic.test FOR EACH ROW EXECUTE PROCEDURE reorg.reorg_trigger(INSERT INTO reorg.log_7441636(pk, row) VALUES( CASE WHEN $1 IS NULL THEN NULL ELSE (ROW($1.test_id)::reorg.pk_7441636) END, $2)) LOG: (query) SELECT reorg.disable_autovacuum(reorg.log_7441636) LOG: (query) COMMIT 30Friday, March 25, 2011
  • pg_reorg in action $ /opt/pgsql/bin/pg_reorg -e -t public.test --order-by=test_id -U postgres -d test LOG: (query) SET statement_timeout = 0 LOG: (query) SET search_path = pg_catalog, pg_temp, public SET LOG: (query) SET client_min_messages = warning LOG: (query) SELECT * FROM reorg.tables WHERE relid = $1::regclass variables LOG: (param:0) = public.test LOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED LOG: (query) SET LOCAL statement_timeout = 100 LOG: (query) LOCK TABLE test IN ACCESS EXCLUSIVE MODE LOG: (query) RESET statement_timeout LOG: (query) SELECT reorg.conflicted_triggers($1) LOG: (param:0) = 7441636 LOG: (query) CREATE TYPE reorg.pk_7441636 AS (test_id integer) LOG: (query) CREATE TABLE reorg.log_7441636 (id bigserial PRIMARY KEY, pk reorg.pk_7441636, row pulbic.test) LOG: (query) CREATE TRIGGER z_reorg_trigger BEFORE INSERT OR DELETE OR UPDATE ON pulbic.test FOR EACH ROW EXECUTE PROCEDURE reorg.reorg_trigger(INSERT INTO reorg.log_7441636(pk, row) VALUES( CASE WHEN $1 IS NULL THEN NULL ELSE (ROW($1.test_id)::reorg.pk_7441636) END, $2)) LOG: (query) SELECT reorg.disable_autovacuum(reorg.log_7441636) LOG: (query) COMMIT 30Friday, March 25, 2011
  • pg_reorg in action $ /opt/pgsql/bin/pg_reorg -e -t public.test --order-by=test_id -U postgres -d test LOG: (query) SET statement_timeout = 0 LOG: (query) SET search_path = pg_catalog, pg_temp, public SET LOG: (query) SET client_min_messages = warning LOG: (query) SELECT * FROM reorg.tables WHERE relid = $1::regclass variables LOG: (param:0) = public.test LOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED LOG: (query) SET LOCAL statement_timeout = 100 LOG: (query) LOCK TABLE test IN ACCESS EXCLUSIVE MODE LOG: (query) RESET statement_timeout LOG: (query) SELECT reorg.conflicted_triggers($1) Place Triggers LOG: (param:0) = 7441636 LOG: (query) CREATE TYPE reorg.pk_7441636 AS (test_id integer) LOG: (query) CREATE TABLE reorg.log_7441636 (id bigserial PRIMARY KEY, pk reorg.pk_7441636, row pulbic.test) LOG: (query) CREATE TRIGGER z_reorg_trigger BEFORE INSERT OR DELETE OR UPDATE ON pulbic.test FOR EACH ROW EXECUTE PROCEDURE reorg.reorg_trigger(INSERT INTO reorg.log_7441636(pk, row) VALUES( CASE WHEN $1 IS NULL THEN NULL ELSE (ROW($1.test_id)::reorg.pk_7441636) END, $2)) LOG: (query) SELECT reorg.disable_autovacuum(reorg.log_7441636) LOG: (query) COMMIT 30Friday, March 25, 2011
  • 31Friday, March 25, 2011
  • pg_reorg in action LOG: (query) BEGIN ISOLATION LEVEL SERIALIZABLE LOG: (query) SELECT set_config(work_mem, current_setting(maintenance_work_mem), true) LOG: (query) SELECT reorg.array_accum(virtualtransaction) FROM pg_locks WHERE locktype = virtualxid AND pid <> pg_backend_pid() LOG: (query) DELETE FROM reorg.log_7441636 LOG: (query) CREATE TABLE reorg.table_7441636 WITH (oids=false) TABLESPACE pg_default AS SELECT * FROM ONLY public.test ORDER BY test_id LOG: (query) SELECT reorg.disable_autovacuum(reorg.table_7441636) LOG: (query) COMMIT LOG: (query) SELECT indexrelid, reorg.reorg_indexdef(indexrelid, indrelid) FROM pg_index WHERE indrelid = $1 LOG: (param:0) = 7441636 LOG: (query) CREATE UNIQUE INDEX index_201337719 ON reorg.table_7441636 USING btree (test_id) LOG: (query) SELECT reorg.reorg_apply($1, $2, $3, $4, $5, $6) LOG: (param:0) = SELECT * FROM reorg.log_7441636 ORDER BY id LIMIT $1 LOG: (param:1) = INSERT INTO reorg.table_7441636 VALUES ($1.*) LOG: (param:2) = DELETE FROM reorg.table_7441636 WHERE (test_id) = ($1.test_id) LOG: (param:3) = UPDATE reorg.table_7441636 SET (test_id, name, lat, lon, city, country, state_abr) = ($2.test_id, $2.name, $2.lat, $2.lon, $2.city, $2.country, $2.state_abr) WHERE (test_id) = ($1.test_id) LOG: (param:4) = DELETE FROM reorg.log_7441636 WHERE id <= $1 LOG: (param:5) = 1000 31Friday, March 25, 2011
  • pg_reorg in action LOG: (query) BEGIN ISOLATION LEVEL SERIALIZABLE LOG: (query) SELECT set_config(work_mem, current_setting(maintenance_work_mem), true) LOG: (query) SELECT reorg.array_accum(virtualtransaction) FROM pg_locks WHERE locktype = virtualxid AND pid <> pg_backend_pid() CTAS LOG: (query) DELETE FROM reorg.log_7441636 Create LOG: (query) CREATE TABLE reorg.table_7441636 WITH (oids=false) TABLESPACE pg_default AS SELECT * FROM ONLY public.test ORDER BY test_id Indexes LOG: (query) SELECT reorg.disable_autovacuum(reorg.table_7441636) LOG: (query) COMMIT LOG: (query) SELECT indexrelid, reorg.reorg_indexdef(indexrelid, indrelid) FROM pg_index WHERE indrelid = $1 LOG: (param:0) = 7441636 LOG: (query) CREATE UNIQUE INDEX index_201337719 ON reorg.table_7441636 USING btree (test_id) LOG: (query) SELECT reorg.reorg_apply($1, $2, $3, $4, $5, $6) LOG: (param:0) = SELECT * FROM reorg.log_7441636 ORDER BY id LIMIT $1 LOG: (param:1) = INSERT INTO reorg.table_7441636 VALUES ($1.*) LOG: (param:2) = DELETE FROM reorg.table_7441636 WHERE (test_id) = ($1.test_id) LOG: (param:3) = UPDATE reorg.table_7441636 SET (test_id, name, lat, lon, city, country, state_abr) = ($2.test_id, $2.name, $2.lat, $2.lon, $2.city, $2.country, $2.state_abr) WHERE (test_id) = ($1.test_id) LOG: (param:4) = DELETE FROM reorg.log_7441636 WHERE id <= $1 LOG: (param:5) = 1000 31Friday, March 25, 2011
  • pg_reorg in action LOG: (query) BEGIN ISOLATION LEVEL SERIALIZABLE LOG: (query) SELECT set_config(work_mem, current_setting(maintenance_work_mem), true) LOG: (query) SELECT reorg.array_accum(virtualtransaction) FROM pg_locks WHERE locktype = virtualxid AND pid <> pg_backend_pid() CTAS LOG: (query) DELETE FROM reorg.log_7441636 Create LOG: (query) CREATE TABLE reorg.table_7441636 WITH (oids=false) TABLESPACE pg_default AS SELECT * FROM ONLY public.test ORDER BY test_id Indexes LOG: (query) SELECT reorg.disable_autovacuum(reorg.table_7441636) LOG: (query) COMMIT LOG: (query) SELECT indexrelid, reorg.reorg_indexdef(indexrelid, indrelid) FROM pg_index WHERE indrelid = $1 LOG: (param:0) = 7441636 LOG: (query) CREATE UNIQUE INDEX index_201337719 ON reorg.table_7441636 USING btree (test_id) Apply LOG: (query) SELECT reorg.reorg_apply($1, $2, $3, $4, $5, $6) LOG: (param:0) = SELECT * FROM reorg.log_7441636 ORDER BY id LIMIT $1 Operational LOG: (param:1) = INSERT INTO reorg.table_7441636 VALUES ($1.*) logs LOG: (param:2) = DELETE FROM reorg.table_7441636 WHERE (test_id) = ($1.test_id) LOG: (param:3) = UPDATE reorg.table_7441636 SET (test_id, name, lat, lon, city, country, state_abr) = ($2.test_id, $2.name, $2.lat, $2.lon, $2.city, $2.country, $2.state_abr) WHERE (test_id) = ($1.test_id) LOG: (param:4) = DELETE FROM reorg.log_7441636 WHERE id <= $1 LOG: (param:5) = 1000 31Friday, March 25, 2011
  • 32Friday, March 25, 2011
  • pg_reorg in action LOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED LOG: (query) SET LOCAL statement_timeout = 100 LOG: (query) LOCK TABLE public.test IN ACCESS EXCLUSIVE MODE LOG: (query) RESET statement_timeout LOG: (query) SELECT reorg.reorg_apply($1, $2, $3, $4, $5, $6) LOG: (param:0) = SELECT * FROM reorg.log_7441636 ORDER BY id LIMIT $1 LOG: (param:1) = INSERT INTO reorg.table_7441636 VALUES ($1.*) LOG: (param:2) = DELETE FROM reorg.table_7441636 WHERE (test_id) = ($1.test_id) LOG: (param:3) = UPDATE reorg.table_7441636 SET (test_id, name, lat, lon, city, country, state_abr) = ($2.test_id, $2.name, $2.lat, $2.lon, $2.city, $2.country, $2.state_abr) WHERE (test_id) = ($1.test_id) LOG: (param:4) = DELETE FROM reorg.log_7441636 WHERE id <= $1 LOG: (param:5) = 0 LOG: (query) SELECT reorg.reorg_swap($1) LOG: (param:0) = 7441636 LOG: (query) COMMIT LOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED LOG: (query) SELECT reorg.reorg_drop($1) LOG: (param:0) = 7441636 LOG: (query) COMMIT LOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED LOG: (query) ANALYZE publoc.test 32 LOG: (query) COMMITFriday, March 25, 2011
  • pg_reorg in action LOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED Lock LOG: (query) SET LOCAL statement_timeout = 100 table and LOG: (query) LOCK TABLE public.test IN ACCESS EXCLUSIVE MODE apply LOG: (query) RESET statement_timeout LOG: (query) SELECT reorg.reorg_apply($1, $2, $3, $4, $5, $6) operational logs LOG: (param:0) = SELECT * FROM reorg.log_7441636 ORDER BY id LIMIT $1 last time LOG: (param:1) = INSERT INTO reorg.table_7441636 VALUES ($1.*) LOG: (param:2) = DELETE FROM reorg.table_7441636 WHERE (test_id) = ($1.test_id) LOG: (param:3) = UPDATE reorg.table_7441636 SET (test_id, name, lat, lon, city, country, state_abr) = ($2.test_id, $2.name, $2.lat, $2.lon, $2.city, $2.country, $2.state_abr) WHERE (test_id) = ($1.test_id) LOG: (param:4) = DELETE FROM reorg.log_7441636 WHERE id <= $1 LOG: (param:5) = 0 LOG: (query) SELECT reorg.reorg_swap($1) LOG: (param:0) = 7441636 LOG: (query) COMMIT LOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED LOG: (query) SELECT reorg.reorg_drop($1) LOG: (param:0) = 7441636 LOG: (query) COMMIT LOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED LOG: (query) ANALYZE publoc.test 32 LOG: (query) COMMITFriday, March 25, 2011
  • pg_reorg in action LOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED Lock LOG: (query) SET LOCAL statement_timeout = 100 table and LOG: (query) LOCK TABLE public.test IN ACCESS EXCLUSIVE MODE apply LOG: (query) RESET statement_timeout LOG: (query) SELECT reorg.reorg_apply($1, $2, $3, $4, $5, $6) operational logs LOG: (param:0) = SELECT * FROM reorg.log_7441636 ORDER BY id LIMIT $1 last time LOG: (param:1) = INSERT INTO reorg.table_7441636 VALUES ($1.*) LOG: (param:2) = DELETE FROM reorg.table_7441636 WHERE (test_id) = ($1.test_id) LOG: (param:3) = UPDATE reorg.table_7441636 SET (test_id, name, lat, lon, city, country, state_abr) = ($2.test_id, $2.name, $2.lat, $2.lon, $2.city, $2.country, $2.state_abr) WHERE (test_id) = ($1.test_id) LOG: (param:4) = DELETE FROM reorg.log_7441636 WHERE id <= $1 LOG: (param:5) = 0 LOG: (query) SELECT reorg.reorg_swap($1) Swapping LOG: (param:0) = 7441636 LOG: (query) COMMIT LOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED LOG: (query) SELECT reorg.reorg_drop($1) LOG: (param:0) = 7441636 LOG: (query) COMMIT LOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED LOG: (query) ANALYZE publoc.test 32 LOG: (query) COMMITFriday, March 25, 2011
  • pg_reorg in action LOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED Lock LOG: (query) SET LOCAL statement_timeout = 100 table and LOG: (query) LOCK TABLE public.test IN ACCESS EXCLUSIVE MODE apply LOG: (query) RESET statement_timeout LOG: (query) SELECT reorg.reorg_apply($1, $2, $3, $4, $5, $6) operational logs LOG: (param:0) = SELECT * FROM reorg.log_7441636 ORDER BY id LIMIT $1 last time LOG: (param:1) = INSERT INTO reorg.table_7441636 VALUES ($1.*) LOG: (param:2) = DELETE FROM reorg.table_7441636 WHERE (test_id) = ($1.test_id) LOG: (param:3) = UPDATE reorg.table_7441636 SET (test_id, name, lat, lon, city, country, state_abr) = ($2.test_id, $2.name, $2.lat, $2.lon, $2.city, $2.country, $2.state_abr) WHERE (test_id) = ($1.test_id) LOG: (param:4) = DELETE FROM reorg.log_7441636 WHERE id <= $1 LOG: (param:5) = 0 LOG: (query) SELECT reorg.reorg_swap($1) Swapping LOG: (param:0) = 7441636 LOG: (query) COMMIT LOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED LOG: (query) SELECT reorg.reorg_drop($1) Drop work and LOG: (param:0) = 7441636 temp tables LOG: (query) COMMIT LOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED Analyze new table LOG: (query) ANALYZE publoc.test 32 LOG: (query) COMMITFriday, March 25, 2011
  • 33Friday, March 25, 2011
  • pg_reorg in Production •Database Size 540GB •OLTP database 2-3K transactions/sec •Largest pre-rebuild table Size was 127GB •Took 5.75 hours to rebuild •Reclaimed 52GB •No outage reported for website! 33Friday, March 25, 2011
  • 34Friday, March 25, 2011
  • pg_reorg in Production Table Size Before Pg_Reorg Bloat Removed Reorg Execution time 35GB 24m 10sec 13GB 27GB 116m 8 Sec 6GB 31GB 64m 32sec 4GB 44GB 128m 44sec 3GB 34Friday, March 25, 2011
  • 35Friday, March 25, 2011
  • 35Friday, March 25, 2011
  • 36Friday, March 25, 2011
  • 36Friday, March 25, 2011
  • 37Friday, March 25, 2011
  • Test Case • Database : PostgreSQL 9.0.2 • OS: Solaris 11 • RAM : 32GB postgres=# d users • work_mem=16MB Table "public.users" Column | Type | Modifiers • Table Name : Users --------------+--------------------------+----------- user_id | integer | not null • Table Size : 1180 MB name | text | creation_tsz | timestamp with time zone | • Indexes Size : 256 MB active | boolean | clicks | double precision | • Rows : 6M Indexes: "users_pkey" PRIMARY KEY, btree (user_id) "users_creation_tsz" btree (creation_tsz) - DELETE 3M ROWS 37 - UPDATE 1M ROWSFriday, March 25, 2011
  • 38Friday, March 25, 2011
  • Comparison CLUSTER VACUUM FULL PG_REORG Execution Time: Execution Time: Execution Time: 2 min 1 min 50 Sec 1 min 4 Sec Table Size: 594MB Table Size: 594MB Table Size: 594MB Index Size: 128 MB Index Size: 128 MB Index Size: 128 MB offline offline online Built - in Built-in External Tool 38Friday, March 25, 2011
  • 39Friday, March 25, 2011
  • Other Tips •Remove unused indexes •Reindex indexes as needed •Partitioned table •Detach child table -> Cluster child table -> Attach •Avoid vacuum full (Pre PostgrSQL 9.0) 39Friday, March 25, 2011
  • 40Friday, March 25, 2011
  • Monitoring •Circonus •Nagios monitoring •Bloat report 40Friday, March 25, 2011
  • 41Friday, March 25, 2011
  • trending •Cacti •Circonus 41Friday, March 25, 2011
  • 42Friday, March 25, 2011
  • trending 42Friday, March 25, 2011
  • trending 42Friday, March 25, 2011
  • 43Friday, March 25, 2011
  • thanks! •OmniTi •Robert Treat •Depesz •Joshua (End Point) •PGEast Conference Committee 43Friday, March 25, 2011
  • 44Friday, March 25, 2011
  • References •http://www.postgresql.org/docs/9.0/ •http://reorg.projects.postgresql.org/ pg_reorg.html •https://labs.omniti.com/pgtreats/trunk/ tools/compact_table •http://blog.endpoint.com/2010/09/ reducing-bloat-without-locking.html 44Friday, March 25, 2011
  • 45Friday, March 25, 2011
  • open discussion •When PostgreSQL will have built in features to rebuild table and index online ? •Is it worth to change MVCC handling to avoid bloating? 45Friday, March 25, 2011
  • 46Friday, March 25, 2011
  • Questions? •Email : denish@omniti.com •Twitter : http://twitter.com/#!/denishpatel 46Friday, March 25, 2011