SlideShare a Scribd company logo
1 of 28
Download to read offline
Recovering From A Damaged
PostgreSQL Cluster
Recovering From A Damaged
PostgreSQL Cluster
@ me
Recovering From A Damaged
PostgreSQL Cluster
Asurion Mobile Applications
http://www.asurion.com/
Robert Bernier
Sr. Database Administrator, Asurion Mobile Applications
robert.bernier@asurion.com
1400 Fashion Island Blvd, San Mateo, CA 94404
Recovering From A Damaged
PostgreSQL Cluster
Recovering From A Damaged
PostgreSQL Cluster
Recovering From A Damaged
PostgreSQL Cluster
Recovering From A Damaged
PostgreSQL Cluster
Recovering From A Damaged
PostgreSQL Cluster
Recovering From A Damaged
PostgreSQL Cluster
Recovering From A Damaged
PostgreSQL Cluster
Just a little bit of
knowledge
Recovering From A Damaged
PostgreSQL Cluster
The secret is...
Recovering From A Damaged
PostgreSQL Cluster
Indexes & Tables are FILES!
Recovering From A Damaged
PostgreSQL Cluster
Recovering From A Damaged
PostgreSQL Cluster
An Example, ver 9.2.4
create table t1 (id serial primary key,x real);
insert into t1 (x) select random() from generate_series(1,1000);
dtis+
List of relations
Schema | Name | Type | Owner | Table | Size | Description
--------+-----------+----------+----------+-------+------------+-------------
public | t1 | table | postgres | | 40 kB |
public | t1_id_seq | sequence | postgres | | 8192 bytes |
public | t1_pkey | index | postgres | t1 | 40 kB |
select relname,relfilenode,relpages,relpages*1024*8 bytes
from pg_class where relname in ('t1','t1_id_seq','t1_pkey');
relname | relfilenode | relpages | bytes
-----------+-------------+----------+-------
t1 | 24628 | 5 | 40960
t1_id_seq | 24626 | 1 | 8192
t1_pkey | 24632 | 5 | 40960
=========================================================================
cd ~/data
ls -l $(find ./|grep 24628) ---> -rw------- 1 rbernier rbernier 40960 2011-03-07 11:53 ./base/11564/24628
-rw------- 1 rbernier rbernier 24576 2011-03-07 11:51 ./base/11564/24628_fsm
ls -l $(find ./|grep 24626) ---> -rw------- 1 rbernier rbernier 8192 2011-03-07 11:53 ./base/11564/24626
ls -l $(find ./|grep 24632) ---> -rw------- 1 rbernier rbernier 40960 2011-03-07 11:53 ./base/11564/24632
Recovering From A Damaged
PostgreSQL Cluster
Damaged files cannot be processed
-- damage the sequence; can't insert records
rm ./base/11564/24626
touch ./base/11564/24626
_______________________________________________________________________
insert into t1(x) values(9990);
ERROR: could not read block 0 of relation base/11564/24626: read only 0 of 8192 bytes
_______________________________________________________________________
ds t1_id_seq
List of relations
Schema | Name | Type | Owner
--------+-----------+----------+----------
public | t1_id_seq | sequence | postgres
_______________________________________________________________________
table t1_id_seq;
sequence_name | last_value | start_value | increment_by | max_value | min_value | cache_value | log_cnt | is_cycled | is_called
---------------+------------+-------------+--------------+-----------+-----------+-------------+---------+-----------+-----------
Recovering From A Damaged
PostgreSQL Cluster
Damaged files cannot be processed
-- damage the index; can't perform a query
rm ./base/11564/24632
touch ./base/11564/24632
-----------------------------
explain select * from t1 where id=100;
QUERY PLAN
------------------------------------------------------------------
Index Scan using t1_pkey on t1 (cost=0.00..8.27 rows=1 width=8)
Index Cond: (id = 100)
postgres=# select * from t1 where id=100;
ERROR: could not read block 0 of relation base/11564/24632: read only 0 of 8192 bytes
Recovering From A Damaged
PostgreSQL Cluster
Damaged files cannot be processed
-- damage the table, can't perform any query
select * from t1;
ERROR: invalid page header in block 0 of relation base/11564/24628
Recovering From A Damaged
PostgreSQL Cluster
Example Invocation
Scenario: Cannot INSERT Due to Damaged Sequence
Solution: DROP and CREATE SEQUENCE
drop sequence t1_id_seq cascade;
create sequence t1_id_seq start with 10001 owned by t1.id;
alter table only t1 alter column id set default nextval('t1_id_seq'::regclass);
select pg_catalog.setval('t1_id_seq', 10002, false);
BEFORE
relname | relfilenode | relpages | bytes
-----------+-------------+----------+-------
t1_id_seq | 24626 | 1 | 8192
AFTER
relname | relfilenode | relpages | bytes
-----------+-------------+----------+-------
t1_id_seq | 189575 | 1 | 8192
Recovering From A Damaged
PostgreSQL Cluster
Example Invocation
Scenario: Rebuild Data Cluster with Corrupted Index
Solution: Rebuild index
reindex index t1_pkey;
BEFORE
relname | relfilenode | relpages | bytes
-----------+-------------+----------+-------
t1_pkey | 24632 | 5 | 40960
AFTER
relname | relfilenode | relpages | bytes
-----------+-------------+----------+-------
t1_pkey | 189578 | 5 | 40960
Recovering From A Damaged
PostgreSQL Cluster
Example Invocation
Scenario: Rebuild Data Cluster with Corrupted System Indexes
Solution: Rebuild indexes under single-user mode
- start up the damaged cluster and reindex with the following invocation
postgres --single postgres -D ~/data <<_eof_
reindex system mydatabase
_eof_
Recovering From A Damaged
PostgreSQL Cluster
About Single UserMode
- Used during bootstrapping by initdb
- Can be used for debugging
- Can be used for disaster recovery
i.e. System indexes can be manipulated
- Can enter queries and the results are printed to the screen
Recovering From A Damaged
PostgreSQL Cluster
About Single UserMode
postgres --single postgres -D ~/data temp
PostgreSQL stand-alone backend 9.2.3
backend> select * from t1 limit 5
1: id (typeid = 23, len = 4, typmod = -1, byval = t)
2: x (typeid = 700, len = 4, typmod = -1, byval = t)
----
1: id = "1" (typeid = 23, len = 4, typmod = -1, byval = t)
2: x = "0.767757" (typeid = 700, len = 4, typmod = -1, byval = t)
----
1: id = "2" (typeid = 23, len = 4, typmod = -1, byval = t)
2: x = "0.253584" (typeid = 700, len = 4, typmod = -1, byval = t)
----
1: id = "3" (typeid = 23, len = 4, typmod = -1, byval = t)
2: x = "0.0912474" (typeid = 700, len = 4, typmod = -1, byval = t)
----
1: id = "4" (typeid = 23, len = 4, typmod = -1, byval = t)
2: x = "0.45276" (typeid = 700, len = 4, typmod = -1, byval = t)
----
1: id = "5" (typeid = 23, len = 4, typmod = -1, byval = t)
2: x = "0.784381" (typeid = 700, len = 4, typmod = -1, byval = t)
----
backend>
Recovering From A Damaged
PostgreSQL Cluster
Example Invocation
Scenario: PostgreSQL Starts Up But Queries To User-Defined Tables Fails
Solution: Generate a data-dump into a newly created data cluster
- start up the damaged cluster with the following invocation
pg_ctl -D ~/data -o '-c zero_damaged_pages=true' -l logfile.txt start
- execute standard data dump with pg_dumpall
Recovering From A Damaged
PostgreSQL Cluster
Example Invocation
Scenario: Rebuild And Reuse An Existing Data Cluster With Corrupted User-Defined Tables
Solution: Rebuild Cluster
- start up the damaged cluster with the following invocation
pg_ctl -D ~/data -o '-c zero_damaged_pages=true' -l logfile.txt start
- login the database and execute the following SQL statement
VACUUM FULL ANALYZE;
REINDEX DATABASE mydatabase;
- restart the newly repaired cluster
pg_ctl -D ~/data -l logfile.txt -m smart restart
Recovering From A Damaged
PostgreSQL Cluster
Caveats
● DROP and CREATE damaged indexes
● Where possible, recreate the cluster on another machine... don't take a chance
● Executing a dump should be the first action i.e. don't VACUUM damaged tables
as you can actually lose data before you reuse/backup/dump
● Damaged system catalogs must be recovered in single user mode
● Tables larger 1GB+ are split
● TOASTED tables add an entirely new level of complexity
Recovering From A Damaged
PostgreSQL Cluster
One last word
Hacking the cluster...
Recovering From A Damaged
PostgreSQL Cluster
References
- http://www.postgresql.org/docs/9.2/static/index.html
- http://www.postgresql.org/docs/9.2/static/app-initdb.html
- http://www.postgresql.org/docs/9.2/static/app-pg-ctl.html
- http://www.postgresql.org/docs/9.2/static/catalog-pg-class.html
- http://www.postgresql.org/docs/9.2/static/runtime-config-developer.html
- http://www.postgresql.org/docs/9.2/static/sql-createindex.html
- http://www.postgresql.org/docs/9.2/static/sql-dropindex.html
- http://www.postgresql.org/docs/9.2/static/sql-reindex.html
- http://www.postgresql.org/docs/9.2/static/sql-vacuum.html
- http://linux.die.net/man/1/find
- http://linux.die.net/man/1/killall
Recovering From A Damaged
PostgreSQL Cluster
Asurion Mobile Applications
http://www.asurion.com/
Robert Bernier
Sr. Database Administrator, Asurion Mobile Applications
robert.bernier@asurion.com
1400 Fashion Island Blvd, San Mateo, CA 94404

More Related Content

What's hot

[PGDay.Seoul 2020] PostgreSQL 13 New Features
[PGDay.Seoul 2020] PostgreSQL 13 New Features[PGDay.Seoul 2020] PostgreSQL 13 New Features
[PGDay.Seoul 2020] PostgreSQL 13 New Featureshyeongchae lee
 
PostgreSQL 9.6 새 기능 소개
PostgreSQL 9.6 새 기능 소개PostgreSQL 9.6 새 기능 소개
PostgreSQL 9.6 새 기능 소개PgDay.Seoul
 
Webinar: Replication and Replica Sets
Webinar: Replication and Replica SetsWebinar: Replication and Replica Sets
Webinar: Replication and Replica SetsMongoDB
 
Lab1-DB-Cassandra
Lab1-DB-CassandraLab1-DB-Cassandra
Lab1-DB-CassandraLilia Sfaxi
 
【Maclean liu技术分享】拨开oracle cbo优化器迷雾,探究histogram直方图之秘 0321
【Maclean liu技术分享】拨开oracle cbo优化器迷雾,探究histogram直方图之秘 0321【Maclean liu技术分享】拨开oracle cbo优化器迷雾,探究histogram直方图之秘 0321
【Maclean liu技术分享】拨开oracle cbo优化器迷雾,探究histogram直方图之秘 0321maclean liu
 
15 MySQL Basics #burningkeyboards
15 MySQL Basics #burningkeyboards15 MySQL Basics #burningkeyboards
15 MySQL Basics #burningkeyboardsDenis Ristic
 
Store and Process Big Data with Hadoop and Cassandra
Store and Process Big Data with Hadoop and CassandraStore and Process Big Data with Hadoop and Cassandra
Store and Process Big Data with Hadoop and CassandraDeependra Ariyadewa
 
Advanced pg_stat_statements: Filtering, Regression Testing & more
Advanced pg_stat_statements: Filtering, Regression Testing & moreAdvanced pg_stat_statements: Filtering, Regression Testing & more
Advanced pg_stat_statements: Filtering, Regression Testing & moreLukas Fittl
 
Replication
ReplicationReplication
ReplicationMongoDB
 
Example R usage for oracle DBA UKOUG 2013
Example R usage for oracle DBA UKOUG 2013Example R usage for oracle DBA UKOUG 2013
Example R usage for oracle DBA UKOUG 2013BertrandDrouvot
 
Parallel R in snow (english after 2nd slide)
Parallel R in snow (english after 2nd slide)Parallel R in snow (english after 2nd slide)
Parallel R in snow (english after 2nd slide)Cdiscount
 
TimesTen in memory database Creation
TimesTen in memory database Creation TimesTen in memory database Creation
TimesTen in memory database Creation Monowar Mukul
 

What's hot (20)

[PGDay.Seoul 2020] PostgreSQL 13 New Features
[PGDay.Seoul 2020] PostgreSQL 13 New Features[PGDay.Seoul 2020] PostgreSQL 13 New Features
[PGDay.Seoul 2020] PostgreSQL 13 New Features
 
PostgreSQL 9.6 새 기능 소개
PostgreSQL 9.6 새 기능 소개PostgreSQL 9.6 새 기능 소개
PostgreSQL 9.6 새 기능 소개
 
Webinar: Replication and Replica Sets
Webinar: Replication and Replica SetsWebinar: Replication and Replica Sets
Webinar: Replication and Replica Sets
 
8 tune tusc
8 tune tusc8 tune tusc
8 tune tusc
 
Lab1-DB-Cassandra
Lab1-DB-CassandraLab1-DB-Cassandra
Lab1-DB-Cassandra
 
【Maclean liu技术分享】拨开oracle cbo优化器迷雾,探究histogram直方图之秘 0321
【Maclean liu技术分享】拨开oracle cbo优化器迷雾,探究histogram直方图之秘 0321【Maclean liu技术分享】拨开oracle cbo优化器迷雾,探究histogram直方图之秘 0321
【Maclean liu技术分享】拨开oracle cbo优化器迷雾,探究histogram直方图之秘 0321
 
15 MySQL Basics #burningkeyboards
15 MySQL Basics #burningkeyboards15 MySQL Basics #burningkeyboards
15 MySQL Basics #burningkeyboards
 
Store and Process Big Data with Hadoop and Cassandra
Store and Process Big Data with Hadoop and CassandraStore and Process Big Data with Hadoop and Cassandra
Store and Process Big Data with Hadoop and Cassandra
 
My sql1
My sql1My sql1
My sql1
 
MySQLinsanity
MySQLinsanityMySQLinsanity
MySQLinsanity
 
Sql analytic queries tips
Sql analytic queries tipsSql analytic queries tips
Sql analytic queries tips
 
Advanced pg_stat_statements: Filtering, Regression Testing & more
Advanced pg_stat_statements: Filtering, Regression Testing & moreAdvanced pg_stat_statements: Filtering, Regression Testing & more
Advanced pg_stat_statements: Filtering, Regression Testing & more
 
Quick reference for Grafana
Quick reference for GrafanaQuick reference for Grafana
Quick reference for Grafana
 
Replication
ReplicationReplication
Replication
 
Example R usage for oracle DBA UKOUG 2013
Example R usage for oracle DBA UKOUG 2013Example R usage for oracle DBA UKOUG 2013
Example R usage for oracle DBA UKOUG 2013
 
Parallel R in snow (english after 2nd slide)
Parallel R in snow (english after 2nd slide)Parallel R in snow (english after 2nd slide)
Parallel R in snow (english after 2nd slide)
 
Mongodb replication
Mongodb replicationMongodb replication
Mongodb replication
 
TimesTen in memory database Creation
TimesTen in memory database Creation TimesTen in memory database Creation
TimesTen in memory database Creation
 
Broker otw.pptx
Broker otw.pptxBroker otw.pptx
Broker otw.pptx
 
MySql:Basics
MySql:BasicsMySql:Basics
MySql:Basics
 

Similar to Robert Bernier - Recovering From A Damaged PostgreSQL Cluster @ Postgres Open

MySQL InnoDB Cluster 미리보기 (remote cluster test)
MySQL InnoDB Cluster 미리보기 (remote cluster test)MySQL InnoDB Cluster 미리보기 (remote cluster test)
MySQL InnoDB Cluster 미리보기 (remote cluster test)Seungmin Yu
 
Introduction to MySQL InnoDB Cluster
Introduction to MySQL InnoDB ClusterIntroduction to MySQL InnoDB Cluster
Introduction to MySQL InnoDB ClusterI Goo Lee
 
Drizzle to MySQL, Stress Free Migration
Drizzle to MySQL, Stress Free MigrationDrizzle to MySQL, Stress Free Migration
Drizzle to MySQL, Stress Free MigrationAndrew Hutchings
 
Inexpensive Datamasking for MySQL with ProxySQL — Data Anonymization for Deve...
Inexpensive Datamasking for MySQL with ProxySQL — Data Anonymization for Deve...Inexpensive Datamasking for MySQL with ProxySQL — Data Anonymization for Deve...
Inexpensive Datamasking for MySQL with ProxySQL — Data Anonymization for Deve...Ontico
 
Lab 2: Classification and Regression Prediction Models, training and testing ...
Lab 2: Classification and Regression Prediction Models, training and testing ...Lab 2: Classification and Regression Prediction Models, training and testing ...
Lab 2: Classification and Regression Prediction Models, training and testing ...Yao Yao
 
PerlApp2Postgresql (2)
PerlApp2Postgresql (2)PerlApp2Postgresql (2)
PerlApp2Postgresql (2)Jerome Eteve
 
Advanced Replication
Advanced ReplicationAdvanced Replication
Advanced ReplicationMongoDB
 
Replication and replica sets
Replication and replica setsReplication and replica sets
Replication and replica setsRandall Hunt
 
Basic Replication in MongoDB
Basic Replication in MongoDBBasic Replication in MongoDB
Basic Replication in MongoDBMongoDB
 
Fatkulin presentation
Fatkulin presentationFatkulin presentation
Fatkulin presentationEnkitec
 
03-inheritance.ppt
03-inheritance.ppt03-inheritance.ppt
03-inheritance.pptSaiM947604
 
Advanced tips of dbms statas
Advanced tips of dbms statasAdvanced tips of dbms statas
Advanced tips of dbms statasLouis liu
 
12c db upgrade from 11.2.0.4
12c db upgrade from 11.2.0.412c db upgrade from 11.2.0.4
12c db upgrade from 11.2.0.4uzzal basak
 
Data Wrangling with dplyr and tidyr Cheat Sheet
Data Wrangling with dplyr and tidyr Cheat SheetData Wrangling with dplyr and tidyr Cheat Sheet
Data Wrangling with dplyr and tidyr Cheat SheetDr. Volkan OBAN
 
From Postgres to Cassandra (Rimas Silkaitis, Heroku) | C* Summit 2016
From Postgres to Cassandra (Rimas Silkaitis, Heroku) | C* Summit 2016From Postgres to Cassandra (Rimas Silkaitis, Heroku) | C* Summit 2016
From Postgres to Cassandra (Rimas Silkaitis, Heroku) | C* Summit 2016DataStax
 
Troubleshooting MySQL Performance
Troubleshooting MySQL PerformanceTroubleshooting MySQL Performance
Troubleshooting MySQL PerformanceSveta Smirnova
 

Similar to Robert Bernier - Recovering From A Damaged PostgreSQL Cluster @ Postgres Open (20)

MySQL InnoDB Cluster 미리보기 (remote cluster test)
MySQL InnoDB Cluster 미리보기 (remote cluster test)MySQL InnoDB Cluster 미리보기 (remote cluster test)
MySQL InnoDB Cluster 미리보기 (remote cluster test)
 
Introduction to MySQL InnoDB Cluster
Introduction to MySQL InnoDB ClusterIntroduction to MySQL InnoDB Cluster
Introduction to MySQL InnoDB Cluster
 
Oracle training in hyderabad
Oracle training in hyderabadOracle training in hyderabad
Oracle training in hyderabad
 
Drizzle to MySQL, Stress Free Migration
Drizzle to MySQL, Stress Free MigrationDrizzle to MySQL, Stress Free Migration
Drizzle to MySQL, Stress Free Migration
 
Inexpensive Datamasking for MySQL with ProxySQL — Data Anonymization for Deve...
Inexpensive Datamasking for MySQL with ProxySQL — Data Anonymization for Deve...Inexpensive Datamasking for MySQL with ProxySQL — Data Anonymization for Deve...
Inexpensive Datamasking for MySQL with ProxySQL — Data Anonymization for Deve...
 
Lab 2: Classification and Regression Prediction Models, training and testing ...
Lab 2: Classification and Regression Prediction Models, training and testing ...Lab 2: Classification and Regression Prediction Models, training and testing ...
Lab 2: Classification and Regression Prediction Models, training and testing ...
 
Sql statement,functions and joins
Sql statement,functions and joinsSql statement,functions and joins
Sql statement,functions and joins
 
PerlApp2Postgresql (2)
PerlApp2Postgresql (2)PerlApp2Postgresql (2)
PerlApp2Postgresql (2)
 
Advanced Replication
Advanced ReplicationAdvanced Replication
Advanced Replication
 
Replication and replica sets
Replication and replica setsReplication and replica sets
Replication and replica sets
 
Basic Replication in MongoDB
Basic Replication in MongoDBBasic Replication in MongoDB
Basic Replication in MongoDB
 
Fatkulin presentation
Fatkulin presentationFatkulin presentation
Fatkulin presentation
 
03-inheritance.ppt
03-inheritance.ppt03-inheritance.ppt
03-inheritance.ppt
 
Advanced tips of dbms statas
Advanced tips of dbms statasAdvanced tips of dbms statas
Advanced tips of dbms statas
 
C3 w2
C3 w2C3 w2
C3 w2
 
12c db upgrade from 11.2.0.4
12c db upgrade from 11.2.0.412c db upgrade from 11.2.0.4
12c db upgrade from 11.2.0.4
 
Data Wrangling with dplyr and tidyr Cheat Sheet
Data Wrangling with dplyr and tidyr Cheat SheetData Wrangling with dplyr and tidyr Cheat Sheet
Data Wrangling with dplyr and tidyr Cheat Sheet
 
Rac nonrac clone
Rac nonrac cloneRac nonrac clone
Rac nonrac clone
 
From Postgres to Cassandra (Rimas Silkaitis, Heroku) | C* Summit 2016
From Postgres to Cassandra (Rimas Silkaitis, Heroku) | C* Summit 2016From Postgres to Cassandra (Rimas Silkaitis, Heroku) | C* Summit 2016
From Postgres to Cassandra (Rimas Silkaitis, Heroku) | C* Summit 2016
 
Troubleshooting MySQL Performance
Troubleshooting MySQL PerformanceTroubleshooting MySQL Performance
Troubleshooting MySQL Performance
 

More from PostgresOpen

Bruce Momjian - Inside PostgreSQL Shared Memory @ Postgres Open
Bruce Momjian - Inside PostgreSQL Shared Memory @ Postgres OpenBruce Momjian - Inside PostgreSQL Shared Memory @ Postgres Open
Bruce Momjian - Inside PostgreSQL Shared Memory @ Postgres OpenPostgresOpen
 
Gurjeet Singh - How Postgres is Different From (Better Tha) Your RDBMS @ Post...
Gurjeet Singh - How Postgres is Different From (Better Tha) Your RDBMS @ Post...Gurjeet Singh - How Postgres is Different From (Better Tha) Your RDBMS @ Post...
Gurjeet Singh - How Postgres is Different From (Better Tha) Your RDBMS @ Post...PostgresOpen
 
Keith Fiske - When PostgreSQL Can't, You Can @ Postgres Open
Keith Fiske - When PostgreSQL Can't, You Can @ Postgres OpenKeith Fiske - When PostgreSQL Can't, You Can @ Postgres Open
Keith Fiske - When PostgreSQL Can't, You Can @ Postgres OpenPostgresOpen
 
David Keeney - SQL Database Server Requests from the Browser @ Postgres Open
David Keeney - SQL Database Server Requests from the Browser @ Postgres OpenDavid Keeney - SQL Database Server Requests from the Browser @ Postgres Open
David Keeney - SQL Database Server Requests from the Browser @ Postgres OpenPostgresOpen
 
Keith Paskett - Postgres on ZFS @ Postgres Open
Keith Paskett - Postgres on ZFS @ Postgres OpenKeith Paskett - Postgres on ZFS @ Postgres Open
Keith Paskett - Postgres on ZFS @ Postgres OpenPostgresOpen
 
Kevin Kempter - PostgreSQL Backup and Recovery Methods @ Postgres Open
Kevin Kempter - PostgreSQL Backup and Recovery Methods @ Postgres OpenKevin Kempter - PostgreSQL Backup and Recovery Methods @ Postgres Open
Kevin Kempter - PostgreSQL Backup and Recovery Methods @ Postgres OpenPostgresOpen
 
Craig Kerstiens - Scalable Uniques in Postgres @ Postgres Open
Craig Kerstiens - Scalable Uniques in Postgres @ Postgres OpenCraig Kerstiens - Scalable Uniques in Postgres @ Postgres Open
Craig Kerstiens - Scalable Uniques in Postgres @ Postgres OpenPostgresOpen
 
Henrietta Dombrovskaya - A New Approach to Resolve Object-Relational Impedanc...
Henrietta Dombrovskaya - A New Approach to Resolve Object-Relational Impedanc...Henrietta Dombrovskaya - A New Approach to Resolve Object-Relational Impedanc...
Henrietta Dombrovskaya - A New Approach to Resolve Object-Relational Impedanc...PostgresOpen
 
Steve Singer - Managing PostgreSQL with Puppet @ Postgres Open
Steve Singer - Managing PostgreSQL with Puppet @ Postgres OpenSteve Singer - Managing PostgreSQL with Puppet @ Postgres Open
Steve Singer - Managing PostgreSQL with Puppet @ Postgres OpenPostgresOpen
 
John Melesky - Federating Queries Using Postgres FDW @ Postgres Open
John Melesky - Federating Queries Using Postgres FDW @ Postgres OpenJohn Melesky - Federating Queries Using Postgres FDW @ Postgres Open
John Melesky - Federating Queries Using Postgres FDW @ Postgres OpenPostgresOpen
 
Koichi Suzuki - Postgres-XC Dynamic Cluster Management @ Postgres Open
Koichi Suzuki - Postgres-XC Dynamic Cluster  Management @ Postgres OpenKoichi Suzuki - Postgres-XC Dynamic Cluster  Management @ Postgres Open
Koichi Suzuki - Postgres-XC Dynamic Cluster Management @ Postgres OpenPostgresOpen
 
Selena Deckelmann - Sane Schema Management with Alembic and SQLAlchemy @ Pos...
Selena Deckelmann - Sane Schema Management with  Alembic and SQLAlchemy @ Pos...Selena Deckelmann - Sane Schema Management with  Alembic and SQLAlchemy @ Pos...
Selena Deckelmann - Sane Schema Management with Alembic and SQLAlchemy @ Pos...PostgresOpen
 
Michael Paquier - Taking advantage of custom bgworkers @ Postgres Open
Michael Paquier - Taking advantage of custom bgworkers @ Postgres OpenMichael Paquier - Taking advantage of custom bgworkers @ Postgres Open
Michael Paquier - Taking advantage of custom bgworkers @ Postgres OpenPostgresOpen
 
Kevin Kempter PostgreSQL Backup and Recovery Methods @ Postgres Open
Kevin Kempter PostgreSQL Backup and Recovery Methods @ Postgres OpenKevin Kempter PostgreSQL Backup and Recovery Methods @ Postgres Open
Kevin Kempter PostgreSQL Backup and Recovery Methods @ Postgres OpenPostgresOpen
 
Michael Bayer Introduction to SQLAlchemy @ Postgres Open
Michael Bayer Introduction to SQLAlchemy @ Postgres OpenMichael Bayer Introduction to SQLAlchemy @ Postgres Open
Michael Bayer Introduction to SQLAlchemy @ Postgres OpenPostgresOpen
 
Robert Haas Query Planning Gone Wrong Presentation @ Postgres Open
Robert Haas Query Planning Gone Wrong Presentation @ Postgres OpenRobert Haas Query Planning Gone Wrong Presentation @ Postgres Open
Robert Haas Query Planning Gone Wrong Presentation @ Postgres OpenPostgresOpen
 
Ryan Jarvinen Open Shift Talk @ Postgres Open 2013
Ryan Jarvinen Open Shift Talk @ Postgres Open 2013Ryan Jarvinen Open Shift Talk @ Postgres Open 2013
Ryan Jarvinen Open Shift Talk @ Postgres Open 2013PostgresOpen
 
Andrew Dunstan 9.3 JSON Presentation @ Postgres Open 2013
Andrew Dunstan 9.3 JSON Presentation @ Postgres Open 2013Andrew Dunstan 9.3 JSON Presentation @ Postgres Open 2013
Andrew Dunstan 9.3 JSON Presentation @ Postgres Open 2013PostgresOpen
 

More from PostgresOpen (18)

Bruce Momjian - Inside PostgreSQL Shared Memory @ Postgres Open
Bruce Momjian - Inside PostgreSQL Shared Memory @ Postgres OpenBruce Momjian - Inside PostgreSQL Shared Memory @ Postgres Open
Bruce Momjian - Inside PostgreSQL Shared Memory @ Postgres Open
 
Gurjeet Singh - How Postgres is Different From (Better Tha) Your RDBMS @ Post...
Gurjeet Singh - How Postgres is Different From (Better Tha) Your RDBMS @ Post...Gurjeet Singh - How Postgres is Different From (Better Tha) Your RDBMS @ Post...
Gurjeet Singh - How Postgres is Different From (Better Tha) Your RDBMS @ Post...
 
Keith Fiske - When PostgreSQL Can't, You Can @ Postgres Open
Keith Fiske - When PostgreSQL Can't, You Can @ Postgres OpenKeith Fiske - When PostgreSQL Can't, You Can @ Postgres Open
Keith Fiske - When PostgreSQL Can't, You Can @ Postgres Open
 
David Keeney - SQL Database Server Requests from the Browser @ Postgres Open
David Keeney - SQL Database Server Requests from the Browser @ Postgres OpenDavid Keeney - SQL Database Server Requests from the Browser @ Postgres Open
David Keeney - SQL Database Server Requests from the Browser @ Postgres Open
 
Keith Paskett - Postgres on ZFS @ Postgres Open
Keith Paskett - Postgres on ZFS @ Postgres OpenKeith Paskett - Postgres on ZFS @ Postgres Open
Keith Paskett - Postgres on ZFS @ Postgres Open
 
Kevin Kempter - PostgreSQL Backup and Recovery Methods @ Postgres Open
Kevin Kempter - PostgreSQL Backup and Recovery Methods @ Postgres OpenKevin Kempter - PostgreSQL Backup and Recovery Methods @ Postgres Open
Kevin Kempter - PostgreSQL Backup and Recovery Methods @ Postgres Open
 
Craig Kerstiens - Scalable Uniques in Postgres @ Postgres Open
Craig Kerstiens - Scalable Uniques in Postgres @ Postgres OpenCraig Kerstiens - Scalable Uniques in Postgres @ Postgres Open
Craig Kerstiens - Scalable Uniques in Postgres @ Postgres Open
 
Henrietta Dombrovskaya - A New Approach to Resolve Object-Relational Impedanc...
Henrietta Dombrovskaya - A New Approach to Resolve Object-Relational Impedanc...Henrietta Dombrovskaya - A New Approach to Resolve Object-Relational Impedanc...
Henrietta Dombrovskaya - A New Approach to Resolve Object-Relational Impedanc...
 
Steve Singer - Managing PostgreSQL with Puppet @ Postgres Open
Steve Singer - Managing PostgreSQL with Puppet @ Postgres OpenSteve Singer - Managing PostgreSQL with Puppet @ Postgres Open
Steve Singer - Managing PostgreSQL with Puppet @ Postgres Open
 
John Melesky - Federating Queries Using Postgres FDW @ Postgres Open
John Melesky - Federating Queries Using Postgres FDW @ Postgres OpenJohn Melesky - Federating Queries Using Postgres FDW @ Postgres Open
John Melesky - Federating Queries Using Postgres FDW @ Postgres Open
 
Koichi Suzuki - Postgres-XC Dynamic Cluster Management @ Postgres Open
Koichi Suzuki - Postgres-XC Dynamic Cluster  Management @ Postgres OpenKoichi Suzuki - Postgres-XC Dynamic Cluster  Management @ Postgres Open
Koichi Suzuki - Postgres-XC Dynamic Cluster Management @ Postgres Open
 
Selena Deckelmann - Sane Schema Management with Alembic and SQLAlchemy @ Pos...
Selena Deckelmann - Sane Schema Management with  Alembic and SQLAlchemy @ Pos...Selena Deckelmann - Sane Schema Management with  Alembic and SQLAlchemy @ Pos...
Selena Deckelmann - Sane Schema Management with Alembic and SQLAlchemy @ Pos...
 
Michael Paquier - Taking advantage of custom bgworkers @ Postgres Open
Michael Paquier - Taking advantage of custom bgworkers @ Postgres OpenMichael Paquier - Taking advantage of custom bgworkers @ Postgres Open
Michael Paquier - Taking advantage of custom bgworkers @ Postgres Open
 
Kevin Kempter PostgreSQL Backup and Recovery Methods @ Postgres Open
Kevin Kempter PostgreSQL Backup and Recovery Methods @ Postgres OpenKevin Kempter PostgreSQL Backup and Recovery Methods @ Postgres Open
Kevin Kempter PostgreSQL Backup and Recovery Methods @ Postgres Open
 
Michael Bayer Introduction to SQLAlchemy @ Postgres Open
Michael Bayer Introduction to SQLAlchemy @ Postgres OpenMichael Bayer Introduction to SQLAlchemy @ Postgres Open
Michael Bayer Introduction to SQLAlchemy @ Postgres Open
 
Robert Haas Query Planning Gone Wrong Presentation @ Postgres Open
Robert Haas Query Planning Gone Wrong Presentation @ Postgres OpenRobert Haas Query Planning Gone Wrong Presentation @ Postgres Open
Robert Haas Query Planning Gone Wrong Presentation @ Postgres Open
 
Ryan Jarvinen Open Shift Talk @ Postgres Open 2013
Ryan Jarvinen Open Shift Talk @ Postgres Open 2013Ryan Jarvinen Open Shift Talk @ Postgres Open 2013
Ryan Jarvinen Open Shift Talk @ Postgres Open 2013
 
Andrew Dunstan 9.3 JSON Presentation @ Postgres Open 2013
Andrew Dunstan 9.3 JSON Presentation @ Postgres Open 2013Andrew Dunstan 9.3 JSON Presentation @ Postgres Open 2013
Andrew Dunstan 9.3 JSON Presentation @ Postgres Open 2013
 

Recently uploaded

activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 

Recently uploaded (20)

activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 

Robert Bernier - Recovering From A Damaged PostgreSQL Cluster @ Postgres Open

  • 1. Recovering From A Damaged PostgreSQL Cluster
  • 2. Recovering From A Damaged PostgreSQL Cluster @ me
  • 3. Recovering From A Damaged PostgreSQL Cluster Asurion Mobile Applications http://www.asurion.com/ Robert Bernier Sr. Database Administrator, Asurion Mobile Applications robert.bernier@asurion.com 1400 Fashion Island Blvd, San Mateo, CA 94404
  • 4. Recovering From A Damaged PostgreSQL Cluster
  • 5. Recovering From A Damaged PostgreSQL Cluster
  • 6. Recovering From A Damaged PostgreSQL Cluster
  • 7. Recovering From A Damaged PostgreSQL Cluster
  • 8. Recovering From A Damaged PostgreSQL Cluster
  • 9. Recovering From A Damaged PostgreSQL Cluster
  • 10. Recovering From A Damaged PostgreSQL Cluster Just a little bit of knowledge
  • 11. Recovering From A Damaged PostgreSQL Cluster The secret is...
  • 12. Recovering From A Damaged PostgreSQL Cluster Indexes & Tables are FILES!
  • 13. Recovering From A Damaged PostgreSQL Cluster
  • 14. Recovering From A Damaged PostgreSQL Cluster An Example, ver 9.2.4 create table t1 (id serial primary key,x real); insert into t1 (x) select random() from generate_series(1,1000); dtis+ List of relations Schema | Name | Type | Owner | Table | Size | Description --------+-----------+----------+----------+-------+------------+------------- public | t1 | table | postgres | | 40 kB | public | t1_id_seq | sequence | postgres | | 8192 bytes | public | t1_pkey | index | postgres | t1 | 40 kB | select relname,relfilenode,relpages,relpages*1024*8 bytes from pg_class where relname in ('t1','t1_id_seq','t1_pkey'); relname | relfilenode | relpages | bytes -----------+-------------+----------+------- t1 | 24628 | 5 | 40960 t1_id_seq | 24626 | 1 | 8192 t1_pkey | 24632 | 5 | 40960 ========================================================================= cd ~/data ls -l $(find ./|grep 24628) ---> -rw------- 1 rbernier rbernier 40960 2011-03-07 11:53 ./base/11564/24628 -rw------- 1 rbernier rbernier 24576 2011-03-07 11:51 ./base/11564/24628_fsm ls -l $(find ./|grep 24626) ---> -rw------- 1 rbernier rbernier 8192 2011-03-07 11:53 ./base/11564/24626 ls -l $(find ./|grep 24632) ---> -rw------- 1 rbernier rbernier 40960 2011-03-07 11:53 ./base/11564/24632
  • 15. Recovering From A Damaged PostgreSQL Cluster Damaged files cannot be processed -- damage the sequence; can't insert records rm ./base/11564/24626 touch ./base/11564/24626 _______________________________________________________________________ insert into t1(x) values(9990); ERROR: could not read block 0 of relation base/11564/24626: read only 0 of 8192 bytes _______________________________________________________________________ ds t1_id_seq List of relations Schema | Name | Type | Owner --------+-----------+----------+---------- public | t1_id_seq | sequence | postgres _______________________________________________________________________ table t1_id_seq; sequence_name | last_value | start_value | increment_by | max_value | min_value | cache_value | log_cnt | is_cycled | is_called ---------------+------------+-------------+--------------+-----------+-----------+-------------+---------+-----------+-----------
  • 16. Recovering From A Damaged PostgreSQL Cluster Damaged files cannot be processed -- damage the index; can't perform a query rm ./base/11564/24632 touch ./base/11564/24632 ----------------------------- explain select * from t1 where id=100; QUERY PLAN ------------------------------------------------------------------ Index Scan using t1_pkey on t1 (cost=0.00..8.27 rows=1 width=8) Index Cond: (id = 100) postgres=# select * from t1 where id=100; ERROR: could not read block 0 of relation base/11564/24632: read only 0 of 8192 bytes
  • 17. Recovering From A Damaged PostgreSQL Cluster Damaged files cannot be processed -- damage the table, can't perform any query select * from t1; ERROR: invalid page header in block 0 of relation base/11564/24628
  • 18. Recovering From A Damaged PostgreSQL Cluster Example Invocation Scenario: Cannot INSERT Due to Damaged Sequence Solution: DROP and CREATE SEQUENCE drop sequence t1_id_seq cascade; create sequence t1_id_seq start with 10001 owned by t1.id; alter table only t1 alter column id set default nextval('t1_id_seq'::regclass); select pg_catalog.setval('t1_id_seq', 10002, false); BEFORE relname | relfilenode | relpages | bytes -----------+-------------+----------+------- t1_id_seq | 24626 | 1 | 8192 AFTER relname | relfilenode | relpages | bytes -----------+-------------+----------+------- t1_id_seq | 189575 | 1 | 8192
  • 19. Recovering From A Damaged PostgreSQL Cluster Example Invocation Scenario: Rebuild Data Cluster with Corrupted Index Solution: Rebuild index reindex index t1_pkey; BEFORE relname | relfilenode | relpages | bytes -----------+-------------+----------+------- t1_pkey | 24632 | 5 | 40960 AFTER relname | relfilenode | relpages | bytes -----------+-------------+----------+------- t1_pkey | 189578 | 5 | 40960
  • 20. Recovering From A Damaged PostgreSQL Cluster Example Invocation Scenario: Rebuild Data Cluster with Corrupted System Indexes Solution: Rebuild indexes under single-user mode - start up the damaged cluster and reindex with the following invocation postgres --single postgres -D ~/data <<_eof_ reindex system mydatabase _eof_
  • 21. Recovering From A Damaged PostgreSQL Cluster About Single UserMode - Used during bootstrapping by initdb - Can be used for debugging - Can be used for disaster recovery i.e. System indexes can be manipulated - Can enter queries and the results are printed to the screen
  • 22. Recovering From A Damaged PostgreSQL Cluster About Single UserMode postgres --single postgres -D ~/data temp PostgreSQL stand-alone backend 9.2.3 backend> select * from t1 limit 5 1: id (typeid = 23, len = 4, typmod = -1, byval = t) 2: x (typeid = 700, len = 4, typmod = -1, byval = t) ---- 1: id = "1" (typeid = 23, len = 4, typmod = -1, byval = t) 2: x = "0.767757" (typeid = 700, len = 4, typmod = -1, byval = t) ---- 1: id = "2" (typeid = 23, len = 4, typmod = -1, byval = t) 2: x = "0.253584" (typeid = 700, len = 4, typmod = -1, byval = t) ---- 1: id = "3" (typeid = 23, len = 4, typmod = -1, byval = t) 2: x = "0.0912474" (typeid = 700, len = 4, typmod = -1, byval = t) ---- 1: id = "4" (typeid = 23, len = 4, typmod = -1, byval = t) 2: x = "0.45276" (typeid = 700, len = 4, typmod = -1, byval = t) ---- 1: id = "5" (typeid = 23, len = 4, typmod = -1, byval = t) 2: x = "0.784381" (typeid = 700, len = 4, typmod = -1, byval = t) ---- backend>
  • 23. Recovering From A Damaged PostgreSQL Cluster Example Invocation Scenario: PostgreSQL Starts Up But Queries To User-Defined Tables Fails Solution: Generate a data-dump into a newly created data cluster - start up the damaged cluster with the following invocation pg_ctl -D ~/data -o '-c zero_damaged_pages=true' -l logfile.txt start - execute standard data dump with pg_dumpall
  • 24. Recovering From A Damaged PostgreSQL Cluster Example Invocation Scenario: Rebuild And Reuse An Existing Data Cluster With Corrupted User-Defined Tables Solution: Rebuild Cluster - start up the damaged cluster with the following invocation pg_ctl -D ~/data -o '-c zero_damaged_pages=true' -l logfile.txt start - login the database and execute the following SQL statement VACUUM FULL ANALYZE; REINDEX DATABASE mydatabase; - restart the newly repaired cluster pg_ctl -D ~/data -l logfile.txt -m smart restart
  • 25. Recovering From A Damaged PostgreSQL Cluster Caveats ● DROP and CREATE damaged indexes ● Where possible, recreate the cluster on another machine... don't take a chance ● Executing a dump should be the first action i.e. don't VACUUM damaged tables as you can actually lose data before you reuse/backup/dump ● Damaged system catalogs must be recovered in single user mode ● Tables larger 1GB+ are split ● TOASTED tables add an entirely new level of complexity
  • 26. Recovering From A Damaged PostgreSQL Cluster One last word Hacking the cluster...
  • 27. Recovering From A Damaged PostgreSQL Cluster References - http://www.postgresql.org/docs/9.2/static/index.html - http://www.postgresql.org/docs/9.2/static/app-initdb.html - http://www.postgresql.org/docs/9.2/static/app-pg-ctl.html - http://www.postgresql.org/docs/9.2/static/catalog-pg-class.html - http://www.postgresql.org/docs/9.2/static/runtime-config-developer.html - http://www.postgresql.org/docs/9.2/static/sql-createindex.html - http://www.postgresql.org/docs/9.2/static/sql-dropindex.html - http://www.postgresql.org/docs/9.2/static/sql-reindex.html - http://www.postgresql.org/docs/9.2/static/sql-vacuum.html - http://linux.die.net/man/1/find - http://linux.die.net/man/1/killall
  • 28. Recovering From A Damaged PostgreSQL Cluster Asurion Mobile Applications http://www.asurion.com/ Robert Bernier Sr. Database Administrator, Asurion Mobile Applications robert.bernier@asurion.com 1400 Fashion Island Blvd, San Mateo, CA 94404