7 Ways To Crash Postgres

8,415 views
7,817 views

Published on

Short presentation giving the seven most popular ways to crash PostgreSQL

Published in: Technology

7 Ways To Crash Postgres

  1. 1. 7 ways tocrashPostgres Josh Berkus PostgreSQL Project LinuxConf.AU 2013
  2. 2. Dont1 Apply Updates
  3. 3. 1. ignore update announcements2. keep running version X.X.0 or X.X.13. wait for disaster to strike
  4. 4. 9. 2. 2
  5. 5. 9. 2. 3
  6. 6. 9.2.2:Prevent database-wide crash andrestart when a new child processis unable to create a pipe for itslatch.9.1.6:Fix misbehavior whendefault_transaction_isolation isset to serializable. Symptomsinclude crashes at process starton Windows, and crashes in hotstandby operation.
  7. 7. Upgrade!EOL after 5 years
  8. 8. 8.3 is now EOL
  9. 9. Out of2 Disk Space
  10. 10. 1. install DB on volume which is 80% full2. dont monitor disk space usage3. let database grow
  11. 11. ERROR: could not writeblock 31840050 oftemporary file: No spaceleft on devicePANIC: could not writeto file"pg_xlog/xlogtemp.49680": No space left ondevice
  12. 12. Out of2a Disk Space Archiving Edition
  13. 13. 1. set up transaction log archiving to a standby2. stop file use/deletion on the standby3. dont monitor
  14. 14. 4. disk on the standby fills up with archive logs5. disk on the master fills up with archive logs6. failover impossible
  15. 15. freeing up spacetakes more space● VACUUM FULL● REINDEX● DELETE● pg_reorg
  16. 16. DROP TABLE;DROP DATABASE;
  17. 17. 3 Deleting Stuff
  18. 18. base           pg_snapshotscold           pg_stat_tmpglobal         pg_subtranshot            pg_tblspcpg_clog        pg_twophasepg_hba.conf    PG_VERSIONpg_ident.conf  pg_xlogpg_log         postgresql.confpg_multixact  pg_notify      postmaster.optspg_serial
  19. 19. base           pg_snapshotscold           pg_stat_tmpglobal         pg_subtranshot            pg_tblspcpg_clog        pg_twophasepg_hba.conf    PG_VERSIONpg_ident.conf  pg_xlogpg_log         postgresql.confpg_multixact  pg_notify      postmaster.optspg_serial
  20. 20. 2012-09-25 08:27:03 UTCERROR: could not accessstatus of transaction5001859032012-09-25 08:27:03 UTCDETAIL: Could not open file"pg_clog/01DD": No such fileor directory.
  21. 21. 4 Out of Memory
  22. 22. 1. set shared_buffers to 80% of RAM2. set work_mem to 1GB3. ramp up to 1000 connections
  23. 23. 1. run Postgres on cloud server2. add a JVM3. add Apache/mod_php4. add MySQL5. add a second Postgres version
  24. 24. 0. run server with default Linux memory management settings
  25. 25. kernel: postgres invokedoom-killer: gfp_mask=0x201d2,order=0, oomkilladj=0LOG: background writerprocess (PID 22999) wasterminated by signal 9:KilledLOG: terminating any otheractive server processes
  26. 26. 5 Bad Hardware
  27. 27. 1. buy brand-new hardware2. install PostgreSQL3. dont test4. deploy in production
  28. 28. 1. deploy production hardware2. dont monitor SMART3. dont monitor syslog4. run for a couple years
  29. 29. ERROR: could not read block1 in file "base/18511/13872":read only 0 of 8192 bytesFATAL: failed to re-findparent key in index"257969064" for split pages8366/12375
  30. 30. Too6 Many Connections
  31. 31. 1. build application with ad-hoc DB connections2. no connection pooling3. program auto-reconnect on timeout4. raise max_connections to 2000
  32. 32. CRITICAL: server postgres2 isnot accepting connections
  33. 33. 7 Zombie Locks
  34. 34. 1. program application so it forgets to close transactions2. run for a few hours or days3. watch the locks pile up
  35. 35. DBI ERROR timeout after300 seconds: "UPDATEsessions SET last_seen =now()WHERE session_key =a16g32b12cd0;"
  36. 36. 1. no updates2. out of disk space3. deleting stuff4. out of RAM5. bad hardware6. too many connections7. zombie locks
  37. 37. more elephants● Postgres Pizza BOF: – Wed Night, 8:30PM, B&G Snooker Hall th● pgDay Melbourne: Feb 5 – 2013.pgday.org.au● pgCon May 21-24 Ottawa● me: www.databasesoup.com – josh@postgresql.org This talk is copyright 2013 Josh Berkus, and is licensed under the Creative Commons Attribution License. Most images are the property of their respective copyright holders, and are used here for purposes of satire.

×