Researching    PostgreSQL Performance      Fernando Ike de Oliveira                                 PGCon 2008            ...
Researching    PostgreSQL Performancehttp://www.midstorm.org/~fike/researching_postgresql.pdf                            
Questions...●   "What version do we use? 8.2 or 8.3? (2007­12)"●     PostgreSQL 8.3 was a beta version●     PostgreSQL 8.3...
Hardware●    2 Server DELL PowerEdge 6850, 16GB RAM, 4 dual­core     processors ●    Storage DELL Clarion of 1TB ●    3 RA...
Tests●   fixed connections = 100●   scaling factor = 100 ●   transactions = 100, 100, 1000000                             ...
Tests●    ~ 340 tests●    initial test with pgbench ●    DBT­2 and pgbench dont attended expected ●    Euler developed/dev...
pgtesttool●   automatic install postgresql and pgbench●   automatic data collection for server during test●   local and re...
     
     
     
     
     
OS Tuningecho "8589934592" > /proc/sys/kernel/shmmax echo "deadline" > /sys/block/sdX/queue/schedulerecho "250 128000 32 2...
/etc/security/limits.confpostgres           soft    nofile           63536postgres           soft    nproc            2047...
basic postgresql.conflisten_addresses = *max_connections = 110max_fsm_pages = 204800effective_cache_size = 10GB           ...
shared_buffers = 4GB (25%)1   -   618.4320 tps, 8.2 w/ default conf2   - 3,822.1502 tps, 8.2 w/ shared_buffers = 4GB3   - ...
4500400035003000                                 8.2 default2500                                 8.2 ­ 4GB2000            ...
fdatasync and opensync            100t1 - 3813.2723 tps, 8.2 w/ opensync2 - 4057.7641 tps, 8.3 w/ fdatasync               ...
41004050400039503900                     8.2 opensync                      8.2 fdatasync38503800375037003650           tps...
autovacuum1   -   618.4320 tps, 8.2 default conf2   -   650.4415 tps, 8.3 default conf, autovac off3   -   660.3667 tps, 8...
500045004000                8.2 default3500                8.3 default avoff                8.3 default avon3000          ...
wal_delay 1,000t1 - 4878.7961 tps, 8.3 better conf, wal_delay = 5002 - 4688.9773 tps, 8.3 better conf, default wal_delay  ...
490048504800               8.3 wal_delay = 4750               500               8.3 wal_delay                default470046...
synchronous_commit           100t1 - 4515.4117 tps, 8.3 synchronous_commit = on2 - 3972.6742 tps, 8.3 synchronous_commit =...
4600450044004300               synchronous_co               mmit on4200               synchronous_co               mmit = ...
wal_method test1 ­ 3915.7527 tps  8.2 w/ opensync  ­ 100t2 ­ 4057.7641 tps  8.2 w/ fdatasync ­ 100t3 ­ 3918.6068 tps  8.2 ...
46004500               8.2 opensync 4400               100t4300               8.2 fdatasync                100t4200       ...
Comparison 8.2 X 8.3      and Filesystem1 ­ 4345.9916 tps  8.2 w/ opensync, ext3 (writeback) ­ 1,000t2 ­ 4470.5250 tps  8....
4800470046004500               8.2 ext3                (writeback)               8.2 xfs4400               8.3 xfs43004200...
8.2 X 8.3 100000t1 - 4839.7800 tps 8.3 100000t2 - 971.7439 tps 8.2 100000t                   1 p/ 2 -398%                 ...
600050004000               8.3 10000t3000               8.2 10000t20001000   0            
autovacuum delay1   -   4821.9402   tps,   8.3   w/   default autovacuum_cost_delay2   -   4816.5119   tps,   8.3   w/   a...
48604850               av_cost_delay                default4840           av_cost_delay                200               a...
Checkpoints 1000t1 - 4774.1278 tps, 8.3 w/ checkpoints_segments = 256,  checkpoint_complementation_target = 052 - 4818.983...
483048204810           Segments 256,                target 54800           Segments 128,                target 74790      ...
Bgwriter, isolated test         (s= 30; t/c = 2000; c = 20)8070                     bgwrite_delay 10    bgwrite_delay 300 ...
Data not registered but     known by us● magic numbers to max_connections- 700 or more?- 1500 connections decrease 3%- 300...
Conclusions of the         tests.●   Ext3 (writeback) ­ better in few transactions●   XFS ­ better in many transactions●  ...
Still to be tested●   Filesystems: Ext4, JFS, Reiserfs●   Scheduling I/O: Anticipatory, Deadline, CFQ, Noop●   Linux Kerne...
Contacts and references    Euler Taveira de Oliveira - euler@timbira.com    Fernando Ike de Oliveira - fike@4linux.com.br ...
Upcoming SlideShare
Loading in...5
×

Researching postgresql

608

Published on

PC

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
608
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
9
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Researching postgresql

  1. 1. Researching PostgreSQL Performance Fernando Ike de Oliveira PGCon 2008   
  2. 2. Researching PostgreSQL Performancehttp://www.midstorm.org/~fike/researching_postgresql.pdf   
  3. 3. Questions...● "What version do we use? 8.2 or 8.3? (2007­12)"●   PostgreSQL 8.3 was a beta version●   PostgreSQL 8.3 was promising to be faster than PostgreSQL 8.2●   PostgreSQL 8.2 doesnt scale well in a high number of transactions   
  4. 4. Hardware●  2 Server DELL PowerEdge 6850, 16GB RAM, 4 dual­core  processors ●  Storage DELL Clarion of 1TB ●  3 RAID 5 (data, index and wal)●  Operation System: Debian Etch 4.0 AMD64.   
  5. 5. Tests● fixed connections = 100● scaling factor = 100 ● transactions = 100, 100, 1000000   
  6. 6. Tests●  ~ 340 tests●  initial test with pgbench ●  DBT­2 and pgbench dont attended expected ●  Euler developed/developing pgtesttool   
  7. 7. pgtesttool● automatic install postgresql and pgbench● automatic data collection for server during test● local and remote pgbench execution● many postgresql.conf tests in one round● generation of graphic results (OS statistics, bwgriter, checkpoints,  database blocks, lock contention, pgbench results, shared buffers)   
  8. 8.    
  9. 9.    
  10. 10.    
  11. 11.    
  12. 12.    
  13. 13. OS Tuningecho "8589934592" > /proc/sys/kernel/shmmax echo "deadline" > /sys/block/sdX/queue/schedulerecho "250 128000 32 256" > /proc/sys/kernel/sem echo "2" > /proc/sys/vm/overcommit_memory echo "16777216" > /proc/sys/net/core/rmem_default echo "16777216" > /proc/sys/net/core/wmem_default echo "16777216" > /proc/sys/net/core/wmem_max echo "16777216" > /proc/sys/net/core/rmem_max     
  14. 14. /etc/security/limits.confpostgres           soft    nofile           63536postgres           soft    nproc            2047postgres           hard    nofile          63536postgres           hard    nproc          16384   
  15. 15. basic postgresql.conflisten_addresses = *max_connections = 110max_fsm_pages = 204800effective_cache_size = 10GB   
  16. 16. shared_buffers = 4GB (25%)1 - 618.4320 tps, 8.2 w/ default conf2 - 3,822.1502 tps, 8.2 w/ shared_buffers = 4GB3 - 660.3667 tps, 8.3 w/ default conf4 - 4,259.6078 tps, 8.3 w/ shared_buffers = 4GB, (full_page_write = off, wal_sync_method = open_sync). 1 p/ 2 - 518,03% 1 p/ 3 - 6.78% 3 p/ 4 - 545.03%   
  17. 17. 4500400035003000  8.2 default2500  8.2 ­ 4GB2000  8.3 default  8.3  ­ 4GB15001000 500 0 shared_buffers    
  18. 18. fdatasync and opensync 100t1 - 3813.2723 tps, 8.2 w/ opensync2 - 4057.7641 tps, 8.3 w/ fdatasync 1 p/ 2 - 6,41%   
  19. 19. 41004050400039503900 8.2 opensync  8.2 fdatasync38503800375037003650 tps    
  20. 20. autovacuum1 - 618.4320 tps, 8.2 default conf2 - 650.4415 tps, 8.3 default conf, autovac off3 - 660.3667 tps, 8.3 default conf, autovac on4 - 4515.4117 tps, 8.3 better conf, (fullpgwrite off), autovac off5 - 4259.6078 tps, 8.3 better conf, (fullpgwrite off), autovac on 1 p/ 2 - 5,17% 1 p/ 3 - 6,78% 2 p/ 3 - 1,52% 4 p/ 5 - 6,00%   
  21. 21. 500045004000  8.2 default3500  8.3 default avoff  8.3 default avon3000  8.3 best avoff 2500  8.3 best avon200015001000 500 0    
  22. 22. wal_delay 1,000t1 - 4878.7961 tps, 8.3 better conf, wal_delay = 5002 - 4688.9773 tps, 8.3 better conf, default wal_delay 1 p/ 2 - 4.04%   
  23. 23. 490048504800 8.3 wal_delay = 4750 500 8.3 wal_delay  default4700465046004550    
  24. 24. synchronous_commit 100t1 - 4515.4117 tps, 8.3 synchronous_commit = on2 - 3972.6742 tps, 8.3 synchronous_commit = off [1] p/ [2] - 13,66%   
  25. 25. 4600450044004300 synchronous_co mmit on4200 synchronous_co mmit = off41004000390038003700    
  26. 26. wal_method test1 ­ 3915.7527 tps  8.2 w/ opensync  ­ 100t2 ­ 4057.7641 tps  8.2 w/ fdatasync ­ 100t3 ­ 3918.6068 tps  8.2 w/ fdatasync ­ 1,000t4 ­ 4470.5250 tps  8.2 w/ opensync  ­ 1,000t [1] p/ [2] ­ 3,62%   
  27. 27. 46004500 8.2 opensync 4400 100t4300 8.2 fdatasync  100t4200 8.2 fdatasync  1000t 8.2 opensync 41004000 1000t3900380037003600    
  28. 28. Comparison 8.2 X 8.3 and Filesystem1 ­ 4345.9916 tps  8.2 w/ opensync, ext3 (writeback) ­ 1,000t2 ­ 4470.5250 tps  8.2 w/ opensync, xfs              ­ 1,000t3 ­ 4688.9773 tps  8.3 w/ opensync, xfs              ­ 1,000t 1 p/ 2 ­ 2,86% 2 p/ 3 ­ 4,88%   
  29. 29. 4800470046004500 8.2 ext3  (writeback) 8.2 xfs4400 8.3 xfs430042004100    
  30. 30. 8.2 X 8.3 100000t1 - 4839.7800 tps 8.3 100000t2 - 971.7439 tps 8.2 100000t 1 p/ 2 -398%   
  31. 31. 600050004000 8.3 10000t3000 8.2 10000t20001000 0    
  32. 32. autovacuum delay1 - 4821.9402 tps, 8.3 w/ default autovacuum_cost_delay2 - 4816.5119 tps, 8.3 w/ autovacuum_cost_delay = 2003 - 4831.7244 tps, 8.3 w/ autovacuum_cost_delay = 104 - 4845.7629 tps, 8.3 w/ autovacuum_cost_delay = 405 - 4848.5798 tps, 8.3 w/ autovacuum_cost_delay = 100 1 p/ 2 - -0.11% 1 p/ 3 - 0.20% 1 p/ 4 - 0.49% 1 p/ 5 - 0.55%   
  33. 33. 48604850 av_cost_delay  default4840 av_cost_delay  200 av_cost_delay 4830 10 av_cost_delay  404820 av_cost_delay  10048104800    
  34. 34. Checkpoints 1000t1 - 4774.1278 tps, 8.3 w/ checkpoints_segments = 256, checkpoint_complementation_target = 052 - 4818.9832 tps, 8.3 w/ checkpoints_segments = 128, checkpoint_complementation_target = 073 - 4821.9402 tps, 8.3 w/ checkpoints_segments = 256, checkpoint_complementation_target = 07 1 p/ 2 - 0.93% 1 p/ 3 - 1.00%   
  35. 35. 483048204810 Segments 256,  target 54800 Segments 128,  target 74790 Segments 256,  target 74780477047604750    
  36. 36. Bgwriter, isolated test (s= 30; t/c = 2000; c = 20)8070 bgwrite_delay 10 bgwrite_delay 300 bgwrite_delay 100 bgwrite_delay 5060 bgwrite_delay  bgwrite_delay 500 100050 bgwrite_delay  bgwrite_delay  100000 500040 bgwrite_delay 20 bgwrite_delay 800 bgwrite_delay 30 20002010     0
  37. 37. Data not registered but known by us● magic numbers to max_connections- 700 or more?- 1500 connections decrease 3%- 3000 connections decrease 5%● opensync is better with pgbench transaction● fdatasync is better with load tables of the pgbench● full_page_write = off is more than faster● deadline I/O scheduler > CFQ   
  38. 38. Conclusions of the tests.● Ext3 (writeback) ­ better in few transactions● XFS ­ better in many transactions● synchronous_commit disabled decrease performance● to turn off autovacuum in the PostgreSQL 8.3 is a bad idea● PostgreSQL 8.3 has better performance compared to PostgreSQL 8.2.   
  39. 39. Still to be tested● Filesystems: Ext4, JFS, Reiserfs● Scheduling I/O: Anticipatory, Deadline, CFQ, Noop● Linux Kernel 2.6.23 >= Completely fair scheduler● OS: FreeBSD, OpenSolaris...● Tests with TPC­C, TPC­H, TPC­E● RAID 10 X RAID 5   
  40. 40. Contacts and references Euler Taveira de Oliveira - euler@timbira.com Fernando Ike de Oliveira - fike@4linux.com.br fike@midstorm.org Result tests: - http://www.inf.ufrgs.br/~etoliveira/pg/resultados/ pgtesttool: PGCon Brasil 2008: http://pgcon.postgresql.org.br   
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×