Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Postgre sql best_practices

761 views

Published on

Overview from the initial setup to an OLTP performance benchmark against Oracle

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Postgre sql best_practices

  1. 1. BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH PostgreSQL Best Practices Overview from the initial setup to an OLTP performance benchmark against Oracle. Emiliano Fusaglia Principal Consultant Jacques Kostic Principal Consultant
  2. 2. 2018 © Trivadis Exadata X7-2 POC with OVM 2 TechEvent September 2018 Specialties: • Database Cloud computing (DBaaS) • Oracle RAC • Grid Infrastructure (CRS, ASM) • Data Guard • Instance and SQL Performance & Tuning • Linux & Virtualization Certifications: • Oracle Certified Professional 9i, 10g, 11g & 12c • Oracle Exadata Administrator X3 –X4 Certified Expert Teaching Courses at Trivadis: • Oracle 11g & 12c Grid Infrastructure & RAC • Oracle 11g & 12c Data Guard • Oracle Exadata • Oracle 12c New Features About me… @EFusaglia
  3. 3. PostgreSQL Best Practices3 14/09/2018 Experience: • Initially C/C++ developer • In touch with Oracle since 1990 from version 4 on SCO Unix! • High Availability and Backup & Recovery Architect • SQL and Instance Performance & Tuning • License Audit and Consolidation Certifications: • Oracle Certified Master 11g & 12c • Oracle 11g Performance Tuning Certified Expert • Oracle RAC 11g and Grid Infrastructure Administration • Oracle Exadata Administrator Certified Expert • Oracle Certified SQL Expert 11g Teaching Courses at Trivadis: • Oracle 11g & 12c Grid Infrastructure & RAC • Oracle 11g & 12c Data Guard • Oracle 11g & 12c Performance & Tuning • Oracle 11g & 12c Administration • SQL & PL-SQL • OEM – 12 & 13 About me… @JKOFR
  4. 4. Agenda PostgreSQL Best Practices9/14/2018 1. PostgreSQL Introduction & Architecture OS Requirements Installation Options Securing PostgreSQL ClusterDB Main parameters to configure Backup and Recovery 2. OLTP performance benchmark PostgreSQL vs Oracle Configuration Results 3. Conclusion Takeaway 4
  5. 5. PostgreSQL Best Practices9/14/2018 PostgreSQL Introduction & Architecture 5
  6. 6. Introduction to PostgreSQL PostgreSQL is an opensource and independent Object-RDBMS developed and maintained by the PostgreSQL Global Development Group. The first version was released in 1996 as INGRES development, and it included support for Object orientation and SQL. Main Characteristics: ACID (Atomicity, Consistency, Isolation, Durability) Multiversion concurrency control (MVCC) Foreign keys, Indexes, Views, Trigger, Functions, Procedural Languages (PL), etc.. Streaming Replication (as of 9.0) Hot Standby (as of 9.0) PostgreSQL Best Practices9/14/20186
  7. 7. PostgreSQL Architecture PostgreSQL Best Practices9/14/20187 pg_crl start Source PostgreSQL documentation
  8. 8. Database Cluster PostgreSQL Best Practices9/14/20188 Source PostgreSQL documentation A cluster is an instance of postgreSQL containing one or many databases – Conceptually similar to MySQL, SQL Server and Oracle Pluggable Databases Server Cluster pgclu01 (port 5438) Cluster pgclu02 (port 5439) postgres template0 template1 postgres template0 template1 ecom01 erp01 sales01 dwh01 hr01 supp01
  9. 9. Database Cluster PostgreSQL Best Practices9/14/20189 postgres template0 template1 System or Master database, it contains system tables, views, procedures, metadata, user and role definitions. Template0 it is a read-only empty database used as seed database. Template1 it is a read-write database, which allows customizations before to be used as default seed database. App Application Database it contains application objects like tables, indexes, views, procedures, constraints etc..
  10. 10. Before Image and Vacuum Process PostgreSQL has no rollback segments, and it guarantees read consistency in the following way: Writing new image in a new location Marking the initial image as OLD, keeping intact the data. Adding a pointer to the OLD image pointing the new one. PostgreSQL Best Practices9/14/201810
  11. 11. Before Image and Vacuum Process PostgreSQL Best Practices9/14/201811 Page x Case 1 The new image remains on the same page. 1,’blue’ 2,’green’ 0,’red’ Case 2 The new image migrates on a new page. 1,’white’ NEW OLD UPDATE app_tab SET col2=‘white’ WHERE col0=1; Page x Page y 1,’bb’,test4 OLD NEW 1,’blue’ 2,’green’ 0,’red’ 1,’white’ UPDATE app_tab SET col2=‘white’ WHERE col0=1;
  12. 12. Before Image and Vacuum Process PostgreSQL Best Practices9/14/201812 VACUUM Process Reclaims space occupied by old tuple images Updates data statistics used by the query planner Updates the visibility map Resets the transaction ID of old blocks to prevent wraparound The standard VACUUM is executed regularly by default Manual VACUUMing is possible
  13. 13. PostgreSQL Best Practices9/14/2018 Hands on set up 13
  14. 14. OS Optimization 1/2 PostgreSQL14 14/09/2018 Kernel optimization – /etc/sysctl.d/90-postgres-sysctl.conf fs.file-max = 6815744 fs.aio-max-nr = 1048576 kernel.sem = 250 32768 1000 128 kernel.shmall = 1073741824 kernel.shmmni = 4096 kernel.shmmax = 4398046511104 #Huge Pages 80GB vm.nr_hugepages= 40960 vm.min_free_kbytes=524288
  15. 15. OS Optimization 2/2 PostgreSQL15 14/09/2018 – /etc/security/limits.d/postgres-limits.conf postgres soft nofile 16384 postgres hard nofile 16384 postgres soft memlock 83886080 postgres hard memlock 83886080 Storage – Binaries, ClusterDB, External Tablespaces and WAL files on T1 Storage – Backups on T2 Storage
  16. 16. Installation Options List of Supported Platforms: Linux (all recent distributions), Windows (Win2000 SP4 and later), FreeBSD, OpenBSD, NetBSD, Mac OS X, AIX, HP/UX, IRIX, Solaris, Tru64 Unix, and UnixWare. P.S.: this presentation focuses on Linux 64-bit PostgreSQL can be installed using one of the following method : Source Code Binary Package Binaries Archive without installer (Advanced users) TVD Recommended Option PostgreSQL Best Practices9/14/201816
  17. 17. Securing PostgreSQL ClusterDB After the installation one of the first tasks to perform is securing the local and remote database connections, defining the open ports and the authentication methods. Those settings can be defined in two different configuration files: postgresql.conf section CONNECTIONS AND AUTHENTICATION pg_hba.conf PostgreSQL Best Practices9/14/201817
  18. 18. Securing PostgreSQL ClusterDB - postgresql.conf 1/2 PostgreSQL Best Practices9/14/201818 #------------------------------------------------------------------------------ # CONNECTIONS AND AUTHENTICATION #------------------------------------------------------------------------------ # - Connection Settings - listen_addresses = '192.168.1.129,localhost' # what IP address(es) to listen on; # comma-separated list of addresses; # defaults to 'localhost'; use '*' for all # (change requires restart) port = 5544 # (change requires restart) max_connections = 100 # (change requires restart) #superuser_reserved_connections = 3 # (change requires restart) #unix_socket_directories = '/tmp' # comma-separated list of directories # (change requires restart) #unix_socket_group = '' # (change requires restart) unix_socket_permissions = 0770 # begin with 0 to use octal notation # (change requires restart) #bonjour = off # advertise server via Bonjour # (change requires restart) #bonjour_name = '' # defaults to the computer name # (change requires restart) # - Security and Authentication - #authentication_timeout = 1min # 1s-600s #ssl = off #ssl_ciphers = 'HIGH:MEDIUM:+3DES:!aNULL' # allowed SSL ciphers #ssl_prefer_server_ciphers = on #ssl_ecdh_curve = 'prime256v1' #ssl_dh_params_file = '' #ssl_cert_file = 'server.crt' ...
  19. 19. Securing PostgreSQL ClusterDB - postgresql.conf 2/2 PostgreSQL Best Practices9/14/201819 # - Security and Authentication - #authentication_timeout = 1min # 1s-600s #ssl = off #ssl_ciphers = 'HIGH:MEDIUM:+3DES:!aNULL' # allowed SSL ciphers #ssl_prefer_server_ciphers = on #ssl_ecdh_curve = 'prime256v1' #ssl_dh_params_file = '' #ssl_cert_file = 'server.crt' #ssl_key_file = 'server.key' #ssl_ca_file = '' #ssl_crl_file = ‚‘ password_encryption = scram-sha-256 # md5 or scram-sha-256 #db_user_namespace = off #row_security = on # GSSAPI using Kerberos #krb_server_keyfile = '' #krb_caseins_users = off # - TCP Keepalives - # see "man 7 tcp" for details #tcp_keepalives_idle = 0 # TCP_KEEPIDLE, in seconds; # 0 selects the system default #tcp_keepalives_interval = 0 # TCP_KEEPINTVL, in seconds; # 0 selects the system default #tcp_keepalives_count = 0 # TCP_KEEPCNT; # 0 selects the system default ...
  20. 20. Securing PostgreSQL ClusterDB - pg_hba.conf PostgreSQL Best Practices9/14/201820 # TYPE DATABASE USER ADDRESS METHOD # "local" is for Unix domain socket connections only local all all trust # IPv4 local connections: # host all all 127.0.0.1/32 trust host all all 192.168.1.1/24 trust # IPv6 local connections: # host all all ::1/128 trust # Allow replication connections from localhost, by a user with the # replication privilege. # local replication all trust # host replication all 127.0.0.1/32 trust # host replication all ::1/128 trust
  21. 21. Main parameters to configure - 1/2 PostgreSQL Best Practices9/14/201821 postgres=# select name,setting from pg_file_settings; name settings ------------------------------------------------------------------------------------------------ external_pid_file | extra_pid.info listen_addresses | 192.168.1.129,localhost port | 5544 max_connections | 100 unix_socket_permissions | 0770 password_encryption | scram-sha-256 shared_buffers | 4096MB huge_pages | on max_stack_depth | 6MB dynamic_shared_memory_type | posix effective_io_concurrency | 80 max_worker_processes | 150 max_parallel_workers | 24 wal_level | replica wal_compression | on archive_mode | on archive_command | test ! -f /home/postgres01/backup_dir/clusterTEST/archives/%f && cp %p /home/postgres01/backup_dir/clusterTEST/archives/%f archive_timeout | 14400 log_destination | stderr,syslog logging_collector | on log_directory | /home/postgres01/clusterTEST/logs
  22. 22. Main parameters to configure - 2/2 PostgreSQL Best Practices9/14/201822 name settings ------------------------------------------------------------------------------------------------ log_filename | alert_cluster_test-%Y-%m-%d_%H%M%S.log log_file_mode | 0600 log_rotation_age | 30d log_rotation_size | 100MB syslog_facility | LOCAL0 syslog_ident | postgres_cluster_test log_lock_waits | on log_timezone | Europe/Vaduz cluster_name | cluster_test log_autovacuum_min_duration | 0 default_tablespace | TS_data01 datestyle | iso, mdy timezone | Europe/Vaduz extra_float_digits | 3 lc_messages | en_US.UTF-8 lc_monetary | en_US.UTF-8 lc_numeric | en_US.UTF-8 lc_time | en_US.UTF-8 default_text_search_config | pg_catalog.english (40 rows) postgres=#
  23. 23. Backup and Recovery PostgreSQL provides the following options regarding the backup/recovery strategy: Logical Backup – Single database dump pg_dump – Cluster database dump pg_dumpall Physical Backup – File System Level Backup – Continuous Archiving and Point-in-Time Recovery (PITR) PostgreSQL Best Practices9/14/201823
  24. 24. Backup and Recovery - Logical Backup PostgreSQL provides the following options regarding the backup/recovery strategy: Logical Backup pg_dump create a TEXT file that can be restored by psql Restore Dump File on database dbtest01_restore PostgreSQL Best Practices9/14/201824 $ pg_dump dbtest01 > /backup_dir/dbtest01_dump_20180716.dmp $ psql --set ON_ERROR_STOP=on dbtest01_restore < /backup_dir/dbtest01_dump_20180716.dmp
  25. 25. Backup and Recovery - Physical Backup PostgreSQL provides the following options regarding the backup/recovery strategy: Physical Backup Continuous Archiving and Point-in-Time Recovery (PITR) PostgreSQL Best Practices9/14/201825 #!/bin/bash db_cluster_base="/u01/PosgreSQL" db_cluster_dir="/u01/PosgreSQL/clusterTEST" bckup_start=$(date +"%Y%m%d%H%M") logfile="/u01/PosgreSQL/backup_dir/logs/clusterTEST_backup_$bckup_start.log" backup_dir="/u01/PosgreSQL/backup_dir/clusterTEST" bck_label="Start_Backup_$bckup_start“ ...
  26. 26. Backup and Recovery - Physical Backup PostgreSQL Best Practices9/14/201826 ... psql postgres -L $logfile << EOF SELECT pg_start_backup('$bck_label', false, false); ! tar zcvf $backup_dir/backup_$bckup_start.tar.gz --warning=no-file-changed --warning=no-file-removed -C /u01/PosgreSQL clusterTEST --exclude='pg_wal/*‘ --exclude='postmaster*' --exclude='pg_replslot/*' SELECT * FROM pg_stop_backup(false, true); q EOF
  27. 27. 9/14/2018 OLTP Performance Benchmark PostgreSQL vs Oracle Oracle DBaaS27
  28. 28. OLTP Test: PostGreSQL vs Oracle PostgreSQL Best Practices9/14/201828 Goal Use the same type of machine Test the same OLTP workload on both databases Test different CPU allocation Compare the results
  29. 29. OLTP Test: PostGreSQL vs Oracle: Configuration PostgreSQL Best Practices9/14/201829 Server details Main host: • 2 * 8 cores CPU E5-2680 0 @ 2.70GHz • OEL 7.2 • 192 GB • Flash Storage volumes on PCIe cards (no NVMe) VMs: • 8 vCPU • 8 GB RAM Concurrent sessions: • 100
  30. 30. OLTP Test: PostGreSQL vs Oracle : Configuration PostgreSQL Best Practices9/14/201830 Hammerdbcli PostGreSQL Test Setup dbset db pg diset tpcc pg_defaultdbase hammerdb loadscript vudestroy vuset delay 5 vuset repeat 5 vuset showoutput 1 vuset timestamps 1 vuset logtotemp 1 vuset vu 100 vucreate vurun
  31. 31. OLTP Test: PostGreSQL vs Oracle : Configuration PostgreSQL Best Practices9/14/201831 Hammerdbcli Oracle Test Setup dbset db ora diset connection instance jko loadscript vudestroy vuset delay 5 vuset repeat 5 vuset showoutput 1 vuset timestamps 1 vuset logtotemp 1 vuset vu 100 vucreate vurun
  32. 32. OLTP Test: PostGreSQL vs Oracle : Results PostgreSQL Best Practices9/14/201832 PostGreSQL Time to complete the full test  4.23 mn Average CPU Usage  88 % Transaction per minutes max  156’222
  33. 33. OLTP Test: PostGreSQL vs Oracle : Results PostgreSQL Best Practices9/14/201833 Oracle Time to complete the full test  Time 4.12 mn Average CPU Usage  74 % Transaction per minutes max  172’268
  34. 34. OLTP Test: PostGreSQL vs Oracle : Results PostgreSQL Best Practices9/14/201834 select sum(value) from v$sysstat where name = 'user commits' or name = 'user rollbacks' select sum(xact_commit + xact_rollback) from pg_stat_database
  35. 35. OLTP Test: PostGreSQL vs Oracle : Configuration PostgreSQL Best Practices9/14/201835 Lets Scale! Main host: • 2 * 8 cores CPU E5-2680 0 @ 2.70GHz • OEL 7.2 • 192 GB • Flash Storage volumes on PCIe cards (no NVMe) VMs: • 16 vCPU • 8 GB RAM Concurrent sessions: • 100
  36. 36. OLTP Test: PostGreSQL vs Oracle : Results PostgreSQL Best Practices9/14/201836 PostGreSQL Time to complete the full test  Time 3.27 mn Average CPU Usage  65 % Transaction per minutes max  194’904
  37. 37. OLTP Test: PostGreSQL vs Oracle : Results PostgreSQL Best Practices9/14/201837 Oracle Time to complete the full test  Time 3.16 mn Average CPU Usage  57 % Transaction per minutes max  251’292
  38. 38. OLTP Test: PostGreSQL vs Oracle : Results PostgreSQL Best Practices9/14/201838 select sum(value) from v$sysstat where name = 'user commits' or name = 'user rollbacks' select sum(xact_commit + xact_rollback) from pg_stat_database
  39. 39. OLTP Test: PostGreSQL vs Oracle : Results PostgreSQL Best Practices9/14/201839 8 vCPU 2.6% Faster 16% Less CPU 9.3% More TPM
  40. 40. OLTP Test: PostGreSQL vs Oracle : Results PostgreSQL Best Practices9/14/201840 16 vCPU 3.4% Faster 12.3% Less CPU 22.43% More TPM
  41. 41. 41 9/14/2018 Conclusion Oracle DBaaS
  42. 42. Conclusions PostgreSQL Best Practices9/14/201842  PostgreSQL is a mature, enterprise RDBMS  PostgreSQL offers good performances  PostgreSQL offers High Availability solution
  43. 43. Emilian Fusaglia, Principal Consultant Tel. +41-79-909 7213 Emiliano.Fusaglia@trivadis.com Jacques Kostic, Principal Consultant Tel. +41-79-909 7263 Jacques.Kostic@trivadis.com 9/14/201843 TechEvent September 2018
  44. 44. Session Feedback – now TechEvent September 201844 14.09.2018 Please use the Trivadis Events mobile app to give feedback on each session Use "My schedule" if you have registered for a session Otherwise use "Agenda" and the search function If the mobile app does not work (or if you have a Windows smartphone), use your smartphone browser – URL: http://trivadis.quickmobileplatform.eu/ – User name: <your_loginname> (such as "svv") – Password: sent by e-mail...

×