This document provides an overview of five steps to improve PostgreSQL performance: 1) hardware optimization, 2) operating system and filesystem tuning, 3) configuration of postgresql.conf parameters, 4) application design considerations, and 5) query tuning. The document discusses various techniques for each step such as selecting appropriate hardware components, spreading database files across multiple disks or arrays, adjusting memory and disk configuration parameters, designing schemas and queries efficiently, and leveraging caching strategies.
Best Practices for Becoming an Exceptional Postgres DBA EDB
Drawing from our teams who support hundreds of Postgres instances and production database systems for customers worldwide, this presentation provides real-real best practices from the nation's top DBAs. Learn top-notch monitoring and maintenance practices, get resource planning advice that can help prevent, resolve, or eliminate common issues, learning top database tuning tricks for increasing system performance and ultimately, gain greater insight into how to improve your effectiveness as a DBA.
This presentation covers all aspects of PostgreSQL administration, including installation, security, file structure, configuration, reporting, backup, daily maintenance, monitoring activity, disk space computations, and disaster recovery. It shows how to control host connectivity, configure the server, find the query being run by each session, and find the disk space used by each database.
Best Practices for Running PostgreSQL on AWS - DAT314 - re:Invent 2017Amazon Web Services
PostgreSQL is an open source database growing in popularity because of its rich features, vibrant community, and compatibility with commercial databases. Learn about ways to run PostgreSQL on AWS including self-managed, and the managed database services from AWS: Amazon Relational Database Service (Amazon RDS) and the Amazon Aurora PostgreSQL-compatible Edition. This talk covers key Amazon RDS for PostgreSQL functionality, availability, and management. We also review general guidelines for common user operations and activities such as migration, tuning, and monitoring for their RDS for PostgreSQL instances.
Spencer Christensen
There are many aspects to managing an RDBMS. Some of these are handled by an experienced DBA, but there are a good many things that any sys admin should be able to take care of if they know what to look for.
This presentation will cover basics of managing Postgres, including creating database clusters, overview of configuration, and logging. We will also look at tools to help monitor Postgres and keep an eye on what is going on. Some of the tools we will review are:
* pgtop
* pg_top
* pgfouine
* check_postgres.pl.
Check_postgres.pl is a great tool that can plug into your Nagios or Cacti monitoring systems, giving you even better visibility into your databases.
Best Practices for Becoming an Exceptional Postgres DBA EDB
Drawing from our teams who support hundreds of Postgres instances and production database systems for customers worldwide, this presentation provides real-real best practices from the nation's top DBAs. Learn top-notch monitoring and maintenance practices, get resource planning advice that can help prevent, resolve, or eliminate common issues, learning top database tuning tricks for increasing system performance and ultimately, gain greater insight into how to improve your effectiveness as a DBA.
This presentation covers all aspects of PostgreSQL administration, including installation, security, file structure, configuration, reporting, backup, daily maintenance, monitoring activity, disk space computations, and disaster recovery. It shows how to control host connectivity, configure the server, find the query being run by each session, and find the disk space used by each database.
Best Practices for Running PostgreSQL on AWS - DAT314 - re:Invent 2017Amazon Web Services
PostgreSQL is an open source database growing in popularity because of its rich features, vibrant community, and compatibility with commercial databases. Learn about ways to run PostgreSQL on AWS including self-managed, and the managed database services from AWS: Amazon Relational Database Service (Amazon RDS) and the Amazon Aurora PostgreSQL-compatible Edition. This talk covers key Amazon RDS for PostgreSQL functionality, availability, and management. We also review general guidelines for common user operations and activities such as migration, tuning, and monitoring for their RDS for PostgreSQL instances.
Spencer Christensen
There are many aspects to managing an RDBMS. Some of these are handled by an experienced DBA, but there are a good many things that any sys admin should be able to take care of if they know what to look for.
This presentation will cover basics of managing Postgres, including creating database clusters, overview of configuration, and logging. We will also look at tools to help monitor Postgres and keep an eye on what is going on. Some of the tools we will review are:
* pgtop
* pg_top
* pgfouine
* check_postgres.pl.
Check_postgres.pl is a great tool that can plug into your Nagios or Cacti monitoring systems, giving you even better visibility into your databases.
This ppt was used by Devrim at pgDay Asia 2017. He talked about some important facts about WAL - Transaction Logs or xlogs in PostgreSQL. Some of these can really come handy on a bad day
Tech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of FacebookThe Hive
This presentation describes the reasons why Facebook decided to build yet another key-value store, the vision and architecture of RocksDB and how it differs from other open source key-value stores. Dhruba describes some of the salient features in RocksDB that are needed for supporting embedded-storage deployments. He explains typical workloads that could be the primary use-cases for RocksDB. He also lays out the roadmap to make RocksDB the key-value store of choice for highly-multi-core processors and RAM-speed storage devices.
PostgreSQL is designed to be easily extensible. For this reason, extensions loaded into the database can function just like features that are built in. In this session, we will learn more about PostgreSQL extension framework, how are they built, look at some popular extensions, management of these extensions in your deployments.
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013mumrah
Apache Kafka is a new breed of messaging system built for the "big data" world. Coming out of LinkedIn (and donated to Apache), it is a distributed pub/sub system built in Scala. It has been an Apache TLP now for several months with the first Apache release imminent. Built for speed, scalability, and robustness, Kafka should definitely be one of the data tools you consider when designing distributed data-oriented applications.
The talk will cover a general overview of the project and technology, with some use cases, and a demo.
PostgreSQL is one of the most advanced relational databases. It offers superb replication capabilities. The most important features are: Streaming replication, Point-In-Time-Recovery, advanced monitoring, etc.
Webinar slides: An Introduction to Performance Monitoring for PostgreSQLSeveralnines
To operate PostgreSQL efficiently, you need to have insight into database performance and make sure it is at optimal levels.
With that in mind, we dive into monitoring PostgreSQL for performance in this webinar replay.
PostgreSQL offers many metrics through various status overviews and commands, but which ones really matter to you? How do you trend and alert on them? What is the meaning behind the metrics? And what are some of the most common causes for performance problems in production?
We discuss this and more in ordinary, plain DBA language. We also have a look at some of the tools available for PostgreSQL monitoring and trending; and we’ll show you how to leverage ClusterControl’s PostgreSQL metrics, dashboards, custom alerting and other features to track and optimize the performance of your system.
AGENDA
- PostgreSQL architecture overview
- Performance problems in production
- Common causes
- Key PostgreSQL metrics and their meaning
- Tuning for performance
- Performance monitoring tools
- Impact of monitoring on performance
- How to use ClusterControl to identify performance issues
- Demo
SPEAKER
Sebastian Insausti, Support Engineer at Severalnines, has loved technology since his childhood, when he did his first computer course (Windows 3.11). And from that moment he was decided on what his profession would be. He has since built up experience with MySQL, PostgreSQL, HAProxy, WAF (ModSecurity), Linux (RedHat, CentOS, OL, Ubuntu server), Monitoring (Nagios), Networking and Virtualization (VMWare, Proxmox, Hyper-V, RHEV).
Prior to joining Severalnines, Sebastian worked as a consultant to state companies in security, database replication and high availability scenarios. He’s also a speaker and has given a few talks locally on InnoDB Cluster and MySQL Enterprise together with an Oracle team. Previous to that, he worked for a Mexican company as chief of sysadmin department as well as for a local ISP (Internet Service Provider), where he managed customers' servers and connectivity.
This webinar builds upon a related blog post by Sebastian: https://severalnines.com/blog/performance-cheat-sheet-postgresql.
How to use histograms to get better performanceMariaDB plc
Sergei Petrunia and Varun Gupta, software engineers MariaDB, show how histograms can be used to improve query performance. They begin by introducing histrograms and explaining why they’re needed by the query optimizer. Next, they discuss how to determine whether or not histrograms are needed, and if so, how to determine which tables and columns they should be applied. Finally, they cover best practices and recent improvements to histograms.
This presentation shortly describes key features of Apache Cassandra. It was held at the Apache Cassandra Meetup in Vienna in January 2014. You can access the meetup here: http://www.meetup.com/Vienna-Cassandra-Users/
Espresso: LinkedIn's Distributed Data Serving Platform (Paper)Amy W. Tang
This paper, written by the LinkedIn Espresso Team, appeared at the ACM SIGMOD/PODS Conference (June 2013). To see the talk given by Swaroop Jagadish (Staff Software Engineer @ LinkedIn), go here:
http://www.slideshare.net/amywtang/li-espresso-sigmodtalk
The paperback version is available on lulu.com there http://goo.gl/fraa8o
This is the first volume of the postgresql database administration book. The book covers the steps for installing, configuring and administering a PostgreSQL 9.3 on Linux debian. The book covers the logical and physical aspect of PostgreSQL. Two chapters are dedicated to the backup/restore topic.
This ppt was used by Devrim at pgDay Asia 2017. He talked about some important facts about WAL - Transaction Logs or xlogs in PostgreSQL. Some of these can really come handy on a bad day
Tech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of FacebookThe Hive
This presentation describes the reasons why Facebook decided to build yet another key-value store, the vision and architecture of RocksDB and how it differs from other open source key-value stores. Dhruba describes some of the salient features in RocksDB that are needed for supporting embedded-storage deployments. He explains typical workloads that could be the primary use-cases for RocksDB. He also lays out the roadmap to make RocksDB the key-value store of choice for highly-multi-core processors and RAM-speed storage devices.
PostgreSQL is designed to be easily extensible. For this reason, extensions loaded into the database can function just like features that are built in. In this session, we will learn more about PostgreSQL extension framework, how are they built, look at some popular extensions, management of these extensions in your deployments.
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013mumrah
Apache Kafka is a new breed of messaging system built for the "big data" world. Coming out of LinkedIn (and donated to Apache), it is a distributed pub/sub system built in Scala. It has been an Apache TLP now for several months with the first Apache release imminent. Built for speed, scalability, and robustness, Kafka should definitely be one of the data tools you consider when designing distributed data-oriented applications.
The talk will cover a general overview of the project and technology, with some use cases, and a demo.
PostgreSQL is one of the most advanced relational databases. It offers superb replication capabilities. The most important features are: Streaming replication, Point-In-Time-Recovery, advanced monitoring, etc.
Webinar slides: An Introduction to Performance Monitoring for PostgreSQLSeveralnines
To operate PostgreSQL efficiently, you need to have insight into database performance and make sure it is at optimal levels.
With that in mind, we dive into monitoring PostgreSQL for performance in this webinar replay.
PostgreSQL offers many metrics through various status overviews and commands, but which ones really matter to you? How do you trend and alert on them? What is the meaning behind the metrics? And what are some of the most common causes for performance problems in production?
We discuss this and more in ordinary, plain DBA language. We also have a look at some of the tools available for PostgreSQL monitoring and trending; and we’ll show you how to leverage ClusterControl’s PostgreSQL metrics, dashboards, custom alerting and other features to track and optimize the performance of your system.
AGENDA
- PostgreSQL architecture overview
- Performance problems in production
- Common causes
- Key PostgreSQL metrics and their meaning
- Tuning for performance
- Performance monitoring tools
- Impact of monitoring on performance
- How to use ClusterControl to identify performance issues
- Demo
SPEAKER
Sebastian Insausti, Support Engineer at Severalnines, has loved technology since his childhood, when he did his first computer course (Windows 3.11). And from that moment he was decided on what his profession would be. He has since built up experience with MySQL, PostgreSQL, HAProxy, WAF (ModSecurity), Linux (RedHat, CentOS, OL, Ubuntu server), Monitoring (Nagios), Networking and Virtualization (VMWare, Proxmox, Hyper-V, RHEV).
Prior to joining Severalnines, Sebastian worked as a consultant to state companies in security, database replication and high availability scenarios. He’s also a speaker and has given a few talks locally on InnoDB Cluster and MySQL Enterprise together with an Oracle team. Previous to that, he worked for a Mexican company as chief of sysadmin department as well as for a local ISP (Internet Service Provider), where he managed customers' servers and connectivity.
This webinar builds upon a related blog post by Sebastian: https://severalnines.com/blog/performance-cheat-sheet-postgresql.
How to use histograms to get better performanceMariaDB plc
Sergei Petrunia and Varun Gupta, software engineers MariaDB, show how histograms can be used to improve query performance. They begin by introducing histrograms and explaining why they’re needed by the query optimizer. Next, they discuss how to determine whether or not histrograms are needed, and if so, how to determine which tables and columns they should be applied. Finally, they cover best practices and recent improvements to histograms.
This presentation shortly describes key features of Apache Cassandra. It was held at the Apache Cassandra Meetup in Vienna in January 2014. You can access the meetup here: http://www.meetup.com/Vienna-Cassandra-Users/
Espresso: LinkedIn's Distributed Data Serving Platform (Paper)Amy W. Tang
This paper, written by the LinkedIn Espresso Team, appeared at the ACM SIGMOD/PODS Conference (June 2013). To see the talk given by Swaroop Jagadish (Staff Software Engineer @ LinkedIn), go here:
http://www.slideshare.net/amywtang/li-espresso-sigmodtalk
The paperback version is available on lulu.com there http://goo.gl/fraa8o
This is the first volume of the postgresql database administration book. The book covers the steps for installing, configuring and administering a PostgreSQL 9.3 on Linux debian. The book covers the logical and physical aspect of PostgreSQL. Two chapters are dedicated to the backup/restore topic.
Data deduplication is a hot topic in storage and saves significant disk space for many environments, with some trade offs. We’ll discuss what deduplication is and where the Open Source solutions are versus commercial offerings. Presentation will lean towards the practical – where attendees can use it in their real world projects (what works, what doesn’t, should you use in production, etcetera).
Kafka on ZFS: Better Living Through Filesystems confluent
(Hugh O'Brien, Jet.com) Kafka Summit SF 2018
You’re doing disk IO wrong, let ZFS show you the way. ZFS on Linux is now stable. Say goodbye to JBOD, to directories in your reassignment plans, to unevenly used disks. Instead, have 8K Cloud IOPS for $25, SSD speed reads on spinning disks, in-kernel LZ4 compression and the smartest page cache on the planet. (Fear compactions no more!)
Learn how Jet’s Kafka clusters squeeze every drop of disk performance out of Azure, all completely transparent to Kafka.
-Striping cheap disks to maximize instance IOPS
-Block compression to reduce disk usage by ~80% (JSON data)
-Instance SSD as the secondary read cache (storing compressed data), eliminating >99% of disk reads and safe across host redeployments
-Upcoming features: Compressed blocks in memory, potentially quadrupling your page cache (RAM) for free
We’ll cover:
-Basic Principles
-Adapting ZFS for cloud instances (gotchas)
-Performance tuning for Kafka
-Benchmarks
Gluster for Geeks: Performance Tuning Tips & TricksGlusterFS
In this Gluster for Geeks technical webinar, Jacob Shucart, Senior Systems Engineer, will provide useful tips and tricks to make a Gluster cluster meet your performance requirements. He will review considerations for all different phases including planning, configuration, implementation, tuning, and benchmarking.
Topics covered will include:
• Protocols (CIFS, NFS, GlusterFS)
• Hardware configuration
• Tuning parameters
• Performance benchmarks
This session will cover performance-related developments in Red Hat Gluster Storage 3 and share best practices for testing, sizing, configuration, and tuning.
Join us to learn about:
Current features in Red Hat Gluster Storage, including 3-way replication, JBOD support, and thin-provisioning.
Features that are in development, including network file system (NFS) support with Ganesha, erasure coding, and cache tiering.
New performance enhancements related to the area of remote directory memory access (RDMA), small-file performance, FUSE caching, and solid state disks (SSD) readiness.
Howdah - An Application using Pylons, PostgreSQL, Simpycity and ExceptableCommand Prompt., Inc
Aurynn Shaw
This mini-tutorial covers building a small application on Howdah, an open source, Python based web development framework by Commandprompt, Inc. We will cover the full process of designing a vertically coherent application on Howdah, integrating DB-level stored procedures, DB exception propagation through Exceptable, DB access through Simpycity, authentication through repoze.who, permissions through VerticallyChallenged, and application views through Pylons. By the end of the talk, we will have covered a full application built on The Stack, and how to cover common pitfalls in using Howdah components.
Scott Bailey
Few things we model in our databases are as complicated as time. The major database vendors have struggled for years with implementing the base data types to represent time. And the capabilities and functionality vary wildly among databases. Fortunately PostgreSQL has one of the best implementations out there. We will look at PostgreSQL's core functionality, discuss temporal extensions, modeling temporal data, time travel and bitemporal data.
Bruce Momjian
Pg_Migrator allows data to be transfered between major Postgres versions
without a dump/restore. This talk explains the internal workings of
pg_migrator and includes a pg_migrator demonstration
Adrian Klaver
An exploration of various Python projects (PyRTF,ReportLab,xlwt) that help with presenting your data in formats (rtf,pdf,xls) that other people want. I will step through a simple data extraction and conversion process using the above software to create an RTF,PDF and XLS file respectively.
PostgreSQL, Extensible to the Nth Degree: Functions, Languages, Types, Rules,...Command Prompt., Inc
Jeff Davis
I'll be showing how the extensible pieces of PostgreSQL fit together to give you the full power of native functionality -- including performance. These pieces, when combined, make PostgreSQL able to do almost anything you can imagine. A variety add-ons have been very successful in PostgreSQL merely by using this extensibility. Examples in this talk will range from PostGIS (a GIS extension for PostgreSQL) to DBI-Link (manage any data source accessible via perl DBI).
Jeff Davis
UNIQUE indexes have long held a unique position among constraints: they are the only way to express a constraint that two tuples in a table conflict without resorting to triggers and locks (which severely impact performance). But what if you want to impose the constraint that one person can't be in two places at the same time? In other words, you have a schedule, and you want to be sure that two periods of time for the same person do not overlap. This is nearly impossible to do efficiently with the current version of PostgreSQL -- and most other database systems. I will be presenting Generalized Index Constraints, which is being submitted for inclusion in the next PostgreSQL release, along with the PERIOD data type (available now from PgFoundry). I will show how these can, together, offer a fast, scalable, and highly concurrent solution to a very common business requirement. A business requirement is still a requirement even if your current database system can't do it!
Implementing the Future of PostgreSQL Clustering with TungstenCommand Prompt., Inc
Robert Hodges
Users have traditionally used database clusters to solve database availability and performance requirements. However, clustering requirements are changing as hardware improvements make performance concerns obsolete for many users. In this talk I will discuss how the Tungsten project uses master/slave replication, group communications, and rules processing to develop easy-to-manage database clusters that solve database availability, protect data, and address hardware utilization. Our implementation is based on existing PostgreSQL capabilities like Londiste and WAL shipping, which we eventually plan to replace with our own log-based replication. Come see the future of database clustering with Tungsten!
Josh Berkus
Most users know that PostgreSQL has a 23-year development history. But did you know that Postgres code is used for over a dozen other database systems? Thanks to our liberal licensing, many companies and open source projects over the years have taken the Postgres or PostgreSQL code, changed it, added things to it, and/or merged it into something else. Illustra, Truviso, Aster, Greenplum, and others have seen the value of Postgres not just as a database but as some darned good code they could use. We'll explore the lineage of these forks, and go into the details of some of the more interesting ones.
Joshua D. Drake
Are you tired of not having a real solution for PITR? Enter PITRTools, a single and secure solution for using Point In Time Recovery for PostgreSQL.
Selena Deckelmann
Bucardo is a mature replication system for PostgreSQL that supports asynchronous replication for both master-slave and multi-master systems. Originally designed for slow and unreliable networks, it has remarkable recovery ability, an easy to use command-line interface and development is active! Uses for Bucardo include: a slave read-only database, multi-master replication, data warehousing and just having fun moving your data around! Will include overview replication for PostgreSQL in general, a tour of features, a basic configuration walk through and the much feared live demo!
Matt Smiley
This is a basic primer aimed primarily at developers or DBAs new to Postgres. The format is a Q/A style tour with examples, based on common questions and pitfalls. Begin with a quick tour of relevant parts of the postgres catalog, with an aim to answer simple but important questions like:
How many rows does the optimizer think my table has?
When was it last analyzed?
Which other tables also have a column named "foo"?
How often is this index used?
Rod Anderson
For the small business support person being able to provide PostgreSQL hosting for a small set of specific applications without having to build and support several Pg installations is necessary. By building a multi-tenant Pg cluster with one tenant per database and each application in it's own schema maintenance and support is much simpler. The issues that present themselves are how to provide and control dba and user access to the database and get the applications into their own schema. With this comes need to make logging in to the database (pg_hba.conf) as non-complex as possible.
Brent Friedman
- Three hour workshop format, 30 minutes or so on history, trends, quick review of normalization, 45 minutes on a normalization walk-through including everyone, then break into small teams (3-6 ppl) to do a 'normalization challenge' on a real world practical problem
Brent Friedman
- Three hour workshop format, 30 minutes or so on history, trends, quick review of normalization, 45 minutes on a normalization walk-through including everyone, then break into small teams (3-6 ppl) to do a 'normalization challenge' on a real world practical problem
Leo Hsu and Regina Obe
We'll demonstrate integrating PostGIS in both PHP and ASP.NET applications.
We'll demonstrate using the new PostGIS 1.5 geography offering to extend existing web applications with proximity analysis.
More advanced use to display maps and stats using OpenLayers, WMS/WFS services and roll your own WFS like service using the PostGIS KML/GML/and or GeoJSON output functions.
6. What Flavor is Your DB? O
1
W ►Web Application (Web)
● DB smaller than RAM
● 90% or more “one-liner” queries
O ►Online Transaction Processing (OLTP)
● DB slightly larger than RAM to 1TB
● 20-40% small data write queries, some large transactions
D ►Data Warehousing (DW)
● Large to huge databases (100GB to 100TB)
● Large complex reporting queries
● Large bulk loads of data
● Also called "Decision Support" or "Business Intelligence"
7. P.E. Tips O
1
►Engineer for the problems you have
● not for the ones you don't
►A little overallocation is cheaper than downtime
● unless you're an OEM, don't stint a few GB
● resource use will grow over time
►Test, Tune, and Test Again
● you can't measure performance by “it seems fast”
►Most server performance is thresholded
● “slow” usually means “25x slower”
● it's not how fast it is, it's how close you are to capacity
9. Hardware Basics
►Four basic components:
● CPU
● RAM
● I/O: Disks and disk bandwidth
● Network
►Different priorities for different applications
● Web: CPU, Netowrk, RAM, ... I/O W
● OLTP: balance all O
● DW: I/O, CPU, RAM D
10. Getting Enough CPU 1
►Most applications today are CPU-bound
● even I/O takes CPU
►One Core, One Query
● PostgreSQL is a multi-process application
▬ Except for IOwaits, each core can only process one query at a
time.
▬ How many concurrent queries do you need?
● Best performance at 1 core per no more than two concurrent
queries
►So if you can up your core count, do
● you don't have to pay for licenses for the extra cores!
11. CPU Tips 1
►CPU
● SMP scaling isn't perfect; fewer faster cores is usually better
than more slower ones
▬ exception: highly cachable web applications W
▬ more processors with less cores each should perform better
● CPU features which matter
▬ Speed
▬ Large L2 cache helps with large data
▬ 64-bit performance can be 5-20% better
– especially since it lets you use large RAM
– but sometimes it isn't an improvement
12. Getting Enough RAM 1
►RAM use is "thresholded"
● as long as you are above the amount of RAM you need, even
1%, server will be fast
● go even 1% over and things slow down a lot
►Critical RAM thresholds
● Do you have enough RAM to keep the database in W
shared_buffers?
▬ Ram 6x the size of DB
● Do you have enough RAM to cache the whole database? O
▬ RAM 2x to 3x the on-disk size of the database
● Do you have enough RAM for sorts & aggregates? D
▬ What's the largest data set you'll need to work with?
▬ For how many users
13. Other RAM Issues 1
►Get ECC RAM
● Better to know about bad RAM before it corrupts your data.
►What else will you want RAM for?
● RAMdisk?
● SWRaid?
● Applications?
14. Getting Enough I/O 1
►Will your database be I/O Bound?
● many writes: bound by transaction log
● database 3x larger than RAM: bound by I/O for every query
►Optimize for the I/O you'll need
● if you DB is terabytes, spend most of your money on disks
● calculate how long it will take to read your entire database
from disk
● don't forget the transaction log!
15. I/O Decision Tree 1
lots of fits in
No Yes mirrored
writes? RAM?
Yes No
afford
terabytes HW RAID
good HW Yes No
of data?
RAID?
Yes
No
mostly
SW RAID SAN/NAS read?
Yes No
RAID 5 RAID 1+0
16. I/O Tips 1
►RAID
● get battery backup and turn your write cache on
● SAS has 2x the real throughput of SATA
● more spindles = faster database
▬ big disks are generally slow
►SAN/NAS
● measure lag time: it can kill response time
● how many channels?
▬ “gigabit” is only 100mb/s
▬ make sure multipath works
● use fiber if you can afford it
17. SSD: Not There Yet 1
►Fast
● 1 SSD as fast as a 4-drive RAID
● low-energy and low-profile
►But not reliable
● MTF in months or weeks
● Mainly good for static data
● Seeks are supposed to be as fast as scans …
▬ but they're not
►Don't rely on SSD now
● but you will be using it next year
18. Network 1
►Network can be your bottleneck
● lag time
● bandwith
● oversubscribed switches
►Have dedicated connections
● between appserver and database server
● between database server and failover server
● multiple interfaces!
►Data Transfers
● Gigabit is 100MB/s
● Calculate capacity for data copies, standby, dumps
19. The Most Important
Hardware Advice:
1
►Quality matters
● not all CPUs are the same
● not all RAID cards are the same
● not all server systems are the same
● one bad piece of hardware, or bad driver, can destroy your
application performance
►High-performance databases means hardware
expertise
● the statistics don't tell you everything
● vendors lie
● you will need to research different models and combinations
● read the pgsql-performance mailing list
20. The Most Important
Hardware Advice:
1
►So Test, Test, Test!
● CPU: PassMark, sysbench, Spec CPU
● RAM: memtest, cachebench, Stream
● I/O: bonnie++, dd, iozone
● Network: bwping, netperf
● DB: pgBench, sysbench
►Make sure you test your hardware before you
put your database on it
● “Try before you buy”
● Never trust the vendor or your sysadmins
22. Spread Your Files Around 1
2
►Separate the transaction log if possible O D
● pg_xlog directory
● on a dedicated disk/array, performs 10-50% faster
● many WAL options only work if you have a separate drive
number of drives/arrays 1 2 3
which partition
OS/applications 1 1 1
transaction log 1 1 2
database 1 2 3
23. Spread Your Files Around 1
2
►Tablespaces for large tables O D
● try giving the most used table/index its own tablespace & disk
▬ if that table gets more transactions than any other
▬ if that table is larger than any other
▬ having tables and indexes in separate tablespaces helps with
very large tables
● however, often not worth the headache for most applications
24. Linux Tuning 1
2
►Filesystems
● XFS & JFS are best in OLTP tests O
▬ but can be unstable on RHEL
● Otherwise, use Ext3
● Reduce logging
▬ data=writeback, noatime, nodiratime
►OS tuning
● must increase shmmax, shmall in kernel
● use deadline scheduler to speed writes O
● check your kernel version carefully for performance issues!
▬ any 2.6 before 2.6.9 is bad
25. Solaris Tuning 1
2
►Filesystems
● ZFS for very large DBs D
● UFS for everything else W O
● Mount the transaction log on a partition forcedirectio
▬ even if it's on the same disk
● turn off full_page_writes with UFS
►OS configuration
● no need to configure shared memory, semaphores in Solaris
10
● compile PostgreSQL with aggressive optimization using Sun
Studio 11/12
26. FreeBSD Tuning 1
2
►Filesystems
● Increase readahead on the FS O D
vfs.read_max = 64
►OS tuning
● need to increase shmall, shmmax and semaphores:
kernel.ipc.shmmax = (1/3 RAM in Bytes)
kernel.ipc.shmall = (1/3 RAM in pages)
kernel.ipc.semmap = 256
kernel.ipc.semmni = 256 W O D
kernel.ipc.semmns = 512
kernel.ipc.semmnu = 256
28. Set up Monitoring! 1
2
►Get warning ahead of time
● know about performance problems before they go critical
● set up alerts
▬ 80% of capacity is an emergency!
● set up trending reports
▬ is there a pattern of steady growth?
►Monitor everything
● cpu / io / network load
● disk space & memory usage
►Use your favorite tools
● nagios, cacti, reconnitor, Hyperic, OpenNMS
30. shared_buffers 3
1
►Increase: how much?
● shared_buffers are usually a minority of RAM
▬ use filesystem cache for data
● but should be large: 1/4 of RAM on a dedicated server
▬ as of 8.1, no reason to worry about too large
● cache_miss statistics can tell you if you need more
● more buffers needed especially for:
▬ many concurrent queries
W O
▬ many CPUs
31. Other memory parameters 3
1
►work_mem
● non-shared
▬ lower it for many connections W O
▬ raise it for large queries D
● watch for signs of misallocation
▬ swapping RAM: too much work_mem
▬ log temp files: not enough work_mem
● probably better to allocate by task/ROLE
32. Other memory parameters 3
1
►maintenance_work_mem
● the faster vacuum completes, the better
▬ but watch out for multiple autovacuum workers!
● raise to 256MB to 1GB for large databases
● also used for index creation
▬ raise it for bulk loads
33. Commits 3
1
►wal_buffers
● raise it to 8MB for SMP systems
►checkpoint_segments
● more if you have the disk: 16, 64, 128
►synchronous_commit W
● response time more important than data integrity?
● turn synchronous_commit = off
● lose a finite amount of data in a shutdown
►effective_io_concurrency
● set to number of disks or channels
34. Query tuning 3
1
►effective_cache_size
● RAM available for queries
● set it to 2/3 of your available RAM
►default_statistics_target D
● raise to 200 to 1000 for large databases
● now defaults to 100
● setting statistics per column is better
35. Maintenance 3
1
►Autovacuum
● turn it on for any application which gets constant writes W O
● not so good for batch writes -- do manual vacuum for bulk
loads D
● make sure to include analyze
● have 100's or 1000's of tables?
multiple_autovacuum_workers
▬ but not more than ½ cores
►Vacuum delay
● 50-100ms
● Makes vacuum take much longer, but have little impact on
performance
37. Schema Design 1
4
►Table design
● do not optimize prematurely
▬ normalize your tables and wait for a proven issue to
denormalize
▬ Postgres is designed to perform well with normalized tables
● Entity-Attribute-Value tables and other innovative designs
tend to perform poorly
● think of when data needs to be updated, as well as read
▬ sometimes you need to split tables which will be updated at
different times
▬ don't trap yourself into updating the same rows multiple times
● BLOBs are slow
▬ have to be completely rewritten, compressed
38. Schema Design 1
4
►Indexing
● index most foreign keys
● index common WHERE criteria
● index common aggregated columns
● learn to use special index types: expressions, full text, partial
►Not Indexing
● indexes cost you on updates, deletes
▬ especially with HOT
● too many indexes can confuse the planner
● don't index: tiny tables, low-cardinality columns
39. Right indexes? 5
1
►pg_stat_user_indexes
● shows indexes not being used
● note that it doesn't record unique index usage
►pg_stat_user_tables
● shows seq scans: index candidates?
● shows heavy update/delete tables: index less
40. Partitioning 5
1
►Partition large or growing tables
● historical data
▬ data will be purged
▬ massive deletes are server-killers
● very large tables
▬ anything over 1GB / 10m rows
▬ partition by active/passive
►Application must be partition-compliant
● every query should call the partition key
● pre-create your partitions
▬ do not create them on demand … they will lock
41. Query design 1
4
►Do more with each query
● PostgreSQL does well with fewer larger queries
● not as well with many small queries
● avoid doing joins, tree-walking in middleware
►Do more with each transaction
● batch related writes into large transactions
►Know the query gotchas (per version)
● try swapping NOT IN and NOT EXISTS for bad queries
● avoid multiple outer joins before 8.2 if you can
● try to make sure that index/key types match
● avoid unanchored text searches "ILIKE '%josh%'"
42. But I use ORM! 1
4
►Object-Relational Management
!= high performance
● ORM is for ease of development
● make sure your ORM allows "tweaking" queries
● applications which are pushing the limits of performance
probably can't use ORM
▬ but most don't have a problem
43. It's All About Caching 1
4
►Use prepared queries W O
►Cache, cache everywhere W O
● plan caching: on the PostgreSQL server
● parse caching: in some drivers
● data caching:
▬ in the appserver
▬ in memcached
▬ in the client (javascript, etc.)
● use as many kinds of caching as you can
►think carefully about cache invalidation
● and avoid “cache storms”
44. Connection Management 1
4
►Connections take resources W O
● RAM, CPU
● transaction checking
►Make sure you're only using connections you
need
● look for “<IDLE>” and “<IDLE> in Transaction”
● log and check for a pattern of connection growth
▬ may indicate a “connecion leak”
● make sure that database and appserver timeouts are
synchronized
● if your app requires > 500 database connections, you need
better pooling
45. Pooling 1
4
►New connections are expensive W
● use persistent connections or connection pooling sofware
▬ appservers
▬ pgBouncer / pgPool
● set pool side to maximum connections needed
▬ establishing hundreds of new connections in a few seconds can
bring down your application
Webserver
Webserver Pool PostgreSQL
Webserver
47. Optimize Your Queries
5
1
in Test
►Before you go production
● simulate user load on the application
● monitor and fix slow queries
● look for worst procedures
►Look for “bad queries”
● queries which take too long
● data updates which never complete
● long-running stored procedures
● interfaces issuing too many queries
● queries which block
49. Finding bad queries 5
1
►Log Analysis
● dozens of logging options
● log_min_duration
● pgfouine
50. Fixing bad queries 5
1
►EXPLAIN ANALYZE
● things to look for:
▬ bad rowcount estimates
▬ sequential scans
▬ high-count loops
● reading explain analyze is an art
▬ it's an inverted tree
▬ look for the deepest level at which the problem occurs
● try re-writing complex queries several ways