Grabbing the PostgreSQL Elephant by the Trunk
by hgimenez
- 3,840 views
Statistics
- Likes
- 2
- Downloads
- 1
- Comments
- 0
- Embed Views
- Views on SlideShare
- 3,801
- Total Views
- 3,840
Grabbing the PostgreSQL Elephant by the Trunk
Their trunk has over 100000 muscle units
They eat and walk most of the day - terrible digestive system. Other animals eat their poop.
One of the big five.
Intimidating. Aggressive. Nobody messes with them.
They change their teeth six times in a lifetime. At age 60 they don’t grow them again and die of starvation.
Four toes.
A: Transactional operations either succeed completely or fail completely. In Postgres DDL operations are also transactional.
C: Database goes from one consistent state to the next.
I: Data that is generated in interim steps of a transaction can never be seen by any other readers/queries/viewers. This is what MVCC is all about.
D: Guarantees that once the user/client is notified of success, the data is persisted and the transaction will not be undonde.
An alternative to MVCC is read-locking. Every time a query reads from the DB, it locks the rows so that no other statement can change the data, and therefore it is reading “real and consistent” data. However, there can be many users reading from the same table. Hundreds of thousands in bigger websites. What if there’s a queue of hundreds of reads, and you need to update the table? The update statement must wait until all reads are done in order to issue the update. Additional reads must wait until the update is completed before retrieving new data.
With MVCC, none of the reads need to lock the rows, and the update doesn’t require a lock either. They all execute immediately.
Postgres keeps multiple versions (MVcc) of the data every time the data changes (to clean up older versions, use VACUUM).
http://www.pgexperts.com/documents.html
Oracle has over 500 config settings.
In general, OS swapping == too much work_mem, while caching sorts in pg_temp == not enough work_mem.
work_mem = 32MB
So, I’ve convinced you and you will port the app to postgres. Here’s what to look after to avoid any pitfalls.
If any one component fails, the whole system suffers.
Hardware: I/O is a very common bottleneck. Many writes cause the transaction log to bring the process down. If the database is 3x larger than RAM -> I/O bound per query.
Use SATA, not SAS (SATA is half duplex). Raid 1+0 over Raid 5, especially for writes. More spindles, the better => Many small drives better than less bigger drives. Move transaction log to a separate disk.
BEWARE OF CLOUD and IO!
Low RAM is also common, and capping on memory makes the server hit the disks. bad. It is ideal to fit entire database in RAM, using shared_buffers. If that’s not possible, cache as much of the database as possible, in which case think about how big is the operation dataset that needs caching. Think about sorting, aggregates and other operations that may benefit from in-ram caching.
CPU: Pick more CPUs over more cores on same processor. L2 cache, speed, and 64 bit.
OS: Use something that supports direct I/O - linux, solaris, freebsd.
* Tablespaces for large tables
* Tuning linux:
* XFS & JFS for database files. Otherwise ext3 if in redhat (no support for XFS). UFS if in solaris.
* Reduce logging - data-writeback, noatime, nodiratime.
* pre 2.6.9 kernels must upgrade.
* use deadline scheduler for write speed
Schema:
* 1NF. Postgres is optimized for normalized lookups. Wait for an actual issue before denormalizing.
* You may benefit from separating out tables (1-to-1s) based on read/write patterns.
* Indexes: FKs, where clauses, aggregations. Use expression indexes, partial indexes. Don’t over index. Don’t index small tables. Look at pg_stat_user_indexes and pg_stat_user_tables.
* Partitioning: historical data, very big tables. Any big deletes. App must know how to deal with it, by querying the partition key as part of the WHERE criteria.
* Query design. Do more on each query.
* Connection pooling: pgPool, pgBouncer.
* pg_fouine: Analyze queries. In 8.4 there’s pg_stat_statement
* EXPLAIN_ANALYZE
Have a mechanism for measuring changes you make. How much CPU, RAM, swapping is happening before and after tuning? Use top, vmstats, etc
* Table Partitioning allows you to manage dead data without overhead, keeping your database healthy. Performance of course.