EnterpriseDB's Best Practices for Postgres DBAs

© 2014 EDB All rights reserved. 1
DBA Best Practices – Becoming
an Exceptional Postgres DBA

© 2014 EnterpriseDB Corporation. All rights reserved. 2
•  EnterpriseDB Overview
•  Best Practice Expertise
•  The Postgres DBA Role
−  Monitoring
−  Maintenance
−  Capacity Planning
−  Configuration Tuning
−  Deployment Planning
•  Professional Development
•  Summary and Resources
•  Q&A
Agenda

Brief EDB Overview

POSTGRES
innovation
ENTERPRISE
reliability
24/7
support
Services
& training
Enterprise-class
features & tools
Indemnification
Product
road-map
Control
Thousands
of developers
Fast
development
cycles
Low cost
No vendor
lock-in
Advanced
features
Enabling commercial
adoption of Postgres

Postgres Plus
Advanced Server Postgres Plus
Cloud Database
High Availability
PerformanceManagement
REMOTE
DBA 24x7
SUPPORT
PROFESSIONAL
SERVICES
TRAINING
EDB Serves
All Your Postgres Needs
PostgreSQL
Security

Why EDB?
More than 3,000 Enterprises and Governments

EnterpriseDB
is a Market
Leader
The Gartner report, Magic
Quadrant for Operational
Database Management
Systems, by Donald Feinberg,
Merv Adrian and Nick
Heudecker, was published
October 16, 2014.

EDB is an Open Source Community
Leader
Amit Kapila
Ashesh Vashi
Bruce Momjian
Dave Page
Devrim Gunduz
Jan Wieck
Kevin Grittner
Korry Douglas
Muhammad Usama
Robert M Haas
Thom Brown

Best Practice Expertise

•  Remote DBA Services
•  Architectural Health Checks
•  Postgres Support Tickets
•  Postgres Training
Hundreds of Postgres Instances Supported
When
•  ~70% support calls are at
deployment post-deployment
•  Remote DBA and AHC are
post-deployment
What ?
•  How to perform DBA tasks
•  Which product tools or
features to use

When and Where do Users Need Help?
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
POC Dev Deployment Maintenance
Tuning
Product Awareness
How to
Enhancements
Corruption
Bug

Where Do Users Need Help
# Tickets Phase
Category POC Dev Deployment Maintenance Grand Total
Connectors 0% 2% 1% 0% 3%
Bug 0% 0% 0% 0% 1%
How to 0% 1% 0% 0% 1%
Product Awareness 0% 1% 0% 0% 1%
Database 10% 9% 17% 37% 73%
Bug 1% 2% 1% 2% 6%
Corruption 0% 0% 0% 2% 2%
Enhancements 0% 0% 0% 0% 1%
How to 4% 3% 6%
10%
23%

Product Awareness 4% 3% 8%
15%
31%

Tuning 0% 0% 1% 7%
9%

Replication 1% 4% 4% 2% 12%
Bug 0% 1% 0% 0% 2%
How to 0% 0% 3% 1% 4%
Tuning 0% 0% 0% 1% 1%
Utilities 1% 3% 6% 3% 12%
Bug 0% 1% 1% 0% 2%
Enhancements 0% 0% 0% 0% 0%
How to 1% 1% 3% 1% 6%
Grand Total 12% 17% 28% 42% 100%

Building Blocks of the DBA Role
Monitoring
Maintenance
Performance
Tuning
Capacity
Management
Security
Deployment
Training &
Professional
Development

The Postgres DBA Role

•  Postgres is very reliable and stable
•  You can translate your existing DBA skills to Postgres
•  Key areas for Postgres DBAs
−  Bloat management
−  Vacuum setup
−  OS configuration
−  Storage sub-system configuration
−  Backup – logical and physical
−  HA Setup
−  pg_hba.conf and other security settings
−  Monitoring
Taking on the Postgres DBA Role

•  Begin with assessing the health
of the DB - Architectural Health Check (AHC)
−  Performance
−  Maintenance Tasks – index management, bloat reduction
−  Availability – Backup strategies, recovery, replication and
failover
−  Scalability and support for business and data growth
−  Upgrade Planning
•  Establish monitoring
•  Focus on list of items uncovered in health check
Postgres DBA Role – An Approach

•  Technical architecture – OS,
application, database
•  Indexes and bloat
•  Configuration tuning – OS,
database
−  Memory
−  Write-ahead log
−  Background worker
−  Query
−  Logging
−  Checkpoint
−  Vacuum
−  Query performance
−  Security
Architectural Health Check Process
•  DR plan
−  Backup strategy – logical and physical
−  HA solutions – warm standby/log
shipping, hot standby, streaming and
cascading replication
•  Performance tuning
−  Reindexing
−  Partitioning
•  Resource management
−  Connections
−  CPU
•  Hardware and OS configuration
−  File system options

Monitoring and Maintenance

Maintenance Highlights from 20+ AHCs
Unused
Indexes
39%
Bloated
Indexes
30%
Bloated Tables
13%
Missing
Indexes
9%
Missing
PK
9%
Bloat management
(43%) is a key
production
problem
specific to
Postgres

General Monitoring Recommendations
Description Frequency Alerting Criteria /
Parameters monitored
Load average, disk
space and checking
for .ready files.
Every 10
minutes
(i) When load average is above threshold
(ii) When disk space is below threshold
(iii) When .ready(pg_xlog/archive_status/*.ready) files
count is beyond threshold
Postgres instance is
running and connection
count
Every minute (i) When postgres instance is not running
(ii) When number of connection is reached to value of
max_connections
(iii) When number of connections crosses threshold
Long running and
waiting queries in
database
Every 10
minutes
(i) When any query is running for longer than threshold
(ii) When any query is in state of "idle in transaction" for
more than threshold
(iii) When any query is waiting for more than threshold
Bloats for tables and
indexes.
Once a day (i) When database tables and/or indexes individually
bloated more than critical threshold
(ii) When transaction ID wrap around is approaching
Is connection pooler
running
Every 10
minutes
If pgbouncer or pgPool process is not running
Is streaming replication
in sync with primary.
Every 5
minutes
(i) When lag detected on streaming cluster exceeds
threshold

•  Open source plug-in check_postgres.pl
−  http://bucardo.org/
•  PostgreSQL Statistics Collector - pg_stat_xxx views
−  pg_stat_activity - details of open connections and running
transactions
−  pg_stat_database - details of databases
−  pg_stat_user_* - details of tables, indexes and functions
−  pg_locks - list of current locks being held
•  Database object size functions
−  pg_*_size - disk space used by a tablespace, database,
relation or total_relation (includes indexes and toasted data)
Monitoring Tools (if you can’t use PEM)

© 2011 EnterpriseDB . All rights reserved.
Postgres Enterprise Manager
•  Postgres Enterprise Manager (PEM) Tool
−  Monitor real-time alerts
−  Historical reports
−  Identify poorly running SQL statements

•  Top Features
−  Capacity Manager
−  Audit Manager
−  Postgres Expert
−  SQL Profiler
−  Alert Manager
−  Log Manager
−  Web Client
−  Graphical Dashboards for I/O, Memory & user activity

© 2011 EnterpriseDB . All rights reserved.
PEM Features
•  Global dashboards with up-to-date status (up/down/
performance) of all your servers
•  Ability to create performance thresholds for each key metric
e.g. memory, storage, etc.
•  Any threshold violation results in an alert being sent to a
centralized dashboard
•  All key performance-related statistics are automatically
collected and retained
•  Forecast resource usage in the future
•  Identify and tune poorly running SQL statements
•  SNMP integration
•  Audit Logging for Postgres Plus Advance server instances

Capacity Management

•  Use cases:
−  Consumption projection:
−  At current consumption rates when will I drop below 100MB of disk?
−  At the consumption rate of the past 3 days when will I drop below 1 GB of
disk space?
−  Point in Time Activity
−  Users were complaining of performance problems from 3 to 4 PM today.
What was going on in the system, the database, how many users were
connected?
−  We get a load spike every day at 11. What was happing at the time of the
load spike?
−  Root cause for slower Query Performance
−  Queries that access the employ table seem to be taking longer and longer.
View the growth of the number of dead tuples in the table, see what kind of
scans have been executing against the table (index or sequential)
−  Activity monitoring
−  Which database on the server is the most active? What kind of growth is it
seeing?
Capacity Management

Sample PEM Report – Disk Space Available
26© 2011 EnterpriseDB . All rights reserved.
Extrapolated
DataActual Data

Security

•  Conceptual Layers
•  Authentication
•  Authorization
•  Auditing
•  Data Security
•  SQL Injection Attacks
Security Auditing
•  Approach
•  Secure physical
access
•  Network access
limitation
•  Host access limitation
•  Database access
limitation
•  Data access limitation

Security Best Practices for PostgreSQLSecurity Best Practices for PostgreSQL and Postgres Plus Advanced Server
Executive Summary ................................................................3
Introduction ….........................................................................4
Postgres Security Features within the Above Framework ......6
Authentication .........................................................6
Authorization ….......................................................7
Accounting/Auditing ................................................8
Data Security ….......................................................9
SQL Injection Attacks ............................................10
Postgres Plus Advanced Server Security Features ..............11
Further Reading and Useful Links ........................................12
About EnterpriseDB ..............................................................12
Table of Contents
http://info.enterprisedb.com/rs/enterprisedb/images/
Whitepapers_Security_BP_PostgreSQL_and_Postgres_Plus_AS.pdf

Parameter Tuning for Performance

Performance Improvement Recommendations in:
•  Optimizing memory usage
−  Adjusting shared_buffers
−  work_mem * max_connections
•  vm.dirty_bytes and vm.dirty_background_bytes
•  Understanding the storage
−  effective_io_concurrency
•  Identifying problem queries - pgbadger and SQL
Profiler
The following analysis is based on a set of 20 AHCs
executed in 2013
Lessons From AHC and RDBA

0%#
10%#
20%#
30%#
40%#
50%#
60%#
70%#
80%#
90%#
100%#
effec0ve_cache_size#
shared_buffers#
bgw
riter_lru_m
axpages#
w
al_buffers#
m
aintenance_w
ork_m
em
#
bgw
riter_lru_m
ul0plier#
w
ork_m
em
#
random
_page_cost#
checkpoint_segm
ents#
cpu_tuple_cost#
checkpoint_com
ple0on_target#
checkpoint_0m
eout#
log_line_prefix#
log_lock_w
aits#
log_m
in_dura0on_statem
ent#
autovacuum
_vacuum
_scale_factor#
checkpoint_w
arning#
autovacuum
_vacuum
_cost_delay#
autovacuum
_vacuum
_threshold#
autovacuum
_nap0m
e#
log_autovacuum
_m
in_dura0on#
Top$Database$Parameter$Adjustments$3$Ordered$by$Frequency$
Red bars:
experts expected top parameters,
including vacuum_cost_limit,
log_line_prefix,
log_min_duration_statement,
log_checkpoints
Top Database Parameter Adjustments – Ordered by Frequency

Kernel Parameters
0
2
4
6
8
10
12
14
16

Deployment Planning

•  Why do you need backup and recovery? Backup and
Recovery Strategies protect you in case of:
−  Catastrophic Device Failure
−  Site Failure
−  Maintenance
−  Operator Error
−  Compliance
−  Data Corruption
•  Multiple components:
−  Logical backup provides granularity in objects (tables, table
spaces, databases)
−  Physical backups provide granularity in time for PITR
−  Cold/off-line backups
−  Hot/on-line backup, with WAL archiving
Backup and Recovery Strategies

Backup and Standby
hot
47%
warm
23%
None
18%
cold
12%
Standby Approach
Physical
Only
36%
Logical
Only
29%
Physical
and
Logical
14%
FS
Snapshot
7%
None
14%
Backup Approach
43% of customers could not execute
PITR (prior to AHC) – some had NO
backup in place
30% of customers would have been
slow or unable to recover from failure

•  Backup and Recovery Strategy
−  Many deployments lack good backup & recovery (43%)
•  OS Considerations
−  (dirty_background_ratio & dirty_ratio) are important (70% &
45%)
•  Storage Systems
−  key deployment consideration - DAS/SAN/NFS
•  High Availability and Standby Approach
−  30% of customers would have been
slow or unable to recover from failure
•  Plan for Ongoing Maintenance
−  Bloat and vacuum setup is a problem (43%)
Deployment Planning Observations from AHCs

•  Backup Strategy Framework
−  Logical backup after structural changes or
major updates of reference/meta data
−  Physical backup (daily…)
−  WAL file archiving
−  Combine Backup/Recovery and Streaming Replication
−  Uses replica to offload hot backup and logical backup
•  Consider
−  Allowable PITR timeframe
−  Data retention policy
−  Test, test, test
−  Periodic backup validation
•  Advanced solutions, such as RHCS, Veritas Clustering, EFM and
EDB BART augment the basic capabilities
Backup and Recovery Best Practices

•  Types of Storage Systems
−  Direct Attached Storage (DAS)
−  Storage Area Network (SAN) w. Fiber Channel
−  Storage Area Network (SAN) w. iSCSI
−  Network File System (NFS)
•  DAS: Fast and low latency
•  SAN: Fast, expensive (w. Fiber Channel), scalable, can
include redundancy and smart file system operations
•  NFS: Not an optimal solution for PostgreSQL data files
File System and Storage Subsystem

•  OLTP Infrastructure
−  ACID Compliant
−  High Performance
−  Scalable
−  Reliable
•  High Availability
−  Protect against hardware
failure
−  Protect against software failure
•  Disaster Recovery
−  Protect against site failure
−  Protect against operator error
•  Data Integration
−  Integration with other Line of
Business systems
−  Heterogeneous integration with
Microsoft, Oracle, etc
•  Monitoring and Management
−  Capacity Planning
−  Event management and
alerting
•  Security
−  Authentication and
Authorization
−  Encryption and data protection
Key Components of a Enterprise Data
Management Architecture

PEM Client
Oracle/SQL
Server/PG
Replication
DB MasterWitness
Offsite
Replica
Virtual IP
DB Client
Backup &
Recovery
PEM ServerHigh Availability
Disaster Recovery
HA Replica
Read/HA
Replica
Robust,
Resilient,
Scalable
Postgres
Enterprise
Architecture
Monitoring
DataIntegration
Simple,
development
focused
setup

Summary and Useful Resources

•  Postgres DBAs need to focus on
−  Config file settings, especially understanding memory configs
−  Storage subsystem configuration
−  OS Configuration
−  Bloat & vacuum
−  Backup - logical and physical
−  HA setup
−  Monitoring - see the Freight Train before it hits you!
•  Tools, tools, tools – Get the right tools
•  Most problems occur when moving to production
−  Get ready early
−  Get trained before MTP
−  New Postgres DBA -- consider RDBA to help you get started
Summary

•  Training & Certification
−  Online classes and certifications
http://www.enterprisedb.com/products-services-training/training
•  Get the team (developer, DBA, Infrastructure) ready to thrive with Postgres
•  EDB Knowledge Base
•  Blogs
−  EDB’s Blog – http://www.enterprisedb.com/postgres-plus-edb-blog
−  Robert Haas – EDB Chief Architect Database Server (http://rhaas.blogspot.com/)
−  Vibhor Kumar – EDB Database Consultant (http://vibhorkumar.wordpress.com/)
•  User groups and Meetups – almost all major cities
−  http://www.postgresql.org/community/user-groups/
−  http://postgresql.meetup.com/
What You Can Do

EnterpriseDB's Best Practices for Postgres DBAs

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (14)

Similar to EnterpriseDB's Best Practices for Postgres DBAs

Similar to EnterpriseDB's Best Practices for Postgres DBAs (20)

More from EDB

More from EDB (20)

Recently uploaded

Recently uploaded (20)

EnterpriseDB's Best Practices for Postgres DBAs