Best Practices for Becoming an Exceptional Postgres DBA

DBA Best Practices –
Becoming an Exceptional
Postgres DBA
Marc Linster | August 27, 2014
© 2014 EDB All rights reserved. 1

Agenda
• EnterpriseDB Overview
• Best Practice Expertise
• The Postgres DBA Role
− Monitoring
− Maintenance
− Capacity Planning
− Configuration Tuning
− Deployment Planning
• Professional Development
• Summary and Resources
• Q&A
© 2014 EnterpriseDB Corporation. All rights reserved. 2
We will take one quick-poll
during the webinar to
gather group data on best
practices of interest

Brief EDB Overview

POSTGRES
innovation
Services
& training
ENTERPRISE
reliability
24/7
support
Enterprise-class
features & tools
Indemnification
Product
road-map
Control
Thousands
of developers
Fast
development
cycles
Low cost
No vendor
lock-in
Advanced
features
Enabling commercial
adoption of Postgres

Postgres Plus
Advanced Server Postgres Plus
Management Performance
Cloud Database
High Availability
REMOTE
DBA 24x7
SUPPORT
PROFESSIONAL
SERVICES
TRAINING
EDB Serves
All Your Postgres Needs
PostgreSQL
Security

EDB Customers
EDB currently has over 2,500 total customers including 50 of the Fortune
500 and 98 of the Forbes Global 2000

Best Practice Expertise

Hundreds of Postgres Instances Supported
• Remote DBA Services
• Architectural Health Checks
• Postgres Support Tickets
• Postgres Training
When
• ~70% support calls are at
deployment, post-deployment
• Remote DBA and AHC are
post-deployment
What
• How to perform DBA tasks
• Which product tools or
features to use

When and Where do Users Need Help?
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
POC Dev Deployment Maintenance
Tuning
Product Awareness
How to
Enhancements
Corruption
Bug

Where Do Users Need Help
# Tickets Phase
Category POC Dev Deployment Maintenance Grand Total
Connectors 0% 2% 1% 0% 3%
Bug 0% 0% 0% 0% 1%
How to 0% 1% 0% 0% 1%
Product Awareness 0% 1% 0% 0% 1%
Database 10% 9% 17% 37% 73%
Bug 1% 2% 1% 2% 6%
Corruption 0% 0% 0% 2% 2%
Enhancements 0% 0% 0% 0% 1%
How to 4% 3% 6%
10%
23%
Product Awareness 4% 3% 8%
15%
31%
Tuning 0% 0% 1% 7%
9%
Replication 1% 4% 4% 2% 12%
Bug 0% 1% 0% 0% 2%
How to 0% 0% 3% 1% 4%
Tuning 0% 0% 0% 1% 1%
Utilities 1% 3% 6% 3% 12%
Bug 0% 1% 1% 0% 2%
Enhancements 0% 0% 0% 0% 0%
How to 1% 1% 3% 1% 6%
Grand Total 12% 17% 28% 42% 100%

Select Top Support and RDBA Questions
(not in rank order)
Deployment/Database/
Product Awareness & How To
• How to set up connection
pooling?
• How to set up hot standby and
streaming replica?
• How to add table spaces?
• How to partition?
• How to upgrade from major
version?
• What should I monitor?
• How do I reduce downtime?
Maintenance/Database/
Product Awareness & How To
• How to find bloat in tables &
indexes?
• How to enable auditing
selectively without performance
impact?
• What are the effects of changes
to the values of postgresql.conf
file parameters
wal_writer_delay,
bgwriter_delay?
• What is the correct
configuration of hba_conf?

Building Blocks of the DBA Role
Performance
Tuning
Monitoring
Maintenance
Capacity
Management
Security
Deployment
Training &
Professional
Development

The Postgres DBA Role

Taking on the Postgres DBA Role
• Postgres is very reliable and stable
• You can translate your existing DBA skills to Postgres
• Key areas for Postgres DBAs
− Bloat management
− Vacuum setup
− OS configuration
− Storage sub-system configuration
− Backup – logical and physical
− HA Setup
− pg_hba.conf and other security settings
− Monitoring

Postgres DBA Role – An Approach
• Begin with assessing the health
of the DB - Architectural Health Check (AHC)
− Performance
− Maintenance Tasks – index management, bloat reduction
− Availability – backup strategies, recovery, replication and
failover
− Scalability and support for business and data growth
− Upgrade planning
• Establish monitoring
• Focus on list of items uncovered in health check

Architectural Health Check Process
• Technical architecture – OS,
application, database
• Indexes and bloat
• Configuration tuning – OS,
database
− Memory
− Write-ahead log
− Background worker
− Query
− Logging
− Checkpoint
− Vacuum
− Query performance
− Security
• DR plan
− Backup strategy – logical and physical
− HA solutions – warm standby/log
shipping, hot standby, streaming and
cascading replication
• Performance tuning
− Reindexing
− Partitioning
• Resource management
− Connections
− CPU
• Hardware and OS configuration
− File system options

Monitoring and Maintenance

Maintenance Highlights from 20+ AHCs
Bloated Tables
Unused
Indexes
39%
Bloated
Indexes
30%
13%
Missing
Indexes
9%
Missing
PK
9%
Bloat management
(43%) is a key
production
problem
specific to
Postgres

General Monitoring Recommendations
Description Frequency Alerting Criteria /
Parameters monitored
Load average, disk
space and checking
for .ready files
Every 10
minutes
(i) When load average is above threshold
(ii) When disk space is below threshold
(iii) When .ready(pg_xlog/archive_status/*.ready) files
count is beyond threshold
Postgres instance is
running and connection
count
Every minute (i) When postgres instance is not running
(ii) When number of connection is reached to value of
max_connections
(iii) When number of connections crosses threshold
Long running and
waiting queries in
database
Every 10
minutes
(i) When any query is running for longer than threshold
(ii) When any query is in state of "idle in transaction" for
more than threshold
(iii) When any query is waiting for more than threshold
Bloats for tables and
indexes
Once a day (i) When database tables and/or indexes individually
bloated more than critical threshold
(ii) When transaction ID wrap around is approaching
Is connection pooler
running
Every 10
minutes
If pgbouncer or pgPool process is not running
Is streaming replication
in sync with primary
Every 5
minutes
(i) When lag detected on streaming cluster exceeds
threshold

Monitoring Tools (if you can’t use PEM)
• Open source plug-in check_postgres.pl!
− http://bucardo.org/
• PostgreSQL Statistics Collector - pg_stat_xxx views
− pg_stat_activity - details of open connections and running
transactions
− pg_stat_database - details of databases
− pg_stat_user_* - details of tables, indexes and functions
− pg_locks - list of current locks being held
• Database object size functions
− pg_*_size - disk space used by a tablespace, database,
relation or total_relation (includes indexes and toasted data)

Postgres Enterprise Manager
• Postgres Enterprise Manager (PEM) Tool
− Monitor real-time alerts
− Historical reports
− Identify poorly running SQL statements
• Top Features
− Capacity Manager
− Audit Manager
− Postgres Expert
− SQL Profiler
− Alert Manager
− Log Manager
− Web Client
− Graphical Dashboards for I/O, Memory & user activity
© 2011 EnterpriseDB . All rights reserved.

PEM Features
• Global dashboards with up-to-date status (up/down/
performance) of all your servers
• Ability to create performance thresholds for each key metric
e.g. memory, storage, etc.
• Any threshold violation results in an alert being sent to a
centralized dashboard
• All key performance-related statistics are automatically
collected and retained
• Forecast resource usage in the future
• Identify and tune poorly running SQL statements
• SNMP integration
• Audit Logging for Postgres Plus Advanced Server instances
© 2011 EnterpriseDB . All rights reserved.

Capacity Management

Capacity Management
• Use cases:
− Consumption projection:
− At current consumption rates when will I drop below 100MB of disk?
− At the consumption rate of the past 3 days when will I drop below 1 GB of
disk space?
− Point in Time Activity
− Users were complaining of performance problems from 3 to 4 PM today.
What was going on in the system, the database, how many users were
connected?
− We get a load spike every day at 11. What was happing at the time of the
load spike?
− Root cause for slower Query Performance
− Queries that access the employ table seem to be taking longer and longer.
View the growth of the number of dead tuples in the table, see what kind of
scans have been executing against the table (index or sequential)
− Activity monitoring
− Which database on the server is the most active? What kind of growth is it
seeing?

Sample PEM Report – Disk Space Available
© 2011 EnterpriseDB . All rights reserved. 25
Extrapolated
Actual Data Data

Security

Security Auditing
• Conceptual Layers
− Authentication
− Authorization
− Auditing
− Data Security
− SQL Injection Attacks
• Approach
− Secure physical access
− Network access
limitation
− Host access limitation
− Database access
limitation
− Data access limitation

Security Best PracSetcuiricty Beest sPra ctficeos forr PoPstgreoSQsL atndg Porstgeres SPlusQ AdvaLnce d Server
http://info.enterprisedb.com/rs/enterprisedb/images/
Whitepapers_Security_BP_PostgreSQL_and_Postgres_Plus_AS.pdf
Table of Contents
Executive Summary ................................................................3
Introduction ….........................................................................4
Postgres Security Features within the Above Framework ......6
Authentication .........................................................6
Authorization ….......................................................7
Accounting/Auditing ................................................8
Data Security ….......................................................9
SQL Injection Attacks ............................................10
Postgres Plus Advanced Server Security Features ..............11
Further Reading and Useful Links ........................................12
About EnterpriseDB ..............................................................12
Disclaimer

Quick Poll
Database Priorities

Audience Quick Poll #1
• Question: What is your current database priority?
• Answer choices:
a. Resolving a current performance issue
b. Architecture – migration, capacity planning, upgrade
c. Preparing for production deployment
d. Monitoring for ongoing database health
e. Establishing routine maintenance practices

Parameter Tuning for Performance

Lessons From AHC and RDBA
Performance Improvement Recommendations in:
• Optimizing memory usage
− Adjusting shared_buffers
− work_mem * max_connections
• vm.dirty_bytes and vm.dirty_background_bytes
• Understanding the storage
− effective_io_concurrency
• Identifying problem queries - pgbadger and SQL
Profiler
The following analysis is based on a set of 20 AHCs
executed in 2013

100%#
90%#
80%#
70%#
60%#
50%#
40%#
30%#
20%#
10%#
0%#
Top Database Parameter Adjustments – Ordered by Frequency
effec0ve_cache_size#
Top$Database$Parameter$Adjustments$3$Ordered$by$Frequency$
wal_buffers#
maintenance_work_mem#
shared_buffers#
bgwriter_lru_maxpages#
work_mem#
random_page_cost#
checkpoint_segments#
cpu_tuple_cost#
checkpoint_comple0on_target#
bgwriter_lru_mul0plier#
Red bars:
experts expected top parameters,
including vacuum_cost_limit,
log_line_prefix,
log_min_duration_statement,
log_checkpoints
checkpoint_warning#
autovacuum_vacuum_cost_delay#
autovacuum_vacuum_threshold#
log_min_dura0on_statement#
autovacuum_vacuum_scale_factor#
checkpoint_0meout#
log_line_prefix#
log_lock_waits#
autovacuum_nap0me#
log_autovacuum_min_dura0on#

Kernel Parameters
16
14
12
10
8
6
4
2
0

Customer Case Study
Use Case: Supply Chain Logistics IT Provider
SaaS Warehouse Management System
Issue:
• Performance issues - speed
Root Causes
• Slow, memory-hungry queries,
improper indexes
Results
Change Areas
• Indexing improvements
• Query tuning or rewrites

Customer Case Study
Use Case: Global MLS Platform – Reporting System
Issue:
• Intermittent timeouts
Root Cause
• Parameter configuration
• Enabled
− vm.dirty_bytes
− vm.dirty_background_bytes
− vm.zone_reclaim_mode
• Disabled
− vm.overcommit_memory
− vm.swappiness
− redhat_transparent_hugepage
Change Areas
• Kernel parameters
• Database memory parameters
• Increased
− work_mem
− maintenance_work_mem
• Reduced
− shared_buffers
• wal_buffers 32MB
• wal_level “archive”

Deployment Planning

Backup and Recovery Strategies
• Why do you need backup and recovery? Backup and
Recovery Strategies protect you in case of:
− Catastrophic Device Failure
− Site Failure
− Maintenance
− Operator Error
− Compliance
− Data Corruption
• Multiple components:
− Logical backup provides granularity in objects (tables, table
spaces, databases)
− Physical backups provide granularity in time for PITR
− Cold/off-line backups
− Hot/on-line backup, with WAL archiving

Backup and Standby
Snapshot
Backup Approach
Standby Approach
hot
47%
None
18%
cold
12%
warm
23%
Physical
Only
36%
Logical
Only
29%
FS
7%
Physical
and
Logical
14%
None
14%
43% of customers could not execute
PITR (prior to AHC) – some had NO
backup in place
30% of customers would have been
slow or unable to recover from failure

Backup and Recovery Best Practices
• Backup Strategy Framework
− Logical backup after structural changes or
major updates of reference/meta data
− Physical backup (daily…)
− WAL file archiving
− Combine Backup/Recovery and Streaming Replication
− Uses replica to offload hot backup and logical backup
• Consider
− Allowable PITR timeframe
− Data retention policy
− Test, test, test
− Periodic backup validation
• Advanced solutions, such as RHCS, Veritas Clustering , EFM and
EDB BART augment the basic capabilities

Customer Case Study
Use Case: on-line advertising service with hosted chat
Stage: post-production
Primary Issue:
• Outage due to replication
failure - insufficient log info to
determine true root cause
Potential Pain Points
• Concerns have reached limits
of scale and meeting
anticipated growth demands
Recommendations
• Backup and recovery strategies
− Implement continuous archive
recovery in addition to
streaming replication
− Revised backup strategy with
PITR
− Tune auto-vacuum settings and
manually vacuum large tables
• Indexes – create concurrent
indexes, REINDEX to reduce bloat
• Parameter tuning – various for
kernel, memory settings and WALs

File System and Storage Subsystem
• Types of Storage Systems
− Direct Attached Storage (DAS)
− Storage Area Network (SAN) w. Fiber Channel
− Storage Area Network (SAN) w. iSCSI
− Network File System (NFS)
• DAS: Fast and low latency
• SAN: Fast, expensive (w. Fiber Channel), scalable, can
include redundancy and smart file system operations
• NFS: Not an optimal solution for PostgreSQL data files

Summary and Useful Resources

Summary
• Postgres DBAs need to focus on
− Config file settings, especially understanding memory configs
− Storage subsystem configuration
− OS Configuration
− Bloat & vacuum
− Backup - logical and physical
− HA setup
− Monitoring - see the Freight Train before it hits you!
• Tools, tools, tools – Get the right tools
• Most problems occur when moving to production
− Get ready early
− Get trained before MTP
− New Postgres DBA -- consider RDBA to help you get started

What You Can Do
• Training & Certification
− Classroom Comprehensive Admin - Bangalore – Sept 15-19
− Comprehensive Admin Live Online – NA- with Bruce Momjian – PostgreSQL Guru
and community leader – Oct 6-10
− Live Online EMEA – Adv. PostgreSQL Admin – with 9.4 Highlights – Oct 13-15
• Get the team (developer, DBA, Infrastructure) ready to thrive with Postgres
• EDB Knowledge Base
• Blogs
− Robert Haas – EDB Chief Architect Database Server (http://rhaas.blogspot.com/)
− Vibhor Kumar – EDB Database Consultant (http://vibhorkumar.wordpress.com/)
• User groups and Meetups – almost all major cities
− http://www.postgresql.org/community/user-groups/
− http://postgresql.meetup.com/
• PostgresOpen 2014 - Sept 17-19 Chicago
• PostgreSQL Conference Europe 2014 - Oct 21-24 Madrid

Architectural
Health Check
• Review of
database,
OS, env.
settings
• Guidance on
tuning,
backup, high
availability
strategies
Remote DBA
Services
• 24 x 7
monitoring
• Supplemental
staff
• Architectural
health check
• Access to
Postgres
expertise
Kick Start
• Training
• Flex
consulting
• Developer
subscription
Training and
Certification
• Flexible
formats:
on-demand,
live virtual,
on-site
• Continuing
education
and Postgres
certification
How We Can Help
Whitepapers and Recorded Webinars
http://www.enterprisedb.com/resources-community

Special Offers for Attendees
For all attendees we are providing a special offer on related DBA
services including training.
Please complete an inquiry form on-line and mention this webinar, or
speak to your account manager
− EDB’s regional sales offices found on enterprisedb.com:
− US +1 781-357-3390 or 1-877-377-4352
− EMEA +31 (0) 70 36 11 774
− JAPAN +81-50-5532-7038
− KOREA +82-2-6007-2500
− PAKISTAN +92-51-8358874
− INDIA +91-20-30589500/01
− Send email to sales@enterprisedb.com
− http://www.enterprisedb.com/general-inquiry-form

Q&A

Best Practices for Becoming an Exceptional Postgres DBA

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (6)

Similar to Best Practices for Becoming an Exceptional Postgres DBA

Similar to Best Practices for Becoming an Exceptional Postgres DBA (20)

More from EDB

More from EDB (20)

Recently uploaded

Recently uploaded (20)

Best Practices for Becoming an Exceptional Postgres DBA