This document summarizes an enterprise deployment of PostgreSQL at a large mobile advertising company. It discusses: 1) The company uses PostgreSQL for transaction processing and reporting across 60 servers storing 10GB to 20TB of data each, with the critical databases averaging 80GB and handling 18,000 queries per second. 2) To achieve 99.99% uptime, the company employs disaster recovery planning including cascaded replication across data centers and regular disaster drills to test backups. 3) The company uses Puppet for configuration management, stored procedures to optimize queries, monitoring tools, and access control and encryption to secure user access and network traffic.

1. Enterprise grade deployment
and security with PostgreSQL
PGConf US 2015
March 25 - 27, 2015
Himanchali
himamahi09@gmail.com

2. #Who Am I
himanchali@inmobi.com
Technical Lead – Production & Infrastructure Engineering

3. www.inmobi.com
 Independent mobile ad network
 Customer engagement platform through a suit of product ranging from app
distribution & monetization to brand advertisement
 Serves billion of ads
 Hosting the first official PGDay India event in April 
http://www.meetup.com/India-PUG/events/220553997/

4. What kind of application do we have..
 OLTP
 OLAP

5. What exactly am I talking today ??
 Production Deployment
 All main transactional and CI reporting DBs
 Replication across globe
 Load balancing and connection pooling through PGBouncer
 Automation through puppet
 Version control for DB level changes
 Data security
 Data security
 User access control
 OS level security

6. Data growth
0
10
20
30
40
50
60
70
80
90
2010 2011 2012 2013 2014 2015
Main Transactional DBs
(GB)
Reporting DB (TB)
Others (GB)

7. Current stats of Postgres DB Servers
 About 60 hosts spread across 4 regions
 DB size ranges from 10 GB – 20 TB
 The critical transactional DBs :
 Average size 80GB
 Average 18K QPS with peak of 54K QPS
 Reporting DB
 Average 200 GB data load everyday

8. Simple expectation of management
 99.99% Uptime for DB servers
 No data loss
 Good performance

9. Achieving 99.99% Uptime
 A proper DB Architecture
 Multiple disaster recovery Plan
 A time to time disaster drill
 Proper data backup
 PITR setup

10. A proper DB Architecture

11. The architecture..
 Proper data modeling
 Use of constraints carefully
 Minimize the dependency between DBs
 Use of proper data types
 Proper retention policies

12. Disaster Recovery plan
 Question is when to touch the trigger ??
 What if standby doesn’t come up ?? (rare condition)
 Are my backups up to date ??
 Do I have an option to revert some drop of a table?

13. Data availability in different regions
 Cascaded SR setup
 A boon for cross region replication
 One node as entry point to the region
 Cascade slaves within LAN from above node
 A little precaution needed with this setup
 When the master is down and its standby is being triggered as master
 Proper monitoring that no slave is out of sync

14. Read distribution across multiple slaves
 Load balancer
 Pgbouncer
 Connection pooling
 Connection mode control , session or transaction
 Maintenance without downtime

15. What about performance ??
 Proper index creation
 Query optimization
 Table partitioning
 Proper vacuum

16. What else …
 How can I forget those many stored procedures in my DB .
 User defined functions for complex business logic
 Saving time between application and DB
 Get rid of dependency of language used by application

17. Practice adopted for DB changes
 Version control of schema changes
 Version control of data changes

18. Minimize the deployment effort
 Using puppet and chef modules
 No direct change on the host manually
 Configuration changes pushed via puppet
 Packages installation using chef-solo

19. Adding spice of monitoring…
 Use of enriched statistics collector
 User stats
 DB stats
 Table stats
 Index stats
 And many more…
 pg_log monitoring
 Log everything if you want to monitor everything

20. Security aspects

21. Control user access
 Limited access to application users
 Give only what they need
 Control at DB, schema or table level
 Use user groups to manage if many users
 Manage and standardize DBA access
 A common user
 Maintain default privilege

22. Some example
REVOKE CREATE ON SCHEMA public FROM PUBLIC;
REVOKE ALL on schema public from write_user,read_user;
GRANT USAGE on schema public to write_user,read_user ;
GRANT all on all tables in schema public to write_user;
GRANT all on all sequences in schema public to write_user;
REVOKE ALL on all functions in schema public from write_user,read_user;
GRANT EXECUTE on all functions in schema public to write_user,read_user;
GRANT SELECT on all sequences in schema public to read_user;
GRANT SELECT on all tables in schema public to read_user;

23. ALTER DEFAULT PRIVILEGES FOR ROLE enterprisedb,postgres IN SCHEMA public GRANT all on tables to write_user;
ALTER DEFAULT PRIVILEGES FOR ROLE enterprisedb,postgres IN SCHEMA public GRANT EXECUTE on functions to
write_user,read_user;
ALTER DEFAULT PRIVILEGES FOR ROLE enterprisedb,postgres IN SCHEMA public GRANT all on sequences to write_user;
ALTER DEFAULT PRIVILEGES FOR ROLE enterprisedb,postgres IN SCHEMA public GRANT SELECT on sequences to
read_user;
ALTER DEFAULT PRIVILEGES FOR ROLE enterprisedb,postgres IN SCHEMA public GRANT SELECT on tables to read_user;

24. Access restriction to DB
 Use pg_hba.conf and pg_ident.conf
 Avoid using trust
 Use MD5 authentication
 IP / Subnet access allocation for DB or user level
 Restrict the connection for any particular subnet for any user/DB
 Use pg_ident.conf if using ident authentication in pg_hba.conf

25. If that’s not enough…
 Use secure TCP/IP Connections with SSL
 IPTables
 Control the traffic by making some rules
 Allow or reject
 SE-Linux - control access at process level
 setenforce=1

26. Myth broken !!

27. Thank You !!
QUESTIONS??