Denish PatelDatabase Architect, OmniTIDeploying PostgreSQL onAmazon EC2A case study1Tuesday, May 7, 13
Story teller?✤ Denish Patel✤ denish@omniti.com✤ Database Architect at OmniTI✤ Heterogeneous Environment✤ PostgreSQL, Oracl...
Situation✤ Startup✤ Ruby on Rails Shop✤ Entire IT architecture on Amazon Cloud✤ 2 Identical Database Environments on EC2✤ ...
Situation✤ Weekly flake out issues on EC2 instances✤ Weekly backups using pg_dump✤ No file system level backup✤ Master-slave...
Situation✤ Prod , QA and Dev ...✤ M1 Extra large instances (m1.xlarge)✤ 15 GB RAM, 4 Virtual Cores with 2 EC2 compute unit...
The Amazon Cloud✤ Business Benefits✤ Offloading heavy lifting✤ Lowers cost✤ Reduces time to market6Tuesday, May 7, 13
Amazon Data Centers7Tuesday, May 7, 13
Amazon Instances✤ High CPU✤ High IO✤ High Memory8Tuesday, May 7, 13
Amazon StorageTypes9Tuesday, May 7, 13
Network and Security✤ Amazon Security Groups✤ Keep your Amazon EC2 firewall permissions as restrictive aspossible.✤ Elastic...
Deployment Strategies✤ Sizing Instances and Storage Types✤ Configuration Management✤ High Availability✤ Replication✤ Failov...
Sizing Instances/Storage✤ Understand your database , schema and data✤ Env 1: 50GB , Env 2: 25GB✤ 2 large archival tables✤ ...
Sizing Instances/Storage✤ m2.2xlarge: 34.2GB RAM, 4 Virtual Core✤ Storage✤ EBS✤ RAID 10✤ Not EBS optimized ( Cost Savings!...
Deployment Phase #1us-east-1AV1AV2us-west-1AppServersdb-master db-slave-backupdb-slaveStreaming ReplicationWAL shipping (O...
Configuration Management✤ Automation✤ Repeatability✤ Maintainability✤ Security15Tuesday, May 7, 13
Configuration Management16Tuesday, May 7, 13
Puppetizing Postgres✤ DB server role✤ PostgreSQL installation✤ Installing common tools✤ 3 Roles : prod-master role , prod-...
High Availability✤ Replication✤ Failover18Tuesday, May 7, 13
Replication✤ Streaming replication✤ WAL shipping , OmniPITR✤ Different AV zones✤ At least 1 slave in different Region19Tue...
Failover✤ EIP - about 3 minutes to propagate changes after failover✤ Failover tools✤ Semi-automated script✤ pgbouncer✤ pup...
Deployment Phase #2us-east-1AV1AV2us-west-1AppServersdb-masterdb-slave-backupdb-slaveStreaming ReplicationWAL shipping (Om...
pgbouncer✤ Session Pooling✤ Supports all Postgres features✤ Transactions Pooling✤ Does NOT support PREPARE transactions22T...
Operations23Tuesday, May 7, 13
Database Upgrade✤ Upgraded from PostgreSQL 9.0 to PostgreSQL 9.2✤ Minimal outage upgrade using Mimeo✤ http://www.pateldeni...
Beware!!✤ Sept 24th, 2012✤ PostgreSQL 9.1.0 to 9.1.5 & 9.2.0✤ VACUUM all tables in your database & Rebuild Indices✤ http:/...
Backups✤ Online Backups✤ Daily, db-slave , OmniPITR✤ Test the backups!!!!✤ Schema Backups✤ pg_extractor✤ Offsite Backup✤ s...
Tools✤ tracking schema in git : pg_extractor✤ pg_bloat_report✤ Pg log errors reporting : tail_n_mail✤ Log analysis: pgbadg...
Monitoring28Tuesday, May 7, 13
Database Checks29Tuesday, May 7, 13
Database Checks30Tuesday, May 7, 13
System Checks31Tuesday, May 7, 13
Business Metrics32Tuesday, May 7, 13
Histograms!33Tuesday, May 7, 13
References✤ http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Storage.html✤ OmniTI labs tools✤ https://github.com/omniti-...
Upcoming SlideShare
Loading in …5
×

Deploying postgre sql on amazon ec2

8,355 views

Published on

1 Comment
0 Likes
Statistics
Notes
  • Be the first to like this

No Downloads
Views
Total views
8,355
On SlideShare
0
From Embeds
0
Number of Embeds
4,652
Actions
Shares
0
Downloads
41
Comments
1
Likes
0
Embeds 0
No embeds

No notes for slide

Deploying postgre sql on amazon ec2

  1. 1. Denish PatelDatabase Architect, OmniTIDeploying PostgreSQL onAmazon EC2A case study1Tuesday, May 7, 13
  2. 2. Story teller?✤ Denish Patel✤ denish@omniti.com✤ Database Architect at OmniTI✤ Heterogeneous Environment✤ PostgreSQL, Oracle , MySQL, NoSQL✤ Scalability, Performance & Simplicity✤ OmniTI is hiring Database Engineer!2Tuesday, May 7, 13
  3. 3. Situation✤ Startup✤ Ruby on Rails Shop✤ Entire IT architecture on Amazon Cloud✤ 2 Identical Database Environments on EC2✤ 4 Production Instances✤ 50GB Each PostgreSQL 9.0 Master - Slave ( 2 Pairs)✤ 2 QA and 2 Dev environments3Tuesday, May 7, 13
  4. 4. Situation✤ Weekly flake out issues on EC2 instances✤ Weekly backups using pg_dump✤ No file system level backup✤ Master-slave wasn’t working for months!✤ Schema mismatch between 2 environments✤ Almost NO monitoring in-place for DBs4Tuesday, May 7, 13
  5. 5. Situation✤ Prod , QA and Dev ...✤ M1 Extra large instances (m1.xlarge)✤ 15 GB RAM, 4 Virtual Cores with 2 EC2 compute units each✤ Configuration Management tool - Puppet5Tuesday, May 7, 13
  6. 6. The Amazon Cloud✤ Business Benefits✤ Offloading heavy lifting✤ Lowers cost✤ Reduces time to market6Tuesday, May 7, 13
  7. 7. Amazon Data Centers7Tuesday, May 7, 13
  8. 8. Amazon Instances✤ High CPU✤ High IO✤ High Memory8Tuesday, May 7, 13
  9. 9. Amazon StorageTypes9Tuesday, May 7, 13
  10. 10. Network and Security✤ Amazon Security Groups✤ Keep your Amazon EC2 firewall permissions as restrictive aspossible.✤ Elastic IP Address (EIP)10Tuesday, May 7, 13
  11. 11. Deployment Strategies✤ Sizing Instances and Storage Types✤ Configuration Management✤ High Availability✤ Replication✤ Failover✤ Operations✤ Upgrade, Backups, Maintenance tools✤ Monitoring11Tuesday, May 7, 13
  12. 12. Sizing Instances/Storage✤ Understand your database , schema and data✤ Env 1: 50GB , Env 2: 25GB✤ 2 large archival tables✤ Working Set about 20-30GB in total✤ Both environments are inter-related.✤ Decided to consolidate both environment into single cluster12Tuesday, May 7, 13
  13. 13. Sizing Instances/Storage✤ m2.2xlarge: 34.2GB RAM, 4 Virtual Core✤ Storage✤ EBS✤ RAID 10✤ Not EBS optimized ( Cost Savings!)✤ 850 GB13Tuesday, May 7, 13
  14. 14. Deployment Phase #1us-east-1AV1AV2us-west-1AppServersdb-master db-slave-backupdb-slaveStreaming ReplicationWAL shipping (OmniPITR)14Tuesday, May 7, 13
  15. 15. Configuration Management✤ Automation✤ Repeatability✤ Maintainability✤ Security15Tuesday, May 7, 13
  16. 16. Configuration Management16Tuesday, May 7, 13
  17. 17. Puppetizing Postgres✤ DB server role✤ PostgreSQL installation✤ Installing common tools✤ 3 Roles : prod-master role , prod-slave role, prod-slave-backup role✤ PostgreSQL Configuration Files: postgresql.conf, pg_hba.conf✤ Scripts, crontabs17Tuesday, May 7, 13
  18. 18. High Availability✤ Replication✤ Failover18Tuesday, May 7, 13
  19. 19. Replication✤ Streaming replication✤ WAL shipping , OmniPITR✤ Different AV zones✤ At least 1 slave in different Region19Tuesday, May 7, 13
  20. 20. Failover✤ EIP - about 3 minutes to propagate changes after failover✤ Failover tools✤ Semi-automated script✤ pgbouncer✤ puppet20Tuesday, May 7, 13
  21. 21. Deployment Phase #2us-east-1AV1AV2us-west-1AppServersdb-masterdb-slave-backupdb-slaveStreaming ReplicationWAL shipping (OmniPITR)pgbouncer21Tuesday, May 7, 13
  22. 22. pgbouncer✤ Session Pooling✤ Supports all Postgres features✤ Transactions Pooling✤ Does NOT support PREPARE transactions22Tuesday, May 7, 13
  23. 23. Operations23Tuesday, May 7, 13
  24. 24. Database Upgrade✤ Upgraded from PostgreSQL 9.0 to PostgreSQL 9.2✤ Minimal outage upgrade using Mimeo✤ http://www.pateldenish.com/2012/10/an-easy-way-to-reduce-outage-window-for-postgresql-upgrade.html24Tuesday, May 7, 13
  25. 25. Beware!!✤ Sept 24th, 2012✤ PostgreSQL 9.1.0 to 9.1.5 & 9.2.0✤ VACUUM all tables in your database & Rebuild Indices✤ http://wiki.postgresql.org/wiki/20120924updaterelease✤ April 4th, 2013✤ PostgreSQL 9.0.* to 9.1.* & 9.2.*✤ Security release✤ http://www.postgresql.org/support/security/faq/2013-04-04/25Tuesday, May 7, 13
  26. 26. Backups✤ Online Backups✤ Daily, db-slave , OmniPITR✤ Test the backups!!!!✤ Schema Backups✤ pg_extractor✤ Offsite Backup✤ sending backups to Amazon S326Tuesday, May 7, 13
  27. 27. Tools✤ tracking schema in git : pg_extractor✤ pg_bloat_report✤ Pg log errors reporting : tail_n_mail✤ Log analysis: pgbadger✤ log_min_duration_statement = 50 ms✤ log_line_prefix= %t [%r] [%p]: [%l-1] user=%u,db=%d,e=%e 27Tuesday, May 7, 13
  28. 28. Monitoring28Tuesday, May 7, 13
  29. 29. Database Checks29Tuesday, May 7, 13
  30. 30. Database Checks30Tuesday, May 7, 13
  31. 31. System Checks31Tuesday, May 7, 13
  32. 32. Business Metrics32Tuesday, May 7, 13
  33. 33. Histograms!33Tuesday, May 7, 13
  34. 34. References✤ http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Storage.html✤ OmniTI labs tools✤ https://github.com/omniti-labs/pgtreats✤ OmniPITR✤ https://github.com/omniti-labs/omnipitr34Tuesday, May 7, 13

×