Deploying postgre sql on amazon ec2
Upcoming SlideShare
Loading in...5
×
 

Deploying postgre sql on amazon ec2

on

  • 6,477 views

 

Statistics

Views

Total Views
6,477
Slideshare-icon Views on SlideShare
2,717
Embed Views
3,760

Actions

Likes
0
Downloads
24
Comments
1

12 Embeds 3,760

http://www.pateldenish.com 3731
http://192.168.1.36 10
http://admin.pateldenish.com 5
http://tbd.dnsdynamic.com 3
http://prlog.ru 2
http://www.365dailyjournal.com 2
https://reader.aol.com 2
http://127.0.0.1 1
http://translate.googleusercontent.com 1
http://localhost 1
http://ww5.pateldenish.com 1
http://cc.bingj.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

11 of 1

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • Great material for Postgres on EC2 Cloud.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Deploying postgre sql on amazon ec2 Deploying postgre sql on amazon ec2 Presentation Transcript

    • Denish PatelDatabase Architect, OmniTIDeploying PostgreSQL onAmazon EC2A case study1Tuesday, May 7, 13
    • Story teller?✤ Denish Patel✤ denish@omniti.com✤ Database Architect at OmniTI✤ Heterogeneous Environment✤ PostgreSQL, Oracle , MySQL, NoSQL✤ Scalability, Performance & Simplicity✤ OmniTI is hiring Database Engineer!2Tuesday, May 7, 13
    • Situation✤ Startup✤ Ruby on Rails Shop✤ Entire IT architecture on Amazon Cloud✤ 2 Identical Database Environments on EC2✤ 4 Production Instances✤ 50GB Each PostgreSQL 9.0 Master - Slave ( 2 Pairs)✤ 2 QA and 2 Dev environments3Tuesday, May 7, 13
    • Situation✤ Weekly flake out issues on EC2 instances✤ Weekly backups using pg_dump✤ No file system level backup✤ Master-slave wasn’t working for months!✤ Schema mismatch between 2 environments✤ Almost NO monitoring in-place for DBs4Tuesday, May 7, 13
    • Situation✤ Prod , QA and Dev ...✤ M1 Extra large instances (m1.xlarge)✤ 15 GB RAM, 4 Virtual Cores with 2 EC2 compute units each✤ Configuration Management tool - Puppet5Tuesday, May 7, 13
    • The Amazon Cloud✤ Business Benefits✤ Offloading heavy lifting✤ Lowers cost✤ Reduces time to market6Tuesday, May 7, 13
    • Amazon Data Centers7Tuesday, May 7, 13
    • Amazon Instances✤ High CPU✤ High IO✤ High Memory8Tuesday, May 7, 13
    • Amazon StorageTypes9Tuesday, May 7, 13
    • Network and Security✤ Amazon Security Groups✤ Keep your Amazon EC2 firewall permissions as restrictive aspossible.✤ Elastic IP Address (EIP)10Tuesday, May 7, 13
    • Deployment Strategies✤ Sizing Instances and Storage Types✤ Configuration Management✤ High Availability✤ Replication✤ Failover✤ Operations✤ Upgrade, Backups, Maintenance tools✤ Monitoring11Tuesday, May 7, 13
    • Sizing Instances/Storage✤ Understand your database , schema and data✤ Env 1: 50GB , Env 2: 25GB✤ 2 large archival tables✤ Working Set about 20-30GB in total✤ Both environments are inter-related.✤ Decided to consolidate both environment into single cluster12Tuesday, May 7, 13
    • Sizing Instances/Storage✤ m2.2xlarge: 34.2GB RAM, 4 Virtual Core✤ Storage✤ EBS✤ RAID 10✤ Not EBS optimized ( Cost Savings!)✤ 850 GB13Tuesday, May 7, 13
    • Deployment Phase #1us-east-1AV1AV2us-west-1AppServersdb-master db-slave-backupdb-slaveStreaming ReplicationWAL shipping (OmniPITR)14Tuesday, May 7, 13
    • Configuration Management✤ Automation✤ Repeatability✤ Maintainability✤ Security15Tuesday, May 7, 13
    • Configuration Management16Tuesday, May 7, 13
    • Puppetizing Postgres✤ DB server role✤ PostgreSQL installation✤ Installing common tools✤ 3 Roles : prod-master role , prod-slave role, prod-slave-backup role✤ PostgreSQL Configuration Files: postgresql.conf, pg_hba.conf✤ Scripts, crontabs17Tuesday, May 7, 13
    • High Availability✤ Replication✤ Failover18Tuesday, May 7, 13
    • Replication✤ Streaming replication✤ WAL shipping , OmniPITR✤ Different AV zones✤ At least 1 slave in different Region19Tuesday, May 7, 13
    • Failover✤ EIP - about 3 minutes to propagate changes after failover✤ Failover tools✤ Semi-automated script✤ pgbouncer✤ puppet20Tuesday, May 7, 13
    • Deployment Phase #2us-east-1AV1AV2us-west-1AppServersdb-masterdb-slave-backupdb-slaveStreaming ReplicationWAL shipping (OmniPITR)pgbouncer21Tuesday, May 7, 13
    • pgbouncer✤ Session Pooling✤ Supports all Postgres features✤ Transactions Pooling✤ Does NOT support PREPARE transactions22Tuesday, May 7, 13
    • Operations23Tuesday, May 7, 13
    • Database Upgrade✤ Upgraded from PostgreSQL 9.0 to PostgreSQL 9.2✤ Minimal outage upgrade using Mimeo✤ http://www.pateldenish.com/2012/10/an-easy-way-to-reduce-outage-window-for-postgresql-upgrade.html24Tuesday, May 7, 13
    • Beware!!✤ Sept 24th, 2012✤ PostgreSQL 9.1.0 to 9.1.5 & 9.2.0✤ VACUUM all tables in your database & Rebuild Indices✤ http://wiki.postgresql.org/wiki/20120924updaterelease✤ April 4th, 2013✤ PostgreSQL 9.0.* to 9.1.* & 9.2.*✤ Security release✤ http://www.postgresql.org/support/security/faq/2013-04-04/25Tuesday, May 7, 13
    • Backups✤ Online Backups✤ Daily, db-slave , OmniPITR✤ Test the backups!!!!✤ Schema Backups✤ pg_extractor✤ Offsite Backup✤ sending backups to Amazon S326Tuesday, May 7, 13
    • Tools✤ tracking schema in git : pg_extractor✤ pg_bloat_report✤ Pg log errors reporting : tail_n_mail✤ Log analysis: pgbadger✤ log_min_duration_statement = 50 ms✤ log_line_prefix= %t [%r] [%p]: [%l-1] user=%u,db=%d,e=%e 27Tuesday, May 7, 13
    • Monitoring28Tuesday, May 7, 13
    • Database Checks29Tuesday, May 7, 13
    • Database Checks30Tuesday, May 7, 13
    • System Checks31Tuesday, May 7, 13
    • Business Metrics32Tuesday, May 7, 13
    • Histograms!33Tuesday, May 7, 13
    • References✤ http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Storage.html✤ OmniTI labs tools✤ https://github.com/omniti-labs/pgtreats✤ OmniPITR✤ https://github.com/omniti-labs/omnipitr34Tuesday, May 7, 13