The document describes ClearCare's migration of their PostgreSQL database architecture to AWS to meet scalability, availability, automation, and HIPAA compliance requirements. Key aspects included setting up a multi-AZ deployment with streaming replication for high availability, auto scaling read replicas, automated backups to EBS snapshots, role-based access control with LDAP, encryption of data at rest and in transit, and centralized logging and auditing for compliance. The new architecture provides improved performance, security, automation, and a cost-effective solution to support ClearCare's growing business needs.

1. HIPAA Compliant, Scalable and
Highly Available PostgreSQL
Architecture in AWS
Yaser Raja
Architect - OpenSCG
Glenn Poston
Manager Devops - ClearCare

2. Overview
Company Overview
Previous DB
Architecture
Requirements
How we got
there

4. 130 employees
25 engineers
Support 3000 agencies
Provide care to 160,000 patients
20,000 requests per minute
1.6 million DB transaction per minute

5. Old Architecture
Evolved
Organically
Single Points
of Failure
Hard Coded
Connections
No
Automation

6. Requirements
Highly Available
Scalable
Compliant
Automated
Cost Effective

7. Requirements - High Availability
No impact from loss of any 1 server
Recover from loss of master DB in <5m

8. Requirements - Scalable
• Scale without code changes
• Architecture is transparent to the app
Decouple Infrastructure
from App
• Easily isolate certain DB trafficBuilt in Flexibility

9. Requirements - HIPAA
Amazon BAA
Requirements
•Dedicated Tenancy
•Encrypted EBS Volumes
Encryption in
Transit
Encryption at Rest Encrypted Backups
Auditability (who,
what, when)

10. Requirements - Automation
Infrastructure
as Code
Repeatable
Processes
Infrastructure
Testing Reduce Risk
Faster cycle times
Environment Parity

11. Requirements - Cost

12. Phase 1
Monitoring, Alerting, and Access

13. Tools

14. Phase 2
Automation Framework

15. Tools

16. Build Pipeline

17. Phase 3
PostgreSQL Upgrade 9.2 to 9.5

18. PostgreSQL Upgrade
PG 9.2 PG 9.2 PG 9.2
PG 9.2
Streaming
Replication
PG 9.5
Bucardo Replication
PG 9.5 PG 9.5 PG 9.5
Streaming
Replication
EBS Snapshot

19. Backups
WAL-E backup was talking 6-12 hours with ~2TB cluster
EBS volume snapshots
Incremental backups
WAL-E only for WAL archiving
New volume created from the latest snapshot for new slaves and PITR

20. Phase 4
Read Tier HA and Scalability

21. High Availability - Read Tier
• Docker best practice… 1 process per container.HAProxy with App Servers
• Connection management and Docker best practicepgBouncers with App Servers
• More infrastructure to manage.pgPool
• More infrastructure to manage.HAProxy servers
• Decreased visibility.
AWS Elastic Load Balancer
(ELB)

22. High Availability - Read Tier
Pro’s:
• Easy to setup and highly available
• Fully managed service provided by AWS (low maintenance)
• Automatically takes care of routing traffic to correct AZ
Con’s:
• Limited visibility
• The DBs will only see the ELB IPs as the source and will not know about
the client IPs
• ELB is an extra hop which will introduce some latency
• ELB cost will be added

23. Scalability
AWS Auto Scaling Groups (ASG)
Integrates with ELB
Easily increase and decrease instance count in read tier
The AMI is generated by Jenkins jobs
Automates Instance Recovery

24. Scalability
Automatically
Handles Instance
Failure
…but new instances
will not have latest
data
… so an additional
process is required
to sync up with
master

25. Read Tier
POSTGRESQL
ASG
ELB
APP
SERVERS
ELB
APP 1 APP 2 APP 3 …
PGB
+
PG
PGB
+
PG
PGB
+
PG
read.db

26. Challenges
• Performance Degradation
Cache Building
Takes Time
• Lazy copy challenge
• Need time to warm up - fio or dd
EBS Takes
Time to Warm

27. Cache Build Up Impact

28. Phase 5
Master DB High Availability

29. High Availability – Master DB
Dedicated
Failover
Server
Cascading
Replication
DNS name
with low TTL

30. Master Tier
POSTGRESQL
WRITE
ASG
APP
SERVERS
APP 1 APP 2 APP 3 …
PGB
+
PG
PGB
+
PG
write.db
DB
Slaves
WAL Streaming WAL Streaming

31. OS and PostgreSQL Security Update
Easy to roll out the changes
Zero downtime required for read tier
Few seconds downtime for master

32. Phase 6
HIPAA - Securing Data

33. AWS BAA - Instance Tenancy

34. Securing the Data at Rest
Encrypted EBS volumes
Create a new encrypted volume the latest unencrypted snapshot
Performance impact less that 5%

35. Securing the Data in Motion
SSL in PostgreSQL
• Traffic from app to pgBouncer is encrypted w/ SSL (starting from v 1.7).
• Traffic from pgBouncer to DB is local so no encryption required.
• Replication traffic is using SSL.
Issue encountered
• Site slowdown
• Large number of “idle in transaction” connections on DB
CPU ‘pegged’
104283 pgbounce 20 0 52860 16m 3604 R 100 0.0 4031:47 /usr/bin/pgbouncer -R -d -q /etc/pgbouncer/pgbouncer.ini

36. Securing the Data in Motion
HAPROXY
PGB 1 PGB 2
POSTGRESQL DB
round robin

37. PGBouncer Utilization
PostgreSQL
Total Network Traffic

38. PGBouncer Load During Peak Hours
HAProxy Load During Peak Hours

39. Phase 7
HIPAA - ACCESS CONTROL

40. DB User Management
LDAP authentication using JumpCloud (pg_ldap_sync)
Postgres roles being used to grant appropriate permissions
AWS Security Group setting to lock down access
SSH access controlled via JumpCloud
Access allowed to select users only from a dedicated server

41. DB Credentials In App
HIPAA: No shared users
Removed
passwords from
source code
KMS used for password encryption
KMS key
protected by IAM
IAM role assigned
only to app
servers
Password
decrypted at
runtime

42. Phase 8
HIPAA - Logging and Auditing

43. DB Logging
HIPAA: Auditability (Who did what? When?)
• App Traffic can be audited from application logs.
• Full query logging for all other DB users
App and Query Logs shipped to Loggly (BAA).
ALTER USER myuser SET log_min_duration_statement=0;

44. Security/Compliance Audit
HIPAA: Document, Configure, Report.
•Role based access set up as we expect?
•Who’s viewing and making changes?
•Are we using dedicated instances?
•Are we using encrypted EBS?
•Is traffic encrypted?
•Do we have backups and are they working?
•Are IAM roles (password decryption) set up correctly?
•Are security groups set up correctly?
Auditing is Time Consuming, so automate!

45. Summary

46. Value

47. Collaboration
Build/Automation Expertise
AWS Expertise
SRE/Ops Expertise
Postgres Expertise
Developer Expertise
Security/Compliance Expertise

48. Cost Effective

49. Questions?