From 0 to ~100: Business Continuity with PostgreSQL

1. WHO WANTS A SERVICE WITH ZERO DOWNTIME?

2. … EVERYBODY

3. IS IT THAT GOOD?

4. NOT JUST TECHNOLOGY. RISKS, PROCEDURES, PEOPLE

5. FROM 0 TO ~100: BUSINESS CONTINUITY WITH POSTGRESQL Gabriele Bartolini Head of Support @ 2ndQuadrant PgDay.IT 2017, Milan

6. 2ndquadrant.com @_GBartolini_ #PGDayIT ABOUT MYSELF ▸ Open Source passionate and programmer since 1995 ▸ First time with Postgres in 1997, regular from ~2000 ▸ Lean and DevOps practitioner ▸ Co-Founder of ITPUG and PostgreSQL Europe ▸ Entrepreneur, with 2ndQuadrant since 2008 ▸ Co-Author of “PostgreSQL Administration Cookbook” ▸ Came up with the name “Barman”

7. 2ndquadrant.com @_GBartolini_ #PGDayIT BUSINESS CONTINUITY ▸ Disaster Recovery ▸ High Availability ▸ Types of disaster/failures ▸ Availability = Uptime / (Uptime + Downtime)

8. 2ndquadrant.com @_GBartolini_ #PGDayIT OBJECTIVES ▸ Recovery Point Objective (RPO) ▸ How much data can I afford to lose? ▸ Recovery Time Objective (RTO) ▸ How long will it take me to recover?

9. 2ndquadrant.com @_GBartolini_ #PGDayIT SERVICE RELIABILITY ▸ Cost of downtime ▸ How many €/$/£/AUD/…? ▸ Risk management ▸ SLI, SLO and SLA

10. 2ndquadrant.com @_GBartolini_ #PGDayIT SOME NOTES FOR THIS PRESENTATION ▸ PostgreSQL on Linux ▸ Servers can be either physical or virtual ▸ Storage must be redundant ▸ RAID is required ▸ VOLUME: redundant disk mounted on a system

11. LET’S START

12. 0. ONE POSTGRES SERVER

13. 2ndquadrant.com @_GBartolini_ #PGDayIT ARCHITECTURE Server name: hope

14. 2ndquadrant.com @_GBartolini_ #PGDayIT RECAP ▸ Why is RPO = ∞? ▸ Why is RTO = n/a? ▸ “Hope is not a strategy” (cit. Google) ▸ More common than you’d expect

15. 10. ONE POSTGRES SERVER + LOGICAL BACKUPS

16. 2ndquadrant.com @_GBartolini_ #PGDayIT ARCHITECTURE Add systematic backups with pg_dump LOGICAL BACKUP LOGICAL BACKUP LOGICAL BACKUP … Day 0  4AM Day -1 4AM Day -2 4AM

17. 2ndquadrant.com @_GBartolini_ #PGDayIT RECAP ▸ How do you feel now? ▸ Still: RPO = ∞ and RTO = n/a. Why? ▸ A backup is valid only if you have tested it ▸ Unfortunately, this is very common

18. 20. ONE POSTGRES SERVER + LOGICAL BACKUPS + LOGICAL RESTORES

19. 2ndquadrant.com @_GBartolini_ #PGDayIT ARCHITECTURE Test your backups with pg_restore LOGICAL BACKUP Day 0  4AM

20. 2ndquadrant.com @_GBartolini_ #PGDayIT DEFINING SOME OBJECTIVES ▸ Measure time for pg_restore ▸ RPO = backup frequency ▸ RTO = maximum time of recovery ▸ Provision another server ▸ Conﬁgure another server (automated, right?) ▸ Time to restore the last backup (measure it)

21. HAVE WE REALLY THOUGHT ABOUT EVERYTHING?

22. TIME OF REACTION

23. 2ndquadrant.com @_GBartolini_ #PGDayIT RECAP ▸ Can this architecture work for you? ▸ We need reliable monitoring ▸ From now on, we assume we have it in place! ▸ We need to reduce both RPO and RTO

24. HOW? POINT-IN-TIME-RECOVERY

25. 2ndquadrant.com @_GBartolini_ #PGDayIT POSTGRESQL’S PITR ▸ Part of core (fully open source) ▸ Rebuild a cluster at a point in time ▸ From crash recovery to sync streamrep (physical/logical) ▸ RPO = 0 (zero data loss) ▸ Hot base backup, continuous WAL archiving, Recovery ▸ API

26. 2ndquadrant.com @_GBartolini_ #PGDayIT BASIC CONCEPTS ▸ Continuous copy of WAL data (continuous archiving) ▸ Physical base backups ▸ Recovery: ▸ copy base backup to another location ▸ recovery mode (replay of WALs until target)

27. 2ndquadrant.com @_GBartolini_ #PGDayIT BARMAN ▸ In this presentation: Barman 2.3 ▸ Open Source (GNU GPL 3) ▸ Written in Python ▸ Developed and maintained by 2ndQuadrant ▸ Available at www.pgbarman.org

28. 40. ONE POSTGRES SERVER + ONE BARMAN SERVER

29. 2ndquadrant.com @_GBartolini_ #PGDayIT ARCHITECTURE Continuous backup

30. 2ndquadrant.com @_GBartolini_ #PGDayIT BASIC CONCEPTS ▸ Remote backup and recovery ▸ Multiple server management ▸ Backup catalogue and WAL archive ▸ Retention policies

31. 2ndquadrant.com @_GBartolini_ #PGDayIT COPY METHOD ▸ PostgreSQL streaming ▸ Practical/Windows/Docker ▸ Rsync/SSH ▸ Incremental backup and recovery (via hard links) ▸ Parallel backup and recovery ▸ Network compression and bandwidth limitation

32. 2ndquadrant.com @_GBartolini_ #PGDayIT WAL SHIPPING METHOD ▸ “archiving”, through “archive_command”: ▸ RPO ~ 16MB of WAL data, or ▸ “archive_timeout” ▸ “streaming”, through streaming replication: ▸ “pg_receivewal” or “pg_receivexlog” ▸ continuous stream, RPO ~ 0 ▸ PostgreSQL 9.2+ required

33. 2ndquadrant.com @_GBartolini_ #PGDayIT EXAMPLE FROM POSTGRESQL.CONF archive_mode = on wal_level = logical max_wal_senders = 10 max_replication_slots = 10 archive_command = 'rsync -a %p barman@HOST:/var/lib/barman/ID/incoming'

34. 2ndquadrant.com @_GBartolini_ #PGDayIT EXAMPLE FROM BARMAN.CONF [angus] description = “Angus Young database" ssh_command = ssh postgres@angus conninfo = user=barman-acdc dbname=postgres host=angus retention_policy = RECOVERY WINDOW OF 6 MONTHS copy_method = rsync reuse_backup = link parallel_jobs = 4 archiver = true streaming_archiver = true slot_name = barman_streaming_acdc

35. 2ndquadrant.com @_GBartolini_ #PGDayIT RECAP ▸ How do you feel now? ▸ Still: RPO = ∞ and RTO = n/a. Why? ▸ A backup is valid only if you have tested it ▸ Barman reduces backup risks, does not exclude them ▸ Systematic tests (especially custom scripts) ▸ Business risk is very high

36. 60. ONE POSTGRES SERVER + ONE BARMAN SERVER + ONE RECOVERY SERVER

37. 2ndquadrant.com @_GBartolini_ #PGDayIT ARCHITECTURE Test your backups with barman recover

38. WHAT A WASTE!

39. TESTING OR BI? HAVE YOU EVER THOUGHT OF USING IT FOR

40. 2ndquadrant.com @_GBartolini_ #PGDayIT HOOK SCRIPTS ▸ Barman has hook scripts: ▸ pre and post backup ▸ pre and post archiving ▸ with retry option (until the script returns SUCCESS)

41. 2ndquadrant.com @_GBartolini_ #PGDayIT EXAMPLE OF RECOVERY SCRIPT ▸ Write a bash script that: ▸ connects to a remote server via SSH ▸ stops the PostgreSQL server ▸ issues a “barman recover” with target “immediate” ▸ starts the PostgreSQL ▸ Set it as post-backup script

42. 2ndquadrant.com @_GBartolini_ #PGDayIT SOME FOOD FOR THOUGHT ▸ Outcomes: ▸ Systematically test your backup ▸ Measure your recovery time ▸ Identical server? This is a backup server ready to start ▸ You can use a different data centre ▸ Be creative, PostgreSQL gives you inﬁnite freedom!

43. 2ndquadrant.com @_GBartolini_ #PGDayIT RECAP ▸ RPO ~ 0 (your backups work, every time) ▸ RTO = Time of reaction + Recovery time ▸ Example: RPO ~0 and RTO < 1 day ▸ Acceptable or not acceptable? ▸ Entry level architecture for business continuity ▸ Priority now: improve RTO

44. HOW? REPLICATION

45. 2ndquadrant.com @_GBartolini_ #PGDayIT POSTGRESQL’S REPLICATION ▸ Part of core (fully open source) ▸ One master, multiple standby servers ▸ Evolution of PITR ▸ Standby server is in continuous recovery mode ▸ Hot standby (read-only) ▸ Both streaming (9.0+) and ﬁle based pulling of WAL ▸ Cascading from a standby

46. 2ndquadrant.com @_GBartolini_ #PGDayIT SYNCHRONOUS REPLICATION ▸ Fine control (from global down to transaction level) ▸ 2-safe replication ▸ COMMIT of a write transactions waits until written on both the master and a standby (or more from 9.6) ▸ More than a synchronous client is required ▸ Read consistency of a cluster ▸ RPO = 0 (zero data loss)

47. 80. TWO POSTGRES SERVERS + ONE BARMAN SERVER + ONE RECOVERY SERVER

48. 2ndquadrant.com @_GBartolini_ #PGDayIT ARCHITECTURE barman_restore_wal barman recover Symmetric Cluster master standbyANGUS MALCOLM

49. 2ndquadrant.com @_GBartolini_ #PGDayIT EXCERPT FROM POSTGRESQL’S CONFIGURATION postgresql.conf: hot_standby = on recovery.conf: standby_mode = ‘on' # Streaming primary_conninfo = 'host=angus user=replica application_name=ha sslmode=require’ # Fallback via Barman restore_command = 'barman-wal-restore -U barman acdc angus %f %p'

50. 2ndquadrant.com @_GBartolini_ #PGDayIT SWITCHOVER (PLANNED) ▸ Applications are paused (start of downtime) ▸ Shut down the master ▸ Allow the standby to catch up with the master ▸ Promote the standby ▸ Switch virtual IPs ▸ Resume applications (end of downtime) ▸ Reconﬁgure the former master as standby

51. 2ndquadrant.com @_GBartolini_ #PGDayIT FAILOVER (UNPLANNED) ▸ The master is down (start of downtime) ▸ Promote the standby ▸ Change the virtual IP ▸ DEGRADED SYSTEM

52. 2ndquadrant.com @_GBartolini_ #PGDayIT MANUAL SWITCHOVER AND FAILOVER ▸ Manual switchover != manual switchover procedure ▸ Manual switchover = manually triggered ▸ Automate the procedure!!! ▸ bash (good) ▸ Ansible (better) ▸ Enhance gradually

53. 2ndquadrant.com @_GBartolini_ #PGDayIT RECAP ▸ RPO ~ 0 (your backups work, every time) ▸ RTO = Time of reaction + Time of promotion ▸ Criticality: manual intervention ▸ Reliable monitoring ▸ Trained people (practice & docs!)

54. 2ndquadrant.com @_GBartolini_ #PGDayIT MANUAL FAILOVER VS AUTOMATED FAILOVER ▸ Risk management ▸ Split brain nightmare ▸ Automated is built on manual (test!) ▸ Your choice ▸ Very good solution for business continuity ▸ Uptime > 99.99% in a year

55. 90. TWO POSTGRES SYNC SERVERS + ONE BARMAN SERVER + ONE RECOVERY SERVER

56. 2ndquadrant.com @_GBartolini_ #PGDayIT ARCHITECTURE barman_restore_wal barman recover Potential synchronous Synchronous ZERO DATA LOSS

57. 2ndquadrant.com @_GBartolini_ #PGDayIT SYNCHRONOUS REPLICATION ▸ Primary: Barman ▸ Zero data loss backup ▸ Primary: Standby ▸ Zero data loss cluster (reduce RTO) ▸ Just one conﬁguration line in PostgreSQL ▸ synchronous_standby_names = '1 (ha, barman_receive_wal)'

58. ~100. TWO POSTGRES SYNC SERVERS + ONE BARMAN SERVER + ONE RECOVERY SERVER + REPMGR (AUTO-FAILOVER)

59. 2ndquadrant.com @_GBartolini_ #PGDayIT ARCHITECTURE Potential synchronous Synchronous repmgr repmgr repmgr witness

60. WHAT’S MORE?

61. 2ndquadrant.com @_GBartolini_ #PGDayIT PUSH THE BOUNDARIES ▸ Repeatable architectures ▸ PgBouncer ▸ Virtual IPs ▸ S3 relay via Barman hook scripts ▸ Multiple standby servers and cascading replication ▸ Docker containers ▸ Logical replication backups

62. 2ndquadrant.com @_GBartolini_ #PGDayIT CONCLUSIONS ▸ Babysteps and KISS ▸ New? Explore and learn ▸ Practice is the only way to mastery (drills) ▸ Plan regular healthy downtimes ▸ Use switchovers to perform PostgreSQL updates ▸ Smart downtimes increase long-term uptime

63. 2ndquadrant.com @_GBartolini_ #PGDayIT ANY QUESTIONS? ▸ PostgreSQL: www.postgresql.org ▸ Barman: www.pgbarman.org ▸ Barman Cli: github.com/2ndquadrant-it/barman-cli ▸ PgBouncer: pgbouncer.github.io ▸ Repmgr: www.repmgr.org ▸ Our blog: blog.2ndquadrant.com

64. 2ndquadrant.com @_GBartolini_ #PGDayIT LICENCE Attribution 4.0 International (CC BY 4.0) You are free to: ▸ Share — copy and redistribute the material in any medium or format ▸ Adapt — remix, transform, and build upon the material for any purpose, even commercially. The licensor cannot revoke these freedoms as long as you follow the license terms.