Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
PostgreSQL on ZFS
Bits Hopefully Structured the way you Left Them:

The scary parts followed by a Happy Ending™.
“All good lightning talks have a little FUD.”
… except it’s not FUD.
… and few care or are aware

(thank you cloud providers!)
Cloud providers have duped a generation of
operators into thinking bitrot does not exist.
It does. The reality has been hi...
–Many (myself included)
"If you are not using ZFS, you are losing data."
This has been studied recently for SSDs…
ECC on Flash works until it doesn’t.
Probability will rip your face if you do not acknowledge
its existence and its contro...
TL;DR: 4.2% -> 34% of SSDs have one UBER per year
TL;DR: 4.2% -> 34% of SSDs have one UBER per year
How many SSDs in that shiny box 'ya got there?
How many boxes are runnin...
TL;DR: 4.2% -> 34% of SSDs have one UBER per year
How many SSDs in that shiny box 'ya got there?
How many boxes are runnin...
External Factors for UBER on SSDs:
• Temperature
• Bus Power Consumption
• Data Written by the System Software
• Workload ...
In a Datacenter…
…no-one can hear your bits scream.
Well, maybe they can…
Take care of your bits.
Your bits are taking care of you.
Take care of your bits.
Your bits are taking care of you.
Answer their cry for help.
Similar studies and research exist for:
• Fibre Channel
• SAS
• SATA
• Tape
• SANs
• Cloud Object Stores
"Groupon’s Deal on FreeBSD"

FreeBSD Journal, July/August 2015
https://www.freebsdfoundation.org/past-issues/freebsd-in-th...
PostgreSQL and ZFS were meant for each other
PostgreSQL and ZFS were meant for each other
• Because bitrot happens
PostgreSQL and ZFS were meant for each other
• Because bitrot happens
• Because its fast
PostgreSQL and ZFS were meant for each other
• Because bitrot happens
• Because its fast
2-4µs/pwrite(2)!!
PostgreSQL and ZFS were meant for each other
• Because bitrot happens
• Because its fast
PostgreSQL and ZFS were meant for each other
• Because bitrot happens
• Because its fast
PostgreSQL and ZFS were meant for each other
• Because bitrot happens
• Because its fast
P.S. This was observed on 10K RPM...
PostgreSQL and ZFS were meant for each other
• Because bitrot happens
• Because its fast
• Because restoring from backups ...
PostgreSQL and ZFS were meant for each other
• Because bitrot happens
• Because its fast
• Because restoring from backups ...
PostgreSQL and ZFS were meant for each other
• Because bitrot happens
• Because its fast
• Because restoring from backups ...
PostgreSQL and ZFS were meant for each other
• Because bitrot happens
• Because its fast
• Because restoring from backups ...
PostgreSQL and ZFS were meant for each other
• Because bitrot happens
• Because its fast
• Because restoring from backups ...
PostgreSQL and ZFS were meant for each other
• Because bitrot happens
• Because its fast
• Because restoring from backups ...
It even runs on Linux!
It even runs on Linux!
Don’t become cranky and
jaded and relearn the
perils of data sadness.
Winning Strategy
Winning Strategy
Be aware.
But take the Blue Pill.
+
Not running PostgreSQL on ZFS is tantamount
to professional negligence. Discuss.
PostgreSQL on ZFS Lightning Talk
PostgreSQL on ZFS Lightning Talk
PostgreSQL on ZFS Lightning Talk
PostgreSQL on ZFS Lightning Talk
PostgreSQL on ZFS Lightning Talk
PostgreSQL on ZFS Lightning Talk
Upcoming SlideShare
Loading in …5
×

PostgreSQL on ZFS Lightning Talk

677 views

Published on

A 5min lightning talk on the whys and benefits of running PostgreSQL on ZFS. TL;DR: Use ZFS, even in the cloud.

Published in: Internet
  • Login to see the comments

PostgreSQL on ZFS Lightning Talk

  1. 1. PostgreSQL on ZFS Bits Hopefully Structured the way you Left Them:
 The scary parts followed by a Happy Ending™.
  2. 2. “All good lightning talks have a little FUD.”
  3. 3. … except it’s not FUD.
  4. 4. … and few care or are aware
 (thank you cloud providers!)
  5. 5. Cloud providers have duped a generation of operators into thinking bitrot does not exist. It does. The reality has been hidden from plain sight, but it’s there… lurking… silently, waiting…
  6. 6. –Many (myself included) "If you are not using ZFS, you are losing data."
  7. 7. This has been studied recently for SSDs…
  8. 8. ECC on Flash works until it doesn’t. Probability will rip your face if you do not acknowledge its existence and its control over your life.
  9. 9. TL;DR: 4.2% -> 34% of SSDs have one UBER per year
  10. 10. TL;DR: 4.2% -> 34% of SSDs have one UBER per year How many SSDs in that shiny box 'ya got there? How many boxes are running?
  11. 11. TL;DR: 4.2% -> 34% of SSDs have one UBER per year How many SSDs in that shiny box 'ya got there? How many boxes are running? (1-(1-uberRate)^(numDisks)) = Probability of UBER/server/year (1-(1-0.042)^(20)) = 58% (1-(1-0.34)^(20)) = 99.975%
  12. 12. External Factors for UBER on SSDs: • Temperature • Bus Power Consumption • Data Written by the System Software • Workload changes due to SSD failure
  13. 13. In a Datacenter… …no-one can hear your bits scream.
  14. 14. Well, maybe they can…
  15. 15. Take care of your bits. Your bits are taking care of you.
  16. 16. Take care of your bits. Your bits are taking care of you. Answer their cry for help.
  17. 17. Similar studies and research exist for: • Fibre Channel • SAS • SATA • Tape • SANs • Cloud Object Stores
  18. 18. "Groupon’s Deal on FreeBSD"
 FreeBSD Journal, July/August 2015 https://www.freebsdfoundation.org/past-issues/freebsd-in-the-enterprise/
 https://www.freebsdfoundation.org/wp-content/uploads/2015/12/vol2_no4_groupon.pdf
  19. 19. PostgreSQL and ZFS were meant for each other
  20. 20. PostgreSQL and ZFS were meant for each other • Because bitrot happens
  21. 21. PostgreSQL and ZFS were meant for each other • Because bitrot happens • Because its fast
  22. 22. PostgreSQL and ZFS were meant for each other • Because bitrot happens • Because its fast 2-4µs/pwrite(2)!!
  23. 23. PostgreSQL and ZFS were meant for each other • Because bitrot happens • Because its fast
  24. 24. PostgreSQL and ZFS were meant for each other • Because bitrot happens • Because its fast
  25. 25. PostgreSQL and ZFS were meant for each other • Because bitrot happens • Because its fast P.S. This was observed on 10K RPM spinning rust.
  26. 26. PostgreSQL and ZFS were meant for each other • Because bitrot happens • Because its fast • Because restoring from backups is AWFUL! # zfsnap snapshot -rv -a 25h tank/pgdata
 # zfs list -r -t snapshot # zfs rollback -r tank/pgdata@hourly-2016-09-14_14.52.00—25h
  27. 27. PostgreSQL and ZFS were meant for each other • Because bitrot happens • Because its fast • Because restoring from backups is AWFUL! # zfsnap snapshot -rv -a 25h tank/pgdata
 # zfs list -r -t snapshot # zfs rollback -r tank/pgdata@hourly-2016-09-14_14.52.00—25h This happens in seconds!
 It’s YUGE people, absolutely YUGE!
  28. 28. PostgreSQL and ZFS were meant for each other • Because bitrot happens • Because its fast • Because restoring from backups is AWFUL! • Because compression is a performance win, even on SSDs # zfs set compression=lz4 tank/pgdata (wtb publishing of benchmarks any year now… you know who you are…)
  29. 29. PostgreSQL and ZFS were meant for each other • Because bitrot happens • Because its fast • Because restoring from backups is AWFUL! • Because compression is a performance win, even on SSDs • Because compression is a space win (2.2:1 compression for most PG data)
  30. 30. PostgreSQL and ZFS were meant for each other • Because bitrot happens • Because its fast • Because restoring from backups is AWFUL! • Because compression is a performance win, even on SSDs • Because compression is a space win (2.2:1 compression for most PG data) • Because zfs snap; zfs send; ssh… ; zfs recv
  31. 31. PostgreSQL and ZFS were meant for each other • Because bitrot happens • Because its fast • Because restoring from backups is AWFUL! • Because compression is a performance win, even on SSDs • Because compression is a space win (2.2:1 compression for most PG data) • Because zfs snap; zfs send; ssh… ; zfs recv • Because caching compressed records is win https://www.illumos.org/issues/6950
  32. 32. It even runs on Linux!
  33. 33. It even runs on Linux! Don’t become cranky and jaded and relearn the perils of data sadness.
  34. 34. Winning Strategy
  35. 35. Winning Strategy Be aware. But take the Blue Pill.
  36. 36. +
  37. 37. Not running PostgreSQL on ZFS is tantamount to professional negligence. Discuss.

×