“We fixed it ourselves”
Re-fixing a corrupted ZODB
@MatthewWilkes
“We had a
hardware failure
back in April”
“We tried fsrecover -p”
“We found this
script online”
“This script online”
http://www.derstappen-it.de/tech-blog/zodb-repair
“Some times you have a broken ZODB database because of
hard disk problems and sadly no suitable backup exist.“
“[We can] see what data is [...] can be saved.”
“It recovered the
ZODB, and it
deleted 1.7GB”
“We need to
delete the
remaining broken
objects”
“Yes, we had a
backup, but
there’s new
content now”
Commit rebasing
● Script takes two ZODBs
● One known good from some point in the past
● One known bad that has transactions to keep
● If fsrecover.py doesn’t work, this might
● Appends subsequent transactions to the known good, rewriting headers
● About 30 megabytes per second
● Only better than fsrecover if all corruption is in redundant section
Recovered with 0
broken or missing
objects
git.io/fpTI6

"We fixed it ourselves" - ZODB corruption recovery