BitRot or Data Rot is when file or object sits un-disturb on the disk or unaccess from day by day or month by month or year by year and when you go to your pen drive, hard disk or take out your DVD and you see that you can't access that data because bit have flipped called as BitRot or DataRot. BitRot detection is a technique used to identify certain “insidious” type of disk errors where data is silently corrupted with no indication from the disk to the storage software layer that an error has occurred. With GlusterFS 3.7, its possible to detect corruption caused due to bitrot and take steps to rectify them. When bitrot detection is enabled on a Gluster volume, files are signed after they have been written. A periodic filesystem scrubber verifies the integrity of signed files are flags (or marks) files which have mismatching signature. Corrupted files are typically denied access to clients unless it's a replicated volume where it's still possible to access the "good" copy and repair the corrupted file.

1. BitRot detection in GlusterFS
Venky Shankar
Gaurav Garg

2. We are, Gluster developers at Red Hat
... participate in meetups, open source events
... hang out on #freenode: gluster, gluster-dev
nick: overclk, ggarg
... interact with community: gluster-devel@gluster.org
gluster-users@gluster.org

3. OK, enough. Let’s get started...

4. GlusterFS Quick Tour

5. Where’s my data?
● Distributed
● Local filesystem (brick)
○ XFS
○ EXT3, EXT4
○ BTRFS
● Prerequisite
○ POSIX compatible
○ Xattr support

6. Understanding data corruption

7. Corruption?
How ?
● Direct “brick” manipulation
○ Script bug
○ Admin
○ Malicious

8. Corruption?
How ?
(cont..)
● Silent corruption
○ Disk itself
■ Firmware bug
■ Mechanical wear
■ Ageing

9. Illustration

11. Solution: Integrity checks

12. Integrity Check
Consistency
● Track data modifications
○ Checksum (signature)
○ Persistent
● Verify during access
○ Recompute and check
● Repair if corrupted

13. Enough of theory, show me how it’s
done.

14. Implementation
Constraints on choices
● Big fat-file story
● Deployments
○ Distribute + Replicate
○ Stripe, now [3.7+] sharding
○ Erasure coded

15. Implementation
Constraints on choices (cotd..)
● In-band data signing
○ Costly
○ RMW cycle
○ Degraded I/O performance
● Verification
○ “Ditto”

16. Implementation
Details
● Out-of-band data signing
○ Daemon
○ Asynchronous
■ Policy
■ Strong hash (reason ?)
● Verification
○ Daemon (scrubber)
○ On-demand
○ Pre-scrubbed

17. Implementation
Details (cotd..)
● Object versioning
○ Versioned upon modification
○ Versioning xattr (64 bit)
○ Reflect “object state”
● Signature
○ xattr
○ Attached to a “version”

18. Implementation
Details (cotd..)
● Integrity checking
○ Periodic
■ daily, weekly, etc..
○ Filesystem scan
■ Signature mismates
■ Matching version
○ QoS
■ Controlled crunching
● Corrupted objects
○ Denies access (EIO)
○ Repairable
■ Replica, Codes

19. Use cases

20. Use Cases
● Small files
● Long lived data
○ Archival storage
○ WORM workload

21. Future
● Replica consistency
● Metadata checksumming
● Offloading
○ BTRFS
● Sharding adaption
● GlusterFS 4.0 [Interesting!]
○ In-band (weaker hash)
○ Checksum everything
○ Default
○ Lost (phantom) writes

22. 3.7
● Bitrot detection
● No recovery
● In comes sharding
3.7.2
● Recovery support
3.7.4
● Bug fixes
● Scrub status
3.8
● Sharding ready
● Bitrot adaption
4.0
● Hell of a change
● Sharding by default
● Checksum everything

23. Q & A