HUG Nov 2010: HDFS Raid - Facebook

HDFS RAID,[object Object],DhrubaBorthakur (dhruba@fb.com),[object Object],Rodrigo Schmidt (rschmidt@fb.com),[object Object],RamkumarVadali (rvadali@fb.com),[object Object],Scott Chen (schen@fb.com),[object Object],Patrick Kling (pkling@fb.com),[object Object]

Agenda,[object Object],What is RAID,[object Object],RAID at Facebook,[object Object],Anatomy of RAID,[object Object],How to Deploy,[object Object],Questions,[object Object]

What Is RAID,[object Object],[object Object],Default HDFS replication is 3,[object Object],Too much at PetaByte scale,[object Object],RAID helps save space in HDFS,[object Object],Reduce replication of “source” data,[object Object],Data safety using “parity” data,[object Object]

Tolerates 2 missing blocks, Storage cost 3x ,[object Object],1,[object Object],2,[object Object],3,[object Object],4,[object Object],5,[object Object],6,[object Object],7,[object Object],8,[object Object],9,[object Object],10,[object Object],1,[object Object],2,[object Object],3,[object Object],4,[object Object],5,[object Object],6,[object Object],7,[object Object],8,[object Object],9,[object Object],10,[object Object],1,[object Object],2,[object Object],3,[object Object],4,[object Object],5,[object Object],6,[object Object],7,[object Object],8,[object Object],9,[object Object],10,[object Object],Tolerates 4 missing blocks, Storage cost 1.4x ,[object Object],1,[object Object],2,[object Object],3,[object Object],4,[object Object],5,[object Object],6,[object Object],7,[object Object],8,[object Object],9,[object Object],10,[object Object],Source file,[object Object],P1,[object Object],P2,[object Object],P3,[object Object],P4,[object Object],Parity file,[object Object],Reed-Solomon Erasure Codes,[object Object]

RAID at Facebook,[object Object],Reduces disk usage in the warehouse,[object Object],Currently saving about 5PB with XOR RAID,[object Object],Gradual deployment,[object Object],Started with few tables,[object Object],Now used with all tables,[object Object],Reed Solomon RAID under way,[object Object]

Saving 5PB at Facebook,[object Object]

Anatomy of RAID,[object Object],Server-side:,[object Object],RaidNode,[object Object],BlockFixer,[object Object],Block placement policy,[object Object],Client-side:,[object Object],DistributedRaidFileSystem,[object Object],Raid Shell,[object Object]

Anatomy of RAID,[object Object],DataNodes,[object Object],NameNode,[object Object],[object Object]

Get files to raidRaidNode,[object Object],[object Object]

Recover files while readingJobTracker,[object Object],Raid File System,[object Object]

RaidNode,[object Object],Daemon that scans filesystem,[object Object],Policy file used to provide file patterns,[object Object],Generate parity files,[object Object],Single thread,[object Object],Map-Reduce job,[object Object],Reduces replication of source file,[object Object],One thread to purge outdated parity files,[object Object],If the source gets deleted,[object Object],One thread to HAR parity files,[object Object],To reduce inode count,[object Object]

Block Fixer,[object Object],Reconstructs missing/corrupt blocks,[object Object],Retrieves a list of corrupt files from NameNode,[object Object],Source blocks are reconstructed by “decoding”,[object Object],Parity blocks are reconstructed by “encoding”,[object Object]

Block Fixer,[object Object],Bonus: Parity HARs,[object Object],One HAR block => multiple parity blocks,[object Object],Reconstructs all necessary blocks,[object Object]

Block Fixer Stats,[object Object]

Erasure Code,[object Object],ErasureCode,[object Object],abstraction for erasure code implementations,[object Object], public void encode(int[] message, int[] parity);,[object Object], public void decode(int[] data, ,[object Object],int[] erasedLocations,,[object Object],int[] erasedValues);,[object Object],Current implementations,[object Object],XOR Code,[object Object],Reed Solomon Code,[object Object],Encoder/Decoder – uses ErasureCode to integrate with RAID framework,[object Object]

Block Placement,[object Object],Replication = 3, Tolerates any 2 errors,[object Object],1,[object Object],2,[object Object],3,[object Object],4,[object Object],5,[object Object],6,[object Object],7,[object Object],8,[object Object],9,[object Object],10,[object Object],1,[object Object],2,[object Object],3,[object Object],4,[object Object],5,[object Object],6,[object Object],7,[object Object],8,[object Object],9,[object Object],10,[object Object],1,[object Object],2,[object Object],3,[object Object],4,[object Object],5,[object Object],6,[object Object],7,[object Object],8,[object Object],9,[object Object],10,[object Object],Dependent Blocks,[object Object],Replication = 1, Parity Length = 4, Tolerates any 4 errors,[object Object],1,[object Object],2,[object Object],3,[object Object],4,[object Object],5,[object Object],6,[object Object],7,[object Object],8,[object Object],9,[object Object],10,[object Object],P1,[object Object],P2,[object Object],P3,[object Object],P4,[object Object],Dependent Blocks,[object Object]

Block Placement,[object Object],Raid introduces new dependency between blocks in source and parity files,[object Object],Default block placement is bad for RAID,[object Object],Source/Parity blocks can be on a single node/rack,[object Object],Parity blocks could co-locate with source blocks,[object Object],Raid Block Policy,[object Object],Source files: After RAIDing, disperse blocks,[object Object],Parity files: Control placement of parity blocks to avoid source blocks and other parity blocks,[object Object]

DistributedRaidFileSystem,[object Object],A filter file system implementation,[object Object],Allows clients to read “corrupt” source files,[object Object],Catches BlockMissingException, ChecksumException,[object Object],Recreates missing blocks on the fly by using parity,[object Object],Does not fix the missing blocks,[object Object],Only allows the reads to succeed,[object Object]

RaidShell,[object Object],Administrator tool,[object Object],Recover blocks,[object Object],Reconstruct missing blocks,[object Object],Send reconstructed block to a data node,[object Object],Raid FSCK,[object Object], Report corrupt files that cannot be fixed by raid,[object Object],Handy tool as a last resort to fix blocks,[object Object]

Deployment,[object Object],Single configuration file “raid.xml”,[object Object],Specifies file patterns to RAID,[object Object],In HDFS config file,[object Object],Specify raid.xml location,[object Object],Specify location of parity files (default: /raid),[object Object],Specify FileSystem, BlockPlacementPolicy,[object Object],Starting RaidNode,[object Object],start-raidnode.sh, stop-raidnode.sh,[object Object],http://wiki.apache.org/hadoop/HDFS-RAID,[object Object]

HUG Nov 2010: HDFS Raid - Facebook

HUG Nov 2010: HDFS Raid - Facebook

Recommended

More Related Content

What's hot

What's hot(20)

Similar to HUG Nov 2010: HDFS Raid - Facebook

Similar to HUG Nov 2010: HDFS Raid - Facebook(20)

More from Yahoo Developer Network

More from Yahoo Developer Network(20)

Recently uploaded

Recently uploaded(20)

HUG Nov 2010: HDFS Raid - Facebook