Facebook's HBase Backups - StampedeCon 2012

HBASE BackupsPritam DamaniaSoftware Engineer, FacebookAug 1, 2012

Agenda 1 Introduction to HBASE and HDFS 2 Introduction to HBASE Backups 3 Facebook’s Backup solution 4 Results 5 Further Work

INTRODUCTION TO HDFS

What is HDFS ?▪  Distributed FileSystem▪  Runs on top of commodity hardware▪  Scale to Petabytes of data▪  Tolerates machine failures

HDFS Data Model▪  Data is logically organized into files and directories▪  Files are divided into uniform-sized blocks▪  Blocks are distributed across the nodes of the cluster and are replicated to handle hardware failure▪  HDFS keeps checksums of data for corruption detection and recovery▪  HDFS exposes block placement so that computation can be migrated to data

HDFS Data Model (2) MetaInfo(Filename, replicationFactor, block-ids, …) /users/user1/data/part-0, repl:2, ids: {1,3}, … /users/user1/data/part-1, repl:3, ids: {2,4,5}, … Block Storage1 2 1 4 2 5 2 3 4 3 4 5 5 7

HDFS Architecture Metadata (Name, #replicas, …): Namenode /users/foo/data, 3, … Metadata ops Block ops Client Metadata opsRead Datanodes Datanodes Replication Blocks Write Rack 2 Rack 1 Client 8

INTRODUCTION TO HBASE

HBase in a nutshell§  distributed, large-scale data store§  can host very large tables, billions of rows x millions of columns§  efficient at random reads/writes§  open source project modeled after Google’s BigTable

HBase Data Model•  An HBase table is: •  a sparse , three-dimensional array of cells, indexed by: RowKey, ColumnKey, Timestamp/Version •  sharded into regions along an ordered RowKey space•  Within each region: •  Data is grouped into column families ▪  Sort order within each column family: •  Row Key (asc), Column Key (asc), Timestamp (desc)

HBase System Overview Database Layer HBASE Master Backup Master Region Region Region ... Server Server Server Storage Layer Coordination ServiceHDFS Zookeeper Quorum Namenode Secondary Namenode ZK ZK ... Peer PeerDatanode Datanode Datanode ...

HBase OverviewHBASE Region Server .... Region #2 Region #1 .... ColumnFamily #2 ColumnFamily #1 Memstore (in memory data structure) HFiles (in HDFS) flush Write Ahead Log ( in HDFS)

INTRODUCTION TO HBASE BACKUPS

Why Backups ?▪  Data Corruption▪  Operational error▪  Hardware failures▪  Disaster

Hbase Backups – The Problem▪  Need a consistent, point in time backup▪  Issues : ▪  Live cluster, with traffic ▪  Data in MemStore ▪  Flushes and Compations in the background ▪  Regionserver death ▪  Regions moving

CURRENT OPTIONS – Export Table▪  Pros : ▪  Can export part or full table ▪  Map-Reduce job downloads data to output path provided ▪  Supports start time, end time and versions so could provide a consistent backup ▪  Can specify which Column Families to export▪  Cons : ▪  Only one table at a time ▪  Full scans and random reads

CURRENT OPTIONS - Copy Table▪  Tool to copy existing table to a intra/inter cluster▪  Pros : ▪  Another parallel replicated setup to switch ▪  Supports start time, end time, and versions ▪  Cluster being copied to could be in different setup ▪  Can specify which Column Families to export▪  Cons : ▪  Keep another HBASE cluster up and ready ▪  Full scans and random reads

Facebook’s Backup Solution

Backups V1 Log(Put A) Application Backup Cluster Log(Put A) Put A Dedup HBase Verify

Backups V1 – Pros and Cons▪  Pros : ▪  Simple solution ▪  Consistency in backup ▪  Point in time restore ▪  Verification of backups▪  Cons : ▪  Requires replay of large amount of transactions ▪  Requires double writes and deduplication

Backups V2 Flush Region RegionServe Get File List Mapper r Flush Copy Files HDFS .regioninfo

Backups V2 – Tuning▪  Locality based mappers▪  Use in rack replication▪  Increase .Trash retention for HDFS▪  Fault tolerant▪  Use Backups V1 for point in time

Backups V2 – Restore▪  Rewrite backed up .regioninfo▪  Move backup copy in place▪  Add regions to .META using .regioninfo

Backups V2 – Pros and Cons▪  Pros : ▪  Faster restore ▪  Backup entire data in hours ▪  Consistency in backup ▪  Point in time restore ▪  Resilient to RS death, region moves▪  Cons : ▪  Affects production cluster ▪  Not scalable with data growth

Backups V2 – HDFS Improvements▪  Overhead of copying large files▪  Use locality of data▪  HDFS HFiles are immutable▪  HDFS blocks are immutable▪  Hardlinks at block level!

Fast Copy workflow Source DestinationB1 B2 ……………….. B1’ B2’ …………………… ….FastCopy Client Add Block Create Destination Get Source NameNode Copy Block B1 B1’ B1 B1’ B1 B1’ B2 B2’ B2 B2’ B2 B2’ Date Node1 Date Node2 Date Node3

FastCopy – Pros and Cons▪  Pros : ▪  Extremely fast ▪  Lots of space saving ▪  Minimal impact to production cluster▪  Cons : ▪  NameNode not aware ▪  Hardlinks lost on datanode death ▪  Balancer not aware.

Operations▪  Messages Use Case : ▪  3 stage (same cluster, off cluster, off data center) ▪  Stage 1 : once/ day ▪  Stage 2 : once / 10 day ▪  Stage 3 : once / 10 day ▪  Retention based on capacity

Results

Backup NumbersExample :▪  40 TB table▪  49 Mappers▪  Normal Copy – 15 hours▪  Fast Copy – 1.5 hours

Disk Savings - FastCopyDiskusage inpercent

Network Traffic - FastCopy

Further Work

Further Work▪  Backup HLogs▪  Point in time backups▪  Namenode level Hard links▪  Code and JIRAs : ▪  HBASE 4618 ▪  HDFS code in github (https://github.com/facebook/hadoop-20)

Acknowledgements▪  Madhuwanti Vaidya▪  Ryan Thiessen▪  Karthik Ranganathan▪  Paul Tuckfield▪  Kannan Muthukkaruppan▪  Hairong Kuang▪  Dhruba Borthakur▪  Amitanand Aiyer▪  Mikhail Bautin

Questions ?

http://play.daumcorp.com	21
http://eventifier.co	8
https://twitter.com	5
http://www.slashdocs.com	3

Facebook's HBase Backups - StampedeCon 2012

by StampedeCon on Aug 11, 2012

Accessibility

Categories

Upload Details

Usage Rights

4 Embeds 37

Statistics