At StampedeCon 2012 in St. Louis, Pritam Damania presents: Reliable backup and recovery is one of the main requirements for any enterprise grade application. HBase has been very well embraced by enterprises needing random, real-time read/write access with huge volumes of data and ease of scalability. As such, they are looking for backup solutions that are reliable, easy to use, and can co-exist with existing infrastructure. HBase comes with several backup options but there is a clear need to improve the native export mechanisms. This talk will cover various options that are available out of the box, their drawbacks and what various companies are doing to make backup and recovery efficient. In particular it will cover what Facebook has done to improve performance of backup and recovery process with minimal impact to production cluster.

2. HBASE Backups
Pritam Damania
Software Engineer, Facebook
Aug 1, 2012

3. Agenda
1 Introduction to HBASE and HDFS
2 Introduction to HBASE Backups
3 Facebook’s Backup solution
4 Results
5 Further Work

4. INTRODUCTION TO HDFS

5. What is HDFS ?
▪ Distributed FileSystem
▪ Runs on top of commodity hardware
▪ Scale to Petabytes of data
▪ Tolerates machine failures

6. HDFS Data Model
▪ Data is logically organized into files and directories
▪ Files are divided into uniform-sized blocks
▪ Blocks
are distributed across the nodes of the cluster and are replicated
to handle hardware failure
▪ HDFS keeps checksums of data for corruption detection and recovery
▪ HDFS exposes block placement so that computation can be migrated to
data

7. HDFS Data Model (2)
MetaInfo(Filename, replicationFactor, block-ids, …)
/users/user1/data/part-0, repl:2, ids: {1,3}, …
/users/user1/data/part-1, repl:3, ids: {2,4,5}, …
Block Storage
1 2
1 4 2 5
2
3 4
3 4
5 5
7

8. HDFS Architecture
Metadata (Name, #replicas, …):
Namenode /users/foo/data, 3, …
Metadata ops
Block ops
Client Metadata ops
Read Datanodes Datanodes
Replication
Blocks
Write Rack 2
Rack 1
Client
8

9. INTRODUCTION TO HBASE

10. HBase in a nutshell
§ distributed, large-scale data store
§ can host very large tables, billions of rows x millions of columns
§ efficient at random reads/writes
§ open source project modeled after Google’s BigTable

11. HBase Data Model
• An HBase table is:
• a sparse , three-dimensional array of cells, indexed by:
RowKey, ColumnKey, Timestamp/Version
• sharded into regions along an ordered RowKey space
• Within each region:
• Data is grouped into column families
▪ Sort order within each column family:
• Row Key (asc), Column Key (asc), Timestamp (desc)

12. HBase System Overview
Database Layer
HBASE
Master Backup Master
Region Region Region ...
Server Server Server
Storage Layer Coordination Service
HDFS Zookeeper Quorum
Namenode Secondary Namenode ZK ZK ...
Peer Peer
Datanode Datanode Datanode ...

13. HBase Overview
HBASE Region Server
....
Region #2
Region #1
....
ColumnFamily #2
ColumnFamily #1 Memstore
(in memory data structure)
HFiles (in HDFS) flush
Write Ahead Log ( in HDFS)

14. INTRODUCTION TO HBASE
BACKUPS

15. Why Backups ?
▪ Data Corruption
▪ Operational error
▪ Hardware failures
▪ Disaster

16. Hbase Backups – The Problem
▪ Need a consistent, point in time backup
▪ Issues :
▪ Live cluster, with traffic
▪ Data in MemStore
▪ Flushes and Compations in the background
▪ Regionserver death
▪ Regions moving

17. CURRENT OPTIONS – Export Table
▪ Pros :
▪ Can export part or full table
▪ Map-Reduce job downloads data to output path provided
▪ Supports start time, end time and versions so could provide a
consistent backup
▪ Can specify which Column Families to export
▪ Cons :
▪ Only one table at a time
▪ Full scans and random reads

18. CURRENT OPTIONS - Copy Table
▪ Tool to copy existing table to a intra/inter cluster
▪ Pros :
▪ Another parallel replicated setup to switch
▪ Supports start time, end time, and versions
▪ Cluster being copied to could be in different setup
▪ Can specify which Column Families to export
▪ Cons :
▪ Keep another HBASE cluster up and ready
▪ Full scans and random reads

19. Facebook’s Backup Solution

20. Backups V1
Log(Put
A)
Application Backup
Cluster
Log(Put
A)
Put A Dedup
HBase Verify

21. Backups V1 – Pros and Cons
▪ Pros :
▪ Simple solution
▪ Consistency in backup
▪ Point in time restore
▪ Verification of backups
▪ Cons :
▪ Requires replay of large amount of transactions
▪ Requires double writes and deduplication

22. Backups V2
Flush Region
RegionServe Get File List Mapper
r
Flush Copy
Files
HDFS
.regioninfo

23. Backups V2 – Tuning
▪ Locality based mappers
▪ Use in rack replication
▪ Increase .Trash retention for HDFS
▪ Fault tolerant
▪ Use Backups V1 for point in time

24. Backups V2 – Restore
▪ Rewrite backed up .regioninfo
▪ Move backup copy in place
▪ Add regions to .META using .regioninfo

25. Backups V2 – Pros and Cons
▪ Pros :
▪ Faster restore
▪ Backup entire data in hours
▪ Consistency in backup
▪ Point in time restore
▪ Resilient to RS death, region moves
▪ Cons :
▪ Affects production cluster
▪ Not scalable with data growth

26. Backups V2 – HDFS Improvements
▪ Overhead of copying large files
▪ Use locality of data
▪ HDFS HFiles are immutable
▪ HDFS blocks are immutable
▪ Hardlinks at block level!

27. Fast Copy workflow
Source Destination
B1 B2 ……………….. B1’ B2’ ……………………
….
FastCopy Client Add Block
Create Destination
Get Source NameNode
Copy Block
B1 B1’ B1 B1’ B1 B1’
B2 B2’ B2 B2’ B2 B2’
Date Node1 Date Node2 Date Node3

28. FastCopy – Pros and Cons
▪ Pros :
▪ Extremely fast
▪ Lots of space saving
▪ Minimal impact to production cluster
▪ Cons :
▪ NameNode not aware
▪ Hardlinks lost on datanode death
▪ Balancer not aware.

29. Operations
▪ Messages Use Case :
▪ 3 stage (same cluster, off cluster, off data center)
▪ Stage 1 : once/ day
▪ Stage 2 : once / 10 day
▪ Stage 3 : once / 10 day
▪ Retention based on capacity

30. Results

31. Backup Numbers
Example :
▪ 40 TB table
▪ 49 Mappers
▪ Normal Copy – 15 hours
▪ Fast Copy – 1.5 hours

32. Disk Savings - FastCopy
Disk
usage in
percent

33. Network Traffic - FastCopy

34. Further Work

35. Further Work
▪ Backup HLogs
▪ Point in time backups
▪ Namenode level Hard links
▪ Code and JIRAs :
▪ HBASE 4618
▪ HDFS code in github (https://github.com/facebook/hadoop-20)

36. Acknowledgements
▪ Madhuwanti Vaidya
▪ Ryan Thiessen
▪ Karthik Ranganathan
▪ Paul Tuckfield
▪ Kannan Muthukkaruppan
▪ Hairong Kuang
▪ Dhruba Borthakur
▪ Amitanand Aiyer
▪ Mikhail Bautin

37. Questions ?

38. (c) 2009 Facebook, Inc. or its licensors. "Facebook" is a registered trademark of Facebook, Inc.. All rights reserved. 1.0