HBASE BackupsPritam DamaniaSoftware Engineer, FacebookAug 1, 2012
Agenda 1   Introduction to HBASE and HDFS 2   Introduction to HBASE Backups 3   Facebook’s Backup solution 4   Results 5  ...
INTRODUCTION TO HDFS
What is HDFS ?▪  Distributed   FileSystem▪  Runs    on top of commodity hardware▪  Scale   to Petabytes of data▪  Tolerate...
HDFS Data Model▪  Data    is logically organized into files and directories▪  Files   are divided into uniform-sized block...
HDFS Data Model (2)            MetaInfo(Filename, replicationFactor, block-ids, …)            /users/user1/data/part-0, re...
HDFS Architecture                                               Metadata (Name, #replicas, …):                          Na...
INTRODUCTION TO HBASE
HBase in a nutshell§  distributed, large-scale data store§  can host very large tables, billions of rows x millions of c...
HBase Data Model•  An HBase table is: •    a sparse , three-dimensional array of cells, indexed by:       RowKey, ColumnKe...
HBase System Overview                            Database Layer   HBASE       Master     Backup Master   Region           ...
HBase OverviewHBASE Region Server             ....          Region #2       Region #1                    ....             ...
INTRODUCTION TO HBASE       BACKUPS
Why Backups ?▪  Data   Corruption▪  Operational   error▪  Hardware   failures▪  Disaster
Hbase Backups – The Problem▪  Need   a consistent, point in time backup▪  Issues   : ▪    Live cluster, with traffic ▪    ...
CURRENT OPTIONS – Export Table▪  Pros   : ▪    Can export part or full table ▪    Map-Reduce job downloads data to output ...
CURRENT OPTIONS - Copy Table▪  Tool   to copy existing table to a intra/inter cluster▪  Pros   : ▪    Another parallel rep...
Facebook’s Backup Solution
Backups V1                Log(Put                A)  Application             Backup                          Cluster      ...
Backups V1 – Pros and Cons▪  Pros   : ▪    Simple solution ▪    Consistency in backup ▪    Point in time restore ▪    Veri...
Backups V2               Flush Region RegionServe   Get File List           Mapper      r     Flush                     Co...
Backups V2 – Tuning▪  Locality   based mappers▪  Use   in rack replication▪  Increase   .Trash retention for HDFS▪  Fault ...
Backups V2 – Restore▪  Rewrite   backed up .regioninfo▪  Move   backup copy in place▪  Add   regions to .META using .regio...
Backups V2 – Pros and Cons▪  Pros   : ▪    Faster restore ▪    Backup entire data in hours ▪    Consistency in backup ▪   ...
Backups V2 – HDFS Improvements▪  Overhead    of copying large files▪  Use   locality of data▪  HDFS    HFiles are immutabl...
Fast Copy workflow           Source                                                       DestinationB1    B2   ………………..  ...
FastCopy – Pros and Cons▪  Pros   : ▪    Extremely fast ▪    Lots of space saving ▪    Minimal impact to production cluste...
Operations▪  Messages    Use Case : ▪    3 stage (same cluster, off cluster, off data center) ▪    Stage 1 : once/ day ▪  ...
Results
Backup NumbersExample :▪    40 TB table▪    49 Mappers▪    Normal Copy – 15 hours▪    Fast Copy – 1.5 hours
Disk Savings - FastCopyDiskusage inpercent
Network Traffic - FastCopy
Further Work
Further Work▪    Backup HLogs▪    Point in time backups▪    Namenode level Hard links▪    Code and JIRAs :     ▪    HBASE ...
Acknowledgements▪    Madhuwanti Vaidya▪    Ryan Thiessen▪    Karthik Ranganathan▪    Paul Tuckfield▪    Kannan Muthukkarup...
Questions ?
(c) 2009 Facebook, Inc. or its licensors. "Facebook" is a registered trademark of Facebook, Inc.. All rights reserved. 1.0
Facebook's HBase Backups - StampedeCon 2012
Upcoming SlideShare
Loading in...5
×

Facebook's HBase Backups - StampedeCon 2012

2,630

Published on

At StampedeCon 2012 in St. Louis, Pritam Damania presents: Reliable backup and recovery is one of the main requirements for any enterprise grade application. HBase has been very well embraced by enterprises needing random, real-time read/write access with huge volumes of data and ease of scalability. As such, they are looking for backup solutions that are reliable, easy to use, and can co-exist with existing infrastructure. HBase comes with several backup options but there is a clear need to improve the native export mechanisms. This talk will cover various options that are available out of the box, their drawbacks and what various companies are doing to make backup and recovery efficient. In particular it will cover what Facebook has done to improve performance of backup and recovery process with minimal impact to production cluster.

Published in: Technology
0 Comments
19 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,630
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
0
Comments
0
Likes
19
Embeds 0
No embeds

No notes for slide

Facebook's HBase Backups - StampedeCon 2012

  1. 1. HBASE BackupsPritam DamaniaSoftware Engineer, FacebookAug 1, 2012
  2. 2. Agenda 1 Introduction to HBASE and HDFS 2 Introduction to HBASE Backups 3 Facebook’s Backup solution 4 Results 5 Further Work
  3. 3. INTRODUCTION TO HDFS
  4. 4. What is HDFS ?▪  Distributed FileSystem▪  Runs on top of commodity hardware▪  Scale to Petabytes of data▪  Tolerates machine failures
  5. 5. HDFS Data Model▪  Data is logically organized into files and directories▪  Files are divided into uniform-sized blocks▪  Blocks are distributed across the nodes of the cluster and are replicated to handle hardware failure▪  HDFS keeps checksums of data for corruption detection and recovery▪  HDFS exposes block placement so that computation can be migrated to data
  6. 6. HDFS Data Model (2) MetaInfo(Filename, replicationFactor, block-ids, …) /users/user1/data/part-0, repl:2, ids: {1,3}, … /users/user1/data/part-1, repl:3, ids: {2,4,5}, … Block Storage1 2 1 4 2 5 2 3 4 3 4 5 5 7
  7. 7. HDFS Architecture Metadata (Name, #replicas, …): Namenode /users/foo/data, 3, … Metadata ops Block ops Client Metadata opsRead Datanodes Datanodes Replication Blocks Write Rack 2 Rack 1 Client 8
  8. 8. INTRODUCTION TO HBASE
  9. 9. HBase in a nutshell§  distributed, large-scale data store§  can host very large tables, billions of rows x millions of columns§  efficient at random reads/writes§  open source project modeled after Google’s BigTable
  10. 10. HBase Data Model•  An HBase table is: •  a sparse , three-dimensional array of cells, indexed by: RowKey, ColumnKey, Timestamp/Version •  sharded into regions along an ordered RowKey space•  Within each region: •  Data is grouped into column families ▪  Sort order within each column family: •  Row Key (asc), Column Key (asc), Timestamp (desc)
  11. 11. HBase System Overview Database Layer HBASE Master Backup Master Region Region Region ... Server Server Server Storage Layer Coordination ServiceHDFS Zookeeper Quorum Namenode Secondary Namenode ZK ZK ... Peer PeerDatanode Datanode Datanode ...
  12. 12. HBase OverviewHBASE Region Server .... Region #2 Region #1 .... ColumnFamily #2 ColumnFamily #1 Memstore (in memory data structure) HFiles (in HDFS) flush Write Ahead Log ( in HDFS)
  13. 13. INTRODUCTION TO HBASE BACKUPS
  14. 14. Why Backups ?▪  Data Corruption▪  Operational error▪  Hardware failures▪  Disaster
  15. 15. Hbase Backups – The Problem▪  Need a consistent, point in time backup▪  Issues : ▪  Live cluster, with traffic ▪  Data in MemStore ▪  Flushes and Compations in the background ▪  Regionserver death ▪  Regions moving
  16. 16. CURRENT OPTIONS – Export Table▪  Pros : ▪  Can export part or full table ▪  Map-Reduce job downloads data to output path provided ▪  Supports start time, end time and versions so could provide a consistent backup ▪  Can specify which Column Families to export▪  Cons : ▪  Only one table at a time ▪  Full scans and random reads
  17. 17. CURRENT OPTIONS - Copy Table▪  Tool to copy existing table to a intra/inter cluster▪  Pros : ▪  Another parallel replicated setup to switch ▪  Supports start time, end time, and versions ▪  Cluster being copied to could be in different setup ▪  Can specify which Column Families to export▪  Cons : ▪  Keep another HBASE cluster up and ready ▪  Full scans and random reads
  18. 18. Facebook’s Backup Solution
  19. 19. Backups V1 Log(Put A) Application Backup Cluster Log(Put A) Put A Dedup HBase Verify
  20. 20. Backups V1 – Pros and Cons▪  Pros : ▪  Simple solution ▪  Consistency in backup ▪  Point in time restore ▪  Verification of backups▪  Cons : ▪  Requires replay of large amount of transactions ▪  Requires double writes and deduplication
  21. 21. Backups V2 Flush Region RegionServe Get File List Mapper r Flush Copy Files HDFS .regioninfo
  22. 22. Backups V2 – Tuning▪  Locality based mappers▪  Use in rack replication▪  Increase .Trash retention for HDFS▪  Fault tolerant▪  Use Backups V1 for point in time
  23. 23. Backups V2 – Restore▪  Rewrite backed up .regioninfo▪  Move backup copy in place▪  Add regions to .META using .regioninfo
  24. 24. Backups V2 – Pros and Cons▪  Pros : ▪  Faster restore ▪  Backup entire data in hours ▪  Consistency in backup ▪  Point in time restore ▪  Resilient to RS death, region moves▪  Cons : ▪  Affects production cluster ▪  Not scalable with data growth
  25. 25. Backups V2 – HDFS Improvements▪  Overhead of copying large files▪  Use locality of data▪  HDFS HFiles are immutable▪  HDFS blocks are immutable▪  Hardlinks at block level!
  26. 26. Fast Copy workflow Source DestinationB1 B2 ……………….. B1’ B2’ …………………… ….FastCopy Client Add Block Create Destination Get Source NameNode Copy Block B1 B1’ B1 B1’ B1 B1’ B2 B2’ B2 B2’ B2 B2’ Date Node1 Date Node2 Date Node3
  27. 27. FastCopy – Pros and Cons▪  Pros : ▪  Extremely fast ▪  Lots of space saving ▪  Minimal impact to production cluster▪  Cons : ▪  NameNode not aware ▪  Hardlinks lost on datanode death ▪  Balancer not aware.
  28. 28. Operations▪  Messages Use Case : ▪  3 stage (same cluster, off cluster, off data center) ▪  Stage 1 : once/ day ▪  Stage 2 : once / 10 day ▪  Stage 3 : once / 10 day ▪  Retention based on capacity
  29. 29. Results
  30. 30. Backup NumbersExample :▪  40 TB table▪  49 Mappers▪  Normal Copy – 15 hours▪  Fast Copy – 1.5 hours
  31. 31. Disk Savings - FastCopyDiskusage inpercent
  32. 32. Network Traffic - FastCopy
  33. 33. Further Work
  34. 34. Further Work▪  Backup HLogs▪  Point in time backups▪  Namenode level Hard links▪  Code and JIRAs : ▪  HBASE 4618 ▪  HDFS code in github (https://github.com/facebook/hadoop-20)
  35. 35. Acknowledgements▪  Madhuwanti Vaidya▪  Ryan Thiessen▪  Karthik Ranganathan▪  Paul Tuckfield▪  Kannan Muthukkaruppan▪  Hairong Kuang▪  Dhruba Borthakur▪  Amitanand Aiyer▪  Mikhail Bautin
  36. 36. Questions ?
  37. 37. (c) 2009 Facebook, Inc. or its licensors. "Facebook" is a registered trademark of Facebook, Inc.. All rights reserved. 1.0

×