Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Mapreduce over snapshots

4,459 views

Published on

Published in: Technology

Mapreduce over snapshots

  1. 1. MapReduce over snapshots HBASE-8369 Enis Soztutar Enis [at] apache [dot] org @enissoz © Hortonworks Inc. 2011 Page 1
  2. 2. About Me • In the Hadoop space since 2007 • Committer and PMC Member in Apache HBase and Hadoop • Working at Hortonworks as member of Technical Staff • Twitter: @enissoz Architecting the Future of Big Data © Hortonworks Inc. 2011 Page 2
  3. 3. Snapshots • Currently a snapshot is a bunch of reference files together with some metadata • A table’ snapshot can contain – Table descriptor – List of regions – References to files in the regions – References to WALs for regionservers • Current snapshot impl is flush based – Forces flush to all regions, so that in-memory data is written to disk Architecting the Future of Big Data © Hortonworks Inc. 2011 Page 3
  4. 4. MR over Snapshots • Idea is do scan’s on the client side bypassing region servers • Use snapshots since they are immutable • Similar to short circuit hdfs reads • TableSnapshotInputFormat works similar to TableInputFormat • TableMapReduceUtil methods to configure the job Architecting the Future of Big Data © Hortonworks Inc. 2011 Page 4
  5. 5. Deployment Options HBase online • Take snaphot while HBase is running • Run MR job over the snapshot HBase offline • Take snapshot while HBase is running • Export Snapshot using ExportSnapshot to a different hdfs • Run MR job over snapshot with or without HBase running Architecting the Future of Big Data © Hortonworks Inc. 2011 Page 5
  6. 6. TableSnapshotInputFormat • Gets a Scan representing the query • Restore the snapshot to a temporary directory • For each region in the snapshot: – Determine whether the region should be scanned (falls between scan start row and stop row) – Create one split per region in the scan range ( # of map tasks) – Each RecordReader will open the region (Hregion) as in HRegionServer – An internal RegionScanner is used for running the scan Architecting the Future of Big Data © Hortonworks Inc. 2011 Page 6
  7. 7. API Architecting the Future of Big Data © Hortonworks Inc. 2011 Page 7
  8. 8. Timeline • Will (hopefully) be committed to trunk next week or so • Interest in bringing this to 0.94 and 0.96 bases as well • Will come in HDP-2.1, which will be based on 0.96 line Architecting the Future of Big Data © Hortonworks Inc. 2011 Page 8
  9. 9. Security Aspects • HBase user owns the files in filesystem • Snapshot files are also owned by the HBase user • Mapreduce job should be able to read the files in the snapshot + actual data files • HDFS only has posix-like perms based on user/group/other – User running MR job has to be either the HBase user, or have group perms – HDFS does not have ACL’s, so there is no easy way to grant read access at filesystem layer • Idea: similar to current short circuit impl, we can implement a FD transfer – User will submit jobs under her own user credentials – Ask HBase daemons to open the files, and pass a handler / token Architecting the Future of Big Data © Hortonworks Inc. 2011 Page 9
  10. 10. Performance ScanTest: • Scan : open a scanner, do full table scan • SnapshotScan : open a client-side scanner, do full table scan • ScanMR : parallel full table scan from MR • SnapshotScanMR : do full table scan • • • • 8 Region servers, 6 disks each HBase trunk Hadoop-2.2 (HDP-2.0.7.0-12) Load data with IntegrationTestBulkLoad – Evenly distributed rows, created as bulk loaded hfiles. 3 column families • # store files per region varies 3,6,9, and 12 (1,2,3,4 file per store) • Data sizes: 6.6G, 13.2G, 19.8G, 26.4G Architecting the Future of Big Data © Hortonworks Inc. 2011 Page 10
  11. 11. Scan speed Architecting the Future of Big Data © Hortonworks Inc. 2011 Page 11
  12. 12. API • We do not want to limit snapshot scanning only to MapReduce • Allow client side scanners over snapshot files Architecting the Future of Big Data © Hortonworks Inc. 2011 Page 12
  13. 13. ResultScanner is main scan API Architecting the Future of Big Data © Hortonworks Inc. 2011 Page 13
  14. 14. API (caution: not final yet) Architecting the Future of Big Data © Hortonworks Inc. 2011 Page 14
  15. 15. To the future and beyond • HBASE-8691 High-Throughput Streaming Scan API • Can we bypass regionservers without taking snapshots? • Bypass memstore data, or stream memstore data, but read directly from hfiles • Secure reading from snapshots • Keep up with the updates at – https://issues.apache.org/jira/browse/HBASE-8369 Architecting the Future of Big Data © Hortonworks Inc. 2011 Page 15
  16. 16. Thanks Questions? Enis Söztutar enis [ at ] apache [dot] org @enissoz Architecting the Future of Big Data © Hortonworks Inc. 2011 Page 16

×