BigTable And Hbase
Upcoming SlideShare
Loading in...5
×
 

BigTable And Hbase

on

  • 17,657 views

A architecture of BigTable/Hbase

A architecture of BigTable/Hbase

Statistics

Views

Total Views
17,657
Views on SlideShare
12,392
Embed Views
5,265

Actions

Likes
16
Downloads
582
Comments
0

13 Embeds 5,265

http://www.hadoop.co.kr 4999
http://www.scoop.it 86
http://www.slideshare.net 58
http://laurel-paradise.tistory.com 51
http://blog.hadoop.co.kr 32
http://www.linkedin.com 11
http://feeds.feedburner.com 7
http://www.hanrss.com 7
http://webcache.googleusercontent.com 4
http://www.slideee.com 4
http://translate.googleusercontent.com 3
http://www.docshut.com 2
http://4351815591165136289_8554d270e86ed802136ed01cf0e39f97482d7a40.blogspot.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

BigTable And Hbase BigTable And Hbase Presentation Transcript

  • BigTable & Hbase A Distributed Storage System for Structured Data Edward J. Yoon edwardyoon@apache.org
  • Three Major Component • Master Server - Responsible for assigning tablets to tablet servers, detecting the addition and expiration of tablet servers, balancing tablet-server load, and garbage collection of files in HDFS - Handles schema changed such as table and CF creations • Tablet Server - Manages a set tablets(10~1000 per tablet server) - Handles read write requests to the tablets - Splits tablets that have grown too large (100-200 MB) • Client Library - Communicate directly with tablet servers for reads and writes
  • Architecture - Assigning tablets - Detecting the addition and expiration of tablet Client - Balancing tablet-server load - Handle schema changed Master Server Chubby GFS master server, CMS - Handle master election client - Store the bootstrap location of Hbase data - Discover region server - Store access control lists CMS server Tablet Server Tablet Server Tablet Server - Scheduling jobs - Managing resources on the cluster GFS chunk GFS chunk GFS chunk - dealing with machine failures server, CMS server, CMS server, CMS client client client
  • Data Model • Doesn’t support a full relational data model • Multi-dimensional sorted map • Indexed by a row, column, timestamp (row: string, column : string, time : int 64)  string • Column-oriented storage - Most queries only involve a few columns out of many, so greatly reduces I/O.
  • Tablet Location • Use three-level hierarchy analogous to that of a B+ tree - Location is ip : port of relevant server - 1st level: Bootstrapped from lock server, points to location of root tablet - 2nd level: Uses META 0 data to find owner of appropriate META 1 tablet - 3rd level: META1 table holds locations of tablets of all other tables
  • Tablet Assignment Tablet servers Master keeps track of the set of live tablet servers the current assignment of tablets to Cluster region servers, including which tablets are manager unassigned. Chubby 1) Start a server 2) Create a lock 8) Acquire and 9) Reassign unassigned Delete the lock tablets 3) Acquire the lock 4) Monitor 5) Assign tablets Region Server Master Server 6) Check lock status
  • Tablet Serving • To recover a tablet - reads its metadata from the METADATA table - metadata contains - the list of SS-Tables that comprise a tablet - a set of a redo points, which are pointers into any commit logs that may contain data for the tablet. - reads the indices of the SSTables into memory - reconstructs the memtable by applying all of the updates that have committed since the redo points
  • Compaction V5.0 Create new memtable Read op memtable Deleted data are removed Frozen Storage can be re- memtable used Memory Major compaction DFS Memtable + all Tablet log SSTables -> to one SSTable Write op V4.0 V3.0 V2.0 V1.0 Merging compaction Memtable + a few SSTables -> A new SSTable SSTable files Periodically done. Minor compaction Deleted data are still alive. Memtable -> a new V6.0 SSTable
  • Compression • Clients can control whether or not SSTables for a locality group are compressed • Tow-pass custom compression scheme - First-pass: long common strings across a large window (BMDiff) - Second-pass: looks for repetitions in a small 16KB window (zippy) - Both compression passes are very fast - Space reduction • Allow to identify large amounts of shared boilerplate in pages from same host - Choose their row names so that similar data ends up clustered and therefore achieve very good performance
  • Caching for read performance • Use two level of caching to improve read performance • Scan cache - Higher-level cache - Most useful for applications that tend to read the same data • Block cache - Lower-level cache - Useful for applications or random read of different columns in same locality group within a hot row
  • Hbase : BigTable clone project • http://hadoop.apache.org/hbase/ • Written in java • we do not have chubby or a CMS server, we have Job- Tracker and zookeeper coming soon. • Since Hadoop (GFS) doesn't provide file-append function, Current Hbase have a problem of data loss when Hbase crashed. - Hadoop 0.19.x provides file append function