SlideShare a Scribd company logo
ADVANCED HBASE
Architecture and Schema Design
JAX UK, October 2012

Lars George
Director EMEA Services
About Me

•  Director EMEA Services @ Cloudera
    •  Consulting on Hadoop projects (everywhere)
•  Apache Committer
    •  HBase and Whirr
•  O’Reilly Author
    •  HBase – The Definitive Guide
      •  Now in Japanese!

•  Contact
    •  lars@cloudera.com                      日本語版も出ました!	
  
    •  @larsgeorge
Agenda

•  HBase Architecture
•  Schema Design
HBASE ARCHITECTURE
HBase Tables
HBase Tables
HBase Tables
HBase Tables and Regions

•  Table is made up of any number if regions
•  Region is specified by its startKey and endKey
    •  Empty table: (Table, NULL, NULL)
    •  Two-region table: (Table, NULL, “com.cloudera.www”)
       and (Table, “com.cloudera.www”, NULL)
•  Each region may live on a different node and is
 made up of several HDFS files and blocks, each
 of which is replicated by Hadoop
Distribution
HBase Tables

•  Tables are sorted by Row in lexicographical order
•  Table schema only defines its column families
    •  Each family consists of any number of columns
    •  Each column consists of any number of versions
    •  Columns only exist when inserted, NULLs are free
    •  Columns within a family are sorted and stored
       together
    •  Everything except table names are byte[]


(Table, Row, Family:Column, Timestamp) -> Value
HBase Architecture
HBase Architecture (cont.)

•  HBase uses HDFS (or similar) as its reliable
 storage layer
  •  Handles checksums, replication, failover
•  Native Java API, Gateway for REST, Thrift, Avro
•  Master manages cluster
•  RegionServer manage data
•  ZooKeeper is used the “neural network”
    •  Crucial for HBase
    •  Bootstraps and coordinates cluster
HBase Architecture (cont.)

•  Based on Log-Structured Merge-Trees (LSM-Trees)
•  Inserts are done in write-ahead log first
•  Data is stored in memory and flushed to disk on
   regular intervals or based on size
•  Small flushes are merged in the background to keep
   number of files small
•  Reads read memory stores first and then disk based
   files second
•  Deletes are handled with “tombstone” markers
•  Atomicity on row level no matter how many columns
   •  keeps locking model easy
MemStores
•  After data is written to the WAL the RegionServer
   saves KeyValues in memory store
•  Flush to disk based on size, see
   hbase.hregion.memstore.flush.size
•  Default size is 64MB
•  Uses snapshot mechanism to write flush to disk
   while still serving from it and accepting new data
   at the same time
•  Snapshots are released when flush has
   succeeded
Compactions
•  General Concepts
    •  Two types: Minor and Major Compactions
    •  Asynchronous and transparent to client
    •  Manage file bloat from MemStore flushes
•  Minor Compactions
    •  Combine last “few” flushes
    •  Triggered by number of storage files
•  Major Compactions
    •  Rewrite all storage files
    •  Drop deleted data and those values exceeding TTL and/or number of
       versions
    •  Triggered by time threshold
    •  Cannot be scheduled automatically starting at a specific time (bummer!)
    •  May (most definitely) tax overall HDFS IO performance

Tip: Disable major compactions and schedule to run manually (e.g.
  cron) at off-peak times
Block Cache
•  Acts as very large, in-memory distributed cache
•  Assigned a large part of the JVM heap in the RegionServer process,
   see hfile.block.cache.size
•  Optimizes reads on subsequent columns and rows
•  Has priority to keep “in-memory” column families in cache
    if(inMemory) {
           this.priority = BlockPriority.MEMORY;
    } else {
           this.priority = BlockPriority.SINGLE;
    }

•  Cache needs to be used properly to get best read performance
    •  Turn off block cache on operations that cause large churn
    •  Store related data “close” to each other
•  Uses LRU cache with threaded (asynchronous) evictions based on
  priorities
Region Splits
•  Triggered by configured maximum file size of any
 store file
   •  This is checked directly after the compaction call to
     ensure store files are actually approaching the
     threshold
•  Runs as asynchronous thread on RegionServer
•  Splits are fast and nearly instant
    •  Reference files point to original region files and
       represent each half of the split
•  Compactions take care of splitting original files
 into new region directories
Auto Sharding
Auto Sharding and Distribution

•  Unit of scalability in HBase is the Region
•  Sorted, contiguous range of rows
•  Spread “randomly” across RegionServer
•  Moved around for load balancing and failover
•  Split automatically or manually to scale with
   growing data
•  Capacity is solely a factor of cluster nodes vs.
   regions per node
Column Family vs. Column

•  Use only a few column families
    •  Causes many files that need to stay open per region
       plus class overhead per family
•  Best used when logical separation between data
   and meta columns
•  Sorting per family can be used to convey
   application logic or access pattern
Storage Separation

•  Column Families allow for separation of data
    •  Used by Columnar Databases for fast analytical
       queries, but on column level only
    •  Allows different or no compression depending on the
       content type
•  Segregate information based on access pattern
•  Data is stored in one or more storage file, called
 HFiles
Column Families
SCHEMA DESIGN
Key Cardinality
Key Cardinality

•  The best performance is gained from using row
   keys
•  Time range bound reads can skip store files
   •  So can Bloom Filters
•  Selecting column families reduces the amount of
   data to be scanned
•  Pure value based filtering is a full table scan
   •  Filters often are too, but reduce network traffic
Fold, Store, and Shift
Fold, Store, and Shift

•  Logical layout does not match physical one
•  All values are stored with the full coordinates,
   including: Row Key, Column Family, Column
   Qualifier, and Timestamp
•  Folds columns into “row per column”
•  NULLs are cost free as nothing is stored
•  Versions are multiple “rows” in folded table
Key/Table Design

•  Crucial to gain best performance
    •  Why do I need to know? Well, you also need to know
       that RDBMS is only working well when columns are
       indexed and query plan is OK
•  Absence of secondary indexes forces use of row
   key or column name sorting
•  Transfer multiple indexes into one
   •  Generate large table -> Good since fits architecture
     and spreads across cluster
DDI

•  Stands for Denormalization, Duplication and
   Intelligent Keys
•  Needed to overcome shortcomings of
   architecture
•  Denormalization -> Replacement for JOINs
•  Duplication -> Design for reads
•  Intelligent Keys -> Implement indexing and
   sorting, optimize reads
Pre-materialize Everything

•  Achieve one read per customer request if
   possible
•  Otherwise keep at lowest number
•  Reads between 10ms (cache miss) and 1ms
   (cache hit)
•  Use MapReduce to compute exacts in batch
•  Store and merge updates live
•  Use incrementColumnValue


            Motto: “Design for Reads”
Tall-Narrow vs. Flat-Wide Tables

•  Rows do not split
    •  Might end up with one row per region
•  Same storage footprint
•  Put more details into the row key
    •  Sometimes dummy column only
    •  Make use of partial key scans
•  Tall with Scans, Wide with Gets
    •  Atomicity only on row level
•  Example: Large graphs, stored as adjacency
 matrix
Example: Mail Inbox

        <userId> : <colfam> : <messageId> : <timestamp> : <email-message>

12345   :   data   :   5fc38314-e290-ae5da5fc375d       :   1307097848   :   "Hi Lars, ..."
12345   :   data   :   725aae5f-d72e-f90f3f070419       :   1307099848   :   "Welcome, and ..."
12345   :   data   :   cc6775b3-f249-c6dd2b1a7467       :   1307101848   :   "To Whom It ..."
12345   :   data   :   dcbee495-6d5e-6ed48124632c       :   1307103848   :   "Hi, how are ..."


                                               or
12345-5fc38314-e290-ae5da5fc375d         :   data   :   :   1307097848   :   "Hi Lars, ..."
12345-725aae5f-d72e-f90f3f070419         :   data   :   :   1307099848   :   "Welcome, and ..."
12345-cc6775b3-f249-c6dd2b1a7467         :   data   :   :   1307101848   :   "To Whom It ..."
12345-dcbee495-6d5e-6ed48124632c         :   data   :   :   1307103848   :   "Hi, how are ..."


                           è   Same Storage Requirements
Partial Key Scans
Key	
                                        Descrip+on	
  
<userId>                                     Scan	
  over	
  all	
  messages	
  
                                             for	
  a	
  given	
  user	
  ID	
  
<userId>-<date>                              Scan	
  over	
  all	
  messages	
  
                                             on	
  a	
  given	
  date	
  for	
  the	
  
                                             given	
  user	
  ID	
  
<userId>-<date>-<messageId>                  Scan	
  over	
  all	
  parts	
  of	
  a	
  
                                             message	
  for	
  a	
  given	
  user	
  
                                             ID	
  and	
  date	
  
<userId>-<date>-<messageId>-<attachmentId>   Scan	
  over	
  all	
  
                                             a8achments	
  of	
  a	
  
                                             message	
  for	
  a	
  given	
  user	
  
                                             ID	
  and	
  date	
  
Sequential Keys

  <timestamp><more key>: {CF: {CQ: {TS : Val}}}

•  Hotspotting on Regions: bad!
•  Instead do one of the following:
     •  Salting
      •  Prefix   <timestamp> with distributed value
      •  Binning or bucketing rows across regions
   •  Key field swap/promotion
       •  Move <more key> before the timestamp (see OpenTSDB
          later)
   •  Randomization
       •  Move <timestamp> out of key
Salting

•    Prefix row keys to gain spread
•    Use well known or numbered prefixes
•    Use modulo to spread across servers
•    Enforce common data stay close to each other for
     subsequent scanning or MapReduce processing
      0_rowkey1, 1_rowkey2, 2_rowkey3
      0_rowkey4, 1_rowkey5, 2_rowkey6

•  Sorted by prefix first
    0_rowkey1
    0_rowkey4
    1_rowkey2
    1_rowkey5
    …
Hashing vs. Sequential Keys

•  Uses hashes for best spread
    •  Use for example MD5 to be able to recreate key
      •  Key = MD5(customerID)
   •  Counter productive for range scans


•  Use sequential keys for locality
    •  Makes use of block caches
    •  May tax one server overly, may be avoided by salting
       or splitting regions while keeping them small
Key Design
Key Design Summary

•  Based on access pattern, either use sequential or
   random keys
•  Often a combination of both is needed
   •  Overcome architectural limitations
•  Neither is necessarily bad
    •  Use bulk import for sequential keys and reads
    •  Random keys are good for random access patterns
Example: Facebook Insights

•  > 20B Events per Day
•  1M Counter Updates per Second
    •  100 Nodes Cluster
    •  10K OPS per Node
•  ”Like” button triggers AJAX request
•  Event written to log file
•  30mins current for website owner


     Web	
  ➜	
  Scribe	
  ➜	
  Ptail	
  ➜	
  Puma	
  ➜	
  HBase	
  
HBase Counters

•  Store counters per Domain and per URL
    •  Leverage HBase increment (atomic read-modify-
       write) feature
•  Each row is one specific Domain or URL
•  The columns are the counters for specific metrics
•  Column families are used to group counters by
 time range
   •  Set time-to-live on CF level to auto-expire counters by
     age to save space, e.g., 2 weeks on “Daily Counters”
     family
Key Design
•  Reversed Domains
    •  Examples: “com.cloudera.www”, “com.cloudera.blog”
    •  Helps keeping pages per site close, as HBase efficiently
       scans blocks of sorted keys
•  Domain Row Key =
 MD5(Reversed Domain) + Reversed Domain
   •  Leading MD5 hash spreads keys randomly across all regions
      for load balancing reasons
   •  Only hashing the domain groups per site (and per subdomain
      if needed)
•  URL Row Key =
 MD5(Reversed Domain) + Reversed Domain + URL ID
   •  Unique ID per URL already available, make use of it
Insights Schema
Summary

•  Design for Use-Case
    •  Read, Write, or Both?
•  Avoid Hotspotting
•  Consider using IDs instead of full text
•  Leverage Column Family to HFile relation
•  Shift details to appropriate position
    •  Composite Keys
    •  Column Qualifiers
Summary (cont.)

•  Schema design is a combination of
    •  Designing the keys (row and column)
    •  Segregate data into column families
    •  Choose compression and block sizes
•  Similar techniques are needed to scale most
 systems
   •  Add indexes, partition data, consistent hashing
•  Denormalization, Duplication, and Intelligent
 Keys (DDI)
Ques+ons?	
  

More Related Content

What's hot

Facebook Messages & HBase
Facebook Messages & HBaseFacebook Messages & HBase
Facebook Messages & HBase
强 王
 
RocksDB Performance and Reliability Practices
RocksDB Performance and Reliability PracticesRocksDB Performance and Reliability Practices
RocksDB Performance and Reliability Practices
Yoshinori Matsunobu
 
Apache phoenix: Past, Present and Future of SQL over HBAse
Apache phoenix: Past, Present and Future of SQL over HBAseApache phoenix: Past, Present and Future of SQL over HBAse
Apache phoenix: Past, Present and Future of SQL over HBAse
enissoz
 
Hadoop World 2011: Advanced HBase Schema Design
Hadoop World 2011: Advanced HBase Schema DesignHadoop World 2011: Advanced HBase Schema Design
Hadoop World 2011: Advanced HBase Schema Design
Cloudera, Inc.
 
hbaseconasia2019 HBCK2: Concepts, trends, and recipes for fixing issues in HB...
hbaseconasia2019 HBCK2: Concepts, trends, and recipes for fixing issues in HB...hbaseconasia2019 HBCK2: Concepts, trends, and recipes for fixing issues in HB...
hbaseconasia2019 HBCK2: Concepts, trends, and recipes for fixing issues in HB...
Michael Stack
 
How to use histograms to get better performance
How to use histograms to get better performanceHow to use histograms to get better performance
How to use histograms to get better performance
MariaDB plc
 
How to build a streaming Lakehouse with Flink, Kafka, and Hudi
How to build a streaming Lakehouse with Flink, Kafka, and HudiHow to build a streaming Lakehouse with Flink, Kafka, and Hudi
How to build a streaming Lakehouse with Flink, Kafka, and Hudi
Flink Forward
 
RocksDB detail
RocksDB detailRocksDB detail
RocksDB detail
MIJIN AN
 
HBaseCon 2015: Taming GC Pauses for Large Java Heap in HBase
HBaseCon 2015: Taming GC Pauses for Large Java Heap in HBaseHBaseCon 2015: Taming GC Pauses for Large Java Heap in HBase
HBaseCon 2015: Taming GC Pauses for Large Java Heap in HBase
HBaseCon
 
HBaseCon 2015: HBase Performance Tuning @ Salesforce
HBaseCon 2015: HBase Performance Tuning @ SalesforceHBaseCon 2015: HBase Performance Tuning @ Salesforce
HBaseCon 2015: HBase Performance Tuning @ Salesforce
HBaseCon
 
HBase replication
HBase replicationHBase replication
HBase replication
wchevreuil
 
Hadoop World 2011: Advanced HBase Schema Design - Lars George, Cloudera
Hadoop World 2011: Advanced HBase Schema Design - Lars George, ClouderaHadoop World 2011: Advanced HBase Schema Design - Lars George, Cloudera
Hadoop World 2011: Advanced HBase Schema Design - Lars George, Cloudera
Cloudera, Inc.
 
MySQL Space Management
MySQL Space ManagementMySQL Space Management
MySQL Space Management
MIJIN AN
 
ORC File - Optimizing Your Big Data
ORC File - Optimizing Your Big DataORC File - Optimizing Your Big Data
ORC File - Optimizing Your Big Data
DataWorks Summit
 
Tuning Apache Kafka Connectors for Flink.pptx
Tuning Apache Kafka Connectors for Flink.pptxTuning Apache Kafka Connectors for Flink.pptx
Tuning Apache Kafka Connectors for Flink.pptx
Flink Forward
 
Hive + Tez: A Performance Deep Dive
Hive + Tez: A Performance Deep DiveHive + Tez: A Performance Deep Dive
Hive + Tez: A Performance Deep Dive
DataWorks Summit
 
Introduction to Redis
Introduction to RedisIntroduction to Redis
Introduction to Redis
Dvir Volk
 
Kafka replication apachecon_2013
Kafka replication apachecon_2013Kafka replication apachecon_2013
Kafka replication apachecon_2013
Jun Rao
 
Node Labels in YARN
Node Labels in YARNNode Labels in YARN
Node Labels in YARN
DataWorks Summit
 
HBase Accelerated: In-Memory Flush and Compaction
HBase Accelerated: In-Memory Flush and CompactionHBase Accelerated: In-Memory Flush and Compaction
HBase Accelerated: In-Memory Flush and Compaction
DataWorks Summit/Hadoop Summit
 

What's hot (20)

Facebook Messages & HBase
Facebook Messages & HBaseFacebook Messages & HBase
Facebook Messages & HBase
 
RocksDB Performance and Reliability Practices
RocksDB Performance and Reliability PracticesRocksDB Performance and Reliability Practices
RocksDB Performance and Reliability Practices
 
Apache phoenix: Past, Present and Future of SQL over HBAse
Apache phoenix: Past, Present and Future of SQL over HBAseApache phoenix: Past, Present and Future of SQL over HBAse
Apache phoenix: Past, Present and Future of SQL over HBAse
 
Hadoop World 2011: Advanced HBase Schema Design
Hadoop World 2011: Advanced HBase Schema DesignHadoop World 2011: Advanced HBase Schema Design
Hadoop World 2011: Advanced HBase Schema Design
 
hbaseconasia2019 HBCK2: Concepts, trends, and recipes for fixing issues in HB...
hbaseconasia2019 HBCK2: Concepts, trends, and recipes for fixing issues in HB...hbaseconasia2019 HBCK2: Concepts, trends, and recipes for fixing issues in HB...
hbaseconasia2019 HBCK2: Concepts, trends, and recipes for fixing issues in HB...
 
How to use histograms to get better performance
How to use histograms to get better performanceHow to use histograms to get better performance
How to use histograms to get better performance
 
How to build a streaming Lakehouse with Flink, Kafka, and Hudi
How to build a streaming Lakehouse with Flink, Kafka, and HudiHow to build a streaming Lakehouse with Flink, Kafka, and Hudi
How to build a streaming Lakehouse with Flink, Kafka, and Hudi
 
RocksDB detail
RocksDB detailRocksDB detail
RocksDB detail
 
HBaseCon 2015: Taming GC Pauses for Large Java Heap in HBase
HBaseCon 2015: Taming GC Pauses for Large Java Heap in HBaseHBaseCon 2015: Taming GC Pauses for Large Java Heap in HBase
HBaseCon 2015: Taming GC Pauses for Large Java Heap in HBase
 
HBaseCon 2015: HBase Performance Tuning @ Salesforce
HBaseCon 2015: HBase Performance Tuning @ SalesforceHBaseCon 2015: HBase Performance Tuning @ Salesforce
HBaseCon 2015: HBase Performance Tuning @ Salesforce
 
HBase replication
HBase replicationHBase replication
HBase replication
 
Hadoop World 2011: Advanced HBase Schema Design - Lars George, Cloudera
Hadoop World 2011: Advanced HBase Schema Design - Lars George, ClouderaHadoop World 2011: Advanced HBase Schema Design - Lars George, Cloudera
Hadoop World 2011: Advanced HBase Schema Design - Lars George, Cloudera
 
MySQL Space Management
MySQL Space ManagementMySQL Space Management
MySQL Space Management
 
ORC File - Optimizing Your Big Data
ORC File - Optimizing Your Big DataORC File - Optimizing Your Big Data
ORC File - Optimizing Your Big Data
 
Tuning Apache Kafka Connectors for Flink.pptx
Tuning Apache Kafka Connectors for Flink.pptxTuning Apache Kafka Connectors for Flink.pptx
Tuning Apache Kafka Connectors for Flink.pptx
 
Hive + Tez: A Performance Deep Dive
Hive + Tez: A Performance Deep DiveHive + Tez: A Performance Deep Dive
Hive + Tez: A Performance Deep Dive
 
Introduction to Redis
Introduction to RedisIntroduction to Redis
Introduction to Redis
 
Kafka replication apachecon_2013
Kafka replication apachecon_2013Kafka replication apachecon_2013
Kafka replication apachecon_2013
 
Node Labels in YARN
Node Labels in YARNNode Labels in YARN
Node Labels in YARN
 
HBase Accelerated: In-Memory Flush and Compaction
HBase Accelerated: In-Memory Flush and CompactionHBase Accelerated: In-Memory Flush and Compaction
HBase Accelerated: In-Memory Flush and Compaction
 

Viewers also liked

HBase Blockcache 101
HBase Blockcache 101HBase Blockcache 101
HBase Blockcache 101
Nick Dimiduk
 
HBase for Architects
HBase for ArchitectsHBase for Architects
HBase for Architects
Nick Dimiduk
 
Intro to HBase - Lars George
Intro to HBase - Lars GeorgeIntro to HBase - Lars George
Intro to HBase - Lars George
JAX London
 
HBase Advanced Schema Design - Berlin Buzzwords - June 2012
HBase Advanced Schema Design - Berlin Buzzwords - June 2012HBase Advanced Schema Design - Berlin Buzzwords - June 2012
HBase Advanced Schema Design - Berlin Buzzwords - June 2012
larsgeorge
 
Intro to HBase Internals & Schema Design (for HBase users)
Intro to HBase Internals & Schema Design (for HBase users)Intro to HBase Internals & Schema Design (for HBase users)
Intro to HBase Internals & Schema Design (for HBase users)
alexbaranau
 
HBase Coprocessor Introduction
HBase Coprocessor IntroductionHBase Coprocessor Introduction
HBase Coprocessor Introduction
Schubert Zhang
 
Hands-on-Lab: Adding Value to HBase with IBM InfoSphere BigInsights and BigSQL
Hands-on-Lab: Adding Value to HBase with IBM InfoSphere BigInsights and BigSQLHands-on-Lab: Adding Value to HBase with IBM InfoSphere BigInsights and BigSQL
Hands-on-Lab: Adding Value to HBase with IBM InfoSphere BigInsights and BigSQL
Piotr Pruski
 
Valerii Moisieienko Apache hbase workshop
Valerii Moisieienko	Apache hbase workshopValerii Moisieienko	Apache hbase workshop
Valerii Moisieienko Apache hbase workshop
Аліна Шепшелей
 
H base key design
H base key designH base key design
H base key design
Sertuğ Kaya
 
HBase In Action - Chapter 04: HBase table design
HBase In Action - Chapter 04: HBase table designHBase In Action - Chapter 04: HBase table design
HBase In Action - Chapter 04: HBase table design
phanleson
 
Update on OpenTSDB and AsyncHBase
Update on OpenTSDB and AsyncHBase Update on OpenTSDB and AsyncHBase
Update on OpenTSDB and AsyncHBase
HBaseCon
 
Strata + Hadoop World 2012: Apache HBase Features for the Enterprise
Strata + Hadoop World 2012: Apache HBase Features for the EnterpriseStrata + Hadoop World 2012: Apache HBase Features for the Enterprise
Strata + Hadoop World 2012: Apache HBase Features for the Enterprise
Cloudera, Inc.
 
Introduction to HBase - Phoenix HUG 5/14
Introduction to HBase - Phoenix HUG 5/14Introduction to HBase - Phoenix HUG 5/14
Introduction to HBase - Phoenix HUG 5/14
Jeremy Walsh
 
A 3 dimensional data model in hbase for large time-series dataset-20120915
A 3 dimensional data model in hbase for large time-series dataset-20120915A 3 dimensional data model in hbase for large time-series dataset-20120915
A 3 dimensional data model in hbase for large time-series dataset-20120915
Dan Han
 
HBase from the Trenches - Phoenix Data Conference 2015
HBase from the Trenches - Phoenix Data Conference 2015HBase from the Trenches - Phoenix Data Conference 2015
HBase from the Trenches - Phoenix Data Conference 2015
Avinash Ramineni
 
Big Data: HBase and Big SQL self-study lab
Big Data:  HBase and Big SQL self-study lab Big Data:  HBase and Big SQL self-study lab
Big Data: HBase and Big SQL self-study lab
Cynthia Saracco
 
HBaseConEast2016: HBase and Spark, State of the Art
HBaseConEast2016: HBase and Spark, State of the ArtHBaseConEast2016: HBase and Spark, State of the Art
HBaseConEast2016: HBase and Spark, State of the Art
Michael Stack
 
Cassandra Data Modeling
Cassandra Data ModelingCassandra Data Modeling
Cassandra Data Modeling
Matthew Dennis
 
Big data hbase
Big data hbase Big data hbase
Big data hbase
ANSHUL GUPTA
 
HBase Client APIs (for webapps?)
HBase Client APIs (for webapps?)HBase Client APIs (for webapps?)
HBase Client APIs (for webapps?)
Nick Dimiduk
 

Viewers also liked (20)

HBase Blockcache 101
HBase Blockcache 101HBase Blockcache 101
HBase Blockcache 101
 
HBase for Architects
HBase for ArchitectsHBase for Architects
HBase for Architects
 
Intro to HBase - Lars George
Intro to HBase - Lars GeorgeIntro to HBase - Lars George
Intro to HBase - Lars George
 
HBase Advanced Schema Design - Berlin Buzzwords - June 2012
HBase Advanced Schema Design - Berlin Buzzwords - June 2012HBase Advanced Schema Design - Berlin Buzzwords - June 2012
HBase Advanced Schema Design - Berlin Buzzwords - June 2012
 
Intro to HBase Internals & Schema Design (for HBase users)
Intro to HBase Internals & Schema Design (for HBase users)Intro to HBase Internals & Schema Design (for HBase users)
Intro to HBase Internals & Schema Design (for HBase users)
 
HBase Coprocessor Introduction
HBase Coprocessor IntroductionHBase Coprocessor Introduction
HBase Coprocessor Introduction
 
Hands-on-Lab: Adding Value to HBase with IBM InfoSphere BigInsights and BigSQL
Hands-on-Lab: Adding Value to HBase with IBM InfoSphere BigInsights and BigSQLHands-on-Lab: Adding Value to HBase with IBM InfoSphere BigInsights and BigSQL
Hands-on-Lab: Adding Value to HBase with IBM InfoSphere BigInsights and BigSQL
 
Valerii Moisieienko Apache hbase workshop
Valerii Moisieienko	Apache hbase workshopValerii Moisieienko	Apache hbase workshop
Valerii Moisieienko Apache hbase workshop
 
H base key design
H base key designH base key design
H base key design
 
HBase In Action - Chapter 04: HBase table design
HBase In Action - Chapter 04: HBase table designHBase In Action - Chapter 04: HBase table design
HBase In Action - Chapter 04: HBase table design
 
Update on OpenTSDB and AsyncHBase
Update on OpenTSDB and AsyncHBase Update on OpenTSDB and AsyncHBase
Update on OpenTSDB and AsyncHBase
 
Strata + Hadoop World 2012: Apache HBase Features for the Enterprise
Strata + Hadoop World 2012: Apache HBase Features for the EnterpriseStrata + Hadoop World 2012: Apache HBase Features for the Enterprise
Strata + Hadoop World 2012: Apache HBase Features for the Enterprise
 
Introduction to HBase - Phoenix HUG 5/14
Introduction to HBase - Phoenix HUG 5/14Introduction to HBase - Phoenix HUG 5/14
Introduction to HBase - Phoenix HUG 5/14
 
A 3 dimensional data model in hbase for large time-series dataset-20120915
A 3 dimensional data model in hbase for large time-series dataset-20120915A 3 dimensional data model in hbase for large time-series dataset-20120915
A 3 dimensional data model in hbase for large time-series dataset-20120915
 
HBase from the Trenches - Phoenix Data Conference 2015
HBase from the Trenches - Phoenix Data Conference 2015HBase from the Trenches - Phoenix Data Conference 2015
HBase from the Trenches - Phoenix Data Conference 2015
 
Big Data: HBase and Big SQL self-study lab
Big Data:  HBase and Big SQL self-study lab Big Data:  HBase and Big SQL self-study lab
Big Data: HBase and Big SQL self-study lab
 
HBaseConEast2016: HBase and Spark, State of the Art
HBaseConEast2016: HBase and Spark, State of the ArtHBaseConEast2016: HBase and Spark, State of the Art
HBaseConEast2016: HBase and Spark, State of the Art
 
Cassandra Data Modeling
Cassandra Data ModelingCassandra Data Modeling
Cassandra Data Modeling
 
Big data hbase
Big data hbase Big data hbase
Big data hbase
 
HBase Client APIs (for webapps?)
HBase Client APIs (for webapps?)HBase Client APIs (for webapps?)
HBase Client APIs (for webapps?)
 

Similar to HBase Advanced - Lars George

HBase in Practice
HBase in Practice HBase in Practice
HBase in Practice
DataWorks Summit/Hadoop Summit
 
HBase in Practice
HBase in PracticeHBase in Practice
HBase in Practice
larsgeorge
 
L6.sp17.pptx
L6.sp17.pptxL6.sp17.pptx
L6.sp17.pptx
SudheerKumar499932
 
01 hbase
01 hbase01 hbase
HBase Low Latency, StrataNYC 2014
HBase Low Latency, StrataNYC 2014HBase Low Latency, StrataNYC 2014
HBase Low Latency, StrataNYC 2014
Nick Dimiduk
 
NoSql
NoSqlNoSql
No sql databases
No sql databasesNo sql databases
No sql databases
swathika rajan
 
Cassandra an overview
Cassandra an overviewCassandra an overview
Cassandra an overview
PritamKathar
 
UNIT I Introduction to NoSQL.pptx
UNIT I Introduction to NoSQL.pptxUNIT I Introduction to NoSQL.pptx
UNIT I Introduction to NoSQL.pptx
Rahul Borate
 
Big Data Architecture Workshop - Vahid Amiri
Big Data Architecture Workshop -  Vahid AmiriBig Data Architecture Workshop -  Vahid Amiri
Big Data Architecture Workshop - Vahid Amiri
datastack
 
NoSQL.pptx
NoSQL.pptxNoSQL.pptx
NoSQL.pptx
RithikRaj25
 
UNIT I Introduction to NoSQL.pptx
UNIT I Introduction to NoSQL.pptxUNIT I Introduction to NoSQL.pptx
UNIT I Introduction to NoSQL.pptx
Rahul Borate
 
Scaling the Web: Databases & NoSQL
Scaling the Web: Databases & NoSQLScaling the Web: Databases & NoSQL
Scaling the Web: Databases & NoSQL
Richard Schneeman
 
Hbase: an introduction
Hbase: an introductionHbase: an introduction
Hbase: an introduction
Jean-Baptiste Poullet
 
HBaseCon 2015: HBase @ Flipboard
HBaseCon 2015: HBase @ FlipboardHBaseCon 2015: HBase @ Flipboard
HBaseCon 2015: HBase @ Flipboard
HBaseCon
 
HBaseCon 2015- HBase @ Flipboard
HBaseCon 2015- HBase @ FlipboardHBaseCon 2015- HBase @ Flipboard
HBaseCon 2015- HBase @ Flipboard
Matthew Blair
 
Cassandra
CassandraCassandra
Cassandra
exsuns
 
Schema Design
Schema DesignSchema Design
Schema Design
QBurst
 
Apache HBase Workshop
Apache HBase WorkshopApache HBase Workshop
Apache HBase Workshop
Valerii Moisieienko
 
Cassandra training
Cassandra trainingCassandra training
Cassandra training
András Fehér
 

Similar to HBase Advanced - Lars George (20)

HBase in Practice
HBase in Practice HBase in Practice
HBase in Practice
 
HBase in Practice
HBase in PracticeHBase in Practice
HBase in Practice
 
L6.sp17.pptx
L6.sp17.pptxL6.sp17.pptx
L6.sp17.pptx
 
01 hbase
01 hbase01 hbase
01 hbase
 
HBase Low Latency, StrataNYC 2014
HBase Low Latency, StrataNYC 2014HBase Low Latency, StrataNYC 2014
HBase Low Latency, StrataNYC 2014
 
NoSql
NoSqlNoSql
NoSql
 
No sql databases
No sql databasesNo sql databases
No sql databases
 
Cassandra an overview
Cassandra an overviewCassandra an overview
Cassandra an overview
 
UNIT I Introduction to NoSQL.pptx
UNIT I Introduction to NoSQL.pptxUNIT I Introduction to NoSQL.pptx
UNIT I Introduction to NoSQL.pptx
 
Big Data Architecture Workshop - Vahid Amiri
Big Data Architecture Workshop -  Vahid AmiriBig Data Architecture Workshop -  Vahid Amiri
Big Data Architecture Workshop - Vahid Amiri
 
NoSQL.pptx
NoSQL.pptxNoSQL.pptx
NoSQL.pptx
 
UNIT I Introduction to NoSQL.pptx
UNIT I Introduction to NoSQL.pptxUNIT I Introduction to NoSQL.pptx
UNIT I Introduction to NoSQL.pptx
 
Scaling the Web: Databases & NoSQL
Scaling the Web: Databases & NoSQLScaling the Web: Databases & NoSQL
Scaling the Web: Databases & NoSQL
 
Hbase: an introduction
Hbase: an introductionHbase: an introduction
Hbase: an introduction
 
HBaseCon 2015: HBase @ Flipboard
HBaseCon 2015: HBase @ FlipboardHBaseCon 2015: HBase @ Flipboard
HBaseCon 2015: HBase @ Flipboard
 
HBaseCon 2015- HBase @ Flipboard
HBaseCon 2015- HBase @ FlipboardHBaseCon 2015- HBase @ Flipboard
HBaseCon 2015- HBase @ Flipboard
 
Cassandra
CassandraCassandra
Cassandra
 
Schema Design
Schema DesignSchema Design
Schema Design
 
Apache HBase Workshop
Apache HBase WorkshopApache HBase Workshop
Apache HBase Workshop
 
Cassandra training
Cassandra trainingCassandra training
Cassandra training
 

More from JAX London

Everything I know about software in spaghetti bolognese: managing complexity
Everything I know about software in spaghetti bolognese: managing complexityEverything I know about software in spaghetti bolognese: managing complexity
Everything I know about software in spaghetti bolognese: managing complexity
JAX London
 
Devops with the S for Sharing - Patrick Debois
Devops with the S for Sharing - Patrick DeboisDevops with the S for Sharing - Patrick Debois
Devops with the S for Sharing - Patrick Debois
JAX London
 
Busy Developer's Guide to Windows 8 HTML/JavaScript Apps
Busy Developer's Guide to Windows 8 HTML/JavaScript AppsBusy Developer's Guide to Windows 8 HTML/JavaScript Apps
Busy Developer's Guide to Windows 8 HTML/JavaScript Apps
JAX London
 
It's code but not as we know: Infrastructure as Code - Patrick Debois
It's code but not as we know: Infrastructure as Code - Patrick DeboisIt's code but not as we know: Infrastructure as Code - Patrick Debois
It's code but not as we know: Infrastructure as Code - Patrick Debois
JAX London
 
Locks? We Don't Need No Stinkin' Locks - Michael Barker
Locks? We Don't Need No Stinkin' Locks - Michael BarkerLocks? We Don't Need No Stinkin' Locks - Michael Barker
Locks? We Don't Need No Stinkin' Locks - Michael Barker
JAX London
 
Worse is better, for better or for worse - Kevlin Henney
Worse is better, for better or for worse - Kevlin HenneyWorse is better, for better or for worse - Kevlin Henney
Worse is better, for better or for worse - Kevlin Henney
JAX London
 
Java performance: What's the big deal? - Trisha Gee
Java performance: What's the big deal? - Trisha GeeJava performance: What's the big deal? - Trisha Gee
Java performance: What's the big deal? - Trisha Gee
JAX London
 
Clojure made-simple - John Stevenson
Clojure made-simple - John StevensonClojure made-simple - John Stevenson
Clojure made-simple - John Stevenson
JAX London
 
HTML alchemy: the secrets of mixing JavaScript and Java EE - Matthias Wessendorf
HTML alchemy: the secrets of mixing JavaScript and Java EE - Matthias WessendorfHTML alchemy: the secrets of mixing JavaScript and Java EE - Matthias Wessendorf
HTML alchemy: the secrets of mixing JavaScript and Java EE - Matthias Wessendorf
JAX London
 
Play framework 2 : Peter Hilton
Play framework 2 : Peter HiltonPlay framework 2 : Peter Hilton
Play framework 2 : Peter Hilton
JAX London
 
Complexity theory and software development : Tim Berglund
Complexity theory and software development : Tim BerglundComplexity theory and software development : Tim Berglund
Complexity theory and software development : Tim Berglund
JAX London
 
Why FLOSS is a Java developer's best friend: Dave Gruber
Why FLOSS is a Java developer's best friend: Dave GruberWhy FLOSS is a Java developer's best friend: Dave Gruber
Why FLOSS is a Java developer's best friend: Dave Gruber
JAX London
 
Akka in Action: Heiko Seeburger
Akka in Action: Heiko SeeburgerAkka in Action: Heiko Seeburger
Akka in Action: Heiko Seeburger
JAX London
 
NoSQL Smackdown 2012 : Tim Berglund
NoSQL Smackdown 2012 : Tim BerglundNoSQL Smackdown 2012 : Tim Berglund
NoSQL Smackdown 2012 : Tim Berglund
JAX London
 
Closures, the next "Big Thing" in Java: Russel Winder
Closures, the next "Big Thing" in Java: Russel WinderClosures, the next "Big Thing" in Java: Russel Winder
Closures, the next "Big Thing" in Java: Russel Winder
JAX London
 
Java and the machine - Martijn Verburg and Kirk Pepperdine
Java and the machine - Martijn Verburg and Kirk PepperdineJava and the machine - Martijn Verburg and Kirk Pepperdine
Java and the machine - Martijn Verburg and Kirk Pepperdine
JAX London
 
Mongo DB on the JVM - Brendan McAdams
Mongo DB on the JVM - Brendan McAdamsMongo DB on the JVM - Brendan McAdams
Mongo DB on the JVM - Brendan McAdams
JAX London
 
New opportunities for connected data - Ian Robinson
New opportunities for connected data - Ian RobinsonNew opportunities for connected data - Ian Robinson
New opportunities for connected data - Ian Robinson
JAX London
 
HTML5 Websockets and Java - Arun Gupta
HTML5 Websockets and Java - Arun GuptaHTML5 Websockets and Java - Arun Gupta
HTML5 Websockets and Java - Arun Gupta
JAX London
 
The Big Data Con: Why Big Data is a Problem, not a Solution - Ian Plosker
The Big Data Con: Why Big Data is a Problem, not a Solution - Ian PloskerThe Big Data Con: Why Big Data is a Problem, not a Solution - Ian Plosker
The Big Data Con: Why Big Data is a Problem, not a Solution - Ian Plosker
JAX London
 

More from JAX London (20)

Everything I know about software in spaghetti bolognese: managing complexity
Everything I know about software in spaghetti bolognese: managing complexityEverything I know about software in spaghetti bolognese: managing complexity
Everything I know about software in spaghetti bolognese: managing complexity
 
Devops with the S for Sharing - Patrick Debois
Devops with the S for Sharing - Patrick DeboisDevops with the S for Sharing - Patrick Debois
Devops with the S for Sharing - Patrick Debois
 
Busy Developer's Guide to Windows 8 HTML/JavaScript Apps
Busy Developer's Guide to Windows 8 HTML/JavaScript AppsBusy Developer's Guide to Windows 8 HTML/JavaScript Apps
Busy Developer's Guide to Windows 8 HTML/JavaScript Apps
 
It's code but not as we know: Infrastructure as Code - Patrick Debois
It's code but not as we know: Infrastructure as Code - Patrick DeboisIt's code but not as we know: Infrastructure as Code - Patrick Debois
It's code but not as we know: Infrastructure as Code - Patrick Debois
 
Locks? We Don't Need No Stinkin' Locks - Michael Barker
Locks? We Don't Need No Stinkin' Locks - Michael BarkerLocks? We Don't Need No Stinkin' Locks - Michael Barker
Locks? We Don't Need No Stinkin' Locks - Michael Barker
 
Worse is better, for better or for worse - Kevlin Henney
Worse is better, for better or for worse - Kevlin HenneyWorse is better, for better or for worse - Kevlin Henney
Worse is better, for better or for worse - Kevlin Henney
 
Java performance: What's the big deal? - Trisha Gee
Java performance: What's the big deal? - Trisha GeeJava performance: What's the big deal? - Trisha Gee
Java performance: What's the big deal? - Trisha Gee
 
Clojure made-simple - John Stevenson
Clojure made-simple - John StevensonClojure made-simple - John Stevenson
Clojure made-simple - John Stevenson
 
HTML alchemy: the secrets of mixing JavaScript and Java EE - Matthias Wessendorf
HTML alchemy: the secrets of mixing JavaScript and Java EE - Matthias WessendorfHTML alchemy: the secrets of mixing JavaScript and Java EE - Matthias Wessendorf
HTML alchemy: the secrets of mixing JavaScript and Java EE - Matthias Wessendorf
 
Play framework 2 : Peter Hilton
Play framework 2 : Peter HiltonPlay framework 2 : Peter Hilton
Play framework 2 : Peter Hilton
 
Complexity theory and software development : Tim Berglund
Complexity theory and software development : Tim BerglundComplexity theory and software development : Tim Berglund
Complexity theory and software development : Tim Berglund
 
Why FLOSS is a Java developer's best friend: Dave Gruber
Why FLOSS is a Java developer's best friend: Dave GruberWhy FLOSS is a Java developer's best friend: Dave Gruber
Why FLOSS is a Java developer's best friend: Dave Gruber
 
Akka in Action: Heiko Seeburger
Akka in Action: Heiko SeeburgerAkka in Action: Heiko Seeburger
Akka in Action: Heiko Seeburger
 
NoSQL Smackdown 2012 : Tim Berglund
NoSQL Smackdown 2012 : Tim BerglundNoSQL Smackdown 2012 : Tim Berglund
NoSQL Smackdown 2012 : Tim Berglund
 
Closures, the next "Big Thing" in Java: Russel Winder
Closures, the next "Big Thing" in Java: Russel WinderClosures, the next "Big Thing" in Java: Russel Winder
Closures, the next "Big Thing" in Java: Russel Winder
 
Java and the machine - Martijn Verburg and Kirk Pepperdine
Java and the machine - Martijn Verburg and Kirk PepperdineJava and the machine - Martijn Verburg and Kirk Pepperdine
Java and the machine - Martijn Verburg and Kirk Pepperdine
 
Mongo DB on the JVM - Brendan McAdams
Mongo DB on the JVM - Brendan McAdamsMongo DB on the JVM - Brendan McAdams
Mongo DB on the JVM - Brendan McAdams
 
New opportunities for connected data - Ian Robinson
New opportunities for connected data - Ian RobinsonNew opportunities for connected data - Ian Robinson
New opportunities for connected data - Ian Robinson
 
HTML5 Websockets and Java - Arun Gupta
HTML5 Websockets and Java - Arun GuptaHTML5 Websockets and Java - Arun Gupta
HTML5 Websockets and Java - Arun Gupta
 
The Big Data Con: Why Big Data is a Problem, not a Solution - Ian Plosker
The Big Data Con: Why Big Data is a Problem, not a Solution - Ian PloskerThe Big Data Con: Why Big Data is a Problem, not a Solution - Ian Plosker
The Big Data Con: Why Big Data is a Problem, not a Solution - Ian Plosker
 

Recently uploaded

GenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizationsGenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizations
kumardaparthi1024
 
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of GermanyPresentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
innovationoecd
 
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdfNunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
flufftailshop
 
AWS Cloud Cost Optimization Presentation.pptx
AWS Cloud Cost Optimization Presentation.pptxAWS Cloud Cost Optimization Presentation.pptx
AWS Cloud Cost Optimization Presentation.pptx
HarisZaheer8
 
dbms calicut university B. sc Cs 4th sem.pdf
dbms  calicut university B. sc Cs 4th sem.pdfdbms  calicut university B. sc Cs 4th sem.pdf
dbms calicut university B. sc Cs 4th sem.pdf
Shinana2
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
Hiroshi SHIBATA
 
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAUHCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
panagenda
 
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfUnlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Malak Abu Hammad
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
Chart Kalyan
 
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and MilvusGenerating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
Zilliz
 
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and MilvusBuilding Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
Zilliz
 
Azure API Management to expose backend services securely
Azure API Management to expose backend services securelyAzure API Management to expose backend services securely
Azure API Management to expose backend services securely
Dinusha Kumarasiri
 
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptxNordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
MichaelKnudsen27
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
Octavian Nadolu
 
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
shyamraj55
 
System Design Case Study: Building a Scalable E-Commerce Platform - Hiike
System Design Case Study: Building a Scalable E-Commerce Platform - HiikeSystem Design Case Study: Building a Scalable E-Commerce Platform - Hiike
System Design Case Study: Building a Scalable E-Commerce Platform - Hiike
Hiike
 
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing InstancesEnergy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Alpen-Adria-Universität
 
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracyGraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
Tomaz Bratanic
 
Deep Dive: Getting Funded with Jason Jason Lemkin Founder & CEO @ SaaStr
Deep Dive: Getting Funded with Jason Jason Lemkin Founder & CEO @ SaaStrDeep Dive: Getting Funded with Jason Jason Lemkin Founder & CEO @ SaaStr
Deep Dive: Getting Funded with Jason Jason Lemkin Founder & CEO @ SaaStr
saastr
 
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Jeffrey Haguewood
 

Recently uploaded (20)

GenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizationsGenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizations
 
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of GermanyPresentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
 
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdfNunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
 
AWS Cloud Cost Optimization Presentation.pptx
AWS Cloud Cost Optimization Presentation.pptxAWS Cloud Cost Optimization Presentation.pptx
AWS Cloud Cost Optimization Presentation.pptx
 
dbms calicut university B. sc Cs 4th sem.pdf
dbms  calicut university B. sc Cs 4th sem.pdfdbms  calicut university B. sc Cs 4th sem.pdf
dbms calicut university B. sc Cs 4th sem.pdf
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
 
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAUHCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
 
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfUnlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
 
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and MilvusGenerating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
 
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and MilvusBuilding Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
 
Azure API Management to expose backend services securely
Azure API Management to expose backend services securelyAzure API Management to expose backend services securely
Azure API Management to expose backend services securely
 
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptxNordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
 
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
 
System Design Case Study: Building a Scalable E-Commerce Platform - Hiike
System Design Case Study: Building a Scalable E-Commerce Platform - HiikeSystem Design Case Study: Building a Scalable E-Commerce Platform - Hiike
System Design Case Study: Building a Scalable E-Commerce Platform - Hiike
 
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing InstancesEnergy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
 
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracyGraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
 
Deep Dive: Getting Funded with Jason Jason Lemkin Founder & CEO @ SaaStr
Deep Dive: Getting Funded with Jason Jason Lemkin Founder & CEO @ SaaStrDeep Dive: Getting Funded with Jason Jason Lemkin Founder & CEO @ SaaStr
Deep Dive: Getting Funded with Jason Jason Lemkin Founder & CEO @ SaaStr
 
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
 

HBase Advanced - Lars George

  • 1. ADVANCED HBASE Architecture and Schema Design JAX UK, October 2012 Lars George Director EMEA Services
  • 2. About Me •  Director EMEA Services @ Cloudera •  Consulting on Hadoop projects (everywhere) •  Apache Committer •  HBase and Whirr •  O’Reilly Author •  HBase – The Definitive Guide •  Now in Japanese! •  Contact •  lars@cloudera.com 日本語版も出ました!   •  @larsgeorge
  • 8. HBase Tables and Regions •  Table is made up of any number if regions •  Region is specified by its startKey and endKey •  Empty table: (Table, NULL, NULL) •  Two-region table: (Table, NULL, “com.cloudera.www”) and (Table, “com.cloudera.www”, NULL) •  Each region may live on a different node and is made up of several HDFS files and blocks, each of which is replicated by Hadoop
  • 10. HBase Tables •  Tables are sorted by Row in lexicographical order •  Table schema only defines its column families •  Each family consists of any number of columns •  Each column consists of any number of versions •  Columns only exist when inserted, NULLs are free •  Columns within a family are sorted and stored together •  Everything except table names are byte[] (Table, Row, Family:Column, Timestamp) -> Value
  • 12. HBase Architecture (cont.) •  HBase uses HDFS (or similar) as its reliable storage layer •  Handles checksums, replication, failover •  Native Java API, Gateway for REST, Thrift, Avro •  Master manages cluster •  RegionServer manage data •  ZooKeeper is used the “neural network” •  Crucial for HBase •  Bootstraps and coordinates cluster
  • 13. HBase Architecture (cont.) •  Based on Log-Structured Merge-Trees (LSM-Trees) •  Inserts are done in write-ahead log first •  Data is stored in memory and flushed to disk on regular intervals or based on size •  Small flushes are merged in the background to keep number of files small •  Reads read memory stores first and then disk based files second •  Deletes are handled with “tombstone” markers •  Atomicity on row level no matter how many columns •  keeps locking model easy
  • 14. MemStores •  After data is written to the WAL the RegionServer saves KeyValues in memory store •  Flush to disk based on size, see hbase.hregion.memstore.flush.size •  Default size is 64MB •  Uses snapshot mechanism to write flush to disk while still serving from it and accepting new data at the same time •  Snapshots are released when flush has succeeded
  • 15. Compactions •  General Concepts •  Two types: Minor and Major Compactions •  Asynchronous and transparent to client •  Manage file bloat from MemStore flushes •  Minor Compactions •  Combine last “few” flushes •  Triggered by number of storage files •  Major Compactions •  Rewrite all storage files •  Drop deleted data and those values exceeding TTL and/or number of versions •  Triggered by time threshold •  Cannot be scheduled automatically starting at a specific time (bummer!) •  May (most definitely) tax overall HDFS IO performance Tip: Disable major compactions and schedule to run manually (e.g. cron) at off-peak times
  • 16. Block Cache •  Acts as very large, in-memory distributed cache •  Assigned a large part of the JVM heap in the RegionServer process, see hfile.block.cache.size •  Optimizes reads on subsequent columns and rows •  Has priority to keep “in-memory” column families in cache if(inMemory) { this.priority = BlockPriority.MEMORY; } else { this.priority = BlockPriority.SINGLE; } •  Cache needs to be used properly to get best read performance •  Turn off block cache on operations that cause large churn •  Store related data “close” to each other •  Uses LRU cache with threaded (asynchronous) evictions based on priorities
  • 17. Region Splits •  Triggered by configured maximum file size of any store file •  This is checked directly after the compaction call to ensure store files are actually approaching the threshold •  Runs as asynchronous thread on RegionServer •  Splits are fast and nearly instant •  Reference files point to original region files and represent each half of the split •  Compactions take care of splitting original files into new region directories
  • 19. Auto Sharding and Distribution •  Unit of scalability in HBase is the Region •  Sorted, contiguous range of rows •  Spread “randomly” across RegionServer •  Moved around for load balancing and failover •  Split automatically or manually to scale with growing data •  Capacity is solely a factor of cluster nodes vs. regions per node
  • 20. Column Family vs. Column •  Use only a few column families •  Causes many files that need to stay open per region plus class overhead per family •  Best used when logical separation between data and meta columns •  Sorting per family can be used to convey application logic or access pattern
  • 21. Storage Separation •  Column Families allow for separation of data •  Used by Columnar Databases for fast analytical queries, but on column level only •  Allows different or no compression depending on the content type •  Segregate information based on access pattern •  Data is stored in one or more storage file, called HFiles
  • 25. Key Cardinality •  The best performance is gained from using row keys •  Time range bound reads can skip store files •  So can Bloom Filters •  Selecting column families reduces the amount of data to be scanned •  Pure value based filtering is a full table scan •  Filters often are too, but reduce network traffic
  • 27. Fold, Store, and Shift •  Logical layout does not match physical one •  All values are stored with the full coordinates, including: Row Key, Column Family, Column Qualifier, and Timestamp •  Folds columns into “row per column” •  NULLs are cost free as nothing is stored •  Versions are multiple “rows” in folded table
  • 28. Key/Table Design •  Crucial to gain best performance •  Why do I need to know? Well, you also need to know that RDBMS is only working well when columns are indexed and query plan is OK •  Absence of secondary indexes forces use of row key or column name sorting •  Transfer multiple indexes into one •  Generate large table -> Good since fits architecture and spreads across cluster
  • 29. DDI •  Stands for Denormalization, Duplication and Intelligent Keys •  Needed to overcome shortcomings of architecture •  Denormalization -> Replacement for JOINs •  Duplication -> Design for reads •  Intelligent Keys -> Implement indexing and sorting, optimize reads
  • 30. Pre-materialize Everything •  Achieve one read per customer request if possible •  Otherwise keep at lowest number •  Reads between 10ms (cache miss) and 1ms (cache hit) •  Use MapReduce to compute exacts in batch •  Store and merge updates live •  Use incrementColumnValue Motto: “Design for Reads”
  • 31. Tall-Narrow vs. Flat-Wide Tables •  Rows do not split •  Might end up with one row per region •  Same storage footprint •  Put more details into the row key •  Sometimes dummy column only •  Make use of partial key scans •  Tall with Scans, Wide with Gets •  Atomicity only on row level •  Example: Large graphs, stored as adjacency matrix
  • 32. Example: Mail Inbox <userId> : <colfam> : <messageId> : <timestamp> : <email-message> 12345 : data : 5fc38314-e290-ae5da5fc375d : 1307097848 : "Hi Lars, ..." 12345 : data : 725aae5f-d72e-f90f3f070419 : 1307099848 : "Welcome, and ..." 12345 : data : cc6775b3-f249-c6dd2b1a7467 : 1307101848 : "To Whom It ..." 12345 : data : dcbee495-6d5e-6ed48124632c : 1307103848 : "Hi, how are ..." or 12345-5fc38314-e290-ae5da5fc375d : data : : 1307097848 : "Hi Lars, ..." 12345-725aae5f-d72e-f90f3f070419 : data : : 1307099848 : "Welcome, and ..." 12345-cc6775b3-f249-c6dd2b1a7467 : data : : 1307101848 : "To Whom It ..." 12345-dcbee495-6d5e-6ed48124632c : data : : 1307103848 : "Hi, how are ..." è Same Storage Requirements
  • 33. Partial Key Scans Key   Descrip+on   <userId> Scan  over  all  messages   for  a  given  user  ID   <userId>-<date> Scan  over  all  messages   on  a  given  date  for  the   given  user  ID   <userId>-<date>-<messageId> Scan  over  all  parts  of  a   message  for  a  given  user   ID  and  date   <userId>-<date>-<messageId>-<attachmentId> Scan  over  all   a8achments  of  a   message  for  a  given  user   ID  and  date  
  • 34. Sequential Keys <timestamp><more key>: {CF: {CQ: {TS : Val}}} •  Hotspotting on Regions: bad! •  Instead do one of the following: •  Salting •  Prefix <timestamp> with distributed value •  Binning or bucketing rows across regions •  Key field swap/promotion •  Move <more key> before the timestamp (see OpenTSDB later) •  Randomization •  Move <timestamp> out of key
  • 35. Salting •  Prefix row keys to gain spread •  Use well known or numbered prefixes •  Use modulo to spread across servers •  Enforce common data stay close to each other for subsequent scanning or MapReduce processing 0_rowkey1, 1_rowkey2, 2_rowkey3 0_rowkey4, 1_rowkey5, 2_rowkey6 •  Sorted by prefix first 0_rowkey1 0_rowkey4 1_rowkey2 1_rowkey5 …
  • 36. Hashing vs. Sequential Keys •  Uses hashes for best spread •  Use for example MD5 to be able to recreate key •  Key = MD5(customerID) •  Counter productive for range scans •  Use sequential keys for locality •  Makes use of block caches •  May tax one server overly, may be avoided by salting or splitting regions while keeping them small
  • 38. Key Design Summary •  Based on access pattern, either use sequential or random keys •  Often a combination of both is needed •  Overcome architectural limitations •  Neither is necessarily bad •  Use bulk import for sequential keys and reads •  Random keys are good for random access patterns
  • 39. Example: Facebook Insights •  > 20B Events per Day •  1M Counter Updates per Second •  100 Nodes Cluster •  10K OPS per Node •  ”Like” button triggers AJAX request •  Event written to log file •  30mins current for website owner Web  ➜  Scribe  ➜  Ptail  ➜  Puma  ➜  HBase  
  • 40. HBase Counters •  Store counters per Domain and per URL •  Leverage HBase increment (atomic read-modify- write) feature •  Each row is one specific Domain or URL •  The columns are the counters for specific metrics •  Column families are used to group counters by time range •  Set time-to-live on CF level to auto-expire counters by age to save space, e.g., 2 weeks on “Daily Counters” family
  • 41. Key Design •  Reversed Domains •  Examples: “com.cloudera.www”, “com.cloudera.blog” •  Helps keeping pages per site close, as HBase efficiently scans blocks of sorted keys •  Domain Row Key = MD5(Reversed Domain) + Reversed Domain •  Leading MD5 hash spreads keys randomly across all regions for load balancing reasons •  Only hashing the domain groups per site (and per subdomain if needed) •  URL Row Key = MD5(Reversed Domain) + Reversed Domain + URL ID •  Unique ID per URL already available, make use of it
  • 43. Summary •  Design for Use-Case •  Read, Write, or Both? •  Avoid Hotspotting •  Consider using IDs instead of full text •  Leverage Column Family to HFile relation •  Shift details to appropriate position •  Composite Keys •  Column Qualifiers
  • 44. Summary (cont.) •  Schema design is a combination of •  Designing the keys (row and column) •  Segregate data into column families •  Choose compression and block sizes •  Similar techniques are needed to scale most systems •  Add indexes, partition data, consistent hashing •  Denormalization, Duplication, and Intelligent Keys (DDI)