Cloudera Solutions Architect, Joey Echeverria, explains Hadoop and HBases architecture and roles in the real world of data management and storage.

1. Apache Hadoop and HBase in
the Real World
Joey Echeverria
@fwiffo
#novahug

2. The Plug
We're Training!
Developer Training July 25 to 27
Admin Training July 28 to 29
http://www.cloudera.com/training
We’re Hiring!
Solution Architects, Trainers,
Distributed Systems Engineers
http://www.cloudera.com/careers
Copyright 2011 Cloudera Inc. All rights reserved 2

3. 1 Minute Hadoop Recap
HDFS
Distributed file system
Optimized for streaming reads and writes
Block-level replication
MapReduce
Distributed processing framework
Reads/writes data in HDFS (typically)
Operates over (key, value) view of data
Copyright 2011 Cloudera Inc. All rights reserved 3

4. Where does HBase come in?
Google
Google invented GFS and MapReduce
GFS optimized for streaming reads and writes
BigTable
Google's answer to random read/write workloads
Copyright 2011 Cloudera Inc. All rights reserved 4

5. HBase: BigTable-like storage (for Hadoop)
Copyright 2011 Cloudera Inc. All rights reserved 5

6. What is HBase?
Key/value column family store
Data stored in HDFS
ZooKeeper for coordination
Access model is get/put/del
Plus range scans and versions
Copyright 2011 Cloudera Inc. All rights reserved 6

7. Architecture
Image courtsey Lars George, Licensed uner Creative Commons
Attribution-Noncommercial-Share Alike 3.0 Germany License.
Copyright 2011 Cloudera Inc. All rights reserved 7

8. Tables and Column Families
Static part of the schema
Column families also form locality groups
One Store per family
Multiple HFiles per Store
Tables split into regions
Continuous range of row keys
Unit of distribution
Automatically split
Pre-split for performance
Copyright 2011 Cloudera Inc. All rights reserved 8

9. Why use HBase?
Variable schema in each record
Collections of data for each key
Atomic control of per-key data
Row access to each column family
Copyright 2011 Cloudera Inc. All rights reserved 9

10. HBase Applications
“Smart Data, at Scale, made Easy”
http://www.lilyproject.org
“Distributed, scalable Time Series Database (TSDB)”
http://opentsdb.net
Copyright 2011 Cloudera Inc. All rights reserved 10

11. Real-time ad optimizations
Capturing impressions and serving ads
HBase front-end – to serve models (via memcached)
HBase back-end – to serve pixels and capture cookies
MapReduce to compute models between the two
Copyright 2011 Cloudera Inc. All rights reserved 11

12. Click stream sessionization
Key on userid and time
Seperate table for significant events (e.g. purchase)
Load data using HBase importtsv tool
Sessionization performed by simple scans
Copyright 2011 Cloudera Inc. All rights reserved 12

13. Mozilla - Soccorro
When Firefox crashes, where do reports go?
The Mozilla team gathers those crashes in HBase
Crashes varry widely and change format often
Processors take each individually and parse it out
http://crash-stats.mozilla.com
http://code.google.com/p/socorro
Copyright 2011 Cloudera Inc. All rights reserved 13

14. Navteq
Location based content serving
All served out of HBase, location makes a great key
Content is variable – Maps, POI, User Data
Preprocessing is all done via MR jobs
Copyright 2011 Cloudera Inc. All rights reserved 14

15. Cloudera
Gathers data about customer clusters
Each customer node is a key with Avro values
Easy to browse, quick to find issues on Nodes
Dump to HDFS and process with Pig
Copyright 2011 Cloudera Inc. All rights reserved 15

16. Copyright 2011 Cloudera Inc. All rights reserved 16