Chicago Data Summit: Geo-based Content Processing Using HBase

Geo-based Content Processing using HBase
Ravi Veeramachaneni
NAVTEQ
1

Agenda
Problem
Solution
Why HBase?
Challenges
Why Cloudera?
2

Problem
Ineffective to scaleout
Cost:Expensive Oracle license cost
Technology: Inherent limitations of RDBMS
Need for real-time data Availability
Content needs to be updated and delivered much faster than before
Unable to deliver better, richer, faster contextual content more efficiently
Need to support both customers with connected and disconnected devices
Need to support flexible functionality
Need to support both structured and unstructured data
Need to decouple content from the map
Need flexibility to quickly add new content providers
Need to support community input
3

Our Data is Constantly Growing
Content Breadth
100s ofmillions of content records
100s of content suppliers + community input
Content Depth
On average, a content record has 120attributes
Certain types of content have more than 400 attributes
Content classified across 270+ categories
Our content is
Sparse and unstructured
Provided in multiple data formats
Ingested, processed and delivered in transactional and batch mode
Constantly growing (higher TB)
4

Why HBase?
HBase Scales
As it runs on top of Hadoop
HBase stores null values for free
Saves both disk space and disk IO time
HBase supports unstructured data through column families
Individual records are called cells
Cells are addressed with row key/column family/cell qualifier/timestamp tuple
HBase has built-in version management
HBase provides fast table scans for time ranges and fast key based lookups
Map Reduce data input
Tables are sorted and have unique keys
Reducer often times optional
Combiner not needed
Strong community support and wider adoption
5

Provide scalable content processing platform to handle spikes in content processing demand
Providing for horizontal scalability (Hadoop/HBase)
Provide business rules management system to adapt to changing processing needs
Flexible business rules based on the supplier and content information
Flexible business flows and processing to meet SLA
Provide high value and high quality content to customers fast
Corroborate multiple sources to produce the best quality information
Utilize open source software whereeverapplicable
Use commercial support for open source solutions to have a piece of mind
6
Our Approach for Content Processing Challenges

7
Content Processing Overview
Batch and Transactional Ingestion
Supplier n
Supplier 1
Supplier 2
Location ID
Permanent ID
Source & Blended Record Management
PUBLISHING
Permanent ID
Location ID
real-time, on-demand

8
System Decomposition

HBase @NAVTEQ
Started in late-2009, hbase 0.19.x (apache)
8-node VMWare Sandbox Cluster
Flaky, unstable, RS Failures
Switched to CDH
No Cloudera support
Early 2010, hbase 0.20.x (CDH2)
10-node Physical Sandbox Cluster
Still had lot of challenges, RS Failures, META corruption
Cluster expanded significantly with multiple environments
Signed Cloudera support
No official Cloudera support for HBase
Late 2010, hbase 0.89 (CDH3b3)
Most stable than any other version in the past
Multiple teams at NAVTEQ exploring to use Hadoop/HBase
Official Cloudera supported version
Current
Waiting on moving to CDH3 official release
Looking into Hive, Oozie, Lucene/Solr integration, Cloudera Enterprise and few other
Several initiatives to use HBase
9

HBase @NAVTEQ
Hardware / Environment
Dual-quad core with HT
64GB RAM (ECC)
4x2TB (JBOD)
RHEL 5.4
500+ CPU cores
200+ TB configured Disk Capacity
Multiple environments
DEVELOPMENT
INTEGRATION
STAGING
PRODUCTION
10

Database/schema design
Transition to Column-oriented or flat schema
Row-key design/implementation
Sequential keys (or UUID)
Suffers from distribution of load but uses the block caches
Can be addressed by pre-splitting the regions
Randomize keys to get better distribution
Suffers range scans
Achieved through hashing on Key Attributes – SHA1
Too many Column Families
Performance was horrible
Initially we had about 100 or so, now downed to 8
Compression
LZO didn’t work out well with CDH2, using default Block compression
Need to revisit with CDH3
11
Challenges/Lessons Learned

Serialization
Everything in HBase stored as Byte Array with the exception of timestamp
AVRO didn’t work well – deserialization issue
Developed configurable serialization mechanism that uses JSON except Date type
Indexing
Were using ITHBase and IHBase from contrib – doesn’t work well
Redesigned schema without need for index
We still need it though
Cluster configuration
Configuration is the key
Many moving parts – typos, out of synchronization
Automating deployments with Puppet (looking in to it)
Hadoop
Adjust block size based on the use case
12

Cluster Configuration
HBase
Needs more memory
ZK on DN – have a separate ZK quorum
No swapping – JVM hates it
GC pauses could cause timeouts or RS failures (read article posted by Todd Lipcon on avoiding full GC)
Write (puts) optimization (Ryan Rawson HUG8 presentation – HBase importing)
hbase.regionserver.global.memstore.upperLimit=0.3
hbase.regionserver.global.memstore.lowerLimit=0.15
hbase.hstore.blockingStoreFiles=25
hbase.hregion.memstore.block.multiplier=8
hbase.regionserver.handler.count=30
Operating System
Number of open files (ulimit) to 32K or even higher
vm.swapiness to lower or 0
13

Performance
Several tunable parameters (Hadoop, HBase, OS, JVM, Networking, Hardware)
Scalability
Interacting with transactional-based (interactive) systems from batch oriented system
Security
Introduced in CDH3b3 but in flux, need robust RBAC
Reliability
Name Node is SPOF
HBase is sensitive
Region Server Failures
14

Had a few on-site training classes
Cluster configurations were reviewed and made some recommendations
Tickets resolved within the SLAs
Knowledgeable support team
Ability to have access to technology experts, if needed
With Cloudera support for almost a year
Our needs and demand increasing and looking towards enterprise support
15
Why Cloudera?

Better operational tools for using Hadoop and HBase
Job management, backup, restore, user provisioning, general administrative tasks, etc.
Support for Secondary Indexes
Indexing and Searching (Lucene/Solr integration?)
HA support for Name Node
Need Data Replication for HA & DR
Best practices and supported material
16
Features Needed

17
Thank You
Q & A

http://www.cloudera.com	776
http://lanyrd.com	42
http://blog.cloudera.com	6
http://test.cloudera.com	1
http://webcache.googleusercontent.com	1

Chicago Data Summit: Geo-based Content Processing Using HBase

by Cloudera, Inc.

Accessibility

Categories

Upload Details

Usage Rights

5 Embeds 826

Statistics

Chicago Data Summit: Geo-based Content Processing Using HBase Presentation Transcript