1. HBaseCon 2012
Applications Track – Case Study
1

2.  Suraj Varma
 Director of Technology Implementation
 Gap Inc Direct (GID), San Francisco, CA
 IRC: svarma
 Gupta Gogula
 Director-IT & Domain Architect of Catalog
Management & Distribution
 Gap Inc Direct (GID), San Francisco, CA
2

3.  Problem Domain
 HBase Schema Specifics
 HBase Cluster Specifics
 Learning & Challenges
3

4. 2009
2008
2005
2007
2010 APPLICATION SERVERS DATABASES
NEWATHLETA
CA & SITE LAUNCH
UNIVERSALITY
PIPERLIME
EU MARKETS
US
CA EU
US
CA EU
INCOMING TRAFFIC
US
CA EU
US
US
4

5.  Evolution of the GID Apparel Catalog
 2005 - Three independent brands in US
 2010 – 5 integrated brands in US, CA, EU
 Rapid Expansion of Apparel Catalog
 However, each brand / market combination
necessitated separate logical catalog
databases
5

6.  Single Catalog store for all brands/markets
 Horizontally scalable over time
 Cross brand business features
 Access data store directly
 To avail of inventory awareness of items
 Minimal Caching – only for optimization
 Keeping caches in sync is a problem.
 Highly Available
6

7.  Sharded RDMBS, MemCached, etc
 Significant effort was required
 Still had scalability limits
 Non-relational alternatives considered
 HBase POC (early-2010)
 Promising results -decided to move ahead
7

8.  Strong Consistency Model
 Server Side Filters
 Automatic Sharding, Distribution, Failover
 Hadoop Integration out of the box
 General Purpose
 Other use cases outside of Catalog
 Strong Community!
8

9. NEAR REAL TIME INVENTORY UPDATES
MUTATIONS
INCOMING
REQUESTS BACKEND HBASE
FOR SERVICES REQUESTS CLUSTER
CATALOG DATA
MUTATIONS MUTATIONS
PRICING UPDATES ITEM UPDATES
9

10.  Read Mostly  Website Traffic
 Sync MR Jobs
 Write / Delete Bursts  Catalog Publish
 Phase out to near real-
time updates from
originating systems
 MR jobs on Live Cluster
 Continuous Writes  Inventory Updates
10

11.  Hierarchical Data (Primarily)
 SKU -> Style Lookups (child -> parent)
 Cross Brand Sell (sibling <-> sibling) Rows:
100KB avg size
1000-5000 cols
 Data Access Patterns Sparse rows
 Full Product Graph in one read
 Single path of graph from root to leaf node
 Search - Secondary Indices
 Large Feed files
11

12. READ FULL GRAPH
READ SINGLE PATH / EDGE
12

13.  Built custom “bean to schema mapper”
 POJO graph < -> HBase qualifiers
 Flexibility to shorten column qualifiers
 Flexibility to change schema qualifiers (per
environment / developer)
<…>
<association>one-to-many</association>
<prefix>SC</prefix>
<uniqueId>colorCd</uniqueId>
<beanName>styleColorBean</beanName>
<…>
13

14.  <PP>_<id1>_QQ_<id2>_RR_<id3>_name
 Where PP is parent, QQ is child, RR is grandchild
Pattern: ANCESTOR IDS EMBEDDED IN QUALIFIER NAME
cf1:VAR_1_SC_0012_colorCd
cf2:VAR_1_SC_0012_SCIMG_10_path
14

15.  Secondary Index
 <id3> => RR ; QQ ; PP
 FilterList with (RR, QQ, PP) ids to get thin slice
path
Pattern: SECONDARY INDEX TO HIERARCHICAL ANCESTORS
KEY_5555 4444 333 22 1
15

16.  “Publish at Midnight”
 Future Dated PUTs
 Get/Scan with time range
 Large Feed Files
 Sharded into smaller chunks < 2MB per cell
Pattern: SHARDED CHUNKS
KEY_nnnn S_1 S_2 S_3 S_4
16

17.  16 Slave (RS + TT + DN) Nodes
 8 & 16 GB RAM
 3 Master (HM,ZK,JT, NN) Nodes
 8 GB RAM
 NN Failover via NFS
17

18.  Block Cache
 Maximize Block Cache
 hfile.block.cache.size: 0.6
 Garbage Collection
 MSLAB enabled
 CMSInitiatingOccupancyFactor
18

19.  Quick Recovery on node failure
 Default timeouts too large
 zookeeper.session.timeout
 Region Server
 hbase.rpc.timeout
 Data Node
 dfs.heartbeat.recheck.interval
 heartbeat.recheck.interval
19

20.  Block Cache Size Tuning
 Block Cache Churn
 Hot Row scenarios
 Perf Tests & Doing Phased Rollouts
 Hot Region issues
 Perf Tests & Pre-split Regions.
 Filters
 CPU Intensive – profiling needed.
20

21.  Monitoring is crucial
 Layer by layer -> what’s the bottleneck
 Metrics to target optimization & tuning
 Troubleshooting
 Non Uniform Hardware
 Sub-optimal region distribution
 Hefty boxes lightly loaded.
21

22.  M/R Jobs running on live cluster
 Has an impact – so cannot run full throttle
 Go easy …
 Feature Enablement – Phase in
 Don’t turn on several features together
 Easier identification of potential hot regions /
rows, overloaded RS, etc
22

23. INVENTORY UPDATES
FEATURE “A” ENABLED:
ADDITIONAL “N” REQ / SEC
INCOMING BACKEND LOT
HBASE
REQUESTS SERVICES MORE CLUSTER
REQUESTS
FEATURE “B” ENABLED:
ADDITIONAL “K” REQ / SEC
PRICING UPDATES ITEM UPDATES
Enable Features individually to measure impact and tune cluster accordingly
23

24.  Search
 No out-of-the-box secondary indexes.
 Custom solution with Solr
 Transactions
 Only row level atomicity
 But … can’t pack all in a single row
 Atomic Cross-Row Put/Delete and HBASE-5229
seem potential partial solves (0.94+)
24

25.  Orthogonal access patterns
 Optimize for most frequently used pattern.
 Filters
 May suffice, with early out configurations
 Impacts CPU usage
 Duplicate data for every access pattern
 Too drastic
 Effort to keep all copies in sync
25

26.  Rebuild from source data
 Takes time … but no data loss
 Export / import based backups
 Faster … but stale
 Another MR on live cluster
 Better options in future releases …
26

27. We’re hiring!
http://www.gapinc.com
27