1. Large Scale Web Apps @Pinterest
(Powered by Apache HBase)
May 5, 2014

2. Pinterest is a visual discovery tool for
collecting the things you love, and discovering
related content along the way.
What is Pinterest ?

4. Scale
Challenges @scale
• 100s of millions of pins/repins per month
• Billions of requests per week
• Millions of daily active users
• Billions of pins
• One of the largest discovery tools on the internet

5. Storage stack @Pinterest
!
• MySQL
• Redis (persistence and for cache)
• MemCache (Consistent Hashing)
App Tier
Manual
Sharding
Sharding
Logic

6. Why HBase ?
!
• High Write throughput
- Unlike MySQL/B-Tree, writes don’t ever seek on Disk
• Seamless integration with Hadoop
• Distributed operation
- Fault tolerance
- Load Balancing
- Easily add/remove nodes
!
Non-Technical Reasons
• Large active community
• Large scale online use cases

7. Outline
!
• Features powered by HBase
• SaaS (Storage as a Service)
- MetaStore
- HFile Service (Terrapin)
• Our HBase setup - optimizing for High availability & Low latency

8. Applications/Features
!
• Offline
- Analytics
- Search Indexing
- ETL/Hadoop worklows
• Online
- Personalized Feeds
- Rich Pins
- Recommendations
!
Why HBase ?

9. Personalized Feeds
WHY HBASE ?
Write Heavy load due
to Pin fanout.
Recommended
Pins
Users I follow

10. Rich Pins
WHY HBASE ?
Negative Hits with Bloom
Filters

11. Recommendations
HADOOP
1.0
HBASE +
HADOOP 2.0
HADOOP
2.0
WHY HBASE ?
Seamless Data Transfer from
Hadoop
Generate
Recommendations
DistCP Jobs
Serving Cluster

12. SaaS
• Large number of feature requests
• 1 Cluster per feature
• Scaling with organizational growth
• Need for “defensive” multi tenant storage
• Previous solutions reaching their limits

13. MetaStore I
• Key Value store on top of HBase
• 1 HBase Table per Feature with salted keys
• Pre split tables
• Table level rate limiting (online/offline reads/writes)
• No Scan support
• Simple client API!
!
string getValue(string feature, string key, boolean online);
void setValue(string feature, string key, string value,
boolean online);

14. MetaStore II
MetaStore
Thrift Server
Primary HBase Secondary HBase
Clients
Master/Master
Replication
Thrift
Salting +
Rate Limiting
ZooKeeper
Issue
Gets/Sets
Notifications
Metastore Config
- Rate Limits
- Primary Cluster

15. HFile Service (Terrapin)
• Solve the Bulk Upload problem
• HBase backed solution
- Bulk upload + major compact
- Major compact to delete old data
• Design solution from scratch using mashup of:
- HFile
- HBase BlockCache
- Avoid compactions
- Low latency key value lookups
!
!
!

16. High Level Architecture I
!
Client Library
/Service
ETL/Batch Jobs
Load/Reload
HFile
Servers
!
HFiles on
Amazon S3
Key/Value
Lookups
Multiple
HFiles/Server

17. High Level Architecture II
• Each HFile server runs 2 processes
- Copier: pulls HFiles from S3 to local disk
- Supershard: serves multiple HFile shards to client
• ZooKeeper
- Detecting alive servers
- Coordinating loading/swapping of new data
- Enabling clients to detect availability of new data
• Loader Module (replaces distcp)
- Trigger new data copy
- Trigger swap through zookeeper
- Update ZooKeeper and notify client
• Client library understands sharding
• Old data deleted by background process
!

18. Salient Features
• Multi tenancy through namespacing
• Pluggable sharding functions - modulus, range & more
• HBase Block Cache
• Multiple clusters for redundancy
• Speculative execution across clusters for low latency
!
!
!

19. Setting up for Success
• Many online usecases/applications
• Optimize for:
- Low MTTR - high availability
- Low latency (performance)
!
!

20. MTTR - I
DEADLIVE STALE
20sec 9min 40sec
!
• Stale nodes avoided
- As candidates for Reads
- As candidate replicas for writes
- During Lease Recovery
• Copying of underreplicated blocks starts when a Node is
marked as “Dead”
DataNode States

21. MTTR - II
Failure Detection
Lease Recovery
Log Split
Recover Regions
30 sec ZooKeeper
session timeout
HDFS 4721
HDFS 3703 +
HDFS 3912
< 2 min
!
• Avoid stale nodes at each point of the recovery process
• Multi minute timeouts ==> Multi second timeouts

22. Simulate, Simulate, Simulate
Simulate “Pull the plug failures” and “tail -f the logs”
• kill -9 both datanode and region server - causes connection
refused errors
• kill -STOP both datanode and region server - causes socket
timeouts
• Blackhole hosts using iptables - connect timeouts + “No
Route to host” - Most representative of AWS failures

23. Performance
Configuration tweaks
• Small Block Size, 4K-16K
• Prefix compression to cache more - when data is in the key,
close to 4X reduction for some data sets
• Separation of RPC handler threads for reads vs writes
• Short circuit local reads
• HBase level checksums (HBASE 5074)
Hardware
• SATA (m1.xl/c1.xl) and SSD (hi1.4xl)
• Choose based on limiting factor
- Disk space - pick SATA for max GB/$$
- IOPs - pick SSD for max IOPs/$$, clusters with heavy reads or
heavy compaction activity

24. Performance (SSDs)
HFile Read Performance
• Turn off block cache for Data Blocks, reduce GC + heap
fragmentation
• Keep block cache on for Index Blocks
• Increase “dfs.client.read.shortcircuit.streams.cache.size” from
100 to 10,000 (with short circuit reads)
• Approx. 3X improvement in read throughput
!
Write Performance
• WAL contention when client sets AutoFlush=true
• HBase 8755

25. In the Pipeline...
!
• Building a graph database on HBase
• Disaster recovery - snapshot + incremental backup + restore
• Off Heap cache - reduce GC overhead and better use of
hardware
• Read path optimizations

26. And we are Hiring !!