HBaseCon 2015: HBase at Scale in an Online and High-Demand Environment

1. HBase 2 Jeremy Carroll & Tian-Ying Chang Online, at at Low Latency

2. 30+ Billion Pins categorized by people into more than  750 Million Boards 3

3. Use Cases Products running on HBase / Zen 4 The SmartFeed service renders the Pinterest landing page for feeds. Pinterest Engineering Provides personal suggestions based on a preﬁx for a user. Pinterest Engineering Messages / Notiﬁcations Pinterest Engineering Interests Pinterest Engineering 1 2 3 4

4. Online Performance

5. Problem Validating the new i2 platform 6 p99.9 zen get nodes in ms

6. Garbage Collection Stop the world events 7 pause time in milliseconds

7. Problem #1: Promotion Failures Issues seen in production Heap fragmentation causing promotion failures BlockCache at high QPS was causing fragmentation Keeping hundreds of billions of rows BloomFilters from being evicted, leading to latency issues Solutions Tuning CombinedCache for space for Memstore + Blooms & Indexes with CombinedCache / OffHeap Monitoring % of LRU Heap for blooms & indexes Tuning & BucketCache 8

8. Problem #2: Pause Time Calculating Deltas Started noticing ‘user + sys’ vs ‘real’ was very different Random spikes of delta time, concentrated on hourly boundaries Found resources online, but none of the ﬁxes seemed to work http://www.slideshare.net/cuonghuutran/gc-andpagescanattacksbylinux http://yoshinorimatsunobu.blogspot.com/2014/03/why-buffered-writes-are-sometimes.html Low user, low sys, high real 9

9. Cloud Problems Noisy neighbors 10 /dev/sda average io in milliseconds

10. Formatting with EXT4 Logging to instance-store volume 11 100% 1251.60 99.99% 1223.52 99.9% 241.33 90% 151.45 Pause time in ms

11. Formatting with XFS PerfDisableSharedMem & logging to epemeral 12 100% 115.64 99.99% 107.88 99.9% 94.01 90% 66.38 Pause time in ms

12. Online Performance JVM Options -XX:+PerfDisableSharedMem  -Djava.net.preferIPv4Stack=true Instance Conﬁguration Treat instance-store as read only  irqbalance >= 1.0.6  kernel 3.13+ for disk performance Changes for success on EC2 13

13. Monitoring

14. Monitoring Zen SLA driven from proxies • Success criteria is composite of two clusters due to backup requests • Wounded cluster due to too many backup requests • Additional query metadata from proxy logs. Aids in hot key analysis 99.99% Uptime 15 table Thrift zen cluster table table zen cluster table Thrift Thrift Backup Request Replication

15. Dashboards Metrics, Metrics, Metrics • Usually low value, except when you need them • Dashboards for all H-Stack daemons (DataNode, NameNode, etc..) • Capture amazing amounts of information due to per-region information • Per table / per region stats are very useful Deep dives 16

16. HBase Alerts Maintenance primary driver • Few clusters cause the majority of the alarms • Mainly driven by lack of capacity planning Hygiene / Capacity notiﬁcations • Replacing failed / terminated nodes • Measuring disk / cpu / network • Canary analysis on code deployments Common alerts • Hot CPU / Disk Space Analytics for HBase On-Call 17

17. HotSpots Spammers • Few users request the same row over and over • Rate limiting / caching Real time analysis • TCPDump is very helpful tcpdump -i eth0 -w - -s 0 tcp port 60020 | strings • Looking at per-region request stats Code Issues • Hard-coded key in product. Ex: Messages launch Debugging imbalanced requests 18

18. Capacity Planning

19. Capacity Planning Managed Splitting UniformSplitAlgo to pre-split regions Salted keys for uniform distribution Name spaced table on a shared cluster Feature Cost Some table attributes are more expensive BlockSize, Bloom Filters, Compression (Preﬁx, FastDiff, Snappy) Start small. Split for growth 20 table_feat Split Split

20. Capacity Planning Distributing Load Balance is important to eliminate hot spots Per-table load balancing Disable auto-splitting and monitor region size Scaling Metrics CPU Utilization HDFS Space Regions per server (Memstore / Flushing) All Others (Memory, Bandwidth, etc..) Balancing regions to servers 21 table table 1 2 4 5 3 1 3 4 2

21. Launching Messages • Started on a NameSpace table on a shared cluster • Ramping out to production with decider (x%) • Split table to get additional region servers serving trafﬁc • Migrated to dedicated cluster • As experiment ramped up, added / removed capacity as feature was adopted From development to production 22 2:50 PM 100% https://www.ﬂickr.com/photos/zachd1_618/13498790545

22. Operations with AWS 23

23. Availability Conditions for Failure Termination notices from the underlying host Default RF of 3 in one zone dangerous Placement Groups may make this worse Stability Patterns Highest numbered instance type in a family Multi Availability Zone + block placement “Air Gaped” change management Strategies for mitigating failure 24 Master Slave US-East-1A US-East-1E Replication

24. Disaster Recovery Recovering Data System copies WALs and Snapshots to local HDFS HDFS keeps <X> hours locally. Rotates to S3 Data can restore from S3, HDFS, or another Cluster w/rate limited copying Used frequently to test new conﬁgurations, and upgrade clusters side-by-side    hbaserecover -t unique_index -ts 2015-04-08-16-01-15 --source-cluster zennotifications & bootstrapping new clusters 25 Master Slave US-East-1A US-East-1E Replication DR SlaveReplication US-East-1D HDFS S3

25. Flow Monitoring Snapshot backup routine • ZooKeeper based conﬁguration for each cluster • Backup metadata is sent to ElasticSearch for integration with dashboards • Monitoring and alerting around WAL copy & snapshot status Reliable cloud backups 26

26. Flow Monitoring DistCP all hlog hourly • Copy from slave cluster to avoid latency impact • Using DistCP V2 which has throttling • Spike in graph means DistCP failed, huge amount of data copied once issued ﬁxed • Using ElasticSearch to store all backup metadata Alert when backup pipeline failed 27

27. 4 28 Maintenance Check Health Get Lock Get Server Locality Check Server Status Region Movement w/Threads Start RegionServer + Cooldown Release Lock Update Status + Cooldown Rolling Restart Change management while retaining availability 1 2 3 Region Movement w/Threads Stop RegionServer + Cooldown Verify Locality Check Server Status

28. Rolling Compaction Only one region per server is selected • Avoid blocked region in queue Controlled concurrency • Control the space spike • Reduce increased network and disk trafﬁc Controlled time to stop • Stop before day time trafﬁc ramp up • Stop if compaction causing perf issue Resume the next night • Filter out the regions that has run compaction Important for online facing clusters 29

29. Next Challenges Upgrade to latest stable (1.x) from 0.94.x w/no downtime Increasing performance • Lower latency • Better compaction throughput Regional Failover • Cross Datacenter is in production now • Need cross regional failover Looking forward 30