Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Storage Infrastructure Behind
Facebook Messages
Using HBase at Scale
Jianfeng Zhang
• HBase is an open source, non-relational,
distributed database modeled after Google's
BigTable and is written in Java. It...
• Facebook is an online social networking service. Its name
comes from a colloquialism for the directory given to
students...
• As of January 2014, Facebook has about 1.2
billion monthly users.
• Based on its 2012 income of US$5 billion,
Facebook j...
Monthly data volume prior to launch

15Billion×1,024byte=14TB

200 Billion×100bytes=11TB
Why picked Hbase for Facebook
Messages
•
•
•
•
•
•

High write throughput
Low latency random reads
Elasticity
Cheap and fa...
Messaging Data
• Small and medium data (Hbase)
Message metadata and bodies
Snapshot of recent messages
Search indices
• At...
HBase Architecture
•

ZooKeeper
Metadata
• Hmaster
Recovery
Balacing
• Region Server
Log>Flush
Store>Compaction
Region>Spl...
Facebook Messages: Quick Stats
• 6B+ messages/day
•
•
•
•
•
•

▪ Traffic to HBase
▪ 75+ Billion R+W ops/day
▪ At peak: 1.5...
Facebook Messages: Quick Stats (contd.)
• 2PB+ of online data in HBase (6PB+ with
replication;
• excludes backups)
• ▪ mes...
Facebook Messages: Quick Stats (contd.)
•
•
•
•
•
•
•
•
•
•

Timeline:
▪ Started in Dec 2009
▪ Roll out started in Nov 201...
Backups (V2)
• Now, does periodic HFile level backups.
• ▪ Working on:
• ▪ Moving to HFile + Commit Log based
backups to b...
Messages Schema & Evolution
•
•
•
•
•
•
•
•
•
•
•

“Actions” (data) Column Family the source of truth
▪ Log of all user ac...
Write Path Overview
Flushes: Memstore -> HFile
Read Path Overview
Compactions
Reliability: Early work
• HDFS sync support for durability of
transactions
• ▪ Multi-CF transaction atomicity
• ▪ Several ...
Availability: Early Work
•
•
•
•
•
•
•
•
•

Common reasons for unavailability:
▪ S/W upgrades
▪ Solution: rolling upgrades...
Performance: Early Work
• Read optimizations:
• ▪ Seek optimizations for rows with large number
of cells
• ▪ Bloom Filters...
Performance: Compactions
•
•
•
•
•
•
•
•
•

Critical for read performance
▪ Old Algorithm:
#1. Start from newest file (fil...
Performance: Compactions
Performance: Compactions
•
•
•
•
•
•
•
•
•
•
•

More problems!
▪ Read performance dips during
peak
▪ Major compaction stor...
Metrics, metrics, metrics…
•
•
•
•
•
•
•
•
•
•
•

Initially, only had coarse level overall metrics (get/put latency/ops;
b...
Metrics (contd.)
•
•
•
•
•
•

HBase Master Statistics:
▪ Number of region servers alive
▪ Number of regions
▪ Load balanci...
Need to keep up as data grows on you!
•
•
•
•
•
•
•
•
•
•
•
•

Rapidly iterated on several new features while in productio...
Scares & Scars!
• Not without our share of scares and incidents:
• ▪ s/w bugs. (e.g., deadlocks, incompatible LZO used for...
Scares & Scars! (contd.)
•
•
•
•
•
•
•
•
•
•

Sometimes, tried things which hadn’t been tested in dark launch!
▪ Added a r...
Choosing HBase
They evaluated and tested various solutions
• Strong consistency model
• Automatic failover
• Multiple shar...
HBase uses HDFS
They get the benefits of HDFS as a storage system for free
• HDFS has attractive features out of the box
E...
Working with HBase and HDFS
Goal of zero data loss
•
•
•
•

HDFS sync support for Write-Ahead-Log
Row level ACID property
...
Stability and performance
• Availability and operational improvements
Rolling restarts-minimal downtime on upgrades
Abilit...
Operational Challenges
• Darklaunch
• Deployments and monitoring
Lots of internal HBase clusters used for various purposes...
Future work
•
•
•
•
•
•
•
•
•
•
•

Reliability, Availability, Scalability!
▪ Lot of new use cases on top of HBase in the w...
Storage Infrastructure Behind Facebook Messages
Storage Infrastructure Behind Facebook Messages
Storage Infrastructure Behind Facebook Messages
Storage Infrastructure Behind Facebook Messages
Storage Infrastructure Behind Facebook Messages
Storage Infrastructure Behind Facebook Messages
Storage Infrastructure Behind Facebook Messages
Storage Infrastructure Behind Facebook Messages
Storage Infrastructure Behind Facebook Messages
Storage Infrastructure Behind Facebook Messages
Upcoming SlideShare
Loading in …5
×

Storage Infrastructure Behind Facebook Messages

470 views

Published on

  • Be the first to comment

Storage Infrastructure Behind Facebook Messages

  1. 1. Storage Infrastructure Behind Facebook Messages Using HBase at Scale Jianfeng Zhang
  2. 2. • HBase is an open source, non-relational, distributed database modeled after Google's BigTable and is written in Java. It is developed as part of Apache Software Foundation's Apache Hadoop project and runs on top of HDFS (Hadoop Distributed Filesystem), providing BigTable-like capabilities for Hadoop.
  3. 3. • Facebook is an online social networking service. Its name comes from a colloquialism for the directory given to students at some American universities. • Facebook was founded in February 2004 by Mark Zuckerberg. • Users must register before using the site, after which they may create a personal profile, add other users as friends, exchange messages, and receive automatic notifications when they update their profile. Additionally, users may join common-interest user groups, organized by workplace, school or college, or other characteristics, and categorize their friends into lists such as "People From Work" or "Close Friends".
  4. 4. • As of January 2014, Facebook has about 1.2 billion monthly users. • Based on its 2012 income of US$5 billion, Facebook joined the Fortune 500 list for the first time on the list published in May 2013, being placed at position 462.
  5. 5. Monthly data volume prior to launch 15Billion×1,024byte=14TB 200 Billion×100bytes=11TB
  6. 6. Why picked Hbase for Facebook Messages • • • • • • High write throughput Low latency random reads Elasticity Cheap and fault tolerant Strong consistency within a data center Experience with HDFS
  7. 7. Messaging Data • Small and medium data (Hbase) Message metadata and bodies Snapshot of recent messages Search indices • Attachments and large messages (Haystack)
  8. 8. HBase Architecture • ZooKeeper Metadata • Hmaster Recovery Balacing • Region Server Log>Flush Store>Compaction Region>Split
  9. 9. Facebook Messages: Quick Stats • 6B+ messages/day • • • • • • ▪ Traffic to HBase ▪ 75+ Billion R+W ops/day ▪ At peak: 1.5M ops/sec ▪ ~ 55% Read vs. 45% Write ops ▪ Avg write op inserts ~16 records across multiple column families
  10. 10. Facebook Messages: Quick Stats (contd.) • 2PB+ of online data in HBase (6PB+ with replication; • excludes backups) • ▪ message data, metadata, search index • ▪ All data LZO compressed • ▪ Growing at 250TB/month
  11. 11. Facebook Messages: Quick Stats (contd.) • • • • • • • • • • Timeline: ▪ Started in Dec 2009 ▪ Roll out started in Nov 2010 ▪ Fully rolled out by July 2011 (migrated 1B+ accounts from legacy messages!) While in production: ▪ Schema changes: not once, but twice! ▪ Implemented & rolled out HFile V2 and numerous other optimizations in an upward compatible manner!
  12. 12. Backups (V2) • Now, does periodic HFile level backups. • ▪ Working on: • ▪ Moving to HFile + Commit Log based backups to be able to recover to • finer grained points in time • ▪ Avoid need to log data to Scribe. • ▪ Zero copy (hard link based) fast backups
  13. 13. Messages Schema & Evolution • • • • • • • • • • • “Actions” (data) Column Family the source of truth ▪ Log of all user actions (addMessage, markAsRead, etc.) ▪ Metadata (thread index, message index, search index) etc. in other column families ▪ Metadata portion of schema underwent 3 changes: ▪ Coarse grained snapshots (early development; rollout up to 1M users) ▪ Hybrid (up to full rollout – 1B+ accounts; 800M+ active) ▪ Fine-grained metadata (after rollout) ▪ MapReduce jobs against production clusters! ▪ Ran in throttled way ▪ Heavy use of HBase bulk import features
  14. 14. Write Path Overview
  15. 15. Flushes: Memstore -> HFile
  16. 16. Read Path Overview
  17. 17. Compactions
  18. 18. Reliability: Early work • HDFS sync support for durability of transactions • ▪ Multi-CF transaction atomicity • ▪ Several bug fixes in log recovery • ▪ New block placement policy in HDFS • ▪ To reduce probability of data loss
  19. 19. Availability: Early Work • • • • • • • • • Common reasons for unavailability: ▪ S/W upgrades ▪ Solution: rolling upgrades ▪ Schema Changes ▪ Applications needs new Column Families ▪ Need to change settings for a CF ▪ Solution: online “alter table” ▪ Load balancing or cluster restarts took forever ▪ Upon investigation: stuck waiting for compactions to finish • ▪ Solution: Interruptible Compactions!
  20. 20. Performance: Early Work • Read optimizations: • ▪ Seek optimizations for rows with large number of cells • ▪ Bloom Filters • ▪ minimize HFile lookups • ▪ Timerange hints on HFiles (great for temporal data) • ▪ Multigets • ▪ Improved handling of compressed HFiles
  21. 21. Performance: Compactions • • • • • • • • • Critical for read performance ▪ Old Algorithm: #1. Start from newest file (file 0); include next file if: ▪ size[i] < size[i-1] * C (good!) #2. Always compact at least 4 files, even if rule #1 isn’t met. Solution: #1. Compact at least 4 files, but only if eligible files found. #2. Also, new file selection based on summation of sizes. size[i+ < (size*0+ + size*1+ + …size*i-1]) * C
  22. 22. Performance: Compactions
  23. 23. Performance: Compactions • • • • • • • • • • • More problems! ▪ Read performance dips during peak ▪ Major compaction storms ▪ Large compactions bottleneck ▪ Enhancements/fixes: ▪ Staggered major compactions ▪ Multi-thread compactions; separate queues for small & big compactions ▪ Aggressive off-peak compactions
  24. 24. Metrics, metrics, metrics… • • • • • • • • • • • Initially, only had coarse level overall metrics (get/put latency/ops; block cache counters). ▪ Slow query logging ▪ Added per Column Family stats for: ▪ ops counts, latency ▪ block cache usage & hit ratio ▪ memstore usage ▪ on-disk file sizes ▪ file counts ▪ bytes returned, bytes flushed, compaction statistics ▪ stats by block type (data block vs. index blocks vs. bloom blocks, etc.) • ▪ bloom filter stats
  25. 25. Metrics (contd.) • • • • • • HBase Master Statistics: ▪ Number of region servers alive ▪ Number of regions ▪ Load balancing statistics ▪ .. ▪ All stats stored in Facebook’s Operational Data Store (ODS). • ▪ Lots of ODS dashboards for debugging issues • ▪ Side note: ODS planning to use HBase for storage pretty soon!
  26. 26. Need to keep up as data grows on you! • • • • • • • • • • • • Rapidly iterated on several new features while in production: ▪ Block indexes upto 6GB per server! Cluster starts taking longer and longer. Block cache hit ratio on the decline. ▪ Solution: HFile V2 ▪ Multi-level block index, Sharded Bloom Filters ▪ Network pegged after restarts ▪ Solution: Locality on full & rolling restart ▪ High disk utilization during peak ▪ Solution: Several “seek” optimizations to reduce disk IOPS ▪ Lazy Seeks (use time hints to avoid seeking into older HFiles) ▪ Special bloom filter for deletes to avoid additional seek ▪ Utilize off-peak IOPS to do more aggressive compactions during
  27. 27. Scares & Scars! • Not without our share of scares and incidents: • ▪ s/w bugs. (e.g., deadlocks, incompatible LZO used for bulk imported data, etc.) • ▪ found a edge case bug in log recovery as recently as last week! • ▪ performance spikes every 6 hours (even off-peak)! • ▪ cleanup of HDFS’s Recycle bin was sub-optimal! Needed code and config fix. • ▪ transient rack switch failures • ▪ Zookeeper leader election took than 10 minutes when one member of the • quorum died. Fixed in more recent version of ZK. • ▪ HDFS Namenode – SPOF • ▪ flapping servers (repeated failures)
  28. 28. Scares & Scars! (contd.) • • • • • • • • • • Sometimes, tried things which hadn’t been tested in dark launch! ▪ Added a rack of servers to help with performance issue ▪ Pegged top of the rack network bandwidth! ▪ Had to add the servers at much slower pace. Very manual  . ▪ Intelligent load balancing needed to make this more automated. ▪ A high % of issues caught in shadow/stress testing ▪ Lots of alerting mechanisms in place to detect failures cases ▪ Automate recovery for a lots of common ones ▪ Treat alerts on shadow cluster as hi-pri too! ▪ Sharding service across multiple HBase cells also paid off
  29. 29. Choosing HBase They evaluated and tested various solutions • Strong consistency model • Automatic failover • Multiple shards per sever for loading balancing Prevents cascading failures • Compression-save disk and network bandwidth • Read modify writer operation support like counter increment • Map Reduce supported out of the box
  30. 30. HBase uses HDFS They get the benefits of HDFS as a storage system for free • HDFS has attractive features out of the box Easy to scale out for additional storage Checksums to detect and recover from corruptions Block placement enhanced to minimize data loss probability • HDFS is battle tested inside Facebook • Currently running petabyte scale clusters • Development and operational experience with the Hadoop
  31. 31. Working with HBase and HDFS Goal of zero data loss • • • • HDFS sync support for Write-Ahead-Log Row level ACID property Early log rolling Various critical bug fixes log recovery region assignments • HBase master redesign Zookeeper integration
  32. 32. Stability and performance • Availability and operational improvements Rolling restarts-minimal downtime on upgrades Ability to interrupt long running operations(e.g., compactions) HBase fsck, Metrics • Performance Various improvements to response time, column seeking, bloom filters • Stability Fixed various timeouts and race conditions Constant region count &controlled rolling splits
  33. 33. Operational Challenges • Darklaunch • Deployments and monitoring Lots of internal HBase clusters used for various purposes A ton of scripts/dashboards/graphs for monitoring Automatic recovery • Moving a ton of data around Migration Incremental snapshots
  34. 34. Future work • • • • • • • • • • • Reliability, Availability, Scalability! ▪ Lot of new use cases on top of HBase in the works. ▪ HDFS Namenode HA ▪ Recovering gracefully from transient issues ▪ Fast hot-backups ▪ Delta-encoding in block cache ▪ Replication ▪ Performance (HBase and HDFS) ▪ HBase as a service Multi-tenancy ▪ Features- coprocessors, secondary indices

×