The document discusses Facebook's use of HBase as the database storage engine for its messaging platform. It provides an overview of HBase, including its data model, architecture, and benefits like scalability, fault tolerance, and simpler consistency model compared to relational databases. The document also describes Facebook's contributions to HBase to improve performance, availability, and achieve its goal of zero data loss. It shares Facebook's operational experiences running large HBase clusters and discusses its migration of messaging data from MySQL to a de-normalized schema in HBase.
Hadoop World 2011: Building Realtime Big Data Services at Facebook with Hadoo...Cloudera, Inc.
Facebook has one of the largest Apache Hadoop data warehouses in the world, primarily queried through Apache Hive for offline data processing and analytics. However, the need for realtime analytics and end-user access has led to the development of several new systems built using Apache HBase. This talk will cover specific use cases and the work done at Facebook around building large scale, low latency and high throughput realtime services with Hadoop and HBase. This includes several significant contributions to existing projects as well as the release of new open source projects.
Hadoop Summit 2012 | HBase Consistency and Performance ImprovementsCloudera, Inc.
The latest Apache HBase releases, 0.92 and 0.94, contain many improvements over prior releases in terms of correctness and performance improvements. We discuss a couple of these improvements from a development and operations perspective. For correctness, we discuss the ACID guarantees of HBase, give a case study of problems with earlier releases, and give an overview of the implementation internals that were improved to fix the issues. For performance, we discuss recent improvements in 0.94 and how to monitor the performance of a cluster with new metrics.
Hadoop World 2011: Advanced HBase Schema DesignCloudera, Inc.
While running a simple key/value based solution on HBase usually requires an equally simple schema, it is less trivial to operate a different application that has to insert thousands of records per second.
This talk will address the architectural challenges when designing for either read or write performance imposed by HBase. It will include examples of real world use-cases and how they can be implemented on top of HBase, using schemas that optimize for the given access patterns.
Hadoop World 2011: Building Realtime Big Data Services at Facebook with Hadoo...Cloudera, Inc.
Facebook has one of the largest Apache Hadoop data warehouses in the world, primarily queried through Apache Hive for offline data processing and analytics. However, the need for realtime analytics and end-user access has led to the development of several new systems built using Apache HBase. This talk will cover specific use cases and the work done at Facebook around building large scale, low latency and high throughput realtime services with Hadoop and HBase. This includes several significant contributions to existing projects as well as the release of new open source projects.
Hadoop Summit 2012 | HBase Consistency and Performance ImprovementsCloudera, Inc.
The latest Apache HBase releases, 0.92 and 0.94, contain many improvements over prior releases in terms of correctness and performance improvements. We discuss a couple of these improvements from a development and operations perspective. For correctness, we discuss the ACID guarantees of HBase, give a case study of problems with earlier releases, and give an overview of the implementation internals that were improved to fix the issues. For performance, we discuss recent improvements in 0.94 and how to monitor the performance of a cluster with new metrics.
Hadoop World 2011: Advanced HBase Schema DesignCloudera, Inc.
While running a simple key/value based solution on HBase usually requires an equally simple schema, it is less trivial to operate a different application that has to insert thousands of records per second.
This talk will address the architectural challenges when designing for either read or write performance imposed by HBase. It will include examples of real world use-cases and how they can be implemented on top of HBase, using schemas that optimize for the given access patterns.
Hadoop World 2011: Advanced HBase Schema Design - Lars George, ClouderaCloudera, Inc.
"While running a simple key/value based solution on HBase usually requires an equally simple schema, it is less trivial to operate a different application that has to insert thousands of records per second.
This talk will address the architectural challenges when designing for either read or write performance imposed by HBase. It will include examples of real world use-cases and how they can be implemented on top of HBase, using schemas that optimize for the given access patterns. "
With the public confession of Facebook, HBase is on everyone's lips when it comes to the discussion around the new "NoSQL" area of databases. In this talk, Lars will introduce and present a comprehensive overview of HBase. This includes the history of HBase, the underlying architecture, available interfaces, and integration with Hadoop.
HBase Advanced Schema Design - Berlin Buzzwords - June 2012larsgeorge
While running a simple key/value based solution on HBase usually requires an equally simple schema, it is less trivial to operate a different application that has to insert thousands of records per second. This talk will address the architectural challenges when designing for either read or write performance imposed by HBase. It will include examples of real world use-cases and how they
http://berlinbuzzwords.de/sessions/advanced-hbase-schema-design
Hadoop World 2011: Apache HBase Road Map - Jonathan Gray - FacebookCloudera, Inc.
This technical session will provide a quick review of the Apache HBase project, looking at it from the past to the future. It will cover the imminent HBase 0.92 release as well as what is slated for 0.94 and beyond. A number of companies and use cases will be used as examples to describe the overall direction of the HBase community and project.
HBaseCon 2012 | Gap Inc Direct: Serving Apparel Catalog from HBase for Live W...Cloudera, Inc.
Gap Inc Direct, the online division for Gap Inc., uses HBase to serve, in real-time, apparel catalog for all its brands’ and markets’ web sites. This case study will review the business case as well as key decisions regarding schema selection and cluster configurations. We will also discuss implementation challenges and insights that were learned.
From: DataWorks Summit 2017 - Munich - 20170406
HBase hast established itself as the backend for many operational and interactive use-cases, powering well-known services that support millions of users and thousands of concurrent requests. In terms of features HBase has come a long way, overing advanced options such as multi-level caching on- and off-heap, pluggable request handling, fast recovery options such as region replicas, table snapshots for data governance, tuneable write-ahead logging and so on. This talk is based on the research for the an upcoming second release of the speakers HBase book, correlated with the practical experience in medium to large HBase projects around the world. You will learn how to plan for HBase, starting with the selection of the matching use-cases, to determining the number of servers needed, leading into performance tuning options. There is no reason to be afraid of using HBase, but knowing its basic premises and technical choices will make using it much more successful. You will also learn about many of the new features of HBase up to version 1.3, and where they are applicable.
Introduction to HBase. HBase is a NoSQL databases which experienced a tremendous increase in popularity during the last years. Large companies like Facebook, LinkedIn, Foursquare are using HBase. In this presentation we will address questions like: what is HBase?, and compared to relational databases?, what is the architecture?, how does HBase work?, what about the schema design?, what about the IT ressources?. Questions that should help you consider whether this solution might be suitable in your case.
Hadoop World 2011: Advanced HBase Schema Design - Lars George, ClouderaCloudera, Inc.
"While running a simple key/value based solution on HBase usually requires an equally simple schema, it is less trivial to operate a different application that has to insert thousands of records per second.
This talk will address the architectural challenges when designing for either read or write performance imposed by HBase. It will include examples of real world use-cases and how they can be implemented on top of HBase, using schemas that optimize for the given access patterns. "
With the public confession of Facebook, HBase is on everyone's lips when it comes to the discussion around the new "NoSQL" area of databases. In this talk, Lars will introduce and present a comprehensive overview of HBase. This includes the history of HBase, the underlying architecture, available interfaces, and integration with Hadoop.
HBase Advanced Schema Design - Berlin Buzzwords - June 2012larsgeorge
While running a simple key/value based solution on HBase usually requires an equally simple schema, it is less trivial to operate a different application that has to insert thousands of records per second. This talk will address the architectural challenges when designing for either read or write performance imposed by HBase. It will include examples of real world use-cases and how they
http://berlinbuzzwords.de/sessions/advanced-hbase-schema-design
Hadoop World 2011: Apache HBase Road Map - Jonathan Gray - FacebookCloudera, Inc.
This technical session will provide a quick review of the Apache HBase project, looking at it from the past to the future. It will cover the imminent HBase 0.92 release as well as what is slated for 0.94 and beyond. A number of companies and use cases will be used as examples to describe the overall direction of the HBase community and project.
HBaseCon 2012 | Gap Inc Direct: Serving Apparel Catalog from HBase for Live W...Cloudera, Inc.
Gap Inc Direct, the online division for Gap Inc., uses HBase to serve, in real-time, apparel catalog for all its brands’ and markets’ web sites. This case study will review the business case as well as key decisions regarding schema selection and cluster configurations. We will also discuss implementation challenges and insights that were learned.
From: DataWorks Summit 2017 - Munich - 20170406
HBase hast established itself as the backend for many operational and interactive use-cases, powering well-known services that support millions of users and thousands of concurrent requests. In terms of features HBase has come a long way, overing advanced options such as multi-level caching on- and off-heap, pluggable request handling, fast recovery options such as region replicas, table snapshots for data governance, tuneable write-ahead logging and so on. This talk is based on the research for the an upcoming second release of the speakers HBase book, correlated with the practical experience in medium to large HBase projects around the world. You will learn how to plan for HBase, starting with the selection of the matching use-cases, to determining the number of servers needed, leading into performance tuning options. There is no reason to be afraid of using HBase, but knowing its basic premises and technical choices will make using it much more successful. You will also learn about many of the new features of HBase up to version 1.3, and where they are applicable.
Introduction to HBase. HBase is a NoSQL databases which experienced a tremendous increase in popularity during the last years. Large companies like Facebook, LinkedIn, Foursquare are using HBase. In this presentation we will address questions like: what is HBase?, and compared to relational databases?, what is the architecture?, how does HBase work?, what about the schema design?, what about the IT ressources?. Questions that should help you consider whether this solution might be suitable in your case.
Big Data and Hadoop - History, Technical Deep Dive, and Industry TrendsEsther Kundin
An overview of the history of Big Data, followed by a deep dive into the Hadoop ecosystem. Detailed explanation of how HDFS, MapReduce, and HBase work, followed by a discussion of how to tune HBase performance. Finally, a look at industry trends, including challenges faced and being solved by Bloomberg for using Hadoop for financial data.
Big Data and Hadoop - History, Technical Deep Dive, and Industry TrendsEsther Kundin
An overview of the history of Big Data, followed by a deep dive into the Hadoop ecosystem. Detailed explanation of how HDFS, MapReduce, and HBase work, followed by a discussion of how to tune HBase performance. Finally, a look at industry trends, including challenges faced and being solved by Bloomberg for using Hadoop for financial data.
Apache HBase™ is the Hadoop database, a distributed, salable, big data store.Its a column-oriented database management system that runs on top of HDFS.
Apache HBase is an open source NoSQL database that provides real-time read/write access to those large data sets. ... HBase is natively integrated with Hadoop and works seamlessly alongside other data access engines through YARN.
At StampedeCon 2012 in St. Louis, Pritam Damania presents: Reliable backup and recovery is one of the main requirements for any enterprise grade application. HBase has been very well embraced by enterprises needing random, real-time read/write access with huge volumes of data and ease of scalability. As such, they are looking for backup solutions that are reliable, easy to use, and can co-exist with existing infrastructure. HBase comes with several backup options but there is a clear need to improve the native export mechanisms. This talk will cover various options that are available out of the box, their drawbacks and what various companies are doing to make backup and recovery efficient. In particular it will cover what Facebook has done to improve performance of backup and recovery process with minimal impact to production cluster.
Data Storage and Management project ReportTushar Dalvi
This paper aims at evaluating the performance of random reads and random writes the information of HBase and Cassandra and compare the results that we got through various ubuntu operation
Speaker: Varun Sharma (Pinterest)
Over the past year, HBase has become an integral component of Pinterest's storage stack. HBase has enabled us to quickly launch and iterate on new products and create amazing pinner experiences. This talk briefly describes some of these applications, the underlying schema, and how our HBase setup stays highly available and performant despite billions of requests every week. It will also include some performance tips for running on SSDs. Finally, we will talk about a homegrown serving technology we built from a mashup of HBase components that has gained wide adoption across Pinterest.
Chicago Data Summit: Apache HBase: An IntroductionCloudera, Inc.
Apache HBase is an open source distributed data-store capable of managing billions of rows of semi-structured data across large clusters of commodity hardware. HBase provides real-time random read-write access as well as integration with Hadoop MapReduce, Hive, and Pig for batch analysis. In this talk, Todd will provide an introduction to the capabilities and characteristics of HBase, comparing and contrasting it with traditional database systems. He will also introduce its architecture and data model, and present some example use cases.
6. Monthly data volume prior to launch
15B x 1,024 bytes = 14TB
120B x 100 bytes = 11TB
7. Messaging Data
▪ Small/medium sized data HBase
▪ Message metadata & indices
▪ Search index
▪ Small message bodies
▪ Attachments and large messages Haystack
▪ Used for our existing photo/video store
8. Open Source Stack
▪ Memcached --> App Server Cache
▪ ZooKeeper --> Small Data Coordination Service
▪ HBase --> Database Storage Engine
▪ HDFS --> Distributed FileSystem
▪ Hadoop --> Asynchronous Map-Reduce Jobs
9. Our architecture
User Directory Service
Clients
(Front End, MTA, etc.)
What’s the cell for
this user?
Cell 2
Cell 1 Cell 3
Cell 1 Application Server
Application Server Application Server
Attachments
HBase/HDFS/Z
HBase/HDFS/Z KMessage, Metadata,
HBase/HDFS/Z
K Search Index
K
Haystack
11. HBase in a nutshell
• distributed, large-scale data store
• efficient at random reads/writes
• initially modeled after Google’s BigTable
• open source project (Apache)
12. When to use HBase?
▪ storing large amounts of data
▪ need high write throughput
▪ need efficient random access within large data sets
▪ need to scale gracefully with data
▪ for structured and semi-structured data
▪ don’t need full RDMS capabilities (cross table transactions, joins, etc.)
13. HBase Data Model
• An HBase table is:
• a sparse , three-dimensional array of cells, indexed by:
RowKey, ColumnKey, Timestamp/Version
• sharded into regions along an ordered RowKey space
• Within each region:
• Data is grouped into column families
▪ Sort order within each column family:
Row Key (asc), Column Key (asc), Timestamp (desc)
14. Example: Inbox Search
• Schema
• Key: RowKey: userid, Column: word, Version: MessageID
• Value: Auxillary info (like offset of word in message)
• Data is stored sorted by <userid, word, messageID>:
User1:hi:17->offset1
Can efficiently handle queries like:
User1:hi:16->offset2
User1:hello:16->offset3 - Get top N messageIDs for a
User1:hello:2->offset4 specific user & word
...
User2:.... - Typeahead query: for a given user,
User2:... get words that match a prefix
...
15. HBase System Overview
Database Layer
HBASE
Master Backup
Master
Region Region Region ...
Server Server Server
Coordination Service
Storage Layer
HDFS Zookeeper Quorum
Namenode Secondary Namenode ZK ZK ...
Peer Peer
Datanode Datanode Datanode ...
16. HBase Overview
HBASE Region Server
....
Region #2
Region #1
....
ColumnFamily #2
ColumnFamily #1 Memstore
(in memory data structure)
HFiles (in HDFS) flush
Write Ahead Log ( in HDFS)
17. HBase Overview
• Very good at random reads/writes
• Write path
• Sequential write/sync to commit log
• update memstore
• Read path
• Lookup memstore & persistent HFiles
• HFile data is sorted and has a block index for efficient retrieval
• Background chores
• Flushes (memstore -> HFile)
• Compactions (group of HFiles merged into one)
19. Horizontal scalability
▪ HBase & HDFS are elastic by design
▪ Multiple table shards (regions) per physical server
▪ On node additions
▪ Load balancer automatically reassigns shards from overloaded
nodes to new nodes
▪ Because filesystem underneath is itself distributed, data for
reassigned regions is instantly servable from the new nodes.
▪ Regions can be dynamically split into smaller regions.
▪ Pre-sharding is not necessary
▪ Splits are near instantaneous!
20. Automatic Failover
▪ Node failures automatically detected by HBase Master
▪ Regions on failed node are distributed evenly among surviving nodes.
▪ Multiple regions/server model avoids need for substantial
overprovisioning
▪ HBase Master failover
▪ 1 active, rest standby
▪ When active master fails, a standby automatically takes over
21. HBase uses HDFS
We get the benefits of HDFS as a storage system for free
▪ Fault tolerance (block level replication for redundancy)
▪ Scalability
▪ End-to-end checksums to detect and recover from corruptions
▪ Map Reduce for large scale data processing
▪ HDFS already battle tested inside Facebook
▪ running petabyte scale clusters
▪ lot of in-house development and operational experience
22. Simpler Consistency Model
▪ HBase’s strong consistency model
▪ simpler for a wide variety of applications to deal with
▪ client gets same answer no matter which replica data is read from
▪ Eventual consistency: tricky for applications fronted by a cache
▪ replicas may heal eventually during failures
▪ but stale data could remain stuck in cache
23. Typical Cluster Layout
▪ Multiple clusters/cells for messaging
▪ 20 servers/rack; 5 or more racks per cluster
▪ Controllers (master/Zookeeper) spread across racks
ZooKeeper Peer ZooKeeper Peer ZooKeeper Peer ZooKeeper Peer ZooKeeper Peer
HDFS Namenode Backup Namenode Job Tracker Hbase Master Backup Master
Region Server Region Server Region Server Region Server Region Server
Data Node Data Node Data Node Data Node Data Node
Task Tracker Task Tracker Task Tracker Task Tracker Task Tracker
19x... 19x... 19x... 19x... 19x...
Region Server Region Server Region Server Region Server Region Server
Data Node Data Node Data Node Data Node Data Node
Task Tracker Task Tracker Task Tracker Task Tracker Task Tracker
Rack #1 Rack #2 Rack #3 Rack #4 Rack #5
26. Goal of Zero Data Loss/Correctness
▪ sync support added to hadoop-20 branch
▪ for keeping transaction log (WAL) in HDFS
▪ to guarantee durability of transactions
▪ Row-level ACID compliance
▪ Enhanced HDFS’s Block Placement Policy:
▪ Original: rack aware, but minimally constrained
▪ Now: Placement of replicas constrained to configurable node groups
▪ Result: Data loss probability reduced by orders of magnitude
27. Availability/Stability improvements
▪ HBase master rewrite- region assignments using ZK
▪ Rolling Restarts – doing software upgrades without a downtime
▪ Interrupt Compactions – prioritize availability over minor perf gains
▪ Timeouts on client-server RPCs
▪ Staggered major compaction to avoid compaction storms
28. Performance Improvements
▪ Compactions
▪ critical for read performance
▪ Improved compaction algo
▪ delete/TTL/overwrite processing in minor compactions
▪ Read optimizations:
▪ Seek optimizations for rows with large number of cells
▪ Bloom filters to minimize HFile lookups
▪ Timerange hints on HFiles (great for temporal data)
▪ Improved handling of compressed HFiles
29. Operational Experiences
▪ Darklaunch:
▪ shadow traffic on test clusters for continuous, at scale testing
▪ experiment/tweak knobs
▪ simulate failures, test rolling upgrades
▪ Constant (pre-sharding) region count & controlled rolling splits
▪ Administrative tools and monitoring
▪ Alerts (HBCK, memory alerts, perf alerts, health alerts)
▪ auto detecting/decommissioning misbehaving machines
▪ Dashboards
▪ Application level backup/recovery pipeline
30. Working within the Apache community
▪ Growing with the community
▪ Started with a stable, healthy project
▪ In house expertise in both HDFS and HBase
▪ Increasing community involvement
▪ Undertook massive feature improvements with community help
▪ HDFS 0.20-append branch
▪ HBase Master rewrite
▪ Continually interacting with the community to identify and fix issues
▪ e.g., large responses (2GB RPC)
33. Move messaging data from MySQL to HBase
▪ In MySQL, inbox data was kept normalized
▪ user’s messages are stored across many different machines
▪ Migrating a user is basically one big join across tables spread over
many different machines
▪ Multiple terabytes of data (for over 500M users)
▪ Cannot pound 1000s of production UDBs to migrate users
34. How we migrated
▪ Periodically, get a full export of all the users’ inbox data in MySQL
▪ And, use bulk loader to import the above into a migration HBase
cluster
▪ To migrate users:
▪ Since users may continue to receive messages during migration:
▪ double-write (to old and new system) during the migration period
▪ Get a list of all recent messages (since last MySQL export) for the
user
▪ Load new messages into the migration HBase cluster
▪ Perform the join operations to generate the new data
▪ Export it and upload into the final cluster
36. Facebook Insights Goes Real-Time
▪ Recently launched real-time analytics for social plugins on top of
HBase
▪ Publishers get real-time distribution/engagement metrics:
▪ # of impressions, likes
▪ analytics by
▪ Domain, URL, demographics
▪ Over various time periods (the last hour, day, all-time)
▪ Makes use of HBase capabilities like:
▪ Efficient counters (read-modify-write increment operations)
▪ TTL for purging old data
37. Future Work
It is still early days…!
▪ Namenode HA (AvatarNode)
▪ Fast hot-backups (Export/Import)
▪ Online schema & config changes
▪ Running HBase as a service (multi-tenancy)
▪ Features (like secondary indices, batching hybrid mutations)
▪ Cross-DC replication
▪ Lot more performance/availability improvements