The document discusses Simply Measured's migration from MongoDB to HBase for storing social media analytics data. Simply Measured initially used MongoDB but encountered scaling issues as their data grew beyond 10TB. They migrated to HBase using a dual write approach to ensure accuracy. HBase provided better performance and scaling capabilities. The migration process and lessons learned are described, including schema design in HBase and ongoing improvements to the system.

1. Rebuilding from MongoDB for Scale on HBase
Robert Roland (@robdaemon)
Lead Software Engineer
rob@simplymeasured.com
http://www.simplymeasured.com

2. © 2013 Simply Measured, Inc
Who Are We
Social Media Analytics
Serving 25% of the Interbrand Top 100 Global Brands
Collecting data from Twitter, Facebook, Instagram, YouTube,
Google Analytics and more
Delivered in a marketer’s favorite format – Excel!
GeekWire’s 2013 Startup of the Year
2

3. © 2013 Simply Measured, Inc
Twitter Account Report, jetBlue Airlines
3

4. © 2013 Simply Measured, Inc
Complete Social Media Snapshot, Red Bull
4

5. © 2013 Simply Measured, Inc
Our Setup
Cloudera CDH 4.2.1 on Ubuntu 12.04
3 “control” nodes (HDFS name node, job tracker, HBase master)
11 “data” nodes (HDFS data nodes, task trackers, HBase region servers)
Bare-metal, using managed colo hosting
14 TB of data across 50 HBase tables (One table per data source -Twitter,
Facebook, plus secondary indexes and operations)
Our customers have tracked 1.5 billion Tweets so far, and growing
More than 5 million rows across 3,000 reports generated daily
5

6. © 2013 Simply Measured, Inc
Why did we start with MongoDB?
• Ease of development
• Lack of schema is a good and a bad thing
• Easy to use libraries in Ruby (mongomatic)
• Lower initial investment
• You can run MongoDB on one server in production (but you shouldn’t)
• Master/slave was easier to start with than the current sharding model
• Active, engaged community
• Really, really effective marketing masks MongoDB's
shortcomings…
6

7. © 2013 Simply Measured, Inc
Why leave MongoDB?
• 10 TB of data was too much for it
• Instability
• No one wants to restart mongos every two days
• Bugs
• Very public, very high profile failures
• Silent failure modes
• What do you mean, the index creation failed and you just attempted
to load an entire collection into RAM?
7

8. © 2013 Simply Measured, Inc
Why HBase, or, How is this better than Mongo?
• More linear scaling
• Add new nodes, run a major compaction
• Our data can easily be modeled as a sparse column store
• We value consistency
• Our users will tell us if we’re missing one Tweet out of 1 million
• Monitoring
• Metrics
• Stability
• Excellent vendor support
8

9. © 2013 Simply Measured, Inc
How did we migrate?
• Implemented the Strangler pattern
• Dual writes to Mongo and HBase
• Migrate older data a few customers at a time, a few data sources at a
time
• Dual report generation platform – enabled us to compare reports off
our MongoDB platform and our HBase platform
• Migrated existing Ruby 1.8 code to JRuby
• Direct access to the HBase cluster
• I’m a better Java developer than Ruby developer
• Great profiler tools
9

10. © 2013 Simply Measured, Inc
Challenges with HBase
• Out of the box configuration is not good enough
• GC Tuning
• Default file size of 256 mb is too small
• Compactions will eat you alive
• Be sure to enable the HDFS trashcan! (fs.trash.interval)
• Configurations can be difficult to manage
• Chef / Puppet if you want to roll your own
• Cloudera Manager
• Schema
• Lack of types means the HBase shell is harder to read
• No native secondary indexes
10

11. © 2013 Simply Measured, Inc
What’s been awesome
• Great user community on the mailing lists
• Source code is easy to follow and submit patches
• Stable, stable, stable
• Unless you configure something wrong, like ulimits or xcievers!
11

12. © 2013 Simply Measured, Inc
Google Analytics schema
• Row key
• salt|dataSourceId|dimension|…|date
• Columns
• CF: metadata
• dataSourceId – hash, identifies a data source mapping to a customer
• timezone – Google Analytics time zone for this data
• dimensions – delimited list of Google Analytics dimensions
• CF: data
• GA data key – Name of metric as defined by Google Analytics, Protocol
Buffer representing value
12

13. © 2013 Simply Measured, Inc
Schema evolution
• Enrichment via Klout, geolocation data, Bit.ly, etc. was stored
in a separate table, and “joined” at query time
• Enrichment is now a column family within each data source
• Started with Protocol Buffers, now writing as qualifiers and
columns
• Ability to use server-side filters, ease of use within HBase shell
• Protocol Buffers still exist in some cases (example coming up)
• Rekeyed several tables over time
• Dual write during migration
• Map/reduce jobs to migrate data
• More column families per table than were necessary
• Lots and lots of memstore pain.
13

14. © 2013 Simply Measured, Inc
Protocol Buffers
• Union-like Protocol Buffer for arbitrary key/value pairs
14

15. © 2013 Simply Measured, Inc
Tips and Tricks
• Don’t expect to get your row keys correct on the first try
• Look for hot spotting
• Does ordering matter during queries?
• Consider your backup strategy
• S3 seems to work for us
• Replication is also an option
• This is not an RDBMS. Don’t JOIN, denormalize!
15

16. © 2013 Simply Measured, Inc
What’s coming up
• Hive
• Easier querying
• Entirely standardized representation of our data
• HCatalog
• Expose our schema to other internal tooling
• A much larger cluster
• Even more data sources
16

17. Thank You
Robert Roland
@robdaemon
rob@simplymeasured.com
We’re hiring!
http://simplymeasured.com/about/careers/
Our Tech Blog: http://engineering.simplymeasured.com/
Our Open Source: http://simplymeasured.github.io/
More sample reports: http://simplymeasured.com/tour/sample-reports/