This document outlines a study on migrating relational database content to NoSQL storage systems like HBase. It discusses challenges in migration and the need for design patterns for HBase schemas. A 3-dimensional data model in HBase is proposed and evaluated using cosmology and bike rental datasets. Experiment results show the 3D model improves performance for queries that use HBase's version dimension. Future work includes further evaluation of the model's scalability and designing models for other dataset types.

1. Dan Han, Eleni Stroulia
University of Alberta
9/20/2012
MESOCA 2012 1

2. Outline
» Background and Motivation
» Related Work
» A 3-Dimensional Data Model in HBase
» Case Study and Experiment Results
» Discussion
» Conclusions and Future Work
9/20/2012
MESOCA 2012 2

3. Migrating Applications
To the Cloud
» Cloud is an attractive computing platform
˃ Elasticity, Excellent Scalability, High Availability, Low Operating
Cost
» Applications are moving to the cloud
˃ Social networking, online shopping, monitoring system
˃ Time-Series data: grows monotonously over time
˃ Analysis of large scale time-series data
+ May lead to new knowledge
+ May lead to improvements of existing services
» Success adoption of this movement paradigm requires a
9/20/2012
new model of storage
MESOCA 2012 3

4. Migrating RDBMS Content
To NoSQL
» From RDBMS to NoSQL storage systems
˃ Enable the storage of big data, in order of row key
˃ Scale horizontally across storage nodes easily
˃ Not much data-organization support
» Migration challenges
˃ Few experiences and principles to follow
˃ Steep learning curve for programming
˃ Much experimentation is required before deployment
+ Much time is spent in designing the data schema
+ The “wrong” schema may lead to inefficient, high-latency queries
9/20/2012
MESOCA 2012 4

5. We need Design Patterns for
HBase Schemas
» Our objective is to develop a systematic method for
˃ Guiding data organization in NoSQL databases, given
˃ the types of data stored
˃ the amount of data
˃ The data-usage patterns
» We start our investigation with HBase
˃ A NoSQL database offering, built on top of Hadoop
˃ Parallel Distributed Computation
+ MapReduce Framework
+ Coprocessor Framework
9/20/2012
MESOCA 2012 5

6. Related Work
» Talks in HBaseCon2012, held in May
˃ Data schema and Coprocessor are two main topics
˃ Experience from 30 enterprises, i.e., Facebook, Yapmap, eBay, Adobe
» Organizing time-series data in period-specific “buckets”
˃ OpenTSDB: a distributed scalable time-series database, on top of
HBase
˃ A data Model in Cassandra, another NoSQL database offering
˃ Applied in our case study
9/20/2012
MESOCA 2012 6

7. Data Organization in HBase
» Cell in HBase
˃(Row, Family: Column, Version) => (X,Y,Z) = value
Y Z
Y
X VS X
Schema/ Row Family: Column Version
dimension
2-D unique id - varying properties current
timestamp timestamp
9/20/2012
3-D unique id varying properties timestamps
MESOCA 2012 7

8. Case study:
The Datasets
» Cosmology Dataset
˃ Product of an N-body simulation
˃ Three types of particles: dark matter, gas and star
˃ Particles evolve over a series of discrete timestamps
˃ Each snapshot records the properties of all particles at
the time of the snapshot
˃ 9 snapshots, consists of 321,065,547 particles
» Bixi Dataset
˃ Data from a bicycle-renting service in the city of
Montreal
˃ Every minute, the statistic information about bike usage
a station is collected by the sensor
9/20/2012
˃ 100,800 timestamps, consists of 404 stations
MESOCA 2012 8

9. Three Schemas
for the Cosmology Dataset
Schema/ Row Family: Version
dimension Column
Schema1 sid-type-pid particle No meaning
Z
properties
Y
Schema2 type-pid particle Snapshot id
X
properties
Schema3 type-reversedpid particle Snapshot id
properties
Schema1 Schema2 Schema3
Region 24-2-33446666 2-33446666 2-00005533
9/20/2012
Region 64-2-33559999 2-33550000 2-66664433
Region 84-2-33550000 2-33559999 2-99995533
MESOCA 2012 9

10. The cosmology dataset
» Dataset called“cosmo50”
˃ 9 snapshots
S-ID Star Particles Total particles
24 1,291 33,555,723
29 5,568 33,559,998
36 20,246 33,574,630
45 67,268 33,620,890
60 259,219 33,800,108
84 907,025 34,369,014
128 2,743,966 35,908,164
9/20/2012
216 6,396,955 38,889,220
512 12,417,544 43,787,800
MESOCA 2012 10

11. Three Schemas
for the Bixi Dataset
Schema/ Row Family: Column Version
dimension
Schema1 hour-sid minutes[0,59] no meaning
Schema2 hour-sid monitoring metrics minutes [0,59]
Schema3 day-sid monitoring metrics minutes [0,1439]
Schema1 Schema2 Schema3
Time
Time metrics Time
X
X metrics
9/20/2012
X
MESOCA 2012 11

12. The Bixi dataset
» A period of 70 days, from Sep 24, 2010 to Dec 1, 2010,
» 100,800 timestamps
» 404 stations involved
» Stored in XML file
9/20/2012
MESOCA 2012 12

13. Experiment Results
» Experiment Environment
˃ A four-node cluster on virtual machines with Ubuntu
˃ Hadoop 0.20, HBase 0.93-snapshot (Coprocessor support)
˃ HBase Configuration
+ The replication factor of 2
+ 5KB Caching Size
» Queries for each dataset
˃ Three queries of Cosmology dataset from related research
˃ One query of Bixi dataset from business requirement
» Query processing Implementation
9/20/2012
˃ Native java API
˃ User-Level Coprocessor Implementation
MESOCA 2012 13

14. Query1 of Cosmology Dataset
» Get all the particles of a type: star
» in a single snapshot
» with a given property: tform
» whose property matches the expression
˃ [>0.01;84]
˃ [>0.08;128]
˃ [>0.05;128]
˃ [>0.08;216]
˃ [>0.08;512]
9/20/2012
MESOCA 2012 14

15. Query2 of Cosmology Dataset
» Get all the particles added/destroyed
» between s1 and s2
˃ [29;24]
˃ [60;24]
˃ [84;24]
˃ [128;24]
˃ [216;128]
˃ [216;24]
˃ [512;24]
˃ [512;128]
˃ [512;216]
9/20/2012
MESOCA 2012 16

16. Query3 of Cosmology Dataset
» Get the values of a property
» for a set of particle IDs
» across the selected snapshots
˃10;[24]
˃10; [24,512],
˃10;[24,60,128,512]
˃10;[24,29,60,84,128,512]
˃10;[24,36,45,60,84,128,216,512]
˃50;[24,29,84,512]
˃50;[24,29,36,45,60,84,128,216,512]
˃100;[24,29,36,45,60,84,128,216,512]
9/20/2012
˃150;[24,29,36,45,60,84,128,216,512
MESOCA 2012 18

17. Query3 of Cosmology Dataset
» Get the values of a property: star:eps
» for a set of particle IDs: a continuous range particle IDs
» across the selected snapshots
˃ 10;[24]
˃ 10; [24,512],
˃ 10;[24,60,128,512]
˃ 10;[24,29,60,84,128,512]
˃ 10;[24,36,45,60,84,128,216,512]
˃ 50;[24,29,84,512]
˃ 50;[24,29,36,45,60,84,128,216,512]
˃ 100;[24,29,36,45,60,84,128,216,512]
9/20/2012
˃ 150;[24,29,36,45,60,84,128,216,512]
MESOCA 2012 19

18. Bixi Query
» For a given list of stations: 200 stations
» get average bike usage in a given period
˃ [1day]
˃ [2day]
˃ [4day]
˃ [8day]
˃ [16day]
9/20/2012
MESOCA 2012 21

19. Discussion
» “Qualitative” versus “Quantitative” Suggestions
» Dynamic Data versus Static Data
» Historical Dataset versus Real-Time Datasets
» Supported versus Non-Supported Datasets
9/20/2012
MESOCA 2012 23

20. Conclusion
» The objective is to make queries local
» To do that, you have to design the right key, so that all
queries traverse a range of keys
˃With all answers in them
˃With not much irrelevant data in it
» But, hotspotting occurs when
˃???
9/20/2012
MESOCA 2012 24

21. Conclusion
» A 3-dimensional data model
˃Improved performance can be got from the data schema
that use the version dimension of HBase
» Fit in “write-once, read-many” system
˃Monitoring system
˃Sensor-based system
˃Version-based analysis
9/20/2012
MESOCA 2012 25

22. Future Work
» More Evaluation of this data model
˃Scalability
˃Elasticity
˃Utilization
» How to design data model for other datasets
˃Spatial dataset
˃Graphic dataset
9/20/2012
MESOCA 2012 26

23. Questions?
Thank you
9/20/2012
MESOCA 2012 27