Timeseries data

Big (Time Series) Data Jonathan Halliday Core Developer, JBoss [email_address]

Big Time Series Data
Concepts
Big Data (and a little NoSQL)
Time Series: sampling, counting, analysis
Code
Existing solutions and why they suck
Sampling: OpenTSDB and HBase
Counting: rainbird and Cassandra

Big Data

Big Data
Data sets whose size is beyond the ability of commonly used software tools to capture, manage, and process the data within a tolerable elapsed time
Techniques and technologies that make handling data at extreme scale economical

Big Data
Volume : Growth in traditional transaction data and new data types causes storage and analysis issues
Variety : transaction data is now joined by social, mobile and other new data. Different formats and contexts
Velocity : speed of data production and analysis
“Pattern-Based Strategy: Getting Value from Big Data” Gartner (June 2011)

Big Data
Volume – exceeds physical limits of vertical systems
Variety – many different formats makes integration expensive
Velocity – decision window small compared to data change rate
Variability – many options or variable interpretations confound analysis
“Expand Your Digital Horizon With Big Data”, Forrester (Sept 2011)

Big Data
Systems of Record
ERP (things); CRM/HRM (people), finance/accounting/payroll (money)
Transaction and data volume low to moderate as tx involve real world activities e.g. Shipping an order.
Highly structured data with carefully constrained state transitions

Big Data
Systems of Engagement
Loosely structured, highly dynamic data
Social data: conversation and reputation
More complex analysis on larger volumes of data, in less time
Not just for internet scale corps (facebook, twitter) and startups. BUT... traditional enterprise users need software maturity and ease of use.

Time Series

Time Series
A time series is a sequence of data points, measured typically at successive time instants spaced at uniform time intervals (sampling)
Frequency of event occurrences over time periods (counting)
Time ordered set of related events (log analysis)

Time Series Examples
Sampling: system administration metrics; environmental monitoring or process control
Counting: sales, page views, social interactions: likes/dislikes/retweets
Log Analysis: to retrospectively extract information you didn't originally anticipate needing

Time Series Data at scale
Volume
Sample lots of metrics, with increased frequency
Preserve full detail of data, forever
Velocity
Data is write once, but has a high, constant ingest rate
Analysis must not lag

Once Upon a Time...
Relational Database
Flexible general purpose tool
Store samples, increment counters, run analysis
Unnecessary overhead
ACID regardless of if you need it or not.
Easy to use, hard (and expensive) to scale

Relational Databases
Data oriented, not query oriented.
Up front data normalization allows later ad-hoc query, but at a cost
Random I/O does not suit HDDs.
Joins are CPU intensive.
Deeply embedded in the education system
If all you know is how to use a hammer...

NoSQL
A variety of non-relational alternatives
Various datamodels: Key/Value; Graph; Document; Wide Column / Column Family
Variety of storage
In-memory; single node; distributed.
Trade off some properties for better scaling and availability
BASE vs. ACID

Query oriented storage
Identify use cases, layout data to match
sequential reads are good
Denormalize as needed to avoid joins
or join in the application code
Distribute over multiple nodes
Replicate nodes for availability
Relax ACID for performance

NoSQL
Many alternatives of varying maturity
Mostly open source with commercial support
Focus on ease of use, ready to run solutions
cheap software, expensive people
Increasing ecosystem of higher level pieces for specific tasks

Big Time Series Data Part 1: Sampling

Traditional SQL
Sampling example: RHQ
JBoss distributed monitoring and management
Agents send metric samples to a server, which stores them in a SQL database
Sample is {timestamp, metric_id, metric_value}
10s to 100s of samples per machine, 10s to 1000s of machines, ~1min frequency.

SQL Data Model
CREATE TABLE samples (
time_stamp BIGINT NOT NULL, -- 8 bytes
metric_id INTEGER NOT NULL, -- 4 bytes
value DOUBLE NOT NULL, -- 8 bytes
);
CREATE INDEX ON samples(metric_id, timestamp) USING btree;
CREATE INDEX ON samples(time_stamp) USING btree;

SQL Data Model
High transaction load
One tx per host metric batch, containing one row write per metric sample
Index update contention
Individual metrics don't update concurrently, but timestamps do
Bottleneck on database write speed
Classic scaling problem

SQL Data Model
Disk is cheap, but not free
20 bytes nominal data size
24 bytes per row header (pgsql)
Actually 58 bytes per sample (8k block)
Then add 76 bytes per row for indexing
Total 134 bytes/sample
6.7x increase in data footprint

Solutions
Reduce log writes
wal_writer_delay
Trade-off durability for throughput
Reduce index updates
Pack multiple samples per index entry
Also reduces index size
Makes custom queries more expensive

SQL Data Model #2
CREATE TABLE samples (
time_stamp BIGINT NOT NULL,
metric_name INTEGER NOT NULL,
value BYTEA
);
CREATE INDEX ON samples(metric_id, timestamp) USING btree;
CREATE INDEX ON samples(time_stamp) USING btree;

SQL Data Model #2
Row is now N samples wide, where N is ~15-60 minutes, 900-3600 samples.
Index structure updates happen every N samples rather than every 1.
Data row overhead is smaller, indexes are smaller
12 bytes per data point – 0.6x raw size

Solutions
Sooner or later the db caps out
CPU, disk sync
So then you shard the samples table
On metric id, NOT timestamp
Operations overhead
Rebalancing, availability (replication)
Sooner or later the DBA caps out

You're not in Kansas any more
RDBMS were designed for single system use
Distribution (sharding, replication) bolted on with varying elegance and success
You can scale a RDBMS, but it's hard work
Better to start over with a system designed for modern day scale out.

Domain specific solution
RRDTool
Specialist storage (and graphing) for sampled time series data
Fixed amount of data
Individual points for recent samples
Aggregates for older data, progressively more coarse grained.
Single system

Domain specific solution
RRDTool
Ingest rate starting exceed what a single system can deal with
SSD may buy some time, but only a little
Data storage assumptions are outdated
Cost/Value balance has changed – we now want to keep the full data (practically) forever.

OpenTSDB
Store, index and serve time series metrics at scale
HBase distributed storage backend
HDFS handles sharding, replication of data
Designed for scale out on commodity hardware
Lack of joins allows for more flexible data locality

OpenTSDB
Data storage is distributed. For more metrics or greater frequency, just add more nodes
TSDs are stateless – just add more for throughput and availability
Designed with the expectation of failure: No SPF. Rebalancing and failover are more automated

OpenTSDB
HBase data model similar to SQL wide row
HBase has built in data compression, unlike RDBMS
~60 bytes/datapoint -> LZO -> 12 bytes
Add application specific data packing:
12 bytes -> 3.3 bytes/datapoint.

OpenTSDB
Scales using cheaper hardware and software
Requires only free, open source software
Scales using less admin effort

Big Time Series Data Part 2: Counting

Counting things really fast
Lots of interesting metrics rely on counting event occurrences
Previously this relied on batch log processing analytics
High latency, long decision/action loop
'Real-time' counting enables new uses
'real-time' = 'without perceivable delay'

Counting things the SQL way
UPDATE counters SET x=x+1 WHERE metric_id = ?;
Requires a row lock and a synchronous disk force.
Has availability problems
So you're back to needing sharding and replication...

Write Amplification
A single event may increment multiple counters
Hierarchic groups: item, category, url
Dimensions: user, geography, supplier
Time period roll-up: hour, day, month
SQL has classic conundrum
Batch to reduce tx log writes
Unbatch to reduce tx lock contention

OLTP / OLAP
Record one event record in real-time
OLTP schema
Denormalize to N counters in batch
ETL
Run analytics against the denormalized data warehouse schema
OLAP schema

Counting with memcached
in-memory store for small data items
K/V with expiry and eviction
distributed but not replicated
Normally used as a cache for web content
Supports CAS, making it suitable for volatile counters

Counting the Cassandra way
Cassandra is a distributed, fault tolerant column family db similar to HBase
Supports a specific data type for counters
Partitioned over replicas
Idempotent repair
May overcount in some write failure cases
May undercount in some read failure cases

Rainbird/Countandra
Daemon using Cassandra for storage
configurable write cache for aggregation
Automatic hierarchic and temporal roll-ups
100,000s events/second write, 10,000 reads/second at <100ms, 100+TB

Almost Done...

Summary
Traditional SQL solutions can scale
Up to a point
At a cost (hardware, software, people )
NoSQL solutions may be better suited, more economic
A variety of specialist solutions layered on NoSQL are becoming available

Is it right for me?
If your existing DB (and DBA) will handle the load, then just use SQL
Is the data valuable enough to warrant the spend in hardware and people?
Revisit past decisions in light of new economics
Can be a valuable learning experience for later, more critical projects

Questions?

Tues April 10 th : Big Time Series Data
Tues May 8 th : NoSQL and real-time analytics
community.jboss.org/groups/jbug-newcastle

Timeseries data

by supermondays

Accessibility

Categories

Upload Details

Usage Rights

2 Embeds 136

Statistics

Timeseries data Presentation Transcript

http://www.supermondays.org	135
http://translate.googleusercontent.com	1