Brisk: more powerful Hadoop powered by Cassandra

Brisk: More Powerful Hadoop Powered by Cassandra jbellis@datastax.comMonday, July 25, 2011

The evolution of Analytics Analytics + RealtimeMonday, July 25, 2011

The evolution of Analytics replication Analytics RealtimeMonday, July 25, 2011

The evolution of Analytics ETLMonday, July 25, 2011

Brisk re-unifies realtime and analyticsMonday, July 25, 2011

The Traditional Hadoop Stack Slave Nodes Master Nodes Data Node Name Node Task Tracker Secondary Name Node Region Server Job Tracker Hbase Master Client Nodes Pig ZooKeeper Hive MetaStore Region ServerMonday, July 25, 2011

7Monday, July 25, 2011

Brisk ArchitectureMonday, July 25, 2011

Brisk Highlights ✤ Easy to deploy and operate ✤ No single points of failure ✤ Scale and change nodes with no downtime ✤ Cross-DC, multi-master clusters ✤ Allocate resources for OLAP vs OLTP ✤ With no ETLMonday, July 25, 2011

Cassandra data model ✤ ColumnFamilies contain rows + columns ✤ (Not really schemaless for a while now) password name site zznate * Nate McCall driftx * Brandon Williams jbellis * Jonathan Ellis datastax.comMonday, July 25, 2011

Sparse password name zznate * Nate McCall password name driftx * Brandon Williams password name site jbellis * Jonathan Ellis datastax.comMonday, July 25, 2011

Rows as containers / materialized views driftx thobbs pcmanus jbellis zznate circle1 xedin mdennis circle2 xedin pcmanus ymorishita circle3Monday, July 25, 2011

Monday, July 25, 2011

CassandraFS ✤ data stored as ByteBuffer internally -- excellent ﬁt for blocks ✤ local reads mmap data directly (no rpc) ✤ blocks are compressed with google snappy ✤ hadoop distcp hdfs:///mydata cfs:///mydataMonday, July 25, 2011

Hive support ✤ Hive MetaStore in Cassandra ✤ Uniﬁed schema view from any node, with no external systems and no SPOF ✤ Automatically maps Cassandra column families to Hive tables ✤ Supports static and dynamic column families (and supercolumns)Monday, July 25, 2011

Hive: CFS and ColumnFamilies CREATE TABLE users (name STRING, zip INT); LOAD DATA LOCAL INPATH kv2.txt OVERWRITE INTO TABLE users; CREATE EXTERNAL TABLE Keyspace1.Users(name STRING, zip INT) STORED BY org.apache.hadoop.hive.cassandra.CassandraStorageHandler; CREATE EXTERNAL TABLE Keyspace1.Users (row_key STRING, column_name STRING, value string) STORED BY org.apache.hadoop.hive.cassandra.CassandraStorageHandler;Monday, July 25, 2011

Pig Support ✤ With standard Cassandra: $ export PIG_HOME=/path/to/pig $ export PIG_INITIAL_ADDRESS=localhost $ export PIG_RPC_PORT=9160 $ export PIG_PARTITIONER=org.apache.cassandra.dht.RandomPartitioner $ contrib/pig/bin/pig_cassandra grunt> ✤ With Brisk: $ bin/brisk pig grunt>Monday, July 25, 2011

Pig: CFS and ColumnFamilies grunt> data = LOAD cfs:///example.txt using PigStorage() as (name:chararray, value:long); data = LOAD cassandra://Demo1/Scores using CassandraStorage() AS (key, columns: {T: tuple(name, value)}); data = LOAD cassandra://Demo1/Scores&slice_start=M&slice_end=S using CassandraStorage() AS (key, columns: {T: tuple(name, value)});Monday, July 25, 2011

19Monday, July 25, 2011

Data model: Realtime LiveStocks last GOOG $95.52 AAPL $186.10 AMZN $112.98 Portfolios GOOG LNKD P AMZN AAPLE Portfolio1 80 20 40 100 20 StockHist 2011-01-01 2011-01-02 2011-01-03 GOOG $79.85 $75.23 $82.11Monday, July 25, 2011

Data model: Analytics HistLoss worst_date loss Portfolio1 2011-07-23 -$34.81 Portfolio2 2011-03-11 -$11432.24 Portfolio3 2011-05-21 -$1476.93Monday, July 25, 2011

Data model: Analytics 10dayreturns ticker rdate return GOOG 2011-07-25 $8.23 GOOG 2011-07-24 $6.14 GOOG 2011-07-23 $7.78 AAPL 2011-07-25 $15.32 AAPL 2011-07-24 $12.68 INSERT OVERWRITE TABLE 10dayreturns SELECT a.row_key ticker, b.column_name rdate, b.value - a.value FROM StockHist a JOIN StockHist b ON (a.row_key = b.row_key AND date_add(a.column_name,10) = b.column_name);Monday, July 25, 2011

2011-01-01 2011-01-02 2011-01-03 GOOG $79.85 $75.23 $82.11 row_key column_name value GOOG 2011-01-01 $8.23 GOOG 2011-01-02 $6.14 GOOG 2011-001-03 $7.78Monday, July 25, 2011

Data model: Analytics portfolio_returns portfolio rdate preturn Portfolio1 2011-07-25 $118.21 Portfolio1 2011-07-24 $60.78 Portfolio1 2011-07-23 -$34.81 Portfolio2 2011-07-25 $2143.92 Portfolio3 2011-07-24 -$10.19 INSERT OVERWRITE TABLE portfolio_returns SELECT row_key portfolio, rdate, SUM(b.return) FROM portfolios a JOIN 10dayreturns b ON (a.column_name = b.ticker) GROUP BY row_key, rdate;Monday, July 25, 2011

Data model: Analytics HistLoss worst_date loss Portfolio1 2011-07-23 -$34.81 Portfolio2 2011-03-11 -$11432.24 Portfolio3 2011-05-21 -$1476.93 INSERT OVERWRITE TABLE HistLoss SELECT a.portfolio, rdate, minp FROM ( SELECT portfolio, min(preturn) as minp FROM portfolio_returns GROUP BY portfolio ) a JOIN portfolio_returns b ON (a.portfolio = b.portfolio and a.minp = b.preturn);Monday, July 25, 2011

Portfolio Demo dataflow Portfolios Web-based Portfolios Historical Prices Live Prices for today Intermediate Results Largest loss Largest lossMonday, July 25, 2011

OpsCenterMonday, July 25, 2011

Where to get it ✤ http://www.datastax.com/briskMonday, July 25, 2011

http://blog.nosqlfan.com	1178
http://feed.feedsky.com	48
http://xianguo.com	17
http://twitter.com	11
https://twitter.com	6
http://zhuaxia.com	4
http://reader.youdao.com	4
http://www.zhuaxia.com	2
http://a0.twimg.com	2
http://cache.baidu.com	1
http://translate.googleusercontent.com	1

Brisk: more powerful Hadoop powered by Cassandra

by jbellis on Aug 01, 2011

Accessibility

Categories

Upload Details

Usage Rights

11 Embeds 1,274

Statistics

Brisk: more powerful Hadoop powered by Cassandra Presentation Transcript