Brisk: more powerful Hadoop powered by Cassandra
Upcoming SlideShare
Loading in...5
×
 

Brisk: more powerful Hadoop powered by Cassandra

on

  • 6,867 views

 

Statistics

Views

Total Views
6,867
Views on SlideShare
5,404
Embed Views
1,463

Actions

Likes
5
Downloads
104
Comments
0

14 Embeds 1,463

http://blog.nosqlfan.com 1359
http://feed.feedsky.com 48
http://xianguo.com 17
http://twitter.com 11
https://twitter.com 6
http://xue.uplook.cn 4
http://zhuaxia.com 4
http://reader.youdao.com 4
http://a0.twimg.com 2
http://www.uplook.cn 2
http://news.uplook.cn 2
http://www.zhuaxia.com 2
http://cache.baidu.com 1
http://translate.googleusercontent.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Brisk: more powerful Hadoop powered by Cassandra Brisk: more powerful Hadoop powered by Cassandra Presentation Transcript

  • Brisk: More Powerful Hadoop Powered by Cassandra jbellis@datastax.comMonday, July 25, 2011
  • The evolution of Analytics Analytics + RealtimeMonday, July 25, 2011
  • The evolution of Analytics replication Analytics RealtimeMonday, July 25, 2011
  • The evolution of Analytics ETLMonday, July 25, 2011
  • Brisk re-unifies realtime and analyticsMonday, July 25, 2011
  • The Traditional Hadoop Stack Slave Nodes Master Nodes Data Node Name Node Task Tracker Secondary Name Node Region Server Job Tracker Hbase Master Client Nodes Pig ZooKeeper Hive MetaStore Region ServerMonday, July 25, 2011
  • 7Monday, July 25, 2011
  • Brisk ArchitectureMonday, July 25, 2011
  • Brisk Highlights ✤ Easy to deploy and operate ✤ No single points of failure ✤ Scale and change nodes with no downtime ✤ Cross-DC, multi-master clusters ✤ Allocate resources for OLAP vs OLTP ✤ With no ETLMonday, July 25, 2011
  • Cassandra data model ✤ ColumnFamilies contain rows + columns ✤ (Not really schemaless for a while now) password name site zznate * Nate McCall driftx * Brandon Williams jbellis * Jonathan Ellis datastax.comMonday, July 25, 2011
  • Sparse password name zznate * Nate McCall password name driftx * Brandon Williams password name site jbellis * Jonathan Ellis datastax.comMonday, July 25, 2011
  • Rows as containers / materialized views driftx thobbs pcmanus jbellis zznate circle1 xedin mdennis circle2 xedin pcmanus ymorishita circle3Monday, July 25, 2011
  • Monday, July 25, 2011
  • CassandraFS ✤ data stored as ByteBuffer internally -- excellent fit for blocks ✤ local reads mmap data directly (no rpc) ✤ blocks are compressed with google snappy ✤ hadoop distcp hdfs:///mydata cfs:///mydataMonday, July 25, 2011
  • Hive support ✤ Hive MetaStore in Cassandra ✤ Unified schema view from any node, with no external systems and no SPOF ✤ Automatically maps Cassandra column families to Hive tables ✤ Supports static and dynamic column families (and supercolumns)Monday, July 25, 2011
  • Hive: CFS and ColumnFamilies CREATE TABLE users (name STRING, zip INT);  LOAD DATA LOCAL INPATH kv2.txt OVERWRITE INTO TABLE users;   CREATE EXTERNAL TABLE Keyspace1.Users(name STRING, zip INT) STORED BY org.apache.hadoop.hive.cassandra.CassandraStorageHandler; CREATE EXTERNAL TABLE Keyspace1.Users (row_key STRING, column_name STRING, value string) STORED BY org.apache.hadoop.hive.cassandra.CassandraStorageHandler;Monday, July 25, 2011
  • Pig Support ✤ With standard Cassandra: $ export PIG_HOME=/path/to/pig $ export PIG_INITIAL_ADDRESS=localhost $ export PIG_RPC_PORT=9160 $ export PIG_PARTITIONER=org.apache.cassandra.dht.RandomPartitioner $ contrib/pig/bin/pig_cassandra grunt> ✤ With Brisk: $ bin/brisk pig grunt>Monday, July 25, 2011
  • Pig: CFS and ColumnFamilies grunt> data = LOAD cfs:///example.txt using PigStorage() as (name:chararray, value:long); data = LOAD cassandra://Demo1/Scores using CassandraStorage() AS (key, columns: {T: tuple(name, value)}); data = LOAD cassandra://Demo1/Scores&slice_start=M&slice_end=S using CassandraStorage() AS (key, columns: {T: tuple(name, value)});Monday, July 25, 2011
  • 19Monday, July 25, 2011
  • Data model: Realtime LiveStocks last GOOG $95.52 AAPL $186.10 AMZN $112.98 Portfolios GOOG LNKD P AMZN AAPLE Portfolio1 80 20 40 100 20 StockHist 2011-01-01 2011-01-02 2011-01-03 GOOG $79.85 $75.23 $82.11Monday, July 25, 2011
  • Data model: Analytics HistLoss worst_date loss Portfolio1 2011-07-23 -$34.81 Portfolio2 2011-03-11 -$11432.24 Portfolio3 2011-05-21 -$1476.93Monday, July 25, 2011
  • Data model: Analytics 10dayreturns ticker rdate return GOOG 2011-07-25 $8.23 GOOG 2011-07-24 $6.14 GOOG 2011-07-23 $7.78 AAPL 2011-07-25 $15.32 AAPL 2011-07-24 $12.68 INSERT OVERWRITE TABLE 10dayreturns SELECT a.row_key ticker, b.column_name rdate, b.value - a.value FROM StockHist a JOIN StockHist b ON (a.row_key = b.row_key AND date_add(a.column_name,10) = b.column_name);Monday, July 25, 2011
  • 2011-01-01 2011-01-02 2011-01-03 GOOG $79.85 $75.23 $82.11 row_key column_name value GOOG 2011-01-01 $8.23 GOOG 2011-01-02 $6.14 GOOG 2011-001-03 $7.78Monday, July 25, 2011
  • Data model: Analytics portfolio_returns portfolio rdate preturn Portfolio1 2011-07-25 $118.21 Portfolio1 2011-07-24 $60.78 Portfolio1 2011-07-23 -$34.81 Portfolio2 2011-07-25 $2143.92 Portfolio3 2011-07-24 -$10.19 INSERT OVERWRITE TABLE portfolio_returns SELECT row_key portfolio, rdate, SUM(b.return) FROM portfolios a JOIN 10dayreturns b ON (a.column_name = b.ticker) GROUP BY row_key, rdate;Monday, July 25, 2011
  • Data model: Analytics HistLoss worst_date loss Portfolio1 2011-07-23 -$34.81 Portfolio2 2011-03-11 -$11432.24 Portfolio3 2011-05-21 -$1476.93 INSERT OVERWRITE TABLE HistLoss SELECT a.portfolio, rdate, minp FROM ( SELECT portfolio, min(preturn) as minp FROM portfolio_returns GROUP BY portfolio ) a JOIN portfolio_returns b ON (a.portfolio = b.portfolio and a.minp = b.preturn);Monday, July 25, 2011
  • Portfolio Demo dataflow Portfolios Web-based Portfolios Historical Prices Live Prices for today Intermediate Results Largest loss Largest lossMonday, July 25, 2011
  • OpsCenterMonday, July 25, 2011
  • Monday, July 25, 2011
  • Where to get it ✤ http://www.datastax.com/briskMonday, July 25, 2011
  • Monday, July 25, 2011