Cassandra + Hadoop = Brisk

London
Our sponsors:
Acunu

But first, a short back story…

9
10
11
12
5
6
7
8
1
2
3
4

21
22
23
24
GC HELL!
17
18
19
20
13
14
15
16

33
34
35
36
29
30
31
32
25
26
27
28

Please volunteer if you would like to give a talk, Internet fame awaits

My experience with Cassandra in production is positive
Analytics is more difficult than it could be
Welcome Brisk!

Brisk combines Hadoop, Hive and Cassandra in a “distribution”

In a nutshell
CassandraFS as HDFS compatible layer; no namenode, no SPOF
Can split cluster for OLAP and OLTP workloads, scaling up either as required

Demonstrating brisk…
Building an Ad Network!

Demonstrating brisk…
Building anAd Network!

The plan:
Simple data model – segment users into buckets
System to put users in buckets via a pixel
Real-time queries
Analytics

We Have Your KidneysThe ad-network for the paranoid generation
Cookie based identification
API provides:
Add user to a bucket (including ability to define expiry time)
Get buckets a user belongs to

Setup Brisk
http://www.datastax.com/docs/0.8/brisk/install_brisk_ami
Step-by-step guide with pictures!
Ubuntu 10.10 image with RAID 0 ephemeral disks
Jairam has been bug-fixing some minor issues

Data model
CF = users
[userUUID] [segmentID] = 1
CF = segments
[segmentID] [userUUID] = 1

Data model
create keyspacewhyk
... with placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy'
... and strategy_options = [{replication_factor:1}];
create column family users
... with comparator = 'AsciiType'
... and rows_cached = 5000;
create column family segments
... with comparator = 'AsciiType'
... and rows_cached = 5000;

Our pixel
http://wehaveyourkidneys.com/add.php? segment=<alphaNumericCode> &expire=<numberOfSeconds>
We’ll use Cassandra’s expiring columns feature

PHP code – uses phpcassa
$pool = new ConnectionPool('whyk', array('localhost'));
$users = new ColumnFamily($pool, 'users');
$segments = new ColumnFamily($pool, 'segments');
$users->insert(
$userUuid,
array($segment => 1),
NULL, // default TS
$expires
);
$segments->insert(
$segment,
array($userUuid => 1),
NULL, // default TS
$expires
);

Real-time access
http://wehaveyourkidneys.com/show.php
$pool = new ConnectionPool('whyk', array('localhost'));
$users = new ColumnFamily($pool, 'users');
// @todo this only gets first 100!
$segments = $users->get($userUuid);
header('Content-Type: application/json');
echo json_encode(array_keys($segments));

Analytics
How many users in each segment?
Launch HIVE (very easy!)
root@brisk-01:~# brisk hive

CREATE EXTERNAL TABLE whyk.users
(userUuid string, segmentId string, value string)
STORED BY 'org.apache.hadoop.hive.cassandra.CassandraStorageHandler’
WITH SERDEPROPERTIES ("cassandra.columns.mapping" = ":key,:column,:value" );
select segmentId, count(1) as total
from whyk.users
group by segmentId
order by total desc;

Summary
http://www.flickr.com/photos/sovietuk/2956044892/sizes/o/in/photostream/

Real time access+
Batch analytics

Easy
Easy to setup
Easy to deploy mixed-modeclustersEasy to query (Hive)

No Single Pointof Failure

Further reading…
Installing the Brisk AMI
http://www.datastax.com/docs/0.8/brisk/install_brisk_ami
Key advantages of Brisk – from Jonathan Ellis
http://hackerne.ws/item?id=2528271
Why I’m very excited about DataStax’s Brisk – by Nathan Milford
http://blog.milford.io/2011/04/why-i-am-very-excited-about-datastaxs-brisk/
The demo code on Github
https://github.com/davegardnerisme/we-have-your-kidneys

http://www.davegardner.me.uk	1159
http://blog.jairam.me	549
http://translate.googleusercontent.com	5
http://www.slideshare.net	3
url_unknown	3

Cassandra + Hadoop = Brisk

by Dave Gardner on May 17, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

5 Embeds 1,719

Statistics

Cassandra + Hadoop = Brisk — Presentation Transcript