The document discusses database operations and functionalities for a large video website, focusing on user and video data management using a structured query language (CQL). It highlights the integration of intravert with Cassandra for efficient data processing, emphasizing server-side filters and multi-processor capabilities to minimize network I/O. Additionally, the text outlines practical filter creation methods in JavaScript and Groovy, showcasing the potential to streamline operations through server-side computations instead of client-side data processing.

1. Before we get into the heavy
stuff, Let's imagine hacking
around with C* for a bit...

2. You run a large video website
● CREATE TABLE videos (
videoid uuid,
videoname varchar,
username varchar,
description varchar, tags varchar,
upload_date timestamp,
PRIMARY KEY (videoid,videoname) );
● INSERT INTO videos (videoid, videoname, username,
description, tags, upload_date) VALUES ('99051fe9-6a9c-
46c2-b949-38ef78858dd0','My funny cat','ctodd', 'My cat
likes to play the piano! So funny.','cats,piano,lol','2012-06-01
08:00:00');

3. You have a bajillion users
● CREATE TABLE users (
username varchar,
firstname varchar,
lastname varchar,
email varchar,
password varchar,
created_date timestamp,
PRIMARY KEY (username));
● INSERT INTO users (username, firstname, lastname, email,
password, created_date) VALUES ('tcodd','Ted','Codd',
'tcodd@relational.com','5f4dcc3b5aa765d61d8327deb882cf99'
,'2011-06-01 08:00:00');

4. You can query up a storm
● SELECT firstname,lastname FROM users WHERE username='tcodd';
firstname | lastname
-----------+----------
Ted | Codd
● SELECT * FROM videos WHERE videoid = 'b3a76c6b-7c7f-4af6-964f-
803a9283c401' and videoname>'N';
videoid | videoname | description
| tags | upload_date | username
b3a76c6b-7c7f-4af6-964f-803a9283c401 | Now my dog plays piano! | My
dog learned to play the piano because of the cat. | dogs,piano,lol | 2012-
08-30 16:50:00+0000 | ctodd

5. That's great! Then you ask
yourself...

6. ● Can I slice a slice (or sub query)?
● Can I do advanced where clauses ?
● Can I union two slices server side?
● Can I join data from two tables without two
request/response round trips?
● What about procedures?
● Can I write functions or aggregation functions?

7. Let's look at the API's we have
http://www.slideshare.net/aaronmorton/apachecon-nafeb2013

8. But none of those API's do what I
want, and it seems simple
enough to do...

9. Intravert joins the “party”
at the API Layer

10. Why not just do it client side?
● Move processing close to data
– Idea borrowed from Hadoop
● Doing work close to the source can result in:
– Less network IO
– Less memory spend encoding/decoding 'throw
away' data
– New storage and access paradigms

11. Vertx + cassandra
● What is vertx ?
– Distributed Event Bus which spans the server and
even penetrates into client side for effortless 'real-
time' web applications
● What are the cool features?
– Asynchronous
– Hot re-loadable modules
– Modules can be written in groovy, ruby, java, java-
script
http://vertx.io

12. Transport, payload, and
batching

13. HTTP Transport
● HTTP is easy to use on firewall'ed networks
● Easy to secure
● Easy to compress
● The defacto way to do everything anyway
● IntraVert attempts to limit round-trips
– Not provide a terse binary format

14. JSON Payload
● Simple nested types like list, map, String
● Request is composed of N operations
● Each operation has a configurable timeout
● Again, IntraVert attempts to limit round-trips
– Not provide a terse message format

15. Why not use lighting fast transport
and serialization library X?
● X's language/code gen issues
● You probably can not TCP dump X
● Net-admins don't like 90 jars for health checks
● IntraVert attempts to limit round-trips:
– Prepared statements
– Server side filtering
– Other cool stuff

16. Sample request and response
{"e": [ { {
"type": "SETKEYSPACE",
"exception":null,
"op": { "keyspace": "myks" }
"exceptionId":null,
}, {
"type": "SETCOLUMNFAMILY", "opsRes": {
"op": { "columnfamily": "mycf" } "0":"OK",
}, { "1":"OK",
"type": "SLICE",
"2":[{
"op": {
"name":"Founders",
"rowkey": "beers",
"start": "Allagash", "value":"Breakfast Stout"
"end": "Sierra Nevada", }]
"size": 9 }}
} }]}

17. Server side filter

18. Imagine your data looks like...
{ "rowkey": "beers", "name":
"Allagash", "value": "Allagash Tripel" }
{ "rowkey": "beers", "name":
"Founders", "value": "Breakfast Stout" }
{ "rowkey": "beers", "name": "Dogfish
Head",
"value": "Hellhound IPA" }

19. Application requirement
● User request wishes to know which beers are
“Breakfast Stout” (s)
● Common “solutions”:
– Write a copy of the data sorted by type
– Request all the data and parse on client side

20. Using an IntraVert filter
● Send a function to the server
● Function is applied to subsequent get or slice
operations
● Only results of the filter are returned to the
client

21. Defining a filter JavaScript
● Syntax to create a filter
{
"type": "CREATEFILTER",
"op": {
"name": "stouts",
"spec": "javascript",
"value": "function(row) { if (row['value'] == 'Breakfast Stout')
return row; else return null; }"
}
},

22. Defining a filter Groovy/Java
● We can define a groovy closure or Java filter
{
"type": "CREATEFILTER",
"op": {
"name": "stouts",
"spec": "groovy",
"{ row -> if (row["value"] == "Breakfast Stout") return row else
return null }"
}
},

23. Filter flow

24. Common filter use cases
● Transform data
● Prune columns/rows like a where clause
● Extract data from complex fields (json, xml,
protobuf, etc)

25. Some light relief

26. Server Side Multi-Processor

27. It's the cure for your “redis envy”

28. Imagine your data looks like...
● { “row key”:”1”, ● { “row key”:”4”,
name:”a” ,val...} name:”a” ,val...}
● { “row key”:”1”, ● { “row key”:”4”,
name:”b” ,val...} name:”z” ,val...}

29. Application Requirements
● User wishes to intersect the column names of
two slices/queries
● Common “solutions”
– Pull all results to client and apply the intersection
there

30. Server Side MultiProcessor
● Send a class that implements MultiProcessor
interface to server
● public List<Map> multiProcess
(Map<Integer,Object> input, Map params);
● Do one or more get/slice operations as input
● Invoke MultiProcessor on input

31. Multi-processor flow

32. Multi-processor use cases
● Union N slices
● Intersection N slices
● Some “Join” scenarios

33. Fat client becomes
the 'Phat client'

34. Imagine you want to insert this data
● User wishes to enter this event for multiple column
families
– 09/10/201111:12:13
– App=Yahoo
– Platform=iOS
– OS=4.3.4
– Device=iPad2,1
– Resolution=768x1024
– Events–videoPlayPercent=38–Taste=great
http://www.slideshare.net/charmalloc/jsteincassandranyc2011

35. Inserting the data
aggregateColumnNames(”AppPlatformOSVersionDeviceResolution") = "app+platform+osversion+device+resolution#”
def ccAppPlatformOSVersionDeviceResolution(c: (String) => Unit) = {
c(aggregateColumnNames(”AppPlatformOSVersionDeviceResolution”) + app + p(platform) + p(osversion) + p(device) + p(resolution))
}
aggregateKeys(KEYSPACE ”ByMonth") = month //201109
aggregateKeys(KEYSPACE "ByDay") = day //20110910
aggregateKeys(KEYSPACE ”ByHour") = hour //2011091012
aggregateKeys(KEYSPACE ”ByMinute") = minute //201109101213
def r(columnName: String): Unit = {
aggregateKeys.foreach{tuple:(ColumnFamily, String) => {
val (columnFamily,row) = tuple
if (row !=null && row.size > 0)
rows add (columnFamily -> row has columnName inc) //increment the counter
}
}
}
ccAppPlatformOSVersionDeviceResolution(r)
http://www.slideshare.net/charmalloc/jsteincassandranyc2011

36. Solution
● Send the data once and compute the N
permutations on the server side
public void process(JsonObject request, JsonObject state, JsonObject response, EventBus eb) {
JsonObject params = request.getObject("mpparams");
String uid = (String) params.getString("userid");
String fname = (String) params.getString("fname");
String lname = (String) params.getString("lname");
String city = (String) params.getString("city");
RowMutation rm = new RowMutation("myks", IntraService.byteBufferForObject(uid));
QueryPath qp = new QueryPath("users", null, IntraService.byteBufferForObject("fname"));
rm.add(qp, IntraService.byteBufferForObject(fname), System.nanoTime());
QueryPath qp2 = new QueryPath("users", null, IntraService.byteBufferForObject("lname"));
rm.add(qp2, IntraService.byteBufferForObject(lname), System.nanoTime());
...
try {
StorageProxy.mutate(mutations, ConsistencyLevel.ONE);
} catch (WriteTimeoutException | UnavailableException | OverloadedException e) {
e.printStackTrace();
response.putString("status", "FAILED");
}
response.putString("status", "OK");
}

37. Service Processor Flow

38. IntraVert status
● Still pre 1.0
● Good docs
– https://github.com/zznate/intravert-ug/wiki/_pages
● Functional equivalent to thrift (mostly features
ported)
● CQL support
● Virgil (coming soon)
● Hbase like scanners (coming soon)

39. Hack at it
https://github.com/zznate/intravert-ug

40. Questions?