Clojure at BackType

Clojure at BackTypeHow we learned to stop worrying and love the parentheses Nathan Marz BackType @nathanmarz

BackTypeData Services (APIs)Social Media Analytics Dashboard

APIs• Conversational graph for url• Comment search• #Tweets / URL• Inﬂuence scores• Top sites• Trending links stream• etc.

URL Proﬁles

Site comparisons

Inﬂuencer Proﬁles

Twitter Account Analytics

Topic Analysis

BackType’s Challenges

BackType’s Challenges Complex analytics

BackType’s Challenges Complex analyticson lots of data (> 30TB)

BackType’s Challenges Complex analyticson lots of data (> 30TB) in realtime

Clojure at BackType• Cascalog• ElephantDB• Storm

Let’s build an app

Cascalog Cascalog Variables and logicAbstraction Cascading Tuples, data workﬂows Key/value pairs, MapReduce aggregation

Cascalog basics The “age” dataset

Cascalog basics

Cascalog basicsDeﬁne andexecute a query

Cascalog basics Where to emit resultsDeﬁne andexecute a query

Cascalog basics Where to emit results Output variablesDeﬁne andexecute a query

Cascalog basics Where to “Predicates”: constrain emit results the output variables Output variablesDeﬁne andexecute a query

Predicates

PredicatesInput ﬁelds

PredicatesInput ﬁelds Output ﬁelds

PredicatesFields can be constants or variables

PredicatesFields can be constants or variables Variables are preﬁxed with ? or !

Predicates

Predicates• Functions• Filters• Aggregators• Generators: ﬁnite sources of tuples

Example #1 Generator Filter

Example #2Generator Function

Example #3Generator Aggregator Filter

Join example

Join example Triggers a join

Join example

Join exampleJoins are an implementation detail

Cascalog demo!

Composability “Predicate macro”

Composability expands toUsing a predicate macro

Contrast to PigPig’s AVG is 300 lines of code

Let’s build an app

Graph Schema Reshare: trueGender: female Property Tweet: 456 Property Reaction Reactor Reactor Tweet: 123 Alice Bob Property Property Content: RT @bob Content: Data is fun! Data is fun!

ElephantDB Shard 0 Shard 1 Shard 2 DistributedKey/Value pairs Shard 3 Filesystem Pre-shard Shard 4 and index in Shard 5 MapReduce Generation of domain of data

ElephantDBDFS ElephantDB ServerShard 0Shard 1Shard 2 ElephantDB ServerShard 3Shard 4Shard 5 ElephantDB Server Serving domain of data

StormStream Processing Distributed RPC

Stream processing• Automatically distributes computation• Horizontally scalable• Fault-tolerant• Guarantees processing of messages

Stream processing DBQueue DB DB Storm cluster

Raw data What is a query? View

Tweets What is a query? # Tweets for a URL

Tweets What is a query? Inﬂuence Score for a person

Raw data Computing a query Fully precompute view DB Query

Raw data Computing a query Do a live compute from scratch Query

Computing a query DBRaw data Precompute subviews Compute query from DB Query intermediate dbs DB

Distributed RPCApplication Queue “I want to know X, and return the results to me at Y”

Distributed RPC DBsQueue App queries Storm cluster

(BackType is hiring)

Questions?

http://paper.li	36
http://twitter.com	5
http://www.onlydoo.com	2
http://www.mefeedia.com	2
http://www.mongodb.org	1
http://a0.twimg.com	1

Clojure at BackType

by nathanmarz

Accessibility

Upload Details

Usage Rights

6 Embeds 47

Statistics

Clojure at BackType Presentation Transcript