An experiment in a distributed approach to processing the real-time data generated by a large scale social media campaign. Presented at Cambridge Geek Nights 13.

1. Small Pieces Loosely
Joined
#cgn13

2. or...
A practical example of processing real-time
data with a distributed agent network
(Warning: does not contain real code)

3. Red Gate

4. 12th October 2011

5. eMail Marketing

6. Mailchimp webhook
"type": "subscribe",
"fired_at": "2009-03-26 21:35:57",
"data[id]": "8a25ff1d98",
"data[list_id]": "a6b5da1054",
"data[email]": "api@mailchimp.com",
"data[email_type]": "html",
"data[merges][EMAIL]": "api@mailchimp.com",
"data[merges][FNAME]": "MailChimp",
"data[merges][LNAME]": "API",
"data[merges][INTERESTS]": "Group1,Group2",
"data[ip_opt]": "10.20.10.30",
"data[ip_signup]": "10.20.10.30"

7. Pump the callbacks into a message bus...

8. Messaging

9. mailchimp-pump.php
$json = json_encode($_POST);
$msg = new AMQPMessage($json);
$channel->basic_publish($msg, 'mailchimp', "morat.campaign.mailchimp.".
$_POST['type']);

10. I’d like to watch the stream on IRC...

11. Valve
Subscribe to mailchimp exchange
morat.campaign.mailchimp.#
Translate to plain english for IRC
Inject into irc exchange with routing key morat.irc.
[channel]

12. mailchimp-irc-valve.rb
case record['type']
when 'subscribe'
output :irc, "'#{record['data']['merges']['FNAME']} #{record['data']
['merges']['LNAME']}' has joined the list"
when 'unsubscribe'
output :irc, "'#{record['data']['merges']['FNAME']} #{record['data']
['merges']['LNAME']}' has left the list"
...

13. Create a Sink to send the messages to IRC...

14. irc-sink.pl
$q = $amq->channel(1)->queue('morat.irc.' . $channel , { passive => 0,
durable => 0, auto_delete => 1, exclusive => 0, })->subscribe( sub {
my ($payload, $meta) = @_;
my ($channel) = $meta->{'queue'} =~ /.([^.]+)$/;
$irc->yield('privmsg', '#'.$channel, GREEN.$payload);
});

15. Where have we got to?
Pump: Mailchimp webhook (HTTP POST) >
morat.[campaign].mailchimp.[type] (JSON)
Valve: morat.campaign.mailchimp.[type] (JSON) >
morat.irc.[campaign] (Text)
Sink: morat.irc.[campaign] (Text) > IRC server

16. That’s cool, but hey it would be great to see
#campaign tweets as well...

17. twitter-search-pump.rb
TweetStream::Client.new.track(keywords.split(',')) do |status|
keywords.split(',').each do |searchterm|
if status.text.match(searchterm)
searchterm.sub!(' ','')
searchterm.sub!('#','')
log.debug "Sending: #{status.user.screen_name} :: #{status.text} ::
morat.twitter.search.#{searchterm}"
broker.exchange.publish JSON.generate(status), :routing_key =>
"morat.twitter.search.#{searchterm}"
end
end
end

18. twitter-irc-valve.rb
case routing_key
when 'morat.twitter.@neildavidson.list.redgaters'
output :irc, "RG chatter: #{record['user']['screen_name']} tweeted:
#{record['text']}", :routing_key => "morat.irc.redgaters"
else
searchterm = routing_key.match(/morat.twitter.search.(.+)/)[1]
output :irc, "#{record['user']['screen_name']} tweeted:
#{record['text']}", :routing_key => "morat.irc.#{searchterm}"

20. I feel the urge to graph...

21. Thanks @garethr

22. Valve
Subscribe to mailchimp exchange morat.
[campaign].mailchimp.#
Translate to Graphite format: [value] [timestamp]
Inject into graphite exchange with routing key
based on sample window: 10sec.
[campaign].mailchimp.[action].count

23. But let’s make it cool...

24. Complex Event Processing

25. mailchimp-graphite-
valve.rb
%w{ subscribe unsubscribe campaign }.each do |action|
[ '10 sec', '1 min', '5 min', '15 min' ].each do |window|
valve.register "SELECT count(*) from
MailchimpEvent(type='#{action}').win:time_batch(#{window})", (
Listener.new(valve) do |agent, event|
valve.output :graphite, "#{event.get('count(*)')}", :routing_key =>
window.delete(' ') + ".morat.#{valve.application}.mailchimp.#{action}"
end
)
end
end

26. Why use CEP?
# find the sum of retweets of last 5 tweets which saw more than 10 retweets
SELECT sum(retweets) from TweetEvent(retweets >= 10).win:length(5)
# find max, min and average number of retweets for a sliding 60 second window of time
SELECT max(retweets), min(retweets), avg(retweets) FROM TweetEvent.win:time(60 sec)
# compute number of retweets for all tweets in 10 second batches
SELECT sum(retweets) from TweetEvent.win:time_batch(10 sec)
# number of retweets, grouped by timezone, buffered in 10 second increments
SELECT timezone, sum(retweets) from TweetEvent.win:time_batch(10 sec) group by timezone
# compute the sum of retweets in sliding 60 second window, and emit count every 30 events
SELECT sum(retweets) from TweetEvent.win:time(60 sec) output snapshot every 30 events
# every 10 seconds, report timezones which accumulated more than 10 retweets
SELECT timezone, sum(retweets) from TweetEvent.win:time_batch(10 sec) group by timezone having
sum(retweets) > 10
Courtesy @igrigorik http://www.igvita.com/2011/05/27/streamsql-event-processing-with-esper/

28. Is there really a correlation?

29. Statistical Computing

30. Valve
Grab raw data for window from graphite via REST
Create scatter graph using R and calculate
correlation
Inject correlation into graphite exchange

31. twitter-correlation-valve.rb
require 'rsruby'
...
r.jpeg(filename)
r.assign('xs', data[1])
r.assign('ys', data[2])
fit = r.lm('ys ~ xs')
r.plot({
'x' => data[1],
'y' => data[2],
'xlab' => label[1],
'ylab' => label[2]
})
cor = r.cor(data[1],data[2]).to_s
r.title("Correlation: " + cor)
r.abline(fit['coefficients']['(Intercept)'],fit['coefficients']['xs'])
r.eval_R("dev.off()")

33. Lets add some realtime visualisation...

34. Websockets

35. Valve
Subscribe to twitter exchange
morat.twitter.search.[keyword]
Extract adjectives using entagger
Inject adjectives into twitter exchange with routing
key morat.twitter.search.[keyword].adjectives as:
[adjective] [count]

36. twitter-sentiment-valve.rb
require 'engtagger'
...
log.debug "Received tweet from #{record['user']['screen_name']} on
#{routing_key}"
adjectives = @parser.add_tags(record['text']).scan(EngTagger::ADJ).map do |
n|
@parser.strip_tags(n)
end
ret = Hash.new(0)
adjectives.each do |n|
n = @parser.stem(n)
ret[n] += 1 unless n =~ /As*z/
end

37. Sink
Subscribe to twitter exchange
morat.twitter.search.[keyword].adjectives
Use node.js and Socket.IO to send data to web
client via Websockets
Visualise with processing.js in web browser

38. twitter-sentiment-sink.js
io.sockets.on('connection', function (socket) {
amqp_connection.on('ready', function () {
var queue = amqp_connection.queue('');
exchange = amqp_connection.exchange('twitter', { type: 'topic',
passive: false, durable: true, autoDelete: true}, function (exchange) {
queue.bind(exchange,routing_key);
queue.subscribe(function (message) {
socket.emit('data', { text: message.data.toString() });
});
});
});
});

39. twitter-sentiment-sink.html
<H1>Twitter Sentiment</H1>
<div id="container">
<canvas id="twitter-sentiment-sink" data-processing-sources="twitter-
sentiment-sink.pde" WIDTH=800 HEIGHT=600></canvas>
</div>
<script src="/socket.io/socket.io.js"></script>
<script type="text/javascript">
var socket = io.connect('http://localhost');
socket.on('data', function (data) {
var pjs = Processing.getInstanceById('twitter-sentiment-sink');
pjs.addDatum(data.text.split(' ')[0]);
});
</script>

41. @ennui2342
www.morat.co.uk
polis.ecafe.org