REDIS FOR YOUR BOSSELENA KOLEVSKA

Who am I ?A nomad earthling. Lead developer @ www.speedtocontact.com
BLOG.ELENAKOLEVSKA.COM
@ELENA_KOLEVSKA

(Really) SHORT
INTRO
No, seriously!

REDIS IS AN OPEN SOURCE (BSD licensed),
IN-MEMORY DATA STRUCTURE STORE, USED
AS DATABASE, CACHE AND MESSAGE
BROKER
The'Definition'on redis.io

BASIC FEATURES:
▸ Different data structures
▸ Keys with a limited time-to-live
▸ Transactions
▸ Pipelining
▸ Lua scripting
▸ Pub/Sub
▸ Built-in replication
▸ Different levels of on-disk persistence

SPEED

AVAILABLE CLIENTS IN:
ActionScript bash C C# C++ Clojure
Common lisp Crystal D Dart Elixir emacs lisp
Erlang Fancy gawk GNU Prolog Go Haskell
Haxe Io Java Javascript Julia Lua
Matlab mruby Nim Node.js Objective-C OCaml
Pascal Perl PHP Pure Data Python R
Racket Rebol Ruby Rust Scala Scheme
Smalltalk Swift Tcl VB VCL

AVAILABLE DATA STRUCTURES
▸ Strings (Binary safe, can be anything from"hello world"to a jpeg file)
▸ Lists (Collections of string elements sorted according to the order of insertion)
▸ Sets (Collections of unique, unsorted string elements)
▸ Sorted sets (It's like Sets with a score)
▸ Hashes (Maps of fields associated to values. Think non-nested json objects)
▸ Bitmaps (Manipulate Strings on a bit level)
▸ HyperLogLogs (Probabilistic data structure used to estimate the cardinality of a set)

Imagine...

TWITTER ANALYSIS TOOL
▸ Track a selected group of hashtags (#gameofthrones, #got, #gotseason7)
▸ Count mentions of certain keywords ('winter is coming', 'tyrion', 'jon snow', 'stark', 'targaryen', 'cersei', 'asha greyjoy', 'Khaleesi', 'sansa', 'arya')
METRICS:
▸ A feed of all tweets containing one of the hashtags
▸ Total number of tweets with one or more of the selected hashtags
▸ A leaderboard of keyword frequency
▸ A feed of tweets per keyword

[1] CONNECTING
TO REDIS

[1] CONNECTING TO REDIS
Install the PRedis package using composer
composer require predis/predis
...
// Initialize the client
$parameters = [
'scheme' => 'tcp',
'host' => '127.0.0.1',
'port' => 6379
];
$client = new PredisClient($parameters, ['prefix' => 'twitter_stats:']);

[2] SET TRACKED
DATA

[2] SET TRACKED DATA
Use sets to store all the hashtags we'll be looking at and all the keywords
as well
$client->sadd('hashtags', 'gameofthrones','got', 'gotseason7');
// hashtags | 'gameofthrones'
// | 'got'
// | 'gotseason7'
$client->sadd('keywords', 'winter is coming', 'winterfell', 'jon snow',
'stark', 'targaryen', 'cersei', 'asha greyjoy', 'dorne', 'Khaleesi'
'hodor', 'sansa', 'arya', 'white walkers', 'the night king');

[3] GET THE DATA

[3] GET THE DATA
Use Twitter Stream API to receive notifications for tweets containing
any of the hashtags we're following
$hashtags = $client->smembers('hashtags');
// array (size=3)
// 0 => string 'got' (length=3)
// 1 => string 'gameofthrones' (length=13)

Save every new tweet from the stream as a separate String.
$keyname = 'tweet_id:' . $tweet_id;
$tweet_contents = "Winter is coming Khaleesi! #gameofthrones";
$client->set($keyname, $tweet_contents)
// 'tweet_id:45645656' | 'Winter is coming Khaleesi! #gameofthrones'
And then push to a queue to be processed asynchronously
// Use the list data structure as a queue
$client->lpush('message_queue', $keyname);
// 'message_queue' | 'tweet_id:45645656'
// | 'tweet_id:44645234'
// | 'tweet_id:43645232'

[4] WORKER TO
PROCESS THE
QUEUED JOBS

[4] WORKER TO PROCESS THE QUEUED JOBS
A separate worker will be grabbing jobs off the top of the queue and processing them:
$message_queue = $client->rpop('message_queue');
// 'message_queue' | 'tweet_id:45645656'
// | 'tweet_id:44645234'
// | 'tweet_id:43645232'
Reliable queue: RPOPLPUSH, BRPOPLPUSH
Blocking queue: BLPOP, BRPOP

[5] PROCESS THE
TWEET CONTENT

[5] PROCESS THE TWEET CONTENT
$tweet_contents = $client->get($keyname);
$keywords = $client->smembers('keywords');
foreach ($keywords as $keyword) {
$tweet_contents = strtolower($tweet_contents);
$keyword = strtolower($keyword);
if (strpos($tweet_contents,$keyword) !== false){
$client->zincrby('mention_counter', 1, $keyword); // Increase the counter for this specific keyword
// mention_counter | 'tyrion' => 9.00
// | 'the wall' => 5.00
// | 'arya' => 4.00
$keyword_feed_keyname = 'keyword_feeds:'. $keyword;
$client->lpush($keyword_feed_keyname, $tweet_contents); // Add the tweet to the keyword's feed
$client->ltrim($keyword_feed_keyname, 0, 50);
}
}
$client->incr('total_count'); // Increase the general tweet count
$client->lpush('main_feed', $tweet_contents);
$client->ltrim('main_feed', 0, 100);

[6] SHOW THE STATS
$total_count = $client->get('total_count');
// 'total_count' | 259
$scores = $client->zrevrangebyscore('mention_counter', '+inf', '-inf', ['withscores'=>1]);
// mention_counter | 'tyrion' => 9.00
// | 'the wall' => 5.00
// | 'arya' => 4.00
// Feed by keyword
foreach ($scores as $keyname => $score) {
$keyword_feeds[$keyname] = $client->lrange('keyword_feeds:' . $keyname, 0, -1);
}
// Feed of all tweets containing one of the specified hashtags
$main_feed = $client->lrange('main_feed', 0, -1);

[7] USEFUL
EXTRAS

[7] USEFUL EXTRAS
API RATE LIMIER
$ip = $_SERVER['REMOTE_ADDR'] ;
$timestamp = time(); //unix timestamp
$key = 'api_rate_limits:' . $timestamp . ':' . $ip; // $key = 'api_rate_limits:1473613000:192.168.10.1 '
$api_requests = $client->get($keyname);
if (!is_null($api_requests) && $api_requests >= 3){
throw new Exception('Too many requests per second');
}else{
$client->multi();
$client->incr($key);
$client->expire($key,10);
$client->exec();
}

[7] USEFUL EXTRAS
PIPELINING
$keyname = 'tweet_id:' . $tweet_id;
$keywords = $client->smembers('keywords');
$pipe = $client->pipeline();
$pipe->set($keyname, $tweet_contents);
foreach ($keywords as $keyword) {
[...]
}
$pipe->incr('total_count');
$pipe->lpush('main_feed', $tweet_contents);
$pipe->ltrim('main_feed', 0, 20);
$replies = $pipe->execute();

[7] USEFUL EXTRAS
LUA SCRIPTING

Thank you!

Questions?@ELENA_KOLEVSKA
HTTPS://JOIND.IN/TALK/C68B2