Starting with v4, modules hold a promise for changing how Redis is used and developed for. Enabling custom data types and commands, Redis Modules build upon and extend the core functionality to handle any use case.
The video of the webinar given with these slides is at: https://youtu.be/EglSYFodaqw
2. 2
Who We Are
The open source home and commercial provider
of Redis
Open source. The leading in-memory database
platform, supporting any high performance
OLTP or OLAP use case.
Chief Developer Advocate at Redis Labs
http://bit.ly/RedisWatch &&(curat||edit||janit||)
itamar@redislabs.com
@itamarhaber
3. 3
“He who can, does;
He who cannot,
teaches.”
– Bernard Shaw
5. 5
1.Redis: REmote DIctionary Server
2./ rɛdɪs/: “red-iss”
3.OSS: http://github.com/antirez/redis
4.3-clause BSD-license: http://redis.io
5.In-memory: (always) read from RAM
6.A database for: 5 data structures
7.And: 4 (+1) more specialized ones
6. 6
8.Developed & maintained: (mostly)
Salvatore Sanfilippo (a.k.a. @antirez)
and his OSS team at @RedisLabs
9.Short history:v1.0 August 9th, 2009
… v3.2 May 6th, 2016
10.“The Leatherman™ of Databases”:
mostly used as a DB, cache & broker
7. 7
11.A couple or so of extra features:
(a) atomicity; (b) blocking wait;
(c) configurable persistence;
(d) data expiration and (e) eviction;
as well as transactions, PubSub, Lua
scripts, high availability & clustering
12.Next version (v4.0): MODULES!
9. 9
Redis 101
1. Redis is “NoSQL”
0. No (explicit) schema, access by key
1. Key -> structure -> data
SIMPL-ICI-TY: simple, I see it, thank you
10. 10
Redis data strata
v1.0 Strings
Lists
Sets
v1.2 Sorted Sets
v2.0 Hashes
v2.2 Bit arrays
v2.8.9 HyperLogLog
v3.2 Geo Sets
Bit fields
v4 Streams (?)
MODULES!
11. 11
How to Redis in 3 steps:
1. 147 OSS clients in 49 languages, e.g:
Java, Node.js, .NET, Python, Ruby…
2. You make a request, i.e.:
PING
3. The server replies, i.e.g:
PONG
14. 14
Flexibility: model (almost) anything
with basic “building blocks” and simple
rules (v0.0.1)
Composability: transactions (v1.2) and
server-embedded scripted logic (v2.6)
Extensibility: modules (v4) for adding
custom data structures and commands
16. 16
First mentioned in release v1.0
https://groups.google.com/forum/#!msg/redis-db/Z0aiVSRAnRU/XezAFFtgyPUJ
“Another interesting idea is
to add support for plugins
implementing specific
commands and associated
data types, and the
embedding of a scripting
language.”
17. 17
Redis before modules:
1. Redis is ubiquitous for fast data, fits
lots of cases (Swiss™ Army knife)
2. Some use cases need special care
3. Open source has its own agenda
So what can you do? FR, PR or fork
18. 18
Redis with modules:
1. Core still fits lots of cases
2. Module extensions for special cases
3. A new community-driven ecosystem
4. “Give power to users to go faster”
What to expect? Nothing’s impossible!
19. 19
Redis modules are:
1. Dynamically (server-)loaded libraries
2. Future-compatible
3. (will be mostly) written in C
4. (nearly) as fast as the core
5. Planned for public release Q3 2016
20. 20
Modules let you:
1. Process: where the data is at
2. Compose: call core & other modules
3. Extend: new structures, commands
4. (planned) Time & keyspace triggers
5. (also) Blocking custom commands
6. (and) Cross-cluster parallelization
23. 23
The API
1. Where most of the effort was made
2. Abstracts & isolates Redis’ internals
3. The server’s (C-) binding contract
4. Will not be broken once released
5. Exposes three conceptual layers
24. 24
Modules API layers
1.Operational: admin, memory, disk,
replication, arguments, replies…
2.High-level: client-like access to core
and modules’ commands
3.Low-level: (almost) native access to
core data structures memory
28. 28
RedisModuleCtx *ctx
1.The module’s call execution context
2.Used by most calls to the API, just
pass it along
3.A black box: internal housekeeping
structure for tracking memory
allocations, objects, opened keys…
29. 29
if (RedisModule_Init(ctx, "example", 1,
REDISMODULE_APIVER_1) == REDISMODULE_ERR)
return REDISMODULE_ERR;
if (RedisModule_CreateCommand(ctx, "example.echo",
Echo, "readonly", 1, 1, 1) == REDISMODULE_ERR)
return REDISMODULE_ERR;
register the
command
register the module
or die trying
30. 30
int Echo(RedisModuleCtx *ctx,
RedisModuleString **argv, int argc)
if (argc != 2) return RedisModule_WrongArity(ctx);
return RedisModule_ReplyWithString(ctx,argv[1]);
validate number
of arguments
&err if needed
arguments
&count
send back
the argument
31. 31
RedisModule_ReplyWith
• Error – duh
• Null – no words
• LongLong – integer
• String – also Simple and Buffer
• Array – Redis array (can be nested)
• CallReply – High-Level API reply
33. 33
RedisModule_Call(…)
• Does: runs a command
• Expects: context, command name,
printf-like format and arguments
• Returns: RedisModuleCallReply *
• Not unlike: Redis’ Lua redis.call
34. 35
int Educational_HighLevelAPI_Echo(RedisModuleCtx *ctx,
RedisModuleString **argv, int argc) {
if (argc != 2) return RedisModule_WrongArity(ctx);
RedisModule_AutoMemory(ctx);
RedisModuleCallReply *rep = RedisModule_Call(ctx,
"ECHO", "s", argv[1]);
return RedisModule_ReplyWithCallReply(ctx, rep);
}
Using the High-Level API to call
the Redis core ‘ECHO’ command...
...is impractical but educational :)
40. 41
With the low-level API you can:
• Manage keys: open, close, type,
length, get/set TTL, delete…
• Manipulate core data structures:
e.g. RedisModule_StringSet(…),
RedisModule_ListPop(…) and
RedisModule_Zset*Range(…)
41. 42
• Fine tune replication:
RedisModule_Replicate*(…)
• Directly access String memory:
RedisModule_StringDMA(…)
• Register custom data types:
RedisModule_CreateDataType(…)
• And much more but…
42. 43
Build it
• Get Redis unstable version
• Read the docs
• You can also use the Redis Labs
Modules SDK to jumpstart:
https://github.com/RedisLabs/Redis
ModulesSDK
47. 48
There are three kinds
of people in the world;
those who can count
and those who can’t.
48. 49
There are three kinds
of data structures…
…and those who both
can count and can’t.
49. 50
Data Structures of the 3rd kind
• Why: accuracy is (in theory) possible
but scale makes it (nearly) impossible
• Example: number of unique visitors
• Alternative: estimate the answer
• Data structure: the HyperLogLog
• Ergo: modules as models for PDSs
50. 51
The “good” PDSs are
1. Efficient: sublinear space-time
2. Accurate: within their parameters
3. Scalable: by merging et al.
4. Suspiciously not unlike: the Infinite
Improbability Drive (The Hitch Hiker
Guide to the Galaxy, Adams D.)
51. 52
Top-K - k most frequent samples
The entire algorithm:
1. Maintain set S of k counters
2. For each sample s:
2.1 If s exists in S, increment S(x)
2.1 Otherwise, if there’s space add x
to S , else decrement all counters
52. 53
Modeling Top-K with Redis
1. Sorted Set -> unique members
2. Member -> element and score
3. ZSCORE: O(1) membership
4. ZADD: O(Log(N)) write
5. ZRANGEBYSCORE: O(Log(N)) seek
Can this be moduled?
53. 54
redis> TOPK.ADD tk 2 a
(integer) 1
redis> TOPK.ADD tk 2 b
(integer) 1
redis> TOPK.ADD tk 2 b
(integer) 0
redis> ZRANGE tk 1 -1 WITHSCORES
1) "a"
2) "1"
3) "b"
4) "2"
redis> TOPK.ADD tk 2 c
(integer) -1
max items (a.k.a k)
the sample
score
1 means added
0 is freq. incr.
indicates eviction
55. 56
topk Redis Module
1.Optimization: a global score offset
2.Eviction: reservoir sampling
3.TOPK.PRANK: percentile rank
4.TOPK.PRANGE: percentile range
5.Where: Redis Module Hub/topk
56. 57
Bloom filter – set membership
1.Answers: “have I seen this?”
2.Good for: avoiding hard work
3.Promises: no false negatives
4.Sometimes: false positives (error)
5.Gist: hash values of the samples are
indexes in an array of counters
57. 58
redis> CBF.ADD bloom a
(integer) 1
redis> CBF.ADD bloom b
(integer) 2
redis> CBF.CHECK bloom a
(integer) 1
redis> CBF.CHECK bloom b
(integer) 1
redis> CBF.CHECK bloom c
(integer) 0
0 1 0 21 0
h1(a), h2(a)
h1(b), h2(b)h1(c), h2(c)
62. 63
Count Min Sketch - item counts
1.Unlike Top-K:
answers about any sample
2.WRT Bloom filters
Like: hashes as indexes to counters
Unlike: array per hash function,
returns the minimum of counters
63. 64
redis> CMS.INCRBY count a 1 b 2
OK
redis> CMS.QUERY count b
(integer) 2
0 1 0 00 2 h1
0 0 0 03 0 h2
collision
min[h1(b), h2(b)]
hi(b) hi(b)
65. 66
tdigest Redis Module
1.Purpose: streaming quantiles
2.Beauty: tiny, fast and parallelizable
3.Award: 1st community data type
4.Author: Usman Masood
5.Redis Module Hub/tdigest