Slides deck for kicking off Redis Labs' Modules Hackathon - https://www.hackerearth.com/sprints/redislabs-hackathon-global Video of the webinar is at: https://youtu.be/LPxx4QPyUPw

1. Developing a Redis Module
Itamar Haber

2. 2
Who We Are
The open source home and commercial provider
of Redis
Open source. The leading in-memory database
platform, supporting any high performance
OLTP or OLAP use case.
Chief Developer Advocate at Redis Labs
itamar@redislabs.com
@itamarhaber

3. 3
Redis Modules Hackathon
http://bit.ly/redishack
• Opens today, ends on November 12th
• Global & local events (SF and TLV)
• Submit your Redis module to participate
• Win $10K in cash prizes! Gain world fame!

4. ~10 Things About Redis

5. 5
1.Redis: REmote DIctionary Server
2./ rɛdɪs/: “red-iss”
3.OSS: http://github.com/antirez/redis
4.3-clause BSD-license: http://redis.io
5.In-memory: all in RAM (Flash option)
6.A database for: ~9 data structures
7.And: 4 (+1) more specialized ones

6. 6
8.Developed & maintained: (mostly)
Salvatore Sanfilippo (a.k.a. @antirez)
and his OSS team at @RedisLabs
9.History: v1.0 August 9th, 2009
… v3.2.4 September 26th, 2016
10.“The Leatherman™ of Databases”:
mostly used as a DB, cache & broker

7. 7
11.A couple or so of extra features:
(a) atomicity; (b) blocking wait;
(c) configurable persistence;
(d) data expiration and (e) eviction;
as well as transactions, PubSub, Lua
scripts, high availability & clustering
12.Next version (v4.0): MODULES!

8. 8
Redis 101
1. Redis is “NoSQL”
0. No (explicit) schema, access by key
1. Key -> structure -> data

9. 9
Redis data strata
v1.0 Strings
Lists
Sets
v1.2 Sorted Sets
v2.0 Hashes
v2.2 Bit arrays
v2.8.9 HyperLogLog
v3.2 Geo Sets
Bit fields
v4 Neural Network
MODULES!

10. 10
How to Redis in 3 steps:
1. 150 OSS clients in 50 languages, e.g:
Java, Node.js, .NET, Python, Ruby…
2. You make a request, i.e.:
PING
3. The server replies, i.e.g:
PONG

11. 11
~$ redis-cli
127.0.0.1:6379> SET counter 1
OK
127.0.0.1:6379> GET counter
"1"
127.0.0.1:6379> INCRBY counter 1
(integer) 2
127.0.0.1:6379> APPEND counter b||!2b
(integer) 7
127.0.0.1:6379> GETSET counter "x00HelloxffWorld"
"2b||!2b"
127.0.0.1:6379>

12. The Evolution of Versatility

13. 13
Flexibility: model (almost) anything
with basic “building blocks” and simple
rules (v0.0.1)
Composability: transactions (v1.2) and
server-embedded Lua scripts (v2.6)
Extensibility: modules (v4) for adding
custom data structures and commands

14. MODULES! (a.k.a plugins)

15. 15
First mentioned in release v1.0
https://groups.google.com/forum/#!msg/redis-db/Z0aiVSRAnRU/XezAFFtgyPUJ
“Another interesting idea is
to add support for plugins
implementing specific
commands and associated
data types, and the
embedding of a scripting
language.”

16. 16
Redis before modules:
1. Redis is ubiquitous for fast data, fits
lots of cases (Swiss™ Army knife)
2. Some use cases need special care
3. Open source has its own agenda
So what can you do? FR, PR or fork

17. 17
Redis with modules:
1. Core still fits lots of cases
2. Module extensions for special cases
3. A new community-driven ecosystem
4. “Give power to users to go faster”
What to expect? Nothing’s impossible!

18. 18
Redis modules are:
1. Dynamically (server-)loaded libraries
2. Future-compatible
3. (will be mostly) written in C
4. (nearly) as fast as the core
5. Planned for public release Q3 2016

19. 19
Modules let you:
1. Process: where the data is at
2. Compose: call core & other modules
3. Extend: new structures, commands

20. 20
redis> ECHO "Alpha"
"Alpha"
redis> MODULE LOAD example.so
OK
redis> EXAMPLE.ECHO "Bravo"
"Bravo"
redis> ^C
~$ wc example.c
13 46 520 example.c
~$ gcc -fPIC -std=gnu99 -c -o example.o example.c
~$ ld -o example.so example.o -shared -Bsymbolic -lc
core command
module library
“new” command

21. 21
Some exemplary modules:
• Neural Redis – a neural network
• RediSearch – full text search
• countminsketch – streaming counter
• redablooms – Bloom filters
• redis-tdigest – rank estimator
• More at the hub: redismodules.com

22. Redis Modules API

23. 23
The API
1. Where most of the effort was made
2. Abstracts & isolates Redis’ internals
3. The server’s (C-) binding contract
4. Will not be broken once released
5. Exposes three conceptual layers

24. 24
Modules API layers
1.Operational: admin, memory, disk,
replication, arguments, replies…
2.High-level: client-like access to core
and modules’ commands
3.Low-level: (almost) native access to
core data structures memory

25. ~$ cat example.c: operational-API-only example

26. 26
#include "redismodule.h"
int Echo(RedisModuleCtx *ctx,
RedisModuleString **argv, int argc) {
if (argc != 2) return RedisModule_WrongArity(ctx);
return RedisModule_ReplyWithString(ctx,argv[1]); }
int RedisModule_OnLoad(RedisModuleCtx *ctx) {
if (RedisModule_Init(ctx, "example", 1,
REDISMODULE_APIVER_1) == REDISMODULE_ERR)
return REDISMODULE_ERR;
if (RedisModule_CreateCommand(ctx, "example.echo",
Echo, "readonly", 1, 1, 1) == REDISMODULE_ERR)
return REDISMODULE_ERR;
return REDISMODULE_OK; }

27. 27
#include "redismodule.h"
int RedisModule_OnLoad(RedisModuleCtx *ctx) {
MUST:
API definitions
MUST:
is called when
module is loaded
pointer to
context

28. 28
RedisModuleCtx *ctx
1.The module’s call execution context
2.Used by most calls to the API, just
pass it along
3.Exposes the client id if needed

29. 29
if (RedisModule_Init(ctx, "example", 1,
REDISMODULE_APIVER_1) == REDISMODULE_ERR)
return REDISMODULE_ERR;
if (RedisModule_CreateCommand(ctx, "example.echo",
Echo, "readonly", 1, 1, 1) == REDISMODULE_ERR)
return REDISMODULE_ERR;
register the
command
register the module
or die trying

30. 30
int Echo(RedisModuleCtx *ctx,
RedisModuleString **argv, int argc)
if (argc != 2) return RedisModule_WrongArity(ctx);
return RedisModule_ReplyWithString(ctx,argv[1]);
validate number
of arguments
&err if needed
arguments
&count
send back
the argument

31. 31
RedisModule_ReplyWith
• Error – duh
• Null – no words
• LongLong – integer
• String – also Simple and Buffer
• Array – Redis array (can be nested)
• CallReply – High-Level API reply

32. High-Level API

33. 33
RedisModule_Call(…)
• Does: runs a command
• Expects: context, command name,
printf-like format and arguments
• Returns: RedisModuleCallReply *
• Not unlike: Redis’ Lua redis.call

34. 35
int Educational_HighLevelAPI_Echo(RedisModuleCtx *ctx,
RedisModuleString **argv, int argc) {
if (argc != 2) return RedisModule_WrongArity(ctx);
RedisModule_AutoMemory(ctx);
RedisModuleCallReply *rep = RedisModule_Call(ctx,
"ECHO", "s", argv[1]);
return RedisModule_ReplyWithCallReply(ctx, rep);
}
Using the High-Level API to call
the Redis core ‘ECHO’ command...
...is impractical but educational :)

35. 36
RedisModule_AutoMemory(…)
Automagically manages memory
• RedisModuleCallReply *
• RedisModuleString *
• RedisModuleKey *
• RedisModule_Alloc() and family

36. High-Level Visualization Of The Low-Level API

37. 38
user app
Redis client
Redis
core
data
GET foo
"bar"
101010
010101
101010

38. 39
user
101010
010101
101010
High
level
API
app
module

39. 40
user
101010
010101
101010
app
Low
level
API

40. 41
With the low-level API you can:
• Manage keys: open, close, type,
length, get/set TTL, delete…
• Manipulate core data structures:
e.g. RedisModule_StringSet(…),
RedisModule_ListPop(…) and
RedisModule_Zset*Range(…)

41. 42
• Directly access String memory:
RedisModule_StringDMA(…)
• Register custom data types:
RedisModule_CreateDataType(…)
• Asynchronous (blocking) commands:
RedisModule_BlockClient(…)
• And much more…

42. Redis Modules Hackathon

43. 44
Redis Modules Hackathon
http://bit.ly/redishack
• Learn new stuff, help the community
• Have an amazing idea? Start coding!
• Open to individuals and teams globally
• Win $10K in cash prizes! Gain world fame!

44. 45
Developing your module
• Tools: a text editor and a compiler
• Get Redis unstable branch
• Docs and code are at src/modules
• Quick jumpstart & SDK:
https://github.com/RedisLabs/Redis
ModulesSDK

45. 46
Guidelines for a module’s API
• Keep it simple
• Make it generic (and composable)
• Implicitly manage data structures
• Commands have predetermined
space/time cost
• Inspiration: Redis’ core API

46. 47
A few more pointers
• Top performance is key
• RAM consumption is also important
• Code isn’t everything: docs, tests,
benchmarks and examples of use
• Make it useful and awesome!

47. Thank you