Redis provides several tools to achieve atomicity of operations. Single commands are atomic by default. Pipelining ensures commands are executed in order but is not fully atomic. Transactions using MULTI and EXEC are fully atomic but don't allow command chaining. Lua scripting allows complex multi-step operations to be run atomically and allows command results to be passed between operations.

1. Atomicity in RedisAtomicity in Redis
Thomas Hunter IIThomas Hunter II

2. RoadmapRoadmap
1.
2.
3.
4.
The Basics
Pipelining
Transactions
Lua
Scripting

3. The BasicsThe Basics

4. What is Atomicity?What is Atomicity?
Series of database operations
Guaranteed to all run or none run
Prevents operations from running partially
The effects of all operations are immediately visible
I.e. another client cannot see partial state
Also referred to as a Transaction

5. How do these tools work?How do these tools work?
Redis is Mostly Single-Threaded
Except for things like Background IO
Node.js is Mostly Single-Threaded
Except for IO and Node v10 Worker Threads
Single Client & Server is simple
Things get complicated with multiple clients

6. EveryEvery SingleSingle Command is AtomicCommand is Atomic

7. MultipleMultiple Commands aren't AtomicCommands aren't Atomic
Scenario: Two clients want to increment counter

8. MultipleMultiple Commands aren't AtomicCommands aren't Atomic
Client #1 reads value of counter

9. MultipleMultiple Commands aren't AtomicCommands aren't Atomic
Client #2 reads value of counter

10. MultipleMultiple Commands aren't AtomicCommands aren't Atomic
Client #1 sets value of counter to 1

11. MultipleMultiple Commands aren't AtomicCommands aren't Atomic
Client #2 sets value of counter to 1

12. Atomic, Multi-Operation CommandsAtomic, Multi-Operation Commands
Common use-cases have single-command variants
INCR key # GET key ; ~value++ ; SET key ~value
SETNX key value # !EXISTS key ; SET key value
LPUSHX key value # EXISTS key ; LPUSH key value
RPOPLPUSH src dest # RPOP src ; LPUSH dest ~value
GETSET key value # GET key ; SET key value

13. INCRINCR is an Atomic Incrementis an Atomic Increment
Scenario: Two clients want to increment counter

14. INCRINCR is an Atomic Incrementis an Atomic Increment
Client #1 atomically increments value of counter

15. INCRINCR is an Atomic Incrementis an Atomic Increment
Client #2 atomically increments value of counter

16. PipeliningPipelining

17. PipeliningPipelining
Ensures commands are run in order per-connection
Sends a batch of commands separated by newlines
Commands are sent in the same message
The Node.js redis module usually does this anyway

18. Pipelining: Example CodePipelining: Example Code
redis.batch()
.zrangebyscore('jobs', 0, now) // get jobs
.zremrangebyscore('jobs', 0, now) // delete jobs
.exec((error, data) => {
let jobList = data[0];
console.log('jobs', jobList); // perform work
});
ZRANGEBYSCORE jobs 0 1000rnZREMRANGEBYSCORE jobs 0 1000

19. Pipelining: Not Atomic, SorryPipelining: Not Atomic, Sorry
It looks atomic
Prevents command interleaving on one connection
A subset of commands can fail
Other client pipelines can interleave commands

20. Pipelining: Not Atomic, SorryPipelining: Not Atomic, Sorry
const redis = require('redis').createClient();
// run 10 instances of this process in parallel
let keys = [];
for (let i = 0; i < 100000; i++) {
keys.push(i);
}
shuffle(keys);
let pipeline = redis.batch();
for (let key of keys) {
pipeline = pipeline.hsetnx('pipeline', `k${key}`, process.pid);
}
pipeline.exec(() => redis.quit());

21. Pipelining: Not Atomic, SorryPipelining: Not Atomic, Sorry
> HGETALL pipeline
...
k46468: 25198
k67664: 25197
k62167: 25197
k5933: 25202
k19146: 25202
k202: 25196
k47418: 25198
k88650: 25202
...

22. Pipelining: What's it for?Pipelining: What's it for?
Reducing network latency
Send several commands in one message
Receive several responses in one message
echo "PINGrnPINGrnPINGrn" | nc localhost 6379
+PONG
+PONG
+PONG

23. TransactionsTransactions

24. MULTI: True AtomicityMULTI: True Atomicity
Atomic regardless of other clients / connections
Client sends MULTI , more commands, EXEC
Other clients can still run commands
Queued commands are run sequentially
Any failures and the entire transaction fails

25. MULTI: Example CodeMULTI: Example Code
redis.multi()
.zrangebyscore('jobs', 0, now) // get jobs
.zremrangebyscore('jobs', 0, now) // delete jobs
.exec((error, data) => {
let jobList = data[0];
console.log('jobs', jobList); // perform work
});
MULTI
ZRANGEBYSCORE jobs 0 1553099335332
ZREMRANGEBYSCORE jobs 0 1553099335332
EXEC

26. MULTI Drawback: No command chainingMULTI Drawback: No command chaining
Can't use command result as argument
E.g., cannot pop from list, assign to new key

27. Lua ScriptingLua Scripting

28. Lua: The Ultimate in AtomicityLua: The Ultimate in Atomicity
There's a simpler, less-efficient EVAL command
Send the entire script every time
Like sending a normal SQL query
Or use SCRIPT LOAD ahead of time
Then use EVALSHA to run code via resulting hash
Like executing a SQL Stored Procedure
Declare key names as arguments for sharding

29. Lua: Server-Side LogicLua: Server-Side Logic
Output of one command can be piped into another
Other processing can happen, too

30. Lua: Game Lobby Example CodeLua: Game Lobby Example Code
-- add-user.lua: add user to lobby, start game if 4 players
local lobby = KEYS[1] -- Set
local game = KEYS[2] -- Hash
local user_id = ARGV[1] -- String
redis.call('SADD', lobby, user_id)
if redis.call('SCARD', lobby) == 4 then
local members = table.concat(redis.call('SMEMBERS',lobby),",")
redis.call('DEL', lobby) -- empty lobby
local game_id = redis.sha1hex(members)
redis.call('HSET', game, game_id, members)
return {game_id, members}
end
return nil

31. Lua: Game Lobby Example CodeLua: Game Lobby Example Code
const redis = require('redis').createClient();
const rEval = require('util').promisify(redis.eval).bind(redis);
const script = require('fs').readFileSync('./add-user.lua');
const LOBBY = 'lobby-elo-1500', GAME = 'game-hash';
(async () => {
await rEval(script, 2, LOBBY, GAME, 'alice');
await rEval(script, 2, LOBBY, GAME, 'bob');
await rEval(script, 2, LOBBY, GAME, 'cindy');
const [gid,plyrs] = await rEval(script, 2, LOBBY, GAME,'tom');
console.log('GAME ID', gid, 'PLAYERS', plyrs.split(','));
})();

32. Lua: DrawbacksLua: Drawbacks
Another language to maintain
Simple language, easy syntax
Increases overhead on Redis server
An infinite loop could lock up server
Need to load scripts before using to be efficient
It's idempotent; load scripts when app starts

33. RecapRecap
Executing singular commands are atomic
Executing multiple commands are not atomic
Pipelining is not atomic, but it's fast
MULTI is atomic, but you can't chain results
Lua scripts are atomic and chainable

34. Intrinsic: Node.js Security PoliciesIntrinsic: Node.js Security Policies
const REDIS = 'redis://redishost:6379/1';
routes.allRoutes(policy => {
policy.redis.allowConnect(REDIS);
});
routes.get('/users/*', policy => {
policy.redis.allowCommandKey(REDIS, 'GET', 'user-*');
});
routes.post('/server/stats', policy => {
policy.redis.allowInfoSection(REDIS, 'memory');
});

35. FinFin
Follow me:
About Intrinsic:
This presentation:
@tlhunter
intrinsic.com
bit.ly/redis-atomicity
@tlhunter@tlhunter