Learn how to develop and deploy embedded Spark analytics running within Redis on an edge IoT device.

1. PRESENTED BY
Developing and Deploying
Edge Analytics
David Rauschenbach
Nubix.io, CTO

2. PRESENTED BY
1 IoT and Edge Architectures
A brief overview of Enterprise Architectures with Cloud, Fog, and Edge components
2 Open Source Projects Used
A brief overview of the open source projects used in this demo and links to them
3 Demo: Streaming Data into Redis on Edge Devices
A stream of sensor data will be collected and stored in Redis on an edge device.
Agenda:
4 Demo: Running Spark on the Edge with Redis Integration
A simple analytic will be developed using Spark & Redis, deployed to the edge, and executed

3. PRESENTED BY
Device Edge
Casually connected
Edge Architectures
Fog
Commonly air-gapped
Server Edge
Tethered

4. PRESENTED BY
Why are Analytics Required at the Source of Data?
Because:
Sense, Infer, Act
“Return to the Edge” (2016)
by Peter Levine, Andreessen Horowitz

5. PRESENTED BY
Cloud and Device Edge: Two Worlds Under Pressure to Meet
Devices
•Under pressure to add intelligence and inference
•Which requires frequent updates (a tectonic shift for
the embedded world)
•Which requires best-of-breed cloud workflows like
aPaaS, IaaS, and Continuous Integration
Cloud
•Under pressure to realize the final 92% of Digital
Transformation, which hasn’t happened yet
because products live outside of datacenters
•Data Scientists don’t use desktop power supplies
and serial cables
•Don’t have tooling to program in 32k of RAM

6. PRESENTED BY
Open Source Projects We Are Going to Use

7. PRESENTED BY
• Use your favorite LuaRocks modules within Redis
• GitHub: github.com/BixData/lua-amalg-redis
• Blog: https://medium.com/nubix-open-source-edge-computing/introducing-the-
lua-amalgamator-for-redis-c498434c2154
Open Source #1: Lua Amalgamator for Redis

8. PRESENTED BY

9. PRESENTED BY
Moses: the Lodash / Underscore of the Lua ecosystem
Using Moses within Redis
$ luarocks install moses
$ vi main.lua
local moses = require 'moses'
local data = {2, 3, 5, 7, 11, 13, 17, 19, 23, 29}
local sum = moses.reduce(data, function(r, n)
return r + n
end)
print(string.format('Sum of first %d primes is %d', #data, sum))
return sum
$ lua –lamalg-redis main.lua
Sum of first 10 primes is 129
$ amalg-redis.lua –s main.lua –o main-with-dependencies.lua –c
$ redis-cli --eval main-with-dependencies.lua 0,0
(integer) 129

10. PRESENTED BY
Open Source #2: Stuart

11. PRESENTED BY
Using Spark ML Vectors within Redis
$ luarocks install stuart-ml
$ vi main.lua
local BLAS = require 'stuart-ml.linalg.BLAS'
local Vectors = require 'stuart-ml.linalg.Vectors'
local a = Vectors.dense({1,2,3})
local b = Vectors.dense({4,-5,6})
local dotProduct = BLAS.dot(a, b)
local isAcuteAngle = dotProduct > 0
print(string.format('The dot product of [{1,2,3}, {4,-5,6}] is %d, which %s an acute angle',
dotProduct, isAcuteAngle and 'is' or 'is not'))
return dotProduct
$ lua –lamalg-redis main.lua
The dot product of [{1,2,3}, {4,-5,6}] is 12, which is an acute angle
$ amalg-redis.lua –s main.lua –o main-with-dependencies.lua –c
$ redis-cli --eval main-with-dependencies.lua 0,0
(integer) 12

12. PRESENTED BY
Using Spark ML Matrices within Redis
$ vi main.lua
local Matrices = require 'stuart-ml.linalg.Matrices'
local matrix = Matrices.sparse(3, 2, {0, 2, 3}, {0, 2, 1}, {0.0, -1.2, 0.0})
local msg = string.format('The sparse matrix %s has %d non-zeros and %d actives',
'(3, 2, {0, 2, 3}, {0, 2, 1}, {0.0, -1.2, 0.0})',
matrix:numNonzeros(),
matrix:numActives())
print(msg)
return msg
$ lua –lamalg-redis main.lua
The sparse matrix (3, 2, {0, 2, 3}, {0, 2, 1}, {0.0, -1.2, 0.0}) has 1 non-zeros and 3 actives
$ amalg-redis.lua –s main.lua –o main-with-dependencies.lua –c
$ redis-cli --eval main-with-dependencies.lua 0,0
"The sparse matrix (3, 2, {0, 2, 3}, {0, 2, 1}, {0.0, -1.2, 0.0}) has 1 non-zeros and 3 actives"

13. PRESENTED BY
Using Spark ML K-means Clustering within Redis
$ vi main.lua
local KMeans = require 'stuart-ml.clustering.KMeans'
local Vectors = require 'stuart-ml.linalg.Vectors'
local VectorWithNorm = require 'stuart-ml.clustering.VectorWithNorm'
local centers = {
VectorWithNorm.new(Vectors.dense(1,2,6)),
VectorWithNorm.new(Vectors.dense(5,3,9)),
VectorWithNorm.new(Vectors.dense(9,4,7))
}
local point = VectorWithNorm.new(Vectors.dense(6,2,5))
local bestIndex, bestDistance = KMeans.findClosest(centers, point)
local msg = string.format('The point %s is closest to cluster center #%d, with a distance of %d',
tostring(point), bestIndex, bestDistance)
print(msg)
return msg
$ lua –lamalg-redis main.lua
The point ((6,2,5),8.0622577482985) is closest to cluster center #3, with a distance of 17
$ amalg-redis.lua –s main.lua –o main-with-dependencies.lua –c
$ redis-cli --eval main-with-dependencies.lua 0,0
"The point ((6,2,5),8.0622577482985) is closest to cluster center #3, with a distance of 17"

14. PRESENTED BY
Open Source #3: Stuart-Redis

15. PRESENTED BY
Reading+Writing Spark RDDs within Redis$ luarocks install stuart-redis
$ vi main.lua
local stuart = require 'stuart'
local stuartRedis = require 'stuart-redis'
local function split(str, pattern)
local res = {}
for s in string.gmatch(str, pattern) do table.insert(res, s) end
return res
end
local sc = stuart.NewContext()
sc = stuartRedis.export(sc)
-- writing
local doc = [[I mean, think about music. Music is all about repetition and patterns.
If you didn’t have repetition in music, it would all just be noise.]]
local words = sc:parallelize(split(doc, '%w+'))
local wordCountsRDD = words
:map(function(word) return {string.lower(word), 1} end)
:reduceByKey(function(r, x) return r+x end)
:map(function(e) return {e[1], e[2]} end)
sc:toRedisZSET(wordCountsRDD, 'wordCounts')
-- reading
local wordFrequencies = sc:fromRedisZSetWithScore('wordCounts')
:map(function(e) return e[1] .. '=' .. e[2] end)
:collect()
local msg = table.concat(wordFrequencies, ', ')
print(msg)
return msg

16. PRESENTED BY
Reading+Writing Spark RDDs within Redis (continued)
$ redis-cli
monitor
1554135760.154290 [0 127.0.0.1:64608] "EVAL" ”(5,300 lines, 177k not shown)"
1554136904.716402 [0 lua] "ZADD" "wordCounts" "1" "it"
1554136904.716415 [0 lua] "ZADD" "wordCounts" "1" "didn"
1554136904.716420 [0 lua] "ZADD" "wordCounts" "1" "mean"
1554136904.716425 [0 lua] "ZADD" "wordCounts" "1" "is"
1554136904.716429 [0 lua] "ZADD" "wordCounts" "2" "repetition"
1554136904.716435 [0 lua] "ZADD" "wordCounts" "1" "i"
1554136904.716439 [0 lua] "ZADD" "wordCounts" "1" "think"
1554136904.716444 [0 lua] "ZADD" "wordCounts" "1" "would"
1554136904.716449 [0 lua] "ZADD" "wordCounts" "1" "noise"
1554136904.716453 [0 lua] "ZADD" "wordCounts" "1" "just"
1554136904.716458 [0 lua] "ZADD" "wordCounts" "1" "t"
1554136904.716463 [0 lua] "ZADD" "wordCounts" "1" "if"
1554136904.716467 [0 lua] "ZADD" "wordCounts" "1" "you"
1554136904.716472 [0 lua] "ZADD" "wordCounts" "1" "in"
1554136904.716506 [0 lua] "ZADD" "wordCounts" "1" "patterns"
1554136904.716511 [0 lua] "ZADD" "wordCounts" "1" "be"
1554136904.716515 [0 lua] "ZADD" "wordCounts" "1" "have"
1554136904.716520 [0 lua] "ZADD" "wordCounts" "2" "about"
1554136904.716525 [0 lua] "ZADD" "wordCounts" "2" "all"
1554136904.716530 [0 lua] "ZADD" "wordCounts" "1" "and"
1554136904.716534 [0 lua] "ZADD" "wordCounts" "3" "music"
1554136904.716541 [0 lua] "KEYS" "wordCounts"
1554136904.716846 [0 lua] "TYPE" "wordCounts"
1554136904.717252 [0 lua] "ZRANGE" "wordCounts" "0" "-1" "WITHSCORES"
$ lua –lamalg-redis main.lua
and=1, be=1, didn=1, have=1, i=1, if=1,
in=1, is=1, it=1, just=1, mean=1,
noise=1, patterns=1, t=1, think=1,
would=1, you=1, about=2, all=2,
repetition=2, music=3
(… manual edits to amalg.cache …)
$ amalg-redis.lua –s main.lua –o main-
with-dependencies.lua –c
$ redis-cli --eval main-with-
dependencies.lua 0,0
"and=1, be=1, didn=1, have=1, i=1, if=1,
in=1, is=1, it=1, just=1, mean=1,
noise=1, patterns=1, t=1, think=1,
would=1, you=1, about=2, all=2,
repetition=2, music=3"

17. PRESENTED BY
Hardware
• Raspberry Pi 3 B+
• BME280 Atmospheric Sensor
• I²C Communication Bus (shown in
orange+yellow)
Software
• Raspbian OS
• Redis 5.0
• Nubix Agent
DEMO 1: Streaming Data into Redis on an Edge Device

18. PRESENTED BY
Hardware
• Raspberry Pi 3 B+
• BME280 Atmospheric Sensor
• I²C Communication Bus (shown in
orange+yellow)
Software
•Redis 5.0 (unmodified, no
plugins)
• Lua submitted via standard
redis-cli
DEMO 2: Spark Analytics within Redis on Edge Device

19. Thank you!
Code examples available at:
https://github.com/BixData/redisconf19

20. PRESENTED BY