This document discusses the potential for distributed data flow platforms to support Internet of Things application development. It summarizes two existing platforms, WoTKit and Node-RED, and proposes a model for distributing data flows across devices. Key aspects include associating subflows with devices, using remote wires to connect nodes across devices, and deploying flows to devices over MQTT. The goal is to allow visual programming of IoT applications that can coordinate actions between multiple physical things.

1. Toward a Distributed Data Flow
Platform for the Web of Things
(Distributed Node-RED)
Michael Blackstock, Rodger Lea
Human Communication Technologies Lab
Electrical and Computer Engineering Department
University of British Columbia

2. Motivation
• Common IoT scenarios require real time interaction between
things, but application development can be difficult:
• integration, variety of protocols and APIs and
programming environments.
• New web-based tools and platforms have emerged for
rapid development e.g. WoTKit and Node-RED
• Current platforms host applications on a single device/
server.
• Exploration of distributed data flow for the Web of Things
2

3. Data Flow Architectures
• Initially to support massive parallelism, then
software engineering
• natural visual representation: nodes and arcs.
• move more easily between design and
implementation
• can reduce development time
• some evidence that they are easier for non-programmers
3

4. Server
Distributed Data Flow
Twitter Input Command Parser
Presence Sensor
TV Controller
Text Filter
User Sensor Presence Monitor
4
Command
Receiver
TV Controller

5. WoTKit Processor
• Multi user service, bundled
with WoTKit platform*
• Real time data processing,
alerting
• Browser-based visual pipe
editor: modules are connected
by wires to form pipes
• Pipe management page to
manage execution.
• Separate script editor window
for creating new modules.
* Blackstock, M.; Lea, R., "IoT mashups with the WoTKit," Internet
of Things (IOT) Conference, 2012, 24-26 Oct. 2012, Wuxi, China
5

6. WoTKit Processor
• Input modules
• Sensor, Twitter inputs
• Functions
• Threshold, Scripting
• Outputs
• Debugging/Monitoring
• Email, Actuator control
• Script modules interact with
flow using input, output and
module state dictionaries.
6

7. Processor Architecture
• ‘Pipes’ are stored as JSON in database
associated with users.
• On ‘start’, JSON pipe is parsed, module
instances (beans) are created in memory,
wires added to global Routing Table.
• Input modules add messages to
execution queue.
• Scheduler looks up downstream module
endpoint from endpoint ‘Routing Table’.
• Creates task using module (bean) and
message for execution using thread pool.
• Pipes from multiple users can be added
or removed from scheduler and routing
table any time without restarting.
7

8. Node-RED

9. Node RED Architecture
Nodes
• Single flow stored per instance in file
system
• On deploy, JSON flow is parsed, nodes
instantiated and wired up directly.
• Each node has a connection to
downstream nodes. Node.js
EventEmitter API to manage listeners.
• By calling EventEmitter API, rather than
downstream node directly, uses
Node.js’ single threaded Event Loop.
• Worker threads are created for long
running async operations (e.g. I/O).
• All ‘tabs’ are part of a single flow.
1. lookup
next
node(s)
9
Nodes
Nodes
2. emit events
Event Loop
(single thread)
5. return results
4. return
results
3. do work
Non-blocking worker
threads
Based on http://www.aaronstannard.com/post/
2011/12/14/Intro-to-NodeJS-for-NET-Developers.aspx

10. Analysis
User Interface!
• Both have an easy to use web based visual programming environment
• Multi-user system separates pipe/flow management from editor.
• Both use monitor nodes to view execution
Architecture!
• Need a way to manage, stop and start pipes in a multi-user system.
• Need system that is suitably ‘light weight’ for small devices.
• nodes may have state; Node-RED supports global flow state.
• Neither are distributed
Extensibility/Community/Sharing!
• should be easy to extend.
• Sharing pipes/flows and modules/nodes is critical to support
community of users
10

11. Distributed Data Flow Model
• Multi-user system - flows and devices need ‘owners’ to be
managed separately, especially in shared cloud services.
• Should be able to associate sub flows with devices.
• ‘Server’ nodes must run on a server, ‘Device’ nodes on a specific
device, ‘Mobile’ nodes on most appropriate device.
• Local wires for nodes on same device, remote wires between
devices.
• Need easy way to distribute (sub)flows between devices.
• Ideally, should be able to visualize these extensions and
execution.
11

12. Work To Date
12
Device boxes
visualize device
association
Device Selector to
specify device sub flows Import Flow from
Master Device
Extend all nodes
with deviceId.
Extend flow parser to
insert ‘remote wires’
between devices

13. Flow Deployment
13
MQTT Broker
Server
send to ‘remote
wire’ topic
Presence Sensor
send to ‘remote wire’
topic
listen on ‘remote
wire’ Node topic
TV Controller
remote wire topic is generated
using connected node ids

14. Conclusions
• Processor, Node-RED and others demonstrate that
visual data flows are a useful programming model
for developing IoT applications.
• Web based distributed data flow promising for
coordinating multiple ‘things’.
• Raises many questions around programming
model, implementation, security and heuristics.
• Looking for collaborators!
14

15. Thanks
• Code: https://github.com/mblackstock/node-red
(default experimental branch)
• Node-RED: IBM and authors/contributors
! ! ! http://nodered.org/!
• WoTKit Processor: Sense Tecnic Systems
! ! ! http://sensetecnic.com
15