Successfully reported this slideshow.
Your SlideShare is downloading. ×

Distributed Data Flow for the Web of Things: Distributed Node-RED

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Loading in …3
×

Check these out next

1 of 15 Ad

Distributed Data Flow for the Web of Things: Distributed Node-RED

Download to read offline

Presentation at the Web of Things Workshop at the Iot 2014 conference at MIT on our proposal to create a distributed data flow architecture where sub flows are distributed between servers, gateways and devices

Presentation at the Web of Things Workshop at the Iot 2014 conference at MIT on our proposal to create a distributed data flow architecture where sub flows are distributed between servers, gateways and devices

Advertisement
Advertisement

More Related Content

Slideshows for you (20)

Similar to Distributed Data Flow for the Web of Things: Distributed Node-RED (20)

Advertisement

Recently uploaded (20)

Advertisement

Distributed Data Flow for the Web of Things: Distributed Node-RED

  1. 1. Toward a Distributed Data Flow Platform for the Web of Things (Distributed Node-RED) Michael Blackstock, Rodger Lea Human Communication Technologies Lab Electrical and Computer Engineering Department University of British Columbia
  2. 2. Motivation • Common IoT scenarios require real time interaction between things, but application development can be difficult: • integration, variety of protocols and APIs and programming environments. • New web-based tools and platforms have emerged for rapid development e.g. WoTKit and Node-RED • Current platforms host applications on a single device/ server. • Exploration of distributed data flow for the Web of Things 2
  3. 3. Data Flow Architectures • Initially to support massive parallelism, then software engineering • natural visual representation: nodes and arcs. • move more easily between design and implementation • can reduce development time • some evidence that they are easier for non-programmers 3
  4. 4. Server Distributed Data Flow Twitter Input Command Parser Presence Sensor TV Controller Text Filter User Sensor Presence Monitor 4 Command Receiver TV Controller
  5. 5. WoTKit Processor • Multi user service, bundled with WoTKit platform* • Real time data processing, alerting • Browser-based visual pipe editor: modules are connected by wires to form pipes • Pipe management page to manage execution. • Separate script editor window for creating new modules. * Blackstock, M.; Lea, R., "IoT mashups with the WoTKit," Internet of Things (IOT) Conference, 2012, 24-26 Oct. 2012, Wuxi, China 5
  6. 6. WoTKit Processor • Input modules • Sensor, Twitter inputs • Functions • Threshold, Scripting • Outputs • Debugging/Monitoring • Email, Actuator control • Script modules interact with flow using input, output and module state dictionaries. 6
  7. 7. Processor Architecture • ‘Pipes’ are stored as JSON in database associated with users. • On ‘start’, JSON pipe is parsed, module instances (beans) are created in memory, wires added to global Routing Table. • Input modules add messages to execution queue. • Scheduler looks up downstream module endpoint from endpoint ‘Routing Table’. • Creates task using module (bean) and message for execution using thread pool. • Pipes from multiple users can be added or removed from scheduler and routing table any time without restarting. 7
  8. 8. Node-RED
  9. 9. Node RED Architecture Nodes • Single flow stored per instance in file system • On deploy, JSON flow is parsed, nodes instantiated and wired up directly. • Each node has a connection to downstream nodes. Node.js EventEmitter API to manage listeners. • By calling EventEmitter API, rather than downstream node directly, uses Node.js’ single threaded Event Loop. • Worker threads are created for long running async operations (e.g. I/O). • All ‘tabs’ are part of a single flow. 1. lookup next node(s) 9 Nodes Nodes 2. emit events Event Loop (single thread) 5. return results 4. return results 3. do work Non-blocking worker threads Based on http://www.aaronstannard.com/post/ 2011/12/14/Intro-to-NodeJS-for-NET-Developers.aspx
  10. 10. Analysis User Interface! • Both have an easy to use web based visual programming environment • Multi-user system separates pipe/flow management from editor. • Both use monitor nodes to view execution Architecture! • Need a way to manage, stop and start pipes in a multi-user system. • Need system that is suitably ‘light weight’ for small devices. • nodes may have state; Node-RED supports global flow state. • Neither are distributed Extensibility/Community/Sharing! • should be easy to extend. • Sharing pipes/flows and modules/nodes is critical to support community of users 10
  11. 11. Distributed Data Flow Model • Multi-user system - flows and devices need ‘owners’ to be managed separately, especially in shared cloud services. • Should be able to associate sub flows with devices. • ‘Server’ nodes must run on a server, ‘Device’ nodes on a specific device, ‘Mobile’ nodes on most appropriate device. • Local wires for nodes on same device, remote wires between devices. • Need easy way to distribute (sub)flows between devices. • Ideally, should be able to visualize these extensions and execution. 11
  12. 12. Work To Date 12 Device boxes visualize device association Device Selector to specify device sub flows Import Flow from Master Device Extend all nodes with deviceId. Extend flow parser to insert ‘remote wires’ between devices
  13. 13. Flow Deployment 13 MQTT Broker Server send to ‘remote wire’ topic Presence Sensor send to ‘remote wire’ topic listen on ‘remote wire’ Node topic TV Controller remote wire topic is generated using connected node ids
  14. 14. Conclusions • Processor, Node-RED and others demonstrate that visual data flows are a useful programming model for developing IoT applications. • Web based distributed data flow promising for coordinating multiple ‘things’. • Raises many questions around programming model, implementation, security and heuristics. • Looking for collaborators! 14
  15. 15. Thanks • Code: https://github.com/mblackstock/node-red (default experimental branch) • Node-RED: IBM and authors/contributors ! ! ! http://nodered.org/! • WoTKit Processor: Sense Tecnic Systems ! ! ! http://sensetecnic.com 15

×