This document discusses StreamPipes, an open source framework that allows non-technical users to connect, explore, and analyze IoT data streams. It presents StreamPipes as a self-service toolbox for IoT data analytics that handles data integration from heterogeneous sources, pre-processing, and reusable algorithm microservices. The document includes demos of StreamPipes in use cases like condition monitoring and advanced image analytics, and discusses its technical architecture, development process, and roadmap for additional features and community growth.

1. Self-Service IoT Data Analytics with
StreamPipes
Dominik Riemer, Philipp Zehnder, Patrick Wiener
ApacheCon North America, Las Vegas, 2019-09-10

2. www.streampipes.org | @streampipes | github.com/streampipes
2
What's StreamPipes?
"A self-service IoT toolbox to enable non-technical users to connect,
explore and analyze IoT data streams"

3. www.streampipes.org | @streampipes | github.com/streampipes
3
What's StreamPipes?
Big Data / Edge
InfrastructureExecute
Reusable
algorithm toolbox
Install
Model pipelines

4. www.streampipes.org | @streampipes | github.com/streampipes
About us
4
Dominik Riemer
Senior Research Scientist
Philipp Zehnder
Research Scientist
Patrick Wiener
Research Scientist
FZI Research Center for Information Technology, Karlsruhe, Germany
Stream Processing, Data Management, Machine Learning
Non-profit research center for applied ICT research (250 employees)
Started StreamPipes in 2014, first OSS release 2018

5. www.streampipes.org | @streampipes | github.com/streampipes
Agenda
The need for self-service IoT data analytics
Demo
1
StreamPipes: Technical Overview
Demo
2
Use Cases3
Getting Started & Roadmap4

6. The need for self-service IoT data analytics
1

7. www.streampipes.org | @streampipes | github.com/streampipes
Conveyor Belts
Pressure
Oil temperature
Dust particles
Production plans
Environmental Data
Gear box drive
Energy consumption
Telematics
Industrial Internet of Things
Data streams everywhere

8. Continuous Monitoring Situational Awareness
Continuous Data
Harmonization
Flexible data integration
from heterogeneous
sources and monitoring
of current system states
Detect time-critical
situations, e.g., by
means of rules or ML
approaches
Continuous pre-
processing and
transformation of input
streams for third party
systems
Industrial Internet of Things
Typical application scenarios

9. www.streampipes.org | @streampipes | github.com/streampipes
From machine to analysis
Open Source IoT projects: Great software stack to ease IIoT use cases
XX
Data Routing & Transformation
9

10. www.streampipes.org | @streampipes | github.com/streampipes
From machine to analysis
Open Source IoT projects: Positioning of StreamPipes
XX
Data Routing & Transformation
10

11. www.streampipes.org | @streampipes | github.com/streampipes
StreamPipes
Open Source framework to easily manage IoT data
Data Access
Data analytics &
harmonization
Data exploration &
exploitation
Generic adapters
Specific adapters
Metadata
Data streams & sets
Pre-processing
Filter/Aggregation
Pattern Detection
ML
Situation detection
Harmonized data sets
Visualizations
Third-party systems
11

12. Demo
First introduction to StreamPipes, Condition monitoring
Rule-based monitoring of flow rate measurements in a multi tank system

13. Demo
First introduction to StreamPipes, Condition monitoring
Rule-based monitoring of flow rate measurements in a multi tank system
Flow
Sensor
Aggregate
Measurements
Detect
Leakage
Visualize
Notify

14. Demo
Advanced analytics & image processing
Offer drinks to drowsy customers in a Las Vegas casino

15. Demo
Advanced analytics & image processing
Offer drinks to drowsy customers in a Las Vegas casino
Camera
Sensor
Detect Faces
Detect
Drowsiness
Extract Faces Notify

16. www.streampipes.org | @streampipes | github.com/streampipes
IoT landscape
Comparison to other flow-based tools
More focused on dev people
JS runtime
Data pre-processing / ETL
16

17. Technical Overview
17
3

18. www.streampipes.org | @streampipes | github.com/streampipes
High-level architecture
Analytics Microservices
Data Integration
Data Sources
Adapter Library
Pipeline Editor
Streaming Engine
18

19. www.streampipes.org | @streampipes | github.com/streampipes
High-level architecture
Analytics Microservices
Data Integration
Data Sources
Adapter Library
Pipeline Editor
Streaming Engine
19

20. Data Access
StreamPipes Connect: Easily connect IoT sources

21. www.streampipes.org | @streampipes | github.com/streampipes
Data Access
StreamPipes Connect: Easily connect IoT sources
• Easily connect data streams
and data sets (sensor + image
data)
• Generic adapters for frequently
used protocols (e.g., REST,
MQTT, OPC-UA, ROS) and
formats (e.g., JSON, XML)
• Semantic metadata (used for
pipeline verification)
• Automatic schema recognition
Features
21

22. Demo
Connect & harmonize data with StreamPipes Connect
Building a harmonized data lake

23. Demo
Connect & harmonize data with StreamPipes Connect
Building a harmonized data lake
MQTT Flow
Sensor
IoTDB
StreamPipes Connect

24. www.streampipes.org | @streampipes | github.com/streampipes
Data Access
Machine-interpretable metadata
100
011
010
001
010
010
100
101
000
111
data stream
{
"tstamp": 1453478160,
"machineId": "ID5",
"temperature": 73.5,
"flowRate": 4.2
}
Semantic
metadata
Data type, runtime name,
semantic type
Frequency, latency,
measurement unit
Format, Protocol
Schema
Quality
Grounding
24

25. www.streampipes.org | @streampipes | github.com/streampipes
Data Access
Machine-interpretable metadata
Example
temperature
schema.org/temperature
schema.org/degreeCelsius
xsd:float
[0,80]
100
011
010
001
010
010
100
101
000
111
data stream
{
"tstamp": 1453478160,
"machineId": "ID5",
"temperature": 73.5,
"flowRate": 4.2
}
Semantic
metadata
25

26. www.streampipes.org | @streampipes | github.com/streampipes
Data Access
StreamPipes Connect: Architecture
Connect Master
Connect Worker 1 Connect Worker 2 Connect Worker n
MySQL
RESTROS
OPC-UAFile
MQTT
Messaging
Edge Worker Cloud Worker
…
register
capabilities
26

27. www.streampipes.org | @streampipes | github.com/streampipes
High-level architecture
Analytics Microservices
Data Sources
Adapter Library
Pipeline Editor
Data Integration
Streaming Engine
27

28. Analytics microservices
Extensible toolbox

29. www.streampipes.org | @streampipes | github.com/streampipes
• Extensible toolbox for pre-
processing & analytics
• Stateful/stateless
• Semantics-based
consistency checking
• Exchangable run-time
wrappers
• Inclusion of ML-models
possible
Features
Analytics microservices
Extensible toolbox
29

30. www.streampipes.org | @streampipes | github.com/streampipes
Analytics microservices
Anatomy of a processing element
Trend Detection
Controller
output eventsinput events
Runtime
30

31. www.streampipes.org | @streampipes | github.com/streampipes
Analytics microservices
How to implement a new processing element
Select Wrapper
Implement
runtime
Describe
controller
Build / Install
Maven
Archetype
StreamPipes
SDK
StreamPipes
SDK
SDK, Docker,
UI
Trend Detection
Controller
Runtime
31

32. www.streampipes.org | @streampipes | github.com/streampipes
Analytics microservices
Runtime Wrapper
Standalone/Edge
Wrapper
Flink Wrapper
Kafka Streams
Wrapper
Python Wrapper
Select
Wrapper
Implement
runtime
Describe
controller
Build /
Install
Trend Detection
Controller
output eventsinput events
Runtime
32

33. www.streampipes.org | @streampipes | github.com/streampipes
Analytics microservices
SDK: Runtime @Override
public void onInvocation (TrendParameters params,
EventProcessorRuntimeContext runtimeContext) {
this.params = params;
this.trendDetector = new TrendDetector(params.getIncrease(),
params.getDuration());
}
@Override
public void onEvent(Event event, SpOutputCollector out) {
boolean trendDetected =
this.trendDetector.detect(event.get(this.params.getField());
if (trendDetected) {
out.collect(event);
}
}
@Override
public void onDetach() {
// clean up
}
Select
Wrapper
Implement
runtime
Describe
controller
Build /
Install
Trend Detection
Controller
Runtime
33

34. www.streampipes.org | @streampipes | github.com/streampipes
Analytics microservices
Processing Element Description
User Configuration Output StrategyInput Requirements
Schema, Quality, Protocol,
Format
Text Input, Selections, Domain
Knowledge, …
Keep, Custom, Transform,
Append, …
Semantic Metadata
Select
Wrapper
Implement
runtime
Describe
controller
Build /
Install
Trend Detection
Controller
Runtime
34

35. www.streampipes.org | @streampipes | github.com/streampipes
Analytics microservices
Development: Maven Archetypes & SDK
@Override
public DataProcessorDescription declareModel() {
return
ProcessingElementBuilder.create("org.streampipes.processors.flink.increase")
.withLocales(Locales.EN)
.category(DataProcessorType.PATTERN_DETECT)
.withAssets(Assets.DOCUMENTATION, Assets.ICON)
.requiredStream(StreamRequirementsBuilder.create()
.requiredPropertyWithUnaryMapping(EpRequirements.numberReq(),
Labels.withId(Mapping),
PropertyScope.MEASUREMENT_PROPERTY)
.build())
.requiredSingleValueSelection(Labels.withId(Operation),
Options.from("Increase", "Decrease"))
.requiredIntegerParameter(Labels.withId(Increase), 0, 500, 1)
.requiredIntegerParameter(Labels.withId(Duration))
.outputStrategy(OutputStrategies.custom())
.build();
}
Select
Wrapper
Implement
runtime
Describe
controller
Build /
Install
Trend Detection
Controller
Runtime
35

36. Analytics microservices
Configuration
User
Configuration
Output
Strategy
Input
Requirements
Select
Wrapper
Implement
runtime
Describe
controller
Build /
Install
Controller
Runtime
Trend Detection
36

37. www.streampipes.org | @streampipes | github.com/streampipes
Analytics microservices
Installation
Pipeline
Management
register
description
trigger
invocation
Select
Wrapper
Implement
runtime
Describe
controller
Build /
Install
Trend Detection
Controller
output eventsinput events
Runtime
37

38. Use Cases
38
4

39. Universal Robots + StreamPipes
Data Exploration & Predictive Maintenance
Monitor health status of UR robot in a production line
Continuously track robot sensor data (ROS) and detect deviations

40. Test Area Autonomous Driving + StreamPipes
Control Center Monitoring
Analyze sensor data in a German test area for autonomous driving (> 120 mi)
Enable control center staff to quickly detect potential issues

41. Getting Started & Roadmap
41
4

42. www.streampipes.org | @streampipes | github.com/streampipes
How to start
Setting up StreamPipes
Docker-based installation
streampipes.org/en/download
Download installer from Github1
./streampipes start2
Finish installation in browser3
42

43. www.streampipes.org | @streampipes | github.com/streampipes
43
What's next?
Data Access
Data analytics &
harmonization
Data exploration &
exploitation
PLC4x adapter
Metadata recognition
Python wrapper
AutoML
Historical Data
Explorer
New features: Current work-in-progress
Infrastructure (Edge / Fog)

44. www.streampipes.org | @streampipes | github.com/streampipes
44
What we're looking for
…grow the user
base
…establish a
community
…become an
Apache project
Your help/experience is appreciated!
How to…

45. Let's connect!
…and if you like StreamPipes, star us on Github 
streampipes.org
docs.streampipes.org
github.com/streampipes/streampipes
twitter.com/streampipes
feedback@streampipes.org