Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Self-Service IoT Data Analytics with StreamPipes

159 views

Published on

ApacheCon North America 2019

StreamPipes is an open source self-service IoT toolbox to enable non-technical users to connect, analyze and explore IoT data streams

https://www.streampipes.org

Published in: Technology
  • Be the first to comment

Self-Service IoT Data Analytics with StreamPipes

  1. 1. Self-Service IoT Data Analytics with StreamPipes Dominik Riemer, Philipp Zehnder, Patrick Wiener ApacheCon North America, Las Vegas, 2019-09-10
  2. 2. www.streampipes.org | @streampipes | github.com/streampipes 2 What's StreamPipes? "A self-service IoT toolbox to enable non-technical users to connect, explore and analyze IoT data streams"
  3. 3. www.streampipes.org | @streampipes | github.com/streampipes 3 What's StreamPipes? Big Data / Edge InfrastructureExecute Reusable algorithm toolbox Install Model pipelines
  4. 4. www.streampipes.org | @streampipes | github.com/streampipes About us 4 Dominik Riemer Senior Research Scientist Philipp Zehnder Research Scientist Patrick Wiener Research Scientist FZI Research Center for Information Technology, Karlsruhe, Germany Stream Processing, Data Management, Machine Learning Non-profit research center for applied ICT research (250 employees) Started StreamPipes in 2014, first OSS release 2018
  5. 5. www.streampipes.org | @streampipes | github.com/streampipes Agenda The need for self-service IoT data analytics Demo 1 StreamPipes: Technical Overview Demo 2 Use Cases3 Getting Started & Roadmap4
  6. 6. The need for self-service IoT data analytics 1
  7. 7. www.streampipes.org | @streampipes | github.com/streampipes Conveyor Belts Pressure Oil temperature Dust particles Production plans Environmental Data Gear box drive Energy consumption Telematics Industrial Internet of Things Data streams everywhere
  8. 8. Continuous Monitoring Situational Awareness Continuous Data Harmonization Flexible data integration from heterogeneous sources and monitoring of current system states Detect time-critical situations, e.g., by means of rules or ML approaches Continuous pre- processing and transformation of input streams for third party systems Industrial Internet of Things Typical application scenarios
  9. 9. www.streampipes.org | @streampipes | github.com/streampipes From machine to analysis Open Source IoT projects: Great software stack to ease IIoT use cases XX Data Routing & Transformation 9
  10. 10. www.streampipes.org | @streampipes | github.com/streampipes From machine to analysis Open Source IoT projects: Positioning of StreamPipes XX Data Routing & Transformation 10
  11. 11. www.streampipes.org | @streampipes | github.com/streampipes StreamPipes Open Source framework to easily manage IoT data Data Access Data analytics & harmonization Data exploration & exploitation Generic adapters Specific adapters Metadata Data streams & sets Pre-processing Filter/Aggregation Pattern Detection ML Situation detection Harmonized data sets Visualizations Third-party systems 11
  12. 12. Demo First introduction to StreamPipes, Condition monitoring Rule-based monitoring of flow rate measurements in a multi tank system
  13. 13. Demo First introduction to StreamPipes, Condition monitoring Rule-based monitoring of flow rate measurements in a multi tank system Flow Sensor Aggregate Measurements Detect Leakage Visualize Notify
  14. 14. Demo Advanced analytics & image processing Offer drinks to drowsy customers in a Las Vegas casino
  15. 15. Demo Advanced analytics & image processing Offer drinks to drowsy customers in a Las Vegas casino Camera Sensor Detect Faces Detect Drowsiness Extract Faces Notify
  16. 16. www.streampipes.org | @streampipes | github.com/streampipes IoT landscape Comparison to other flow-based tools More focused on dev people JS runtime Data pre-processing / ETL 16
  17. 17. Technical Overview 17 3
  18. 18. www.streampipes.org | @streampipes | github.com/streampipes High-level architecture Analytics Microservices Data Integration Data Sources Adapter Library Pipeline Editor Streaming Engine 18
  19. 19. www.streampipes.org | @streampipes | github.com/streampipes High-level architecture Analytics Microservices Data Integration Data Sources Adapter Library Pipeline Editor Streaming Engine 19
  20. 20. Data Access StreamPipes Connect: Easily connect IoT sources
  21. 21. www.streampipes.org | @streampipes | github.com/streampipes Data Access StreamPipes Connect: Easily connect IoT sources • Easily connect data streams and data sets (sensor + image data) • Generic adapters for frequently used protocols (e.g., REST, MQTT, OPC-UA, ROS) and formats (e.g., JSON, XML) • Semantic metadata (used for pipeline verification) • Automatic schema recognition Features 21
  22. 22. Demo Connect & harmonize data with StreamPipes Connect Building a harmonized data lake
  23. 23. Demo Connect & harmonize data with StreamPipes Connect Building a harmonized data lake MQTT Flow Sensor IoTDB StreamPipes Connect
  24. 24. www.streampipes.org | @streampipes | github.com/streampipes Data Access Machine-interpretable metadata 100 011 010 001 010 010 100 101 000 111 data stream {  "tstamp": 1453478160, "machineId": "ID5", "temperature": 73.5, "flowRate": 4.2 } Semantic metadata Data type, runtime name, semantic type Frequency, latency, measurement unit Format, Protocol Schema Quality Grounding 24
  25. 25. www.streampipes.org | @streampipes | github.com/streampipes Data Access Machine-interpretable metadata Example temperature schema.org/temperature schema.org/degreeCelsius xsd:float [0,80] 100 011 010 001 010 010 100 101 000 111 data stream {  "tstamp": 1453478160, "machineId": "ID5", "temperature": 73.5, "flowRate": 4.2 } Semantic metadata 25
  26. 26. www.streampipes.org | @streampipes | github.com/streampipes Data Access StreamPipes Connect: Architecture Connect Master Connect Worker 1 Connect Worker 2 Connect Worker n MySQL RESTROS OPC-UAFile MQTT Messaging Edge Worker Cloud Worker … register capabilities 26
  27. 27. www.streampipes.org | @streampipes | github.com/streampipes High-level architecture Analytics Microservices Data Sources Adapter Library Pipeline Editor Data Integration Streaming Engine 27
  28. 28. Analytics microservices Extensible toolbox
  29. 29. www.streampipes.org | @streampipes | github.com/streampipes • Extensible toolbox for pre- processing & analytics • Stateful/stateless • Semantics-based consistency checking • Exchangable run-time wrappers • Inclusion of ML-models possible Features Analytics microservices Extensible toolbox 29
  30. 30. www.streampipes.org | @streampipes | github.com/streampipes Analytics microservices Anatomy of a processing element Trend Detection Controller output eventsinput events Runtime 30
  31. 31. www.streampipes.org | @streampipes | github.com/streampipes Analytics microservices How to implement a new processing element Select Wrapper Implement runtime Describe controller Build / Install Maven Archetype StreamPipes SDK StreamPipes SDK SDK, Docker, UI Trend Detection Controller Runtime 31
  32. 32. www.streampipes.org | @streampipes | github.com/streampipes Analytics microservices Runtime Wrapper Standalone/Edge Wrapper Flink Wrapper Kafka Streams Wrapper Python Wrapper Select Wrapper Implement runtime Describe controller Build / Install Trend Detection Controller output eventsinput events Runtime 32
  33. 33. www.streampipes.org | @streampipes | github.com/streampipes Analytics microservices SDK: Runtime @Override public void onInvocation (TrendParameters params,  EventProcessorRuntimeContext runtimeContext) { this.params = params; this.trendDetector = new TrendDetector(params.getIncrease(),  params.getDuration()); } @Override public void onEvent(Event event, SpOutputCollector out) { boolean trendDetected = this.trendDetector.detect(event.get(this.params.getField()); if (trendDetected) { out.collect(event); } } @Override public void onDetach() { // clean up } Select Wrapper Implement runtime Describe controller Build / Install Trend Detection Controller Runtime 33
  34. 34. www.streampipes.org | @streampipes | github.com/streampipes Analytics microservices Processing Element Description User Configuration Output StrategyInput Requirements Schema, Quality, Protocol, Format Text Input, Selections, Domain Knowledge, … Keep, Custom, Transform, Append, … Semantic Metadata Select Wrapper Implement runtime Describe controller Build / Install Trend Detection Controller Runtime 34
  35. 35. www.streampipes.org | @streampipes | github.com/streampipes Analytics microservices Development: Maven Archetypes & SDK @Override public DataProcessorDescription declareModel() { return ProcessingElementBuilder.create("org.streampipes.processors.flink.increase") .withLocales(Locales.EN) .category(DataProcessorType.PATTERN_DETECT) .withAssets(Assets.DOCUMENTATION, Assets.ICON) .requiredStream(StreamRequirementsBuilder.create() .requiredPropertyWithUnaryMapping(EpRequirements.numberReq(),  Labels.withId(Mapping),  PropertyScope.MEASUREMENT_PROPERTY) .build()) .requiredSingleValueSelection(Labels.withId(Operation), Options.from("Increase", "Decrease")) .requiredIntegerParameter(Labels.withId(Increase), 0, 500, 1) .requiredIntegerParameter(Labels.withId(Duration)) .outputStrategy(OutputStrategies.custom()) .build(); } Select Wrapper Implement runtime Describe controller Build / Install Trend Detection Controller Runtime 35
  36. 36. Analytics microservices Configuration User Configuration Output Strategy Input Requirements Select Wrapper Implement runtime Describe controller Build / Install Controller Runtime Trend Detection 36
  37. 37. www.streampipes.org | @streampipes | github.com/streampipes Analytics microservices Installation Pipeline Management register description trigger invocation Select Wrapper Implement runtime Describe controller Build / Install Trend Detection Controller output eventsinput events Runtime 37
  38. 38. Use Cases 38 4
  39. 39. Universal Robots + StreamPipes Data Exploration & Predictive Maintenance Monitor health status of UR robot in a production line Continuously track robot sensor data (ROS) and detect deviations
  40. 40. Test Area Autonomous Driving + StreamPipes Control Center Monitoring Analyze sensor data in a German test area for autonomous driving (> 120 mi) Enable control center staff to quickly detect potential issues
  41. 41. Getting Started & Roadmap 41 4
  42. 42. www.streampipes.org | @streampipes | github.com/streampipes How to start Setting up StreamPipes Docker-based installation streampipes.org/en/download Download installer from Github1 ./streampipes start2 Finish installation in browser3 42
  43. 43. www.streampipes.org | @streampipes | github.com/streampipes 43 What's next? Data Access Data analytics & harmonization Data exploration & exploitation PLC4x adapter Metadata recognition Python wrapper AutoML Historical Data Explorer New features: Current work-in-progress Infrastructure (Edge / Fog)
  44. 44. www.streampipes.org | @streampipes | github.com/streampipes 44 What we're looking for …grow the user base …establish a community …become an Apache project Your help/experience is appreciated! How to…
  45. 45. Let's connect! …and if you like StreamPipes, star us on Github  streampipes.org docs.streampipes.org github.com/streampipes/streampipes twitter.com/streampipes feedback@streampipes.org

×