Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Orientation in the fog: differences between stream processing in edge and cloud

3,954 views

Published on

Original (german) title: Orientierung im Nebel: Unterschiede zwischen Stream Processing in Edge und Cloud.
See https://www.buildingiot.de/veranstaltung-7812-orientierung-im-nebel%3A-unterschiede-zwischen-stream-processing-in-edge-und-cloud.html?source=0&id=7812

Published in: Technology
  • DOWNLOAD FULL eBOOK INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF eBook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB eBook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. doc eBook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. PDF eBook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB eBook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. doc eBook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, CookeBOOK Crime, eeBOOK Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Orientation in the fog: differences between stream processing in edge and cloud

  1. 1. Orientierung im Nebel: Unterschiede zwischen Stream Processing in Edge und Cloud
  2. 2. • IIoT vs IoT • (Stream) processing at the Edge • How Apache Edgent can help What I’ll talk about
  3. 3. • Julian Feinauer • Phd in mathematics • Founder of pragmatic minds GmbH and pragmatic industries GmbH • Apache PLC4X & Apache Edgent (P)PMC member • Twitter: @jfeinauer2 Who am I?
  4. 4. Industry 4.0, Digitalization and the Industrial Internet of Things?!
  5. 5. • Everybody wants Industrie 4.0 or Digitalization • „Can we do something with Big Data?“ • „We need a Data Lake“ • Can AI solve this for us?“ • This means that the Shopfloor is confrontend with • ... connectivity • ... modern IT infrastructure • ... Cloud technology Current Trend in the Industry
  6. 6. • #Neuland for the Industry • Industry is always conservative regarding IT infrastructure • Suppliers try to close the gap but are themselves faced with a completly new situation and new technologies • Very dynamic situation but Industry likes to plan • In many industrial sites we have bad bandwidths But...
  7. 7. • „Universal Protocol Adapter“ • Connectivity for PLCs • Bridging the GAP between Shopfloor and IT However...
  8. 8. The Industrial Internet of Things!
  9. 9. The Industrial Internet of Things PLCs Sensors Actors
  10. 10. The Industrial Internet of Things PLCs Sensors Actors Hundreds of internal variables • 10-100 Sensors • Scaling • Data types • 10-100 Actors • Different technologies • Metadata is important
  11. 11. The Industrial Internet of Things PLCs Sensors Actors • No discrete Events • Timeseries! • Events have to be extracted based on • ... absolute values (thresholds) • ... (bit-)triggers / value changes • ... derivatives • ... • è Stream Processing at the Edge!
  12. 12. Stream Processing at the Edge
  13. 13. (Classic-)Definition: “A data stream can be seen as a continuous and potentially infinite stochastic process in which events occur independently from another”
  14. 14. Streaming is easy!
  15. 15. The Industrial Internet of Things PLCs Sensors Actors • No discrete Events • Timeseries! • Events have to be extracted based on • ... absolute values (thresholds) • ... (bit-)triggers / value changes • ... derivatives • ... • è Stream Processing at the Edge!
  16. 16. A simple Example…
  17. 17. Example: Cycle Detection • A (boolean) marker is true during cycle, false otherwise • We want to detect cycles as events, and record their start / end
  18. 18. Example: Cycle Detection
  19. 19. Example: Cycle Detection • A (boolean) marker is true during cycle, false otherwise • We want to detect cycles as events, and record their start / end • „Oh, we sometimes jitter, so a cycle is only a real cycle if its longer than 10 seconds“
  20. 20. Example: Cycle Detection
  21. 21. Example: Cycle Detection • A (boolean) marker is true during cycle, false otherwise • We want to detect cycles as events, and record their start / end • „Oh, we sometimes jitter, so a cycle is only a real cycle if its longer than 10 seconds“ • „Ah, we forgot, it would be good to also store the pressure at the end of the cycle“
  22. 22. Example: Cycle Detection
  23. 23. Example: Cycle Detection • A (boolean) marker is true during cycle, false otherwise • We want to detect cycles as events, and record their start / end • „Oh, we sometimes jitter, so a cycle is only a real cycle if its longer than 10 seconds“ • „Ah, we forgot, it would be good to also store the pressure at the end of the cycle“ • „In fact, we also need the current, rpm and temperature for the cycle“
  24. 24. Example: Cycle Detection ...
  25. 25. Example: Error Logging • Exceptions are stored as bit mask • About 2300 bits / exceptions on the machine • Task • Read all bits • See which toggled • Generate Event for Exception on / off
  26. 26. ... Example: Error Logging
  27. 27. Example: Complex Actions • We need production parameters, at defined points • First, wait until bit0 is set, then take the start time • Then, wait until bit1 is set, then take the pressure • Then, wait until bit2 is set, then take the rpm • ...
  28. 28. Example: Complex Actions ...
  29. 29. Part II – Which Engine to use?
  30. 30. Streaming is easy!
  31. 31. Stream Processing is not Stream Processing Property Cloud Edge Distribution Yes No Scalable Yes No Fast Yes Easy Stateless Yes* No Message Delivery Exactly-Once At least once sufficient
  32. 32. From Imperative to Declarative
  33. 33. CRUNCH it! • CRUNCH - A framework for declarative industrial data analysis • Not another streaming framework, ABOVE the streaming layer • Created by pragmatic minds GmbH • Open Source sind end of 2018 • Apache 2.0 License • Visit: https://github.com/pragmaticminds/crunch
  34. 34. CRUNCH it! • CRUNCH - A framework for declarative industrial data analysis • Not another streaming framework, ABOVE the streaming layer • Created by pragmatic minds GmbH • Open Source sind end of 2018 • Apache 2.0 License • Visit: https://github.com/pragmaticminds/crunch
  35. 35. Introducing Apache Edgent (incubating)
  36. 36. Best practices... λ-Architecture
  37. 37. λ-Architecture at the Edge?
  38. 38. λ-Architecture at the Edge?
  39. 39. • Incubating project at the Apache Software Foundation • Established 2016 • Edge runtime with • a streaming runtime • real-time analytics capabilities • and cloud communication for constrained devices (i.e., with small footprint), at the edge. Apache Edgent
  40. 40. Apache Edgent
  41. 41. Apache Edgent
  42. 42. Edge to Cloud...
  43. 43. But... • All have their own specific API • Partly open source, but usually very vendor driven • Vendor-Lock in • Entrance barrier is low but this should be a strategic decision • Can become very expensive, long term
  44. 44. Edgent to the rescue! • All Open Source (Apache 2.0 Licence) • Cloud Connectors for different providers, no vendor lock-in • Tight integration with other Apache Frameworks, like • Apache PLC4X (incubating) • Apache IoTDB (coming soon) • Complete toolkit to build analytic applications at the edge!
  45. 45. What‘s (probably) Next? • More Cloud Connectors! • Tight integration of CRUNCH • Edge Integration Patterns! • Improve Routing / Rule Engine at the Edge („Camel at the Edge“) • Revive the Community
  46. 46. Interested? • Visit www.edgent.apache.org • Sign up to our mailing-list: dev@edgent.apache.org • Follow us on twitter: @ApacheEdgent • Start using Edgent • File bugs: https://issues.apache.org/jira/projects/EDGENT • Source code: https://github.com/apache/incubator-edgent • Examples: https://github.com/apache/incubator-edgent-samples • Join our meetup in Frankfurt on Mai 6th in Frankfurt (am Main)
  47. 47. Questions? Julian Feinauer T +49 7022 24933 01 M +49 176 2488 3977 E j.feinauer@pragmaticminds.de PRAGMATICMINDS GmbH Zementwerk 1 72622 Nürtingen www.pragmaticminds.de

×