Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

0

Share

Download to read offline

zenoh: The Edge Data Fabric

Download to read offline

Zenoh is rapidly growing Eclipse project that unifies data in motion, data at rest and computations. It elegantly blends traditional pub/sub with geo distributed storage, queries and computations, while retaining a level of time and space efficiency that is well beyond any of the mainstream stacks. This presentation will provide an introduction to Eclipse Zenoh along with a crisp explanation of the challenges that motivated the creation of this project. We will go through a series of real-world use cases that demonstrate the advantages brought by Zenoh in enabling and optimising typical edge scenarios and in simplifying the development of any scale distributed applications.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to like this

zenoh: The Edge Data Fabric

  1. 1. Angelo Corsaro, PhD Chief Technology Officer Advanced Technology Office angelo@adlink-labs.tech Data The Edge Fabric <
  2. 2. Abstract Zenoh is rapidly growing Eclipse project that uni fi es data in motion, data at rest and computations. It elegantly blends traditional pub/sub with geo distributed storage, queries and computations, while retaining a level of time and space ef fi ciency that is well beyond any of the mainstream stacks. This presentation will provide an introduction to Eclipse Zenoh along with a crisp explanation of the challenges that motivated the creation of this project. We will go through a series of real-world use cases that demonstrate the advantages brought by Zenoh in enabling and optimising typical edge scenarios and in simplifying the development of any scale distributed applications.
  3. 3. Speaker Bio Angelo Corsaro, Ph.D. is Chief Technology Of fi cer (CTO) at ADLINK Technology Inc. where he looks after corporate technology strategy and innovation, leads the Advanced Technology Of fi ce and the Software and Technology Business Unit. Angelo is a world top expert in edge/fog computing and a well known researcher in the area of high performance and large scale distributed systems. Angelo has over 100 publications on referred journals, conferences, workshops, and magazines. Angelo has co-authored over ten international standards. Specialties: Fog/Edge Computing, Industrial and Consumer Internet of Things, Innovation and Innovation Management, Product Strategy, Open Source, High Performance Computing, Large Scale Mission/Business Critical Distributed Systems, Real-Time Systems, Software Patterns, Functional Programming Languages
  4. 4. Context
  5. 5. The Data Journey Store Distribute Produce Compute … … … …
  6. 6. Moving and Resting Technologies for dealing with data in motion and data at rest have belonged historically to different families Publish/Subscribe is today the leading paradigm for dealing with with data in motion Databases (SQL and NoSQL) are the leading paradigm to deal with data at rest Data in Motion Data at Rest
  7. 7. Pushing and Pulling Technologies for dealing with data in motion and data at rest also distinguish in another dimension: Data in motion is Pushed to interested parties Data at rest is Pulled when needed Push Data at Rest Pull
  8. 8. Technological Fragmentation The technological fragmentation exist in several Data Distribution, Data Storage and the integration of the two Push Data at Rest Pull
  9. 9. Decentralisation The increasing availability of and storage, compute capabilities on devices is creating new opportunities for computing and storing and data much closer its production Existing technologies for data in motion and data at rest fall short in supporting this scenario. More importantly fail to provide a uni fi ed data management.
  10. 10. Example
  11. 11. Robotics Robotics applications are quickly evolving to require swarm coordination, Internet-Scale management and teleoperation Robots are increasingly operating in swarms and over constantly expanding geographical regions
  12. 12. Computation Offloading Next generation robotics (and autonomous driving) applications need to leverage surrounding infrastructure to of fl oad computations and facilitate coordination
  13. 13. Key Differences • Many • Moving • Geo-Distributed • Collaborative • Internet Scale • Open Environment • Distributed Computing • One • Fixed • Geo-localised • Stand-Alone • LAN Scale • Closed Environment • Cloud Computing
  14. 14. One More Example
  15. 15. Smart Home Today Data produced locally is sent to the cloud where it is processed and stored The core of the application logic runs on the cloud. Most if not all of the interactions with devices that are close to you are through the cloud This leads to several problems, including energy waste, availability in case of connectivity issues, privacy concerns…
  16. 16. Exploiting Locality Ideally we would want communication to be local whenever possible. Ideally we would want to place computations closer to data sources Ideally we would want most of the data to be kept in our house… But still access it from anywhere — if I have the rights to do so Some could be still processed or stored on the cloud — but that should be a choice not the only option.
  17. 17. Managing a Residence Let’s assume for a moment that we want to exploit data and computation locality at each house, yet we would like to easily monitor or query any kind of data — for which we have the rights. How can I do that?
  18. 18. Traditional Approach #1 Replicate all data on the cloud and use that as the location to access information on the houses The drawbacks of this solution is that all data is duplicated, energy is wasted to send data across the cloud, and privacy is again at risk …
  19. 19. Traditional Approach #2 Data is kept on the house and when needing to access it the house of interest is addressed The drawbacks of this solution is there is no location transparency. What if I want to keep some of the data on an edge server? Or even the cloud? …
  20. 20. Wouldn’t be nice if… We could keep data where it makes sense an retrieve it when needed in a location transparent manner — just naming the data Wouldn’t it be nice if we could provision application logic wherever it made sense on this computing fabric?
  21. 21. Status Quo
  22. 22. Technological Gap The ecosystem of technologies available today for data plane are unable to cover the needs of these large scale distributed systems because either cannot work at the proper scale, e.g. DDS, or are inherently depending on broker technologies, e.g. MQTT, AMQP Additionally none of this technologies help with dealing with geo-distributed data at rest
  23. 23. Filling the Gap
  24. 24. Uni fi es data in motion, data in-use, data at rest and computations. It carefully blends traditional pub/sub with distributed queries, while retaining a level of time and space ef fi ciency that is well beyond any of the mainstream stacks. It provides built-in support for geo-distributed storages and distributed computations
  25. 25. Provides a high level API for pub/sub and distributed queries, data representation transcoding, an implementation of geo-distributed storage and distributed computed values zenoh Data Link Network Transport Physical zenoh zenoh.net Implements a networking layer capable of running above a Data Link, Network or Transport Layer. This protocol provides primitives for ef fi cient pub/sub and distributed queries. It supports fragmentation and ordered reliable delivery. zenoh.net
  26. 26. Communication Models
  27. 27. Peer-to-peer communication Peer Peer Peer Peer Peer Peer Peer Peer Peer Peer Peers Scouting: • Multicast • Gossip Clique Mesh
  28. 28. Brokered Communication Router and peers can help with brokering communication between clients as well as between clients and mesh of peers Router Client Client Client Peer Peer Peer Peer Peer Client Client Client
  29. 29. Peer Peer Peer Peer Router Router Router Router Router Router Peer Peer Peer Peer Peer Peer Peer Peer Peer Peer Router Generalised Topology Client Client Client Client Client Client Client Client Client Client Client Client Client
  30. 30. Abstractions
  31. 31. Naming Data Following the tradition of Named Data Networking protocols, data is named by a sequence of byte arrays — called key — such as: /home/kitchen/sensors/temp /home/kitchen/sensors/C202 Data interest and intents are expressed by means of keys regular expressions, such as: /home/*/sensors/temp /home/**/C202
  32. 32. Selecting Data Uses selector to de fi nes data sets. A selector is composed by a key expression, and optionally a predicate, a projection and a set of properties /myhome/*/sensor/temp?value>25 /mycar/dynamics?speed>25#acceleration The key-expression is used to route the query, while predicate, properties, projection, etc., are interpreted only by the entity that executes the query. It also provide different policies to control query consolidation and completeness and potentially quorums
  33. 33. Primitives: Entities Resource. A named data, in other term a (key,value) Publisher. A spring of values for a key expression Subscriber. A sink of values for a key expression Queryable. A well of values for a key expression (e.g. /home/kitchen/sensor/temp, 21.5 (e.g. /home/kitchen/sensor/temp /home/kitchen/sensor/hum, 0.67) /home/kitchen/sensor/* ) (e.g. /home/kitchen/sensor/temp /home/kitchen/sensor/*) (e.g. /home/**)
  34. 34. Primitives: Operations open/close — Open/Close a zenoh.net session scout — Looks for zenoh entities, the kinds of relevant nodes, e.g. peers, router, etc., is speci fi ed by a bit-mask. declare/undeclare — Declare/Undeclare resource, publisher, subscriber and queryable. Declarations are used for discovery and various optimisations. For subscribers the declare primitive registers a user provided call-back that will be triggered when data is available. For queryable, the declare primitive register a user provided call-back triggered whenever a query needs to be answered.
  35. 35. Primitives: Operations write — Writes data for a key expression query — Issues a distributed query and returns a stream of results. The query target, coverage and consolidation depends on policies pull — Pulls data for a pull subscriber.
  36. 36. Storage A storage is de fi ned by: Selector. De fi nes the set of resources keys that stores this storage Back-end. De fi nes the storage technology used /myhome/status/** … Storage Back-end Storage Selector zenoh storages can be created via the administration API anywhere on the network and back-ends are dynamically loaded plugins. zenoh storages automatically align their initial state, but can also be bound to existing data-bases
  37. 37. Eval An eval is de fi ned by: Selector. De fi nes the set of resources keys that will trigger this computation Implementation. The user code implementing the computation Eval Implementation /myhome/energy-cons Eval Selector
  38. 38. Data Link Network Transport Physical zenoh zenoh.net Queryable Subscriber Subscriber Write Queryable Storage Eval Put Subscriber Query Get zenoh protocol
  39. 39. zZ Publisher Storage Querier Pull Subscriber Subscriber Subscriber /louvre/**/sensor/temp /louvre/1/42/sensor/temp /louvre/1/42/anomaly /louvre/2/42/sensor/temp /louvre/1/42/sensor/temp Publisher /louvre/2/42/sensor/temp /louvre/2/** /louvre/1/** Storage Eval
  40. 40. zZ Publisher Queryable + Subscriber Queryable Querier Pull Subscriber Subscriber Subscriber Queryable + Subscriber /louvre/**/sensor/temp /louvre/1/42/sensor/temp /louvre/2/** /louvre/1/** /louvre/1/42/anomaly /louvre/2/42/sensor/temp /louvre/1/42/sensor/temp Publisher /louvre/2/42/sensor/temp
  41. 41. zZ Publisher Queryable + Subscriber Queryable Querier Pull Subscriber Subscriber Subscriber Queryable + Subscriber /louvre/**/sensor/temp /louvre/1/42/sensor/temp /louvre/1/42/anomaly /louvre/2/42/sensor/temp /louvre/1/42/sensor/temp Publisher /louvre/2/42/sensor/temp /louvre/2/** /louvre/1/**
  42. 42. zZ Publisher Queryable + Subscriber Queryable Querier Pull Subscriber Subscriber Subscriber Queryable + Subscriber /louvre/**/sensor/temp /louvre/1/42/sensor/temp /louvre/1/42/anomaly /louvre/2/42/sensor/temp /louvre/1/42/sensor/temp Publisher /louvre/2/42/sensor/temp /louvre/2/** /louvre/1/**
  43. 43. zZ Publisher Queryable + Subscriber Queryable Querier Pull Subscriber Subscriber Subscriber Queryable + Subscriber /louvre/**/sensor/temp /louvre/1/42/sensor/temp /louvre/1/42/anomaly /louvre/2/42/sensor/temp /louvre/1/42/sensor/temp Publisher /louvre/2/42/sensor/temp /louvre/2/** /louvre/1/** get /Louvre/*/42/sensor/temp
  44. 44. zZ Publisher Queryable + Subscriber Queryable Querier Pull Subscriber Subscriber Subscriber Queryable + Subscriber /louvre/**/sensor/temp /louvre/1/42/sensor/temp /louvre/1/42/anomaly /louvre/2/42/sensor/temp /louvre/1/42/sensor/temp Publisher /louvre/2/42/sensor/temp /louvre/2/** /louvre/1/** get /Louvre/*/42/sensor/temp
  45. 45. Protocol Summary Highlights Most wire/power/memory ef fi cient protocol in the market to provide connectivity to extremely constrained targets Supports push and pull pub/sub along with distributed queries Resource keys are represented as integers on the wire, these integer are local to a session => good for wire ef fi ciency Supports for peer-to-peer and routed communication. Support for zero-copy. Ordered reliable data delivery and fragmentation. Minimal wire overhead for user data is 4-6 bytes Data Link Network Transport Physical zenoh zenoh.net
  46. 46. Performance
  47. 47. Throughput (msg/s) Test ran on 11/03/2021 on Centos 8 AMD Ryzen 32GB RAM Pub Sub Host
  48. 48. Throughput (Gb/s) Test ran on 11/03/2021 on Centos 8 AMD Ryzen 32GB RAM Pub Sub Host
  49. 49. Round Trip Time (us) Test ran on 11/03/2021 on Centos 8 AMD Ryzen 32GB RAM Pub Sub Host
  50. 50. Zenoh vs MQTT (msg/s) Test ran on 11/03/2021 on Centos 8 AMD Ryzen 32GB RAM Pub Rtr Host Sub
  51. 51. Zenoh vs MQTT (Gb/s) Test ran on 11/03/2021 on Centos 8 AMD Ryzen 32GB RAM Pub Rtr Host Sub
  52. 52. Code Lab
  53. 53. Comi ng Up In progr ess APIs zenoh runs on any RUST supported platform plus a few embedded targets such as Zephyr. Zenoh also offers a REST API for programming and administration.
  54. 54. 
 /demo/us-west/** 
 /demo/us-east/** 
 /demo/eu/** /demo/ap/** Example: • Put data: curl -X PUT -d 'Hello World!' http://us-west.zenoh.io:8000/demo/eu/test • Get data: curl http://ap.zenoh.io:8000/demo/*/test us-west.zenoh.io us-east.zenoh.io eu.zenoh.io ap.zenoh.io
  55. 55. Greetings from zenoh import Zenoh # Get a zenoh session zs = Zenoh({‘peer’: ‘tcp/eu.zenoh.io:7447’}) z = zs.workspace() # play around z.put(“/demo/eu/greet/italian”, “Ciao!”)
  56. 56. More Greetings… z.put(“/demo/us-east/greet/american”, “Hi!") z.put(“/demo/us-west/greet/american”, “What’s Up!”) z.put(“/demo/ap/greet/japanese”, “Aisatsu!”)
  57. 57. Getting Greetings from zenoh import Zenoh, ChangeKind # Define the listener def listener(change): print("{} : {} (encoding: {} , timestamp: {})” .format(change.path, "DELETED" if change.kind == ChangeKind.DELETE else change.value.get_content(), "none" if change.kind == ChangeKind.DELETE else change.value.encoding_descr(), change.timestamp)) z.subscribe(“/demo/**/greet/*“, listener)
  58. 58. Finding out Greetings # How do people greet in EU? workspace.get(“/demo/eu/**/greet”) # How about American? workspace.get(“/demo/us-*/**/greet”) # Just get me all you know about greeting… workspace.get(“/demo/**/greet”)
  59. 59. 
 /demo/us-west/** 
 /demo/us-east/** 
 /demo/eu/** /demo/ap/** us-west.zenoh.io us-east.zenoh.io eu.zenoh.io ap.zenoh.io workspace.get(“/demo/eu/**/greet”)
  60. 60. 
 /demo/us-west/** 
 /demo/us-east/** 
 /demo/eu/** /demo/ap/** us-west.zenoh.io us-east.zenoh.io eu.zenoh.io ap.zenoh.io workspace.get(“/demo/eu/**/greet”) workspace.get(““/demo/us-*/**/greet””)
  61. 61. workspace.get(“/demo/us-*/**/greet”) 
 /demo/us-west/** 
 /demo/us-east/** 
 /demo/eu/** /demo/ap/** us-west.zenoh.io us-east.zenoh.io eu.zenoh.io ap.zenoh.io workspace.get(“/demo/eu/**/greet”) workspace.get(“/demo/**/greet”)
  62. 62. Greeting of the Day Imagine you want to do a greeting of the day that each time somebody tries to query it generates a random quote, or a daily quote, etc. We could do that with an eval, here is how: def quote_eval(request): make_a_cute_quote(request) z.register_eval(“/demo/*/greet/*/daily”, quote_eval)
  63. 63. 
 /demo/us-west/** 
 /demo/us-east/** 
 /demo/eu/** /demo/ap/** us-west.zenoh.io us-east.zenoh.io eu.zenoh.io ap.zenoh.io workspace.get(““/demo/*/greet/italian/daily””) workspace.get(“/demo/*/greet/american/daily”) To resolve this query zenoh will pick the eval that happens to be “closer” to the querier. This is true in general as queries can target at the same time evals and storages.
  64. 64. Application Domains
  65. 65. ROS2 and ROS2 based robots can leverage zenoh into two ways (1) by leveraging a ROS2 RMW for zenoh, or (2) by leveraging the zenoh-bridge-dds which transparently moves R2X communication over zenoh The latter case does not require any change to your robot, not even a recompile / re-link Zenoh also supports full interoperability with ROS2 in the sense than you can read/write data from/into ROS2 via native zenoh API
  66. 66. Discovery Traffic Reduction Zenoh drastically reduces DDS discovery overhead – from 97% to 99,9%
  67. 67. Internet Scale Robotics Zenoh enables for mesh peer- to-peer communication when useful, routed communication when necessary and in general enables ef fi cient Internet-scale Additionally, it does not require any changes to your existing ROS2 systems.
  68. 68. In Action
  69. 69. Indy Autonomous Challenge Zenoh used for R2X communication
  70. 70. Final Thoughts
  71. 71. zenoh is an innovative and performant protocol that solves some of they problems at the very core of IoT and Edge Computing Its open architecture enables to easily expand both storage back-ends as well as protocols that are routed and integrated into the zenoh world If you like zenoh, star our repo and start hacking some code!
  72. 72. References
  73. 73. “Patience, persistence and perspiration make an unbeatable combination for success.”

Zenoh is rapidly growing Eclipse project that unifies data in motion, data at rest and computations. It elegantly blends traditional pub/sub with geo distributed storage, queries and computations, while retaining a level of time and space efficiency that is well beyond any of the mainstream stacks. This presentation will provide an introduction to Eclipse Zenoh along with a crisp explanation of the challenges that motivated the creation of this project. We will go through a series of real-world use cases that demonstrate the advantages brought by Zenoh in enabling and optimising typical edge scenarios and in simplifying the development of any scale distributed applications.

Views

Total views

5,572

On Slideshare

0

From embeds

0

Number of embeds

25

Actions

Downloads

7

Shares

0

Comments

0

Likes

0

×