Marcin Bielak - presentation from RUSTMEET.EU - PL, Gliwice 02/2025 This presentation highlights the creation of the IoText protocol for IoT data streams, supporting various architectures like ARMs, ESP32s, and x86. You'll learn Rust practically while understanding the protocol's evolution, its transition from Python to Rust, and performance comparisons with JSON, Protobuf, and AVRO.

1. Let’s write open
IoText data protocol
for time-series data in Rust
Marcin Bielak
github.com/bieli
RUSTMEET
PL, Gliwice, 15 March 2025

2. / agenda
/ … / motivations
- data streams traceability for Internet of Things devices
- multi architecture targets (ARMs, ESP32s, x86, …)
- Rust learning in practice
/ … / showtime
- First challenges, when IoText specification was born
- Python + Rust PyO3 bridge, next switching to Rust
- Performance measures (compare to JSON, Protobuf, AVRO)
- Tip: How to adopt RUST and IoText for IoT/embedded systems?
/ … / about_me -> @bieli

3. / about_me -> @bieli
/ … / 24+ years in IT
- from electronics, automation, SCADA design, microcontrollers
- to Big Data domain -> data solutions architect and teams leader
- Linux kernel active user 20+ years & self-studies every day!
- connected with 3rd biggest automotive company on the globe
- 50+ PetaBytes scale data streams challenges every day
/ … / hobbies
- IoT embedded systems
- IoT protocols researching 1001000101001001101001110101010010010
- IoT sharing knowledge on “IoT, Hardware and Robotics” meetup
- traveling 50+ [km] on electric scooter (at summer time)

4. / … / motivations

5. / motivations / IoT / data / time-series
Time series data is dataset that is recorded over
consistent intervals of time (e.g., real time).
- needs
- industrial safety / stable toolchains
- performance / reliability
- handle large-scale IoT data streams or batches
- challenges
- scale (TB / PB) -> automotive, medical, space/aero.
- data in the fly
- how to find details/issues inside data streams?!

6. / motivations / IoT / data / traceability
trace & ability - “easy way” how to discovery answers from
data (i.e. streams from Internet of Things).
- looking for
- fast finding details/issues inside data records
- clear representing data types and full data structure
- sth. “in the middle” for IoT enthusiasts - like me -
for review time-series dataset on-demand (on the fly!)
-
-

7. / motivations / multi architecture / targets
- Cross-Platform Compatibility: Rust supports multiple
architectures, including ARM, x64, RISC-V, LX6/ESP32, and
RBPi making it versatile for various IoT devices.
- Efficient Embedded Systems: Rust's zero-cost abstractions and
efficient memory usage make it suitable for
resource-constrained embedded systems.
- Interoperability: Rust's FFI (Foreign Function Interface)
allows seamless integration with existing C/C++ codebases,
enabling interoperability across different systems.

8. / motivations / learning / Rust
As always, nice to check, write some code, failed, return
to right path and again, and again -> dev./eng. story!
- measures
- how fast I can write library inside Rust ecosystem?
- how flexible is Rust ecosystem?
- how stable is non-critical safety code in Rust?
- how cross-platform development in Rust works?
- is it good for IoT enthusiasts and data engineers?
- what exactly performance we have in data exchanges?

9. / … / showtime

10. / showtime / challenge / IoText specification
Approx. 2 years ago, I decide to write my own IoT simplistic data
protocol with a few design check points:
- human readable & data traceability without special tools
- text based, easy extensible - new versions with new features
- time-series data in mind (for data streams and IoT domains)
- easy to compress/pack datasets like simple types and lists
- not limited in case of dataset row (exception: embedded
systems)
- Open Source solution with full specification:
https://github.com/bieli/IoText-data-protocol

11. / showtime / challenge / IoText specification
IoText data protocol data row example & spec. explains:
- t|3900237526042,d|device_name_001,m|water_lvl=i:42
- where data items are splitted by “,” char & items:
- <t|d>|<value> - generic data items TS & DEVICE_ID
- <m>|<metric_name>=<metric_type>:<metric_value> -
metrics items
- with only 4 special characters (to learn!):
- , | = :

12. / showtime / challenge / IoText specification
IoText data protocol data row example & spec. explains:
- t|3900237526042,d|d01,m|water_lvl=i:42,m|door_open=b:0,
m|tank2_lvl=d:12.43,m|warn=t:tank2 has too low pressure
- <m>|<metric_name>=<metric_type>:<metric_value>
- with metric_types:
- i - signed integers 64-bit
- b - boolean (1 or 0)
- d - signed decimals: 96-bit integers + 1-bit sign
(not a float type!)
- t - text messages

13. / showtime / demo for IoText data protocol
- src: https://github.com/bieli/IoText-rs/tree/wind_turbine_demo/examples (on branch, waiting PR)
- steps to run:
- $ git clone https://github.com/bieli/IoText-rs.git
- $ cd IoText-rs
- $ git checkout -b wind_turbine_demo origin/wind_turbine_demo
Please open two Linux terminals:
- term 1) run visual frontend (listening on TCP port)
- $ cd examples/wind_turbine_animation/
- $ cargo run
- discovery changes in metrics values in real-time
- term 2) run basic client emulator (sending to TCP port)
- $cd examples/wind_turbine_animation_client/
- $cargo run
- us arrows keys to changing metrics values
t|1741559223294,d|wind_turbine_01,m|speed
=d:42,m|wind_dir=t:S,m|warnings_cnt=i:0

14. / showtime / challenge / Python + Rust PyO3
We can build modules/code in RUST and add as modules into
Python.
- OpenAI/tiktoken inspiration for Python + Rust!
- PyO3: Library bindings & compilation from Python -> RUST
- PR is here:
https://github.com/bieli/IoText-data-protocol/pull/13
- I dropped below PR -> reasons no exp. with RUST ~2 years
ago … still waiting in todo!(“for experiments”).

15. / showtime / challenge / switching to Rust
I switched to pure Rust and decided to build implementation
from scratch for educational reasons.
- Learning RUST in practice - recommended path!
- Discovery RUST ecosystem
- Experiments with data types, crates and patterns
- My first Crate iotext_rs deployment for Open Source
community!
- Link to repository:
https://github.com/bieli/IoText-rs

16. / showtime / performance / IoText reference example
t|3900237526042,d|device_name_001,m|val_water_001=i:63,m|va
l_water_002=i:56,m|bulb_state=b:1,m|connector_state=b:0,m|t
emp_01=d:23.507712775653197,m|temp_02=d:35.298492988320945,
m|temp_03=d:11.516517204808512,m|pwr=d:38.6922791883438,m|c
urrent=d:3.2369317071470496,m|current_battery=d:17.37696765
476996

17. / showtime / performance / IoText compare to JSON
JSON is text based semi-structured data format, very
popular in web development, data engineering and REST APIs.
- Default without schema - schema inference possible from
data types (optional JSONSchema)
- Human readable - but in case of thousands of lines it’s
need special tools like jq to filtering records
- Very popular in IT today, so all modern programming
languages have their native (de)serializers.
- Resources consumed format - CPU intensive and RAM usage.

18. / showtime / performance / IoText compare to JSON

19. / showtime / performance / IoText compare to JSON
Based on JSON serialization over 864_000 records (100ms per record).
Machine: Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz; 32GB RAM; busy DBUS
Load average: 1,03, 1,21, 1,17
Without JSON writer to file: ~804 MB as output.json TEXT file
IoText serialization time:
~17 sec. (~19µs per rec.)
JSON serialization time:
~16 sec. (~18µs per rec.)

20. / showtime / performance / IoText compare to ProtoBuf
Protobuf was originally developed by Google for use in
their internal systems, including Google Search, Google
Maps, and Gmail. Connected with gRPC for microservices
communication.
- Financial institutions for high-frequency trading and fraud detection.
- Used by game developers to optimize network performance and reduce
bandwidth usage.
- IoT applications send data between devices, such as sensors, gateways,
and cloud servers.
- Automotive applications to exchange data between cars and infrastructure,
such as traffic lights and parking meters.

21. / showtime / performance / IoText compare to ProtoBuf

22. / showtime / performance / IoText compare to ProtoBuf
Based on ProtoBuf serialization over 864_000 records (100ms per record).
Machine: Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz; 32GB RAM; busy DBUS
Load average: 1,03, 1,21, 1,17
Without ProtoBuf writer to file: ~208 MB as output.protobuf BINARY file
IoText serialization time:
~17 sec. (~19µs per rec.)
JSON serialization time:
~6 sec. (~7µs per rec.)

23. / showtime / performance / IoText compare to AVRO
AVRO is used by Apache Hadoop, an open-source big data platform,
for data serialization and exchange between Hadoop ecosystem
components.
- Messaging systems, such as Apache Kafka and Apache Pulsar, for streaming data
between applications and services.
- Analytics platforms, such as Apache Spark and Apache Flink, for data processing and
analysis.
- Machine learning frameworks, such as TensorFlow and PyTorch, for data serialization and
exchange between training and inference stages.
- Log collection tools, such as Apache Flume and Apache NiFi, for streaming log data
from various sources to a centralized location.
- Dedicated AVRO schema file is required.

24. / showtime / performance / IoText compare to AVRO

25. / showtime / performance / IoText compare to AVRO

26. / showtime / performance / IoText compare to AVRO
Based on AVRO serialization over 864_000 records (100ms per record).
Machine: Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz; 32GB RAM; busy DBUS
Load average: 0,43, 0,88, 1,04
Without AVRO writer to file: ~164 MB as output.avro BINARY file
IoText serialization time:
~17 sec. (~19µs per rec.)
AVRO serialization time:
~11 sec. (~13µs per rec.)

27. / showtime / performance / protocols compare table
Based on serialization over 864_000 records (100ms per record). Without writing to
file. Machine: Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz; 32GB RAM; busy DBUS
Summary for serialization time in seconds for tested data protocols:
IoText
~17
~19
~224
AVRO
~11
~13
~164
JSON
~16
~18
~804
protocol name
duration [s]
per rec.[µs]
payload [MB]
ProtoBuf
~6
~7
~208

28. / showtime / performance / IoText compare to MessagePack
Based on MesagePack serialization over 864_000 records (100ms per record).
Machine: Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz; 32GB RAM; busy DBUS
Load average: 1,45, 1,90, 1,82
Without MessagePack writer to file: ~179 MB as output.msgpack.bin BINARY file
IoText serialization time:
~17 sec. (~19µs per rec.)
MsgPack serialization time:
~7 sec. (~8µs per rec.)
ADDITIONAL EXTRA PERF. TEST !!!

29. / showtime / embedded systems / IoText + RUST
What I discovered here:
- a few frameworks exists - IDN if maturity level is for real production? todo!(“check”)
- blinking LEDs + WiFi connection + UDP/TCP client -> possible scenario
- no_std instead of std usage in RUST code
- memory allocation care + dedicated no_std allocators
- limited arrays/lists/vectors + dedicated no_std implementations
- all embedded device has memory.x definition of internal memory area - supported by
embedded RUST, but depends what architecture and version of target silicon
- RUST features are our friends, but code isn’t clean with all directives line by line!

30. / lessons learned
From RUST ecosystem, I have some feelings from my experience:
- very good:
- compiler is supportive and informative, more then i.e. Scala and Python,
much more than GCC/G++! Mature and stable toolchain and ecosystem!
- macros extends dev. power and give speed up in development!
- easy steps to publish your lib/package/crate!
- good low-level programming materials and HALs/targets for embedded
systems like ESP32, STM32, Raspberry Pico, nRF, etc…
- very good support for multiple libraries like e.g. RayLib, for building
games and desktop or WebAssembly UIs apps.
- quite nice ecosystem for building web services
- great performance for data engineering and for any data
flows/transformations
- “features” flags give you flexibility and cross-platform features!
- quite nice community and visible Open Source support!

31. / lessons learned
From RUST ecosystem, I have some feelings from my
experience:
- mixed:
- RUST is an easy language, when you are “an expert” in a
few programming languages ;)
- reflections are not on mature level - not intuitive API
and hard for newbies to operate e.g. on structs fields!
- for std & no_std we need different mindsets required to
write universal libraries/crates for servers and
embedded systems (in one code base)

32. / new challenges
Maybe in this year ;-)
- plugin for Grafana for easy plot IoText data from IoText
data streams (needs dedicated Grafana backend/plugin)
- extends IoText protocol (schema versioning, …)
- return to PyO3 + Rust bridge with new experience
- publish multiple hardware/targets specific tests
- find next commercial usage for IoText data protocol
- collects more feedbacks from Open Source community
- … and who knows ;) maybe with Elon we will be on Mars!

33. / rust / new_beginning
Thanks for watching!
“Live long and prosper, friends!”
IoText lib. repository: https://github.com/bieli/IoText-rs
IoT presentations: https://www.slideshare.net/bieli
contact: https://www.linkedin.com/in/marcin-bielak/
🖖