Fluvio is an open source, cloud-native data streaming platform built using Rust for high performance and reliability. It treats streams as first-class citizens and uses WebAssembly to enable intelligent stream processing. Fluvio provides a modern data stack for real-time applications with low latency, reliability, flexibility and scalability.

1. Introduction to Fluvio
Data engineers lunch
3/14/2022

2. Fluvio is the engine for Real-time Economy
Real-time data stack
v Low Latency
v Reliability
v Flexibility
v Scale

3. Modern Data Stack: Is this enough?

4. Problem: Data in Motion
● Pipeline of Tools is not abstraction
● SQL is not good abstraction for data in motion
● Time and History
● As developer, I want API

5. Stream: Abstraction for data in motion
Stream is an unbounded sequence of data

6. Fluvio: Intelligent Data Streaming Platform
● Open source project
● Built on Rust.
● Stream as first class citizen.
● Cloud-native data streaming
architecture.
● Intelligent stream processing
using WebAssembly.

7. Why Rust?
● High Performance with Low Latency
○ Compiled language
○ No Garbage collection
○ Zero cost abstraction
● Safety and Reliability
○ No Null Pointer
○ High level typed language (Similar to Haskell and Scalar)
○ If it compiles, it works
● Batteries built-in
○ Package Manager
○ Well established ecosystem
○ Outstanding IDE support
● Stream as first class citizen.
● Inter-operate with other languages
○ C / C++
○ Java
○ Python
○ JavaScript
“About 70% of all CVEs at Microsoft are memory safety
issues“
Microsoft

8. Rust Performance
Memory
Latency
p99.999
Kafka 1GB 215 ms
Fluvio 20MB 50 ms

9. Data Stream as first class citizen
Rust
Stream
Storage
Durable
Stream
Smart
Stream
Networking I/O
Server
Networking
Python Client
Rust
Stream
Python
Stream
Networking
Rust
Stream
Node.js
Stream
Node.js Client

10. Cloud Native Architecture
Streaming
Controller Metadata
Stream
Processing
Unit
Producer Consumer

11. Immutable Durable Stream
Temperature
Time
10
Old
New
20 3
Consumer 1
6 11 5 9 Consumer 2
Consumer 3
Producer 1
Producer 2

12. Partitioning and Replication of Durable Stre
Stream
Processing
Unit
Stream
Processing
Unit
Stream
Processing
Unit
p1
p2
Stock Transactions
p0
Web Clicks
ML Model
P
0
p0
p0

13. Producer and Consumer Client
Rust Async runtime
OS (Mac, Window, Linux, Browser)
Fluvio Rust API & C Wrapper
CLI Node API Python API Java API Go API

14. Language Binding
Rust
Python
Java

15. SmartModule based transformation
Temperature
10 20 3
Consumer
6 11 5 9
SmartModule
Network connection

16. WebAssembly Technology
● Memory Isolation
● Efficient
● Support multiple Languages
○ Rust
○ C++/C
○ AssemblyScript
○ C#
○ Python
○ Etc.

17. Type of SmartModule

18. SmartModule integration
SmartModule
Client Connectors
SPU

19. Demo