Pulsar Function is a succinct computing abstraction Apache Pulsar provides users to express simple ETL and streaming tasks. The simplicity comes in two folds: Simple Interface and Simple Deployment. As it has been adopted, we realized that the native support of organizing multiple functions into integrity will be very beneficial. With such support, people can express and manage multi-stage jobs easily. In addition, this support also provides the possibility of higher-level abstraction DSL to further simplify the job composition. We call this new feature -- Function Mesh. This talk aims to provide a thorough walkthrough of this new Function Mesh Feature, including its design, implementation, use cases, and examples, to help people seeking simple streaming solutions understand this newly created powerful tool in Apache Pulsar.

1. Function Mesh
Complex Streaming Jobs In A Simple Way
Neng Lu

2. Data Processing With Apache Pulsar

3. Pulsar Functions
Pulsar Functions are lightweight compute processes that:
● consume messages from Pulsar topics
● apply a user-supplied processing logic to each message
● publish results to another Pulsar topic

4. Pulsar Functions
Overview

5. Pulsar Functions
IS & IS NOT
Pulsar Functions IS NOT:
● Another Full-Power Streaming Processing Engine
● A New Computation Abstraction Layer
Pulsar Functions IS:
● Lambda-style functions that are specifically designed to integrate with Pulsar

6. Pulsar Functions
Use Case
● ETL Jobs
● Real-time Aggregation
● Microservices
● Reactive Services
● Event Routing
● ...

7. Pulsar Functions
API
Hello
Hello!
Pulsar
Function

8. Pulsar Functions
Semantics
● ATMOST_ONCE
○ Message is ACKed to Pulsar once received
● ATLEAST_ONCE
○ Message is ACKed to Pulsar after the function completes -- Default
● EFFECTIVELY_ONCE
○ Utilizes Pulsar’s Effectively Once Semantics

9. Pulsar Functions
State
● Built-In State Management
○ Provide `Context` object for users to access State
○ Stores state in Bookkeeper
○ Support Server-Side Operations like Counters

10. Pulsar Functions
CLI

11. Pulsar Functions
Summary
● Developer productivity
○ Intuitive API: `func apply(input) output {}`
○ Multiple Language Support: Java, Python, Golang
● Operational simplicity
○ Fully Integrated with Pulsar
○ No Extra System/Service Setup Needed
● Easy troubleshooting
○ Convenient Local Runtime
○ Easy to Use log topics

12. Function Mesh
Function Mesh is a collection of functions collaborate together to:
● Accomplish a final data processing goal
● With clearly defined stages
Function Mesh IS NOT full power Streaming Engines

13. Function Mesh
Pulsar
Function
1
Pulsar
Function
3
Pulsar
Function
4
Pulsar
Function
5
Pulsar
Function
6
Pulsar
Function
2

14. Function Mesh -- BUT
��
● Redundant to manage
● Hard to track as an
integrity
● Really difficulty to know
upstream/downstream
functions

15. Function Mesh
Function Mesh
Pulsar
Function
1
Pulsar
Function
3
Pulsar
Function
4
Pulsar
Function
5
Pulsar
Function
6
Pulsar
Function
2

16. Pulsar Native Function Mesh
🙂
PIP-66

17. Function Mesh -- Scheduling

18. Kubernetes Function Mesh
😀

19. Kubernetes Function Mesh
- Custom Resource
- Function
- Mesh
- Source
- Sink

20. Kubernetes Function Mesh -- Scheduling

21. Pulsar v.s. Kubernetes
● Utilizes Kubernetes’ Scheduling Power
● Function as a First Class Citizen in Cloud Environment
● Open the potential talking to different messaging system

22. Demo

23. Function & Function
Mesh

24. Connector - Source

25. Future Planning
● Cloud Native Support
● Self-Contained Function Runtime
● Function registry for reusing function unit
● Better Tools/Frontend to Manage & Inspect Function Meshes
● Smartly Group Function Unit Together If No Shuffle Used
● Auto-scaling based on Pulsar metrics

26. Thank You!