In this talk I will present a technique for deploying machine learning models to provide real-time predictions using Apache Pulsar Functions. In order to provide a prediction in real-time, the model usually receives a single data point from the caller, and is expected to provide an accurate prediction within a few milliseconds. Throughout this talk, I will demonstrate the steps required to deploy a fully-trained ML that predicts the delivery time for a food delivery service based upon real-time traffic information, the customer's location, and the restaurant that will be fulfilling the order.

1. Real-Time Machine Learning on
the Edge with Pulsar Functions
David Kjerrumgaard| Developer Advocate

2. streamnative.io
David Kjerrumgaard
Developer Advocate
● Author of Pulsar in Action
● Committer on the Apache Pulsar
project.
● Over a decade of experience
working with customers to solve
Big Data / Streaming use cases

3. Real-Time Machine Learning on the Edge
Using Pulsar Functions
Provide the data to the ML Model in
order to produce the prediction.
Step 4
Create an execution environment for the
ML model as close to the source of the
data as possible.
Step 1
Capture the data needed by the ML
model directly from the source.
Step 2
Provide a mechanism for deploying
the ML Model to the Edge.
Step 3

4. Edge Computing
Environments
Characteristics, Challenges, &
Opportunities

5. ● Any computation happening
outside the cloud at the edge
of the network
● Operates on real-time data
generated by sensors or users
● Improves response times in
applications where real-time
processing of data is required
Edge Computing

6. streamnative.io
Edge Computing Topology
Note: There is an inverse relationship between computational resources and proximity to
the data source.

7. streamnative.io
● The perception layer is responsible for perceiving the physical conditions of the
physical environment as sensed through numerous sensors.
● The transportation layer delivers the signals from the perception layer to the data
processing layer and acts as a bridge between short-range and long-range
communication protocols.
● The data processing layer performs the filtering, aggregation, and more complex
analytical processing including machine learning.
Logical Layers

8. Apache Pulsar
● A high performance solution for
server-to-server messaging.
● Specifically designed to run in
containerized environments in the
cloud.
● Key architectural differentiator is
separate storage and compute
layers.
● Provides Serverless computing
framework.
An open-source, cloud native
distributed messaging and streaming
platform

9. streamnative.io
Apache Pulsar’s Multi-tiered Architecture

10. streamnative.io
● Apache Pulsar’s two-tier architecture separates the compute and storage layers, and
interact with one another over a TCP/IP connection. This allows us to run the
computing layer (Broker) on either Edge servers or IoT Gateway devices.
● Pulsar’s serverless computing framework, know as Pulsar Functions, can run inside
the Broker as threads. Effectively “stretching” the data processing layer.
Edge Computing with Pulsar

11. streamnative.io
● Pulsar’s Serverless computing framework can run inside the Pulsar Broker as a
thread pool. This framework can be used as the execution environment for ML
models.
● The Apache Pulsar Broker supports the MQTT protocol and therefore can directly
receive incoming data from the sensor hubs and store it in a topic.
Benefits of Running Pulsar Broker on the Edge
Step 1
Step 2

12. Pulsar Functions
● Lightweight computation
similar to AWS Lambda.
● Specifically designed to use
Apache Pulsar as a message
bus.
● Function runtime can be
located within Pulsar
Broker.
A serverless event streaming
framework

13. streamnative.io
● Consume messages from one or
more Pulsar topics.
● Apply user-supplied processing
logic to each message.
● Publish the results of the
computation to another topic.
● Support multiple programming
languages (Java, Python, Go)
● Can leverage 3rd-party libraries
to support the execution of ML
models on the edge.
Pulsar Functions

14. streamnative.io
Pulsar Function API
/**
* This is the core interface of the function API. The
* process method is called for every message of the input
* topic of the function. The incoming input bytes are
* converted to the input type I for simple Java
* types(String, Integer, Boolean, Map, and JSON types)
*/
public interface Function<I, O> {
O process(I input, Context context) throws Exception;
}

15. streamnative.io
Pulsar Function Context Object
public interface Context {
void putState(String key, ByteBuffer value);
void putStateAsync(String key, ByteBuffer value);
void deleteState(String key);
ByteBuffer getState(String key);
ByteBuffer getStateAsync(String key);
. . .
}
● The Context interface provides a number of methods that can be used to access
Pulsar’s internal state store which can store arbitrary bytes, which is perfect for
storing ML models.
● Pulsar Functions can then access these models dynamically at runtime through
this interface.
Step 3

16. streamnative.io
Pulsar Function – Third Party Library Support
● You can leverage 3rd party
libraries within Pulsar Functions
● DeepLearning4J
● JPMML
● DJL.AI
● Keras
● Pulsar Functions are able to
support:
● A variety of ML model types.
● Models developed with
different languages and
toolkits
Step 3

17. Real-Time
Machine Learning
With Pulsar Functions

18. streamnative.io
Problem Identification
Preventative Maintenance
● Oil rigs have several different components,
each with various sensors that measure
things such as torque & drag, hydraulics,
vibration, temperature, etc.
● Goal is to use these readings to predict
potential problems and avoid them before
they occur by recommending specific
remedial operations.

19. streamnative.io
Data Acquisition
• There is existing Edge
infrastructure in place to
capture and transmit the
sensor readings to an IoT
Gateway device.
• The IoT Gateway device has
an MQTT-enabled Pulsar
Broker with a Pulsar
Function runtime
environment.

20. streamnative.io
Model Deployment
• Once the models have been trained, they
are exported to a transferable format such
as PMML, etc.
• These exported models are “pushed” to the
state store when the data science team
decides they are ready for production.
• Publishing the model to the StateStore is
done via a REST call, so it can be executed
as part of a CI/CD pipeline or scheduled to
run periodically to change the model based
on the time, which is what we need in our
case.

21. streamnative.io
Model Deployment
This PMML file was uploaded to the StateStore
using a command like the following :
./bin/pulsar-admin functions putstate
--name MyMLEnabledFunction
--namespace MyNamespace
--tenant MyTenant
--state "{"key":"ML-Model",
"byteValue": <contents of remediatation.pmml
>}"
Push the Model to the Edge

22. streamnative.io
Model Execution
• We configure the Pulsar Function to listen to multiple sensor topics using a regex
subscription, e.g., “persistent://drilling/platform123/sensor*topic”
• Whenever a new sensor reading comes in over MQTT to any of the topics, the following
method is invoked, and a prediction is made.

23. streamnative.io
Model Execution
• Inside the Pulsar Function, we retrieve the PMML from the state store and initialize the
ML Model evaluator we will use to execute the model:
Load the Model & Initialize the Evaluator

24. streamnative.io
Model Execution
• Inside the Pulsar Function, we can also retrieve user-specified configuration values that
allow us to map the input topic name to the corresponding sensor value it contains.
• We can also specify the prediction field and threshold which indicates the need for
some preventative action.
Load the Model & Initialize the Evaluator

25. streamnative.io
Model
Deployment
• We record the sensor reading
• Next we populate the feature
vector from the map of most
recent sensor readings.
• Next, we pass the populated
feature vector to the ML model for
evaluation
• Finally, we compare the predicted
failure value to the user-defined
threshold.

26. streamnative.io
Summary

27. Summary
• Apache Puslar can be used to create an execution environment for ML
Models inside an Edge environment.
• Apache Pulsar’s support for the MQTT protocol allows it to receive
incoming sensor readings and directly publish them to a topic, which
then triggers a Pulsar Function
• Technology-agnostic design pattern for deploying ML models inside of
Pulsar Functions that can be used to deploy models regardless of the
underlying algorithm or development language in four basic steps:
• Load the ML Model definition from the state store
• Initialize the Model execution engine with the model definition
• Retrieve the input data for the model
• Execute the model against the input data to produce a prediction

28. Summary cont.
• Pulsar Functions are a great tool for deploying ML Models for online
operation because they allow you to execute the model near the source
of the data and on a per-event basis.
• Pulsar Functions enable you to dynamically swap out the trained model
that is deployed without any code changes or downtime. This is critical
when you are required to rotate the ML model periodically or need to
support multiple versions of an ML model.
• Pulsar Functions enable you to leverage existing ML execution
frameworks which allows you to support the broadest range of model
types. This ensures that you don’t impose any limitations on your Data
Science team tooling.

29. streamnative.io
Now Available
On-Demand
Pulsar Training
Academy.StreamNative.io

30. streamnative.io
We’re Hiring
streamnative.io/careers/

31. Let’s Keep
in Touch!
David Kjerrumgaard
Developer Advocate
@Dkjerrumg1
https://www.linkedin.com/davidkj
https://github.com/tspannhw

32. Questions