This document summarizes a presentation about serving machine learning models with Apache Flink. It discusses Flink's ML roadmap, modeling concepts, and model serving considerations like customer needs, governance, and the model lifecycle. It proposes using Flink's streaming capabilities to dynamically update models without interruption. A prototype exploits Flink joins to merge prediction requests with cached models. Next steps include finalizing specifications, integrating TensorFlow and PMML models, and addressing governance.

1. Let’s Talk ML on Flink
Boris Lublinsky
Stavros Kontopoulos
Lightbend
Apache Flink London Meetup (15/11/2017)

2. Talk’s agenda
• Flink ML Roadmap
• Machine learning & Models
• Model Serving with Flink
• Results and next steps
2

3. Flink ML
Flink ML Roadmap:
https://docs.google.com/document/d/1afQbvZBTV15qF3vobVWUjxQc49h3
Ud06MIRhahtJ6dw
Model Serving
Online Learning
Incremental learning
Batch api (https://issues.apache.org/jira/browse/FLINK-2396)?
3

4. How people see ML
4

5. Reality
5

6. What is the Model?
6
A model is a function transforming inputs to outputs
- y = f(x),for example:
Linear regression:
y = ac + a1*x1 + … + an*xn
Neural network:
Such a definition of the model allows for an easy
implementation of model’s composition. From the
implementation point of view it is just function
composition

7. Model learning pipeline
UC Berkeley AMPLab introduced machine learning pipelines as a graph defining the
complete chain of data transformation.
7

8. Models proliferation
Data Scientist Software engineer
8

9. Model standardization
9

10. Model lifecycle considerations
• Models tend to change
• Update frequencies vary greatly - from hourly to
quarterly/yearly
• Model version tracking
• Model release practices
• Model update process 10

11. Customer SLA
• Response time - time to calculate prediction
• Throughput - predictions per second
• Support for running multiple models
• Model update - how quickly and easily can the model be
updated
• Uptime/reliability 11

12. Model Governance
Models should be:
• governed by the company’s policies and procedures, laws and regulations and
organization’s goals
• be transparent, explainable, traceable and interpretable for auditors and
regulators.
• have approval and release process.
• have reason code for explanation of decision.
• be versioned. Reasoning for introduction of new version should be clearly
specified. 12

13. What are we building
We want to build a streaming system allowing to update
models without interruption of execution (dynamically
controlled stream).
13

14. Model Serving with Apache Flink
Idea: Exploit Flink’s low latency capabilities for serving models. Focus on offline
models loaded from a permanent storage and update them without interruption.
FLIP Proposal:
(https://docs.google.com/document/d/1ON_t9S28_2LJ91Fks2yFw0RYyeZvIvndu8
oGRPsPuk8)
Combines different efforts: https://github.com/FlinkML
● https://github.com/FlinkML/flink-jpmml (https://radicalbit.io/)
● https://github.com/FlinkML/flink-modelServer (Boris Lublinsky)
● https://github.com/FlinkML/flink-tensorflow (Eron Wright)
14
14

15. Pros
● Live model updating
● No Flink runtime changes
● Latency guarantees
● Not all systems cover the requirements. For example TF
Serving:
○ Metadata not available.
(https://github.com/tensorflow/serving/issues/612)
○ No new models at runtime:
(https://github.com/tensorflow/serving/issues/422)
15

16. Flink Low Level Join
• Create a state object for one input (or both)
• Update the state upon receiving elements from its input
• Upon receiving elements from the other input, probe the state and
produce the joined result
16

17. Exporting Model
Different ML packages have different export capabilities. We used
the following:
• For Spark ML export to PMML -
https://github.com/jpmml/jpmml-sparkml
• For Tensorflow:
• Normal graph export
• Saved model format
17

18. Model representationOn the wire
syntax = "proto3";
// Description of the trained model.
message ModelDescriptor {
// Model name
string name = 1;
// Human readable description.
string description = 2;
// Data type for which this model is applied.
string dataType = 3;
// Model type
enum ModelType {
TENSORFLOW = 0;
TENSORFLOWSAVED = 2;
PMML = 2;
};
ModelType modeltype = 4;
oneof MessageContent {
// Byte array containing the model
bytes data = 5;
string location = 6;
}
} 18
Internal
trait Model {
def score(input : AnyVal) : AnyVal
def cleanup() : Unit
def toBytes() : Array[Byte]
def getType : Long
}
trait ModelFactoryl {
def create(input : ModelDescriptor) :
Model
def restore(bytes : Array[Byte]) : Model
}

19. Key based join
Flink’s CoProcessFunction allows key-based merge of 2 streams. When using this API,
data is key-partitioned across multiple Flink executors. Records from both streams are
routed (based on key) to the appropriate executor that is responsible for the actual
processing.
19

20. Partition based join
Flink’s RichCoFlatMapFunction allows merging of 2 streams in parallel (based on
parallelization parameter). When using this API, on the partitioned stream, data from
different partitions is processed by dedicated Flink executor.
20

21. Monitoring
case class ModelToServeStats(
name: String, // Model name
description: String, // Model descriptor
modelType: ModelDescriptor.ModelType, // Model type
since : Long, // Start time of model usage
var usage : Long = 0, // Number of servings
var duration : Double = .0, // Time spent on serving
var min : Long = Long.MaxValue, // Min serving time
var max : Long = Long.MinValue // Max serving time
)
Model monitoring should provide information about usage, behavior,
performance and lifecycle of the deployed models
21

22. Queryable state
Queryable state (interactive queries) is an approach, which allows to get more from
streaming than just the processing of data. This feature allows to treat the stream
processing layer as a lightweight embedded database and, more concretely, to directly
query the current state of a stream processing application, without needing to materialize
that state to external databases or external storage first.
22

23. Queryable state
Limitations
Operator state (non keyed) is not queryable
https://issues.apache.org/jira/browse/FLINK-7771
23

24. Where are we now?
• Proposed solution does not require modification of Flink proper. It
rather represents a framework build leveraging Flink capabilities.
Although extension of the queryable state will be helpful
• Flink improvement proposal (Flip23) - model serving is close to
completion -
https://docs.google.com/document/d/1ON_t9S28_2LJ91Fks2yF
w0RYyeZvIvndu8oGRPsPuk8).
• Fully functioning prototype of solution supporting PMML and
Tensorflow is currently in the FlinkML Git repo
(https://github.com/FlinkML) 24

25. Next steps
• Complete Flip23 document
• Integrate Flink Tensorflow and Flink PMML
implementations to provide PMML and Tensorflow
specific model implementations - work in progress
• Look at bringing in additional model implementations
• Look at model governance and management
implementation 25

26. Model Serving Beyond
Flink
Author: Boris Lublinsky
Download from here:
https://info.lightbend.com/ebook-serving-machine-learning-models-register.html
26

27. Thank you
Any Questions?
27