The document discusses FlinkML, an effort to enhance Apache Flink with large-scale machine learning capabilities, emphasizing scalable implementations and ease of use. It covers various aspects such as supervised and unsupervised learning, optimization frameworks, recommendation systems, and techniques for managing stragglers in distributed machine learning. Additionally, the document outlines current tools and algorithms being developed for FlinkML, including streaming machine learning techniques.

1. FlinkML: Large-scale Machine
Learning with Apache Flink
Theodore Vasiloudis, Swedish Institute of Computer Science (SICS)
Big Data Application Meetup
July 27th, 2016

2. Large-scale Machine Learning

3. What do we mean?

4. What do we mean?
● Small-scale learning ● Large-scale learning
Source: Léon Bottou

5. What do we mean?
● Small-scale learning
○ We have a small-scale learning problem
when the active budget constraint is the
number of examples.
● Large-scale learning
Source: Léon Bottou

6. What do we mean?
● Small-scale learning
○ We have a small-scale learning problem
when the active budget constraint is the
number of examples.
● Large-scale learning
○ We have a large-scale learning problem
when the active budget constraint is the
computing time.
Source: Léon Bottou

7. Apache Flink

8. What is Apache Flink?
● Distributed stream and batch data processing engine
● Easy and powerful APIs for batch and real-time streaming analysis
● Backed by a very robust execution backend
○ true streaming dataflow engine
○ custom memory manager
○ native iterations
○ cost-based optimizer

9. What is Apache Flink?

10. What does Flink give us?
● Expressive APIs
● Pipelined stream processor
● Closed loop iterations

11. Expressive APIs
● Main bounded data abstraction: DataSet
● Program using functional-style transformations, creating a dataflow.
case class Word(word: String, frequency: Int)
val lines: DataSet[String] = env.readTextFile(...)
lines.flatMap(line => line.split(“ “).map(word => Word(word, 1))
.groupBy(“word”).sum(“frequency”)
.print()

12. Pipelined Stream Processor

13. Iterate in the dataflow

14. Iterate by looping
● Loop in client submits one job per iteration step
● Reuse data by caching in memory or disk

15. Iterate in the dataflow

16. Delta iterations

17. Performance
Extending the Yahoo Streaming Benchmark

18. FlinkML

19. FlinkML
● New effort to bring large-scale machine learning to Apache Flink

20. FlinkML
● New effort to bring large-scale machine learning to Apache Flink
● Goals:
○ Truly scalable implementations
○ Keep glue code to a minimum
○ Ease of use

21. FlinkML: Overview

22. FlinkML: Overview
● Supervised Learning
○ Optimization framework
○ Support Vector Machine
○ Multiple linear regression

23. FlinkML: Overview
● Supervised Learning
○ Optimization framework
○ Support Vector Machine
○ Multiple linear regression
● Recommendation
○ Alternating Least Squares (ALS)

24. FlinkML: Overview
● Supervised Learning
○ Optimization framework
○ Support Vector Machine
○ Multiple linear regression
● Recommendation
○ Alternating Least Squares (ALS)
● Pre-processing
○ Polynomial features
○ Feature scaling

25. FlinkML: Overview
● Supervised Learning
○ Optimization framework
○ Support Vector Machine
○ Multiple linear regression
● Recommendation
○ Alternating Least Squares (ALS)
● Pre-processing
○ Polynomial features
○ Feature scaling
● Unsupervised learning
○ Quad-tree exact kNN search

26. FlinkML: Overview
● Supervised Learning
○ Optimization framework
○ Support Vector Machine
○ Multiple linear regression
● Recommendation
○ Alternating Least Squares (ALS)
● Pre-processing
○ Polynomial features
○ Feature scaling
● Unsupervised learning
○ Quad-tree exact kNN search
● sklearn-like ML pipelines

27. FlinkML API
// LabeledVector is a feature vector with a label (class or real value)
val trainingData: DataSet[LabeledVector] = ...
val testingData: DataSet[Vector] = ...

28. FlinkML API
// LabeledVector is a feature vector with a label (class or real value)
val trainingData: DataSet[LabeledVector] = ...
val testingData: DataSet[Vector] = ...
val mlr = MultipleLinearRegression()
.setStepsize(0.01)
.setIterations(100)
.setConvergenceThreshold(0.001)

29. FlinkML API
// LabeledVector is a feature vector with a label (class or real value)
val trainingData: DataSet[LabeledVector] = ...
val testingData: DataSet[Vector] = ...
val mlr = MultipleLinearRegression()
.setStepsize(0.01)
.setIterations(100)
.setConvergenceThreshold(0.001)
mlr.fit(trainingData)

30. FlinkML API
// LabeledVector is a feature vector with a label (class or real value)
val trainingData: DataSet[LabeledVector] = ...
val testingData: DataSet[Vector] = ...
val mlr = MultipleLinearRegression()
.setStepsize(0.01)
.setIterations(100)
.setConvergenceThreshold(0.001)
mlr.fit(trainingData)
// The fitted model can now be used to make predictions
val predictions: DataSet[LabeledVector] = mlr.predict(testingData)

31. FlinkML Pipelines
val scaler = StandardScaler()
val polyFeatures = PolynomialFeatures().setDegree(3)
val mlr = MultipleLinearRegression()

32. FlinkML Pipelines
val scaler = StandardScaler()
val polyFeatures = PolynomialFeatures().setDegree(3)
val mlr = MultipleLinearRegression()
// Construct pipeline of standard scaler, polynomial features and multiple linear
// regression
val pipeline = scaler.chainTransformer(polyFeatures).chainPredictor(mlr)

33. FlinkML Pipelines
val scaler = StandardScaler()
val polyFeatures = PolynomialFeatures().setDegree(3)
val mlr = MultipleLinearRegression()
// Construct pipeline of standard scaler, polynomial features and multiple linear
// regression
val pipeline = scaler.chainTransformer(polyFeatures).chainPredictor(mlr)
// Train pipeline
pipeline.fit(trainingData)
// Calculate predictions
val predictions = pipeline.predict(testingData)

34. FlinkML: Focus on scalability

35. Alternating Least Squares
R ≅ X Y✕Users
Items

36. Naive Alternating Least Squares

37. Blocked Alternating Least Squares

38. Blocked ALS performance
FlinkML blocked ALS performance

39. Going beyond SGD in large-scale
optimization

40. ● Beyond SGD → Use Primal-Dual framework
● Slow updates → Immediately apply local updates
CoCoA: Communication Efficient Coordinate
Ascent

41. Primal-dual framework
Source: Smith
(2014)

42. Primal-dual framework
Source: Smith
(2014)

43. Immediately Apply Updates
Source: Smith
(2014)

44. Immediately Apply Updates
Source: Smith
(2014)
Source: Smith
(2014)

45. CoCoA: Communication Efficient Coordinate
Ascent

46. CoCoA performance
Source:
Jaggi
(2014)

47. CoCoA performance
Available on FlinkML
SVM

48. Dealing with stragglers: SSP Iterations

49. ● BSP: Bulk Synchronous parallel
○ Every worker needs to wait for the others to finish before starting the next iteration
Dealing with stragglers: SSP Iterations

50. ● BSP: Bulk Synchronous parallel
○ Every worker needs to wait for the others to finish before starting the next iteration
● ASP: Asynchronous parallel
○ Every worker can work individually, update model as needed.
Dealing with stragglers: SSP Iterations

51. ● BSP: Bulk Synchronous parallel
○ Every worker needs to wait for the others to finish before starting the next iteration
● ASP: Asynchronous parallel
○ Every worker can work individually, update model as needed.
○ Can be fast, but can often diverge.
Dealing with stragglers: SSP Iterations

52. ● BSP: Bulk Synchronous parallel
○ Every worker needs to wait for the others to finish before starting the next iteration
● ASP: Asynchronous parallel
○ Every worker can work individually, update model as needed.
○ Can be fast, but can often diverge.
● SSP: State Synchronous parallel
○ Relax constraints, so slowest workers can be up to K iterations behind fastest ones.
Dealing with stragglers: SSP Iterations

53. ● BSP: Bulk Synchronous parallel
○ Every worker needs to wait for the others to finish before starting the next iteration
● ASP: Asynchronous parallel
○ Every worker can work individually, update model as needed.
○ Can be fast, but can often diverge.
● SSP: State Synchronous parallel
○ Relax constraints, so slowest workers can be up to K iterations behind fastest ones.
○ Allows for progress, while keeping convergence guarantees.
Dealing with stragglers: SSP Iterations

54. Dealing with stragglers: SSP Iterations
Source: Ho et al.
(2013)

55. SSP Iterations in Flink: Lasso Regression
Source: Peel et
al. (2015)

56. SSP Iterations in Flink: Lasso Regression
Source: Peel et
al. (2015)
PR submitted

57. Challenges in developing an
open-source ML library

58. Challenges in open-source ML libraries
● Depth or breadth
● Design choices
● Testing

59. Challenges in open-source ML libraries
● Attracting developers
● What to commit
● Avoiding code rot

60. Current and future work on
FlinkML

61. Current work
● Tooling
○ Evaluation & cross-validation framework
○ Distributed linear algebra
○ Streaming predictors
● Algorithms
○ Implicit ALS
○ Multi-layer perceptron
○ Efficient streaming decision trees
○ Colum-wise statistics, histograms

62. Future of Machine Learning on Flink
● Streaming ML
○ Flink already has SAMOA bindings.
○ Preliminary work already started, implement SOTA algorithms and develop new
techniques.

63. Future of Machine Learning on Flink
● Streaming ML
○ Flink already has SAMOA bindings.
○ Preliminary work already started, implement SOTA algorithms and develop new
techniques.
● “Computation efficient” learning
○ Utilize hardware and develop novel systems and algorithms to achieve large-scale learning
with modest computing resources.

64. Check it out:
@thvasilo
tvas@sics.se
flink.apache.org
ci.apache.org/projects/flink/flink-docs-master/libs/ml

65. “Demo”

67. “Demo”

68. “Demo”

69. “Demo”

70. “Demo”

71. “Demo”

72. “Demo”

74. “Demo”

75. Thank you
@thvasilo
tvas@sics.se
flink.apache.org
ci.apache.org/projects/flink/flink-docs-master/libs/ml

76. References
● Flink Project: flink.apache.org
● FlinkML Docs: https://ci.apache.org/projects/flink/flink-docs-master/libs/ml/
● Leon Botou: Learning with Large Datasets
● Smith (2014): CoCoA AMPCAMP Presentation
● Jaggi (2014): “Communication-efficient distributed dual coordinate ascent." NIPS 2014.
● Ho (2013): "More effective distributed ML via a stale synchronous parallel parameter server." NIPS
2013.
● Peel (2015): “Distributed Frank-Wolfe under Pipelined Stale Synchronous Parallelism”, IEEE BigData
2015
● Recent INRIA paper examining Spark vs. Flink (batch only)
● Extending the Yahoo streaming benchmark (And winning the Twitter Hack-week with Flink)
● Also interesting: Bayesian anomaly detection in Flink