This document discusses using machine learning with Spring Cloud Data Flow and TensorFlow. It introduces machine learning concepts and how they differ from classical programming. It then describes using TensorFlow for deep learning inference in Java applications and building real-time predictive pipelines with Spring Cloud Data Flow. Several TensorFlow processors for tasks like object detection, image recognition, and sentiment analysis are presented.

1. Machines Can Learn - a Practical Take
on Machine Intelligence Using Spring
Cloud Data Flow and TensorFlow
By Christian Tzolov
@christzolov

2. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
Disclaimer

3. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
Safe Harbor Statement
The following is intended to outline the general direction of Pivotal's offerings. It is intended for information
purposes only and may not be incorporated into any contract. Any information regarding pre-release of Pivotal
offerings, future updates or other planned modifications is subject to ongoing evaluation by Pivotal and is subject to
change. This information is provided without warranty or any kind, express or implied, and is not a commitment to
deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions regarding
Pivotal's offerings. These purchasing decisions should only be based on features currently available. The
development, release, and timing of any features or functionality described for Pivotal's offerings in this
presentation remain at the sole discretion of Pivotal. Pivotal has no obligation to update forward looking
information in this presentation.
3

4. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
Does It Matter?

5. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
Rise of the machines
• Machine Learning brings unprecedented
abilities to the Software Engineering field
• Provides different way to reason about
problems
• Solves un-programmable tasks
5
• There is a technological Gap ( e.g. BLAS/CPU, CUDA/GPU, C++/Python)
• And cultural differences 
• How Machine Learning can enable us deliver richer business solutions?

6. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
Real-Time Predictions with TensorFlow and Spring
Cloud Data Flow (SCDF)
Focus is on
• Supervised, Deep Learning (DL)
• TensorFlow and DL4J as DL toolkits
• Model Inference (e.g. evaluation, prediction)
• (Near) Real-Time - Streaming
• Java and Spring
6

7. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
How to Build SCDF applications like
7

8. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
Can Machine Learn?

9. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
Classical Programming
9
Classical
Programming
Rules
Data
Answers

10. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
Machine Learning Paradigm
10
Machine
Learning
Rules’
Data*
Answers*
* Historical, annotated Datasets

11. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
ML Lifecycle
11
Model Inference
Output Predictions
MODEL
(e.g. Rules)
New Data
Model Training
Annotated answers
Historical Data
Offline Online (Production)
Software Engineering &
Ops
Data Science

12. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
Remote Server
Model Serving & Embedding
12
Data Source
HTTP/REST, RPC
Processor(s)
Data Sink
Data Source Processor(s)
Model
Data Sink
Runtime
Model
Remote
Serving
In-Pipeline Embedded

13. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
Deep Learning Inference with Java
• Deep Learning Java Runtime
• Portable Models
13
DL Java Runtime
Input Data
Output
Predictions
Pre-trained Model
(e.g. Rules)

14. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
Deep Learning Frameworks

15. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
Deep Learning Frameworks
• Tensorflow (Google)
• PyTorch
• CNTK (Microsoft)
• Caffe2 (Facebook)
• MXNet (Amazon)
• DeepLearning4J
• Keras
15
Deep Learning Java Runtime

16. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
Real-Time Model Evaluation with
Spring Cloud Data Flow

17. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
Transport Middleware
Stream A
Spring Cloud Data Flow (SCDF)
Toolkit for building data integration, real-time, and batch data processing pipelines
• Multi-platform support (PCF, Kubernetes)
• Comprehensive app lifecycle (Skipper)
• Pipes & Filters like DSL: Source | Processor … | Sink
Source Processor Processor Sink

18. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
Localizing and identifying multiple objects in a single image
• TF Object Detection API: http://bit.ly/tensorflow-object-detection-api
• Pre-trained models: http://bit.ly/object-detection-api-model-zoo
• Object Detection Processor Doc: http://bit.ly/scdf-object-detection-processor
• Blog: https://content.pivotal.io/blog/real-time-object-detection-with-spring-cloud-stream
Object Detection Processor
file --directory=‘/tmp/scdf/od/input’
| object-detection
--mode=header
--model='file:/model.pb’
--labels='file:/label_map.pbtxt’
| image-viewer

19. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
TensorFlow SCDF Processor
20
TensorFlowService
Message
payload
headers
OutputConverter
InputConverter
Message
payload
headers
Map<String, Tensor>
Load from file:// , http://, classpath://
TensorFlow
Build/Train/Export
TensorFlow
Runtime
TF Model
Offline
Real
Time

20. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
Object Detection - RT Demo

21. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
Available TensorFlow Processors
• Object Detection – TF Object Detection API: [12] [13]
• Instance Segmentation, TF Object Detection API: [11]
• Semantic Segmentation, TF API: [8][9][10]
• Image Recognition (Inception V3), TensorFlow API: [7]
• Pose Estimation (OpenPose), TensorFlow API [5], [6]
• Twitter Sentiment Analysis, TensorFlow API: [4]
• Face Detection (MTCNN), ND4J,
ND4J-Tensorflow, JavaCV: [1], [2]
• Face Recognition (FaceNet), TensorFlow API: [3]

22. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
Quick Starters
• TensorFlow Spring Cloud Stream App Starters:
https://github.com/spring-cloud-stream-app-starters/tensorflow
• Computer Vision Projects (experimental):
https://github.com/tzolov/computer-vision
• SCDF TensorFlow Demos (Docker):
https://github.com/tzolov/scdf-tensorflow-demos
• Real-time Object-Detection With Spring Cloud Stream
https://content.pivotal.io/blog/real-time-object-detection-with-spring-cloud-stream
• Real-time Object-Detection With Spring Cloud Stream
https://content.pivotal.io/blog/real-time-object-detection-with-spring-cloud-stream
• Real-Time Multi-Person, Human Pose Estimation with Spring Cloud Data Flow
and TensorFlow
http://bit.ly/scdf-pose-estimationn-blog

23. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
Conclusions

24. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
Convinced?
• Data Science and Deep Learning mature at a rapid pace
• Pivotal is committed to make them easily accessible for Spring/Java developers.
• Way to start: Check the TensorFlow processors for Spring Cloud Data Flow
• You feedback is appreciated!

25. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
Next

26. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
Unofficial agenda
• Top level TensorFlow (or DL) project – move the utilities and existing
implementations (such as Image Recognition, Object Detection, Sentiment
Analysis) out of SCSt into a top level, generic projects.
• Use with Spring Cloud Stream, Spring Cloud Function, Spring Cloud Batch
• Research “Transfer Learning” possibilities
• Evaluate Deeplearning4J, ND4J as Runtime alternative
• ONNX Runtime to use models from other DL frameworks
• REST, Spring Boot as remote DL serving application
31

27. Unless otherwise indicated, these slides are © 2013 -2018 Pivotal Software, Inc. and licensed under a Creative Commons
Attribution-NonCommercial license: http://creativecommons.org/licenses/by -nc/3.0/
References
[1] Face Detection Lib: https://github.com/tzolov/mtcnn-java
[2] Face Detection Processor: https://github.com/tzolov/computer-vision/blob/master/spring-cloud-starter-stream-processor-face-detection-mtcnn
[3] Face Recognition: https://github.com/tzolov/computer-vision/tree/master/spring-cloud-starter-stream-processor-face-recognition
[4] Twitter Sentiment Analysis: https://github.com/spring-cloud-stream-app-starters/tensorflow/blob/master/spring-cloud-starter-stream-processor-twitter-sentiment
[5] Pose Estimation Processor: https://github.com/spring-cloud-stream-app-starters/tensorflow/tree/master/spring-cloud-starter-stream-processor-pose-estimation
[6] Pose Estimation Blog: https://content.pivotal.io/blog/making-data-science-accessible-to-developers-real-time-multi-person-human-pose-estimation-with-spring-
cloud-data-flow-and-tensorflow
[7] Image Recognition: https://github.com/spring-cloud-stream-app-starters/tensorflow/tree/master/spring-cloud-starter-stream-processor-image-recognition
[8] Semantic Segmentation Blog: https://goo.gl/dtwAfo
[9] Semantic Segmentation - Java2D: https://github.com/tzolov/computer-vision/tree/master/spring-cloud-starter-stream-processor-semantic-segmentation
[10] Semantic Segmentation - JavaCV: https://github.com/tzolov/computer-vision/tree/master/spring-cloud-starter-stream-processor-semantic-segmentation-cv
[11] Instance Segmentation - Object Detection Processor with Masks: https://github.com/spring-cloud-stream-app-starters/tensorflow/tree/master/spring-cloud-starter-
stream-processor-object-detection
[12] Object Detection Processor: https://github.com/spring-cloud-stream-app-starters/tensorflow/tree/master/spring-cloud-starter-stream-processor-object-detection
[13] Object Detection Blog: https://content.pivotal.io/blog/real-time-object-detection-with-spring-cloud-stream
32

28. > Stay Connected.