Google’s TensorFlow drew considerable attention in recent months as it enabled rapid Machine Learning development. In response, Google developed TensorFlow Lite - a solution for deploying intelligent applications on mobile and embedded devices. We will show how to use TensorFlow Lite to develop an app which transforms the style of pictures using a deep neural network. Style-transferring apps can transform summer into winter, horses into zebras, or a Monet into a Van Gogh. In this talk, we see rapid development of intelligent mobile device products which may or may not have a lasting impact.

2. NOVEMBER 2018
Style-transfer
with TensorFlow
Transforming the style of images with
TensorFlow (lite) for mobile devices.
Martin Musiol

3. What is style-
transfer?
3

4. 4

5. Why ?
5

6. Art

7. $ 432,000

8. Data
Augmentation
8
WHY?

9. WHY?
Product Design

10. WHY?
Product Design

11. What is Generative AI?
Generative vs. Discriminative
11

12. 12
WHAT IS GENERATIVE AI?
This bird is red
with white and
has a very short
beak.

13. Who knows that person?

14. Where is it located?
14

15. Where are we?
AI - Artificial Intelligence
ML - Machine Learning
DL - Deep Learning
Gen. AI - Generative AI
Style-transfer
A
I
ML
DL
Gen. AI
15

16. Self-Learning
Algorithms
Non-Self-
Learning
Algorithm
s
ML
DL Self-Learning Algorithms
16

17. How does this
work?
17

18. Neural Style
Transfer
18

19. 19
NEURAL STYLE-TRANSFER
The Three Ingredients
1. Autoencoder for image reconstruction

20. 20
NEURAL STYLE-TRANSFER
1. Autoencoder for image reconstruction
2. Compressed representation
The Three Ingredients

21. 21
NEURAL STYLE-TRANSFER
1. Autoencoder for image reconstruction
2. Compressed representation
3. Transforming features
The Three Ingredients

22. 22
NEURAL STYLE-TRANSFER
1) Autoencoder for image reconstruction

23. 23
NEURAL STYLE-TRANSFER
2) Compressed representation

24. 24
NEURAL STYLE-TRANSFER
2) Compressed representation

25. 25
NEURAL STYLE-TRANSFER
3) Transforming features

26. 26
NEURAL STYLE-TRANSFER
3) Transforming features

27. 27
NEURAL STYLE-TRANSFER
3) Transforming features
Content
Style

28. 28
NEURAL STYLE-TRANSFER
3) Transforming features
Content
Style
Feature
Transforms

29. 29
NEURAL STYLE-TRANSFER
3) Transforming features
Content
Style
Feature
Transforms
1) Whitening
Whitening: keep only shapes
Coloring: keep the colors
Stylization: how much shape from
content and how much shape from
style picture.
EXTRACT SHAPE

30. 30
NEURAL STYLE-TRANSFER
3) Transforming features
Content
Style
Feature
Transforms
1) Whitening
2) Coloring
Whitening: keep only shapes
Coloring: keep the colors
Stylization: how much shape from
content and how much shape from
style picture.
EXTRACT COLORS

31. 31
NEURAL STYLE-TRANSFER
3) Transforming features
Content
Style
Feature
Transforms
1) Whitening
2) Coloring
3) Stylization
Whitening: keep only shapes
Coloring: keep the colors
Stylization: how much shape from
content and how much shape from
style picture.
CONTENT SHAPE/ STYLE SHAPE

32. TensorFlow Lite
32

33. TENSORFLOW LITE
• Lower latency, no server calls
Why on-device ML?

34. TENSORFLOW LITE
• Lower latency, no server calls
• Works offline
Why on-device ML?

35. TENSORFLOW LITE
• Lower latency, no server calls
• Works offline
• Data stays on-device
Why on-device ML?

36. TENSORFLOW LITE
• Lower latency, no server calls
• Works offline
• Data stays on-device
• Power efficient
Why on-device ML?

37. TENSORFLOW LITE
• Lower latency, no server calls
• Works offline
• Data stays on-device
• Power efficient
• All sensor data accessible on-device
Why on-device ML?

38. • Core operations tuned for mobile platforms
Highlights
TENSORFLOW LITE

39. • Core operations tuned for mobile platforms
• FlatBuffers-based model file format
Highlights
TENSORFLOW LITE
The primary benefit of FlatBuffers
comes from the fact that they can be
memory-mapped, and used directly
from disk without being loaded and
parsed. This gives much faster
startup times, and gives the
operating system the option of
loading and unloading the required
pages from the model file, instead of
killing the app when it is low on
memory.
SERIALIZATION LIBRARY FOR :
• JAVA
• C/C++
• PYTHON
• ETC.

40. • Core operations tuned for mobile platforms
• FlatBuffers-based model file format
• On-device interpreter with kernels optimised
for fast execution on mobile
Highlights
TENSORFLOW LITE

41. • Core operations tuned for mobile platforms
• FlatBuffers-based model file format
• On-device interpreter with kernels optimised
for fast execution on mobile
• Small: TF Lite interpreter < 100 kB
Highlights
TENSORFLOW LITE

42. • Core operations tuned for mobile platforms
• FlatBuffers-based model file format
• On-device interpreter with kernels optimised
for fast execution on mobile
• Small: TF Lite interpreter < 100 kB
• Java and C++ API support
Highlights
TENSORFLOW LITE

43. Highlights
TENSORFLOW LITE

44. • Java API for convenience
Highlights
TENSORFLOW LITE

45. • Java API for convenience
• C++ API: loads model and invokes interpreter
Highlights
TENSORFLOW LITE

46. • Java API for convenience
• C++ API: loads model and invokes interpreter
• Interpreter executes model using set of
kernels
Highlights
TENSORFLOW LITE

47. The new graph converter in TF Lite is called
TensorFlow Lite Optimizing Converter aka
TOCO
How to transfer TF to
TF Lite?
TENSORFLOW LITE
How to call it in demo shortly!

48. Demo
48

49. FROG IS A COMPANY OF ARICENT © 2017 FROG DESIGN INC.
Thank you.
Martin Musiol