In this talk, I will share my journey of using machine learning in Java to build a visual recognition system that can identify Lego blocks. As a Java developer, I wanted to use Java for this project rather than Python, which is more commonly used for machine learning projects. I will explain the basics of machine learning and give an overview of the current Java libraries for machine learning and transferring pre-trained models. I will demonstrate how to train and modify existing models using transfer learning. The goal of the project is to create a Java solution that can identify the top 1000 most popular Lego bricks. I will explain all of this without using any complex mathematical formulas, making it accessible to those with no prior knowledge of machine learning.

1. Lessons learned using Machine Learning in Java
Jago de Vreede
# jfall.

6. PYTHON MACHINE LEARNING FRAMEWORKS
• Keras
• SciPy
• PyTorch
• Tensor
fl
ow
• Theano
• Pandas

7. JAVA MACHINE LEARNING FRAMEWORKS*
• deeplearning4j
• Elki
• DeepJavaLibrary (djl)
• Apache MXNet
• PyTorch
• Tensor
fl
ow
• ONNX Runtime
* With a release in the last year and free

8. BENCHMARKS
DJL DL4J

10. BENCHMARKS - TRAINING (LEGO)
Tensor
fl
ow 2.4 (py) PyTorch 2.0.1 (py) PyTorch 2.0.1 (Java) MxNet (Java)
Ran on AWS: g4dn.xlarge on GPU github.com/jagodevreede/ml-banchmark

11. BENCHMARKS - PREDICTION
Python Java
github.com/jagodevreede/ml-banchmark
Ran on laptop

12. BENCHMARKS - RESULTS

13. BENCHMARKS - RESULTS

16. HUMAN

17. HUMAN

20. Chihuahua
Muf
fi
n

21. Chihuahua
Muf
fi
n

22. DATASET
$ pip install simple_image_download
import simple_image_download.simple_image_download as simp
my_downloader = simp.Downloader()
my_downloader.download('Chihuahua', limit=2000)
my_downloader.download('Muffin', limit=2000)

23. DATASET

24. DATASET - CLEANUP
$ pip install ImageHash

25. DATASET - CLEANUP

26. DATASET - CLEANUP
86 images left…

27. $ pip install flickrapi

28. DATASET - FLICKR

29. DATASET - ARGUMENTING

30. DATASET - ARGUMENTING
$ pip install keras
datagen = ImageDataGenerator(
rotation_range=20,
width_shift_range=0.1,
height_shift_range=0.1,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True,
fill_mode='nearest'
)

31. TIME TO BUILD

36. The number of samples that are processed at once
during each iteration of training.
BATCH SIZES
The presented results con
fi
rm that using small
batch sizes achieves the best training stability and
generalization performance, for a given
computational cost, across a wide range of
experiments. In all cases the best results have
been obtained with batch sizes m = 32 or
smaller, often as small as m = 2 or m = 4.
arxiv.org/abs/1804.07612

37. 1 2 4 8 16 32 64
BATCH SIZES - SPEED (SECONDS)

45. ACCURACY
The lower the value of the entropy loss, the better the
model's predictions are aligned with the labels, and
take into account the con
fi
dence of the predictions
accuracy measures the proportion of
correctly classi
fi
ed instances out of the total
number of instances
VS
ENTROPYLOSS

56. Muf
fi
n Chihuahua
Muf
fi
n 5 3
Chihuahua 1 7
CONFUSION MATRIX

58. VALIDATION ACCURACY

62. Only works in DJL >= 0.24.0

63. Only works in DJL >= 0.24.0

64. LEARNING RATE

65. LEARNING RATE

66. LEARNING RATE

67. LEARNING RATE

68. LEARNING RATE - OPTIMIZERS

70. Muf
fi
n Chihuahua
Muf
fi
n 7 1
Chihuahua 0 8
CONFUSION MATRIX

71. TRANSFER LEARNING
Muf
fi
n Chihuahua

72. TRANSFER LEARNING
Muf
fi
n Chihuahua

75. PYTHON AND JAVA TOGETHER

76. CONVERT THE MODEL
import tensorflow as tf
import tensorflow.keras as keras
loaded_model = keras.models.load_model("flowers.h5")
tf.saved_model.save(loaded_model, "model/1")

82. GENERATE IMAGES

83. GENERATE IMAGES
Warning
Next slide contains
fl
ashing images

88. ZOOM OR NOT TO ZOOM

91. Sim
SIM - VS - REAL

92. Sim Real
SIM - VS - REAL

93. 3001 3002 3004
3001 31 0 0
3002 28 0 0
3004 29 0 0
CONFUSION MATRIX EPOCH 17

94. 3001 3002 3004
3001 4 27 0
3002 5 23 0
3004 8 21 0
CONFUSION MATRIX EPOCH 47

95. youtube.com/watch?v=3hOlZpZrsXY

97. SIM - VS - REAL

98. VALIDATION ACCURACY
B
a
s
e
l
i
n
e
H
i
g
h
-
R
e
s

99. GRAYSCALE VS COLOR

101. GRAYSCALE VS COLOR
Gray Color
Ran on AWS: g4dn.xlarge

102. VALIDATION ACCURACY
B
a
s
e
l
i
n
e
H
i
g
h
-
R
e
s
G
r
a
y
s
c
a
l
e

103. VALIDATION ACCURACY
B
a
s
e
l
i
n
e
H
i
g
h
-
R
e
s
G
r
a
y
s
c
a
l
e
L
o
w
C
o
n
t
r
a
s
t
V
a
l
i
d
a
t
i
o
n
o
n
l
y

104. VALIDATION ACCURACY
B
a
s
e
l
i
n
e
H
i
g
h
-
R
e
s
G
r
a
y
s
c
a
l
e
L
o
w
C
o
n
t
r
a
s
t
V
a
l
i
d
a
t
i
o
n
o
n
l
y
L
o
w
C
o
n
t
r
a
s
t

105. VALIDATION ACCURACY
B
a
s
e
l
i
n
e
H
i
g
h
-
R
e
s
G
r
a
y
s
c
a
l
e
L
o
w
C
o
n
t
r
a
s
t
V
a
l
i
d
a
t
i
o
n
o
n
l
y
L
o
w
C
o
n
t
r
a
s
t
H
/
V
fl
i
p
p
i
n
g

109. VALIDATION ACCURACY
B
a
s
e
l
i
n
e
H
i
g
h
-
R
e
s
G
r
a
y
s
c
a
l
e
L
o
w
C
o
n
t
r
a
s
t
V
a
l
i
d
a
t
i
o
n
o
n
l
y
L
o
w
C
o
n
t
r
a
s
t
H
/
V
fl
i
p
p
i
n
g
T
r
a
i
n
s
e
t
7
5
0
(
L
a
r
g
e
s
t
i
m
a
g
e
s
)

110. VALIDATION ACCURACY
B
a
s
e
l
i
n
e
H
i
g
h
-
R
e
s
G
r
a
y
s
c
a
l
e
L
o
w
C
o
n
t
r
a
s
t
V
a
l
i
d
a
t
i
o
n
o
n
l
y
L
o
w
C
o
n
t
r
a
s
t
H
/
V
fl
i
p
p
i
n
g
T
r
a
i
n
s
e
t
7
5
0
(
L
a
r
g
e
s
t
i
m
a
g
e
s
)
A
c
t
u
a
l
i
m
a
g
e
s
(
1
2
%
)

114. TAKEAWAYS
• Creating the dataset is the hardest part
• Java can be used for ML
• Java is a little bit faster than python
• Python has a bigger ecosystem
• Models in made with python can be
used in Java

115. @jagovreede
github.com/jagodevreede/
java-djl-dog-vs-muf
fi
n
ml-banchmark
lego-sorter
slideshare.net/JagodeVreede1/
Javamllegojfall