U-Net is a convolutional neural network used for biomedical image segmentation. It takes in an input image and outputs a segmentation map identifying nuclei pixels. The U-Net architecture consists of a contracting path to capture context and a symmetric expanding path that enables precise localization. The model was trained on microscopy images annotated with nuclei masks to achieve an intersection over union score of 85% after data augmentation was applied.

1. Automating Nuclei Detection by U-Net
Convolutional Networks for Biomedical Image Segmentation
Ding Li 2018.07

2. 2
Microscope
Image
Three Types
Nuclei
Mask
2018 Data Science Bowl: Find the nuclei in divergent images to advance medical discovery

3. 3
256
x
256
8
1 8
16 16
8
16 32
32
32 64 64
64 128 128
64
64 64 64
32
32 32
32
16 16 16 16
8
8 8 1
256
x
256
Input
Image
File
Output
Segmentation
Map
conv 3x3, ReLU
copy & concatenate
max pool 2x2
up-conv 2x2
conv 1x1 Softmax
Learn
Context
Structure
Combine
Context &
Local Details
High
Resolution
Shallow
Low
Resolution
Deep
High
Resolution
Shallow
Low
Resolution
Deep
U-Net: Convolutional Networks for Biomedical Image Segmentation

4. 4
10 10 10 0 0 0
10 10 10 0 0 0
10 10 10 0 0 0
10 10 10 0 0 0
10 10 10 0 0 0
10 10 10 0 0 0
*
1 0 -1
1 0 -1
1 0 -1
10 10 0
10 10 0
10 10 0
*
1 0 -1
1 0 -1
1 0 -1
10 10 0
10 10 0
10 10 0
1 0 -1
1 0 -1
1 0 -1
∑ ∑
10 0 0
10 0 0
10 0 0
= 30
0 30 30 0
0 30 30 0
0 30 30 0
0 30 30 0
Detected edge
No edge
https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/

5. 5
3 8 2 1
7 9 1 1
4 5 2 3
5 6 1 2
9 2
6 3
Max Pool
Pool after convolutional layer, reduce noise and data size.
Local information are concentrated into higher level information.
Besides max pool, there is also average pool.
Size: 2 x 2

6. 6
1 2
3 4
Input
padding
0 0 0 0 0
0 1 0 2 0
0 0 0 0 0
0 3 0 4 0
0 0 0 0 0
*
1 2
2 1
Filter
1 ? ? ?
? ? ? ?
? ? ? ?
6 ? ? ?
Result
0 0
0 1 *
1 2
2 1
∑
0 0
0 1
1 2
2 1
∑
0 0
0 1
1
0 3
0 0 *
1 2
2 1
∑
0 3
0 0
1 2
2 1
∑
0 6
0 0
6
Resolution increased; features of input and filter combined
Convolution arithmetic tutorial An Introduction to different Types of Convolutions in Deep Learning Up-sampling with Transposed Convolution

7. 7
1 2
3 4
Input
padding
0 0 0 0 0
0 1 0 2 0
0 0 0 0 0
0 3 0 4 0
0 0 0 0 0
*
1 2
2 1
Filter
1 2 2 4
2 1 4 2
3 6 4 8
6 3 8 4
Result
0 0
0 1 *
1 2
2 1
∑
0 0
0 1
1 2
2 1
∑
0 0
0 1
1
0 3
0 0 *
1 2
2 1
∑
0 3
0 0
1 2
2 1
∑
0 6
0 0
6
Resolution increased; features of input and filter combined
Convolution arithmetic tutorial An Introduction to different Types of Convolutions in Deep Learning Up-sampling with Transposed Convolution

8. 8
2018 Data Science Bowl State 1 Training data
Training samples: 603
Evaluation samples: 67
In this initial stage, the marks of individual nuclei are
merged into one flatten file to simply the evaluation.
IOU is calculated from all the pixels predicted as nuclei vs
all the pixels marked. All images are converted to
256x256 resolution first.
IOU = 82%
After 100 epochs of training (6 hours for one CPU),
using U-Net implemented with Keras (Tensorflow
backend)
Grand Truth Prediction
False Negative (FN) False Positive (FP)
True Positive (TP)
𝐼𝑂𝑈 =
𝐼𝑛𝑡𝑒𝑟𝑠𝑒𝑐𝑡𝑖𝑜𝑛
𝑈𝑛𝑖𝑜𝑛
=
𝑇𝑃
𝑇𝑃 + 𝐹𝑁 + 𝐹𝑃
541 (81%)
107 (16%)
16 (2%)
Other: 6 (1%)
Image Mosaic

9. 9

10. 1
0

11. 1
1
Flip
Shearing
Translation
Rotation
Scale
IOU = 85%
After 100 epochs of training, applying Keras ImageDataGenerator
(rotation_range=45., width_shift_range=0.1, height_shift_range=0.1,
shear_range=0.2, zoom_range=0.2, horizontal_flip=True, vertical_flip=True,
fill_mode='reflect')
Data Augmentation | How to use Deep Learning when you have Limited Data

12. 1
2

13. 1
3
• U-NET is an efficient Neural Network for image segmentation.
• The first half network reduces the resolution and gets context information.
• The second half network applies the context to the higher resolution data copied from first half network.
• The nuclei boundary info was extracted into context first and applied to each pixel later.
• The symmetry of U-Net is elegant.
• How to improve?
• Images with low IOU are usually uncommon images. (Applying data augmentation already helped.)
• Add the functionality to separate individual nucleus in each image.
Python code of the project at kaggle: https://www.kaggle.com/dingli/keras-u-net-for-nuclei-segmentation