Topic of presentation: Deep learning for satellite imagery colorization and distance measuring. The main points of the presentation: Using modern techniques we compared existing methods for colorization from the perspective of satelite maps. After this we built our own engine for measuring the distances on the maps. http://dataconf.com.ua/index.php#agenda #dataconf #AIBDConference

1. Deep learning for
satellite imagery
colorization and
distance measuring
Lviv AI&BigData Day, November 4
Volodymyr Getmanskyi
ELEKS data science team
skype: paradoxx_xx

2. Automatic image colorization is the task of adding colors to a new grayscale image without any
user intervention. This problem is ill-posed in the sense that there is no unique colorization of a
grayscale image without any prior knowledge. Indeed, many objects can have different colors:
artificial objects, such as plastic objects which can have random colors, but also for natural objects
such as tree leaves which can have various nuances of green and brown in different seasons,
without significant change of shape.

3. Most of the modern methods uses pretrained convolutional
neural networks and extracts the information about
features/objects from the image. Then there is a possibility to
upscale them to original size and concatenating them all
together.
Also most of them uses some simplification to train the models to
produce two color channels which can be concatenated with the
grayscale input channel to produce the YUV/CIELUV image.

4. The most obvious loss function is a Euclidean distance function between the network RGB output image
and the true color RGB image (distance in UV space – from previous slide).
We, however, notice that in numerous cases, the images colorized with such approach are mostly sepia-
toned and muted in color.
To understand why, consider a pixel that exists in a flower petal across multiple images that are identical,
save for the color of the flower petals. Depending on the picture, this pixel can take on various tones of
red, yellow, blue, and more. With a regression-based system that uses an loss function, the predicted
pixel value that minimizes the loss for this particular pixel is the mean pixel value. Accordingly, the
predicted pixel ends up being a mixture of the possible colors.

5. Another problem is that pre-trained CNN weights, that was used for feature extraction, are from
estimators that were trained on reсognizing different objects but not the objects from satellite images.
Also all trained models for colorizing (with RGB/UV distance minimizing), were trained to restore
colors for different objects but not for satellite images.

6. We tried to test existing solutions but most of them are facing the problems that we mentioned earlier
and demonstrates low performance:
* http://demos.algorithmia.com/colorize-photos, http://www.colorizephoto.com/converter
http://pinetools.com/colorize-image, https://github.com/richzhang/colorization

7. After the first steps we found more efficient ways (with no or small sepia effects) and also try to train
the estimator based on maps.
Below you can see our basic results with adding color histograms and color mapping as additional map
features:

8. Below you can see our basic results with models training (black&white original, colorized, original):
First iteration of our model (sepia warning): Weak trained model (1K iteration, 20K
samples without augmentation):

9. Measuring distance is a key tool in map reading and is especially useful for hikers and cyclists who want
to measure how far they have travelled or how far they wish to go based on raw maps. BTW, such
distance measuring is helpful for understanding map scale that is the first step for camera height
evaluation (the distance from sea level to eye level) or altitude of the plane/drone above sea / terrain level
(ASL).
So the main task here is to get the scale. If you understand the map scale number that's great and you
can still measure distances accurately.
So based on such thoughts we decided to build the estimator that can measure distances on real
unlabeled maps using some patterns like shadows, trees’ crowns, roofs and roads.

10. For data gathering we used typical browser and google maps with ImageGrab and win32api python
libraries the same as for colorization task (20K samples), but there are few image processing methods
before the DL usage. So here we need to extract/recognize the scale that will be our labels. Scale is
located on the right bottom of the image, so we need to detect the defining point from which we can move
scale detection. Here we can simply slides the template image over the input image (as in cascade
methods) and compares the template and patch of input image with some distance metric.

11. After the successful detection we can extract two elements that we need for labeling:
- Current measurement units (using simple OCR like tesseract and further rule-based engine)
- Scale Line length (simply calculated the location of rows’ mode with continuous range of white
columns)

12. After the validation comparison with VGG architecture we choose AlexNet architecture with all
convolutional layers frozen (transfer learning for efficient feature extraction):
# Real-time data preprocessing
img_prep = ImagePreprocessing()
img_prep.add_featurewise_zero_center()
img_prep.add_featurewise_stdnorm()
# Real-time data augmentation
img_aug = ImageAugmentation()
img_aug.add_random_flip_leftright()
img_aug.add_random_rotation(max_angle=135.)

13. Results (examples from testset, mae≈25%):
Image Side length Predicted length Image Side length Predicted length
168 m 213 m 45 m 69 m
1333 m 954 m 4970 m 4720 m

14. Through our small experiment, we have checked the efficiency of using deep neural networks to colorize
black and white satellite images the same as for measuring maps’ scale. So our conclusions are the
following.
In particular, formulating the maps colorization task as a classification problem can yield colorized maps
that are arguably much more aesthetically-pleasing than those generated by a baseline regression-based
model (with sepia), and thus shows much promise for further development.
Also, redesigning the system around an adversarial network may improve results, since instead of
focusing on minimizing the loss on a per-pixel deviation, the system would learn to generate/colorize
pictures that compare well with real-world maps.
From distance measuring approach we recommend to use similar feature extraction (transfer learning)
that helps to feed the real objects (trees, roofs, roads) into distance result.

15. Inspired by Technology.
Driven by Value.
Find us at eleks.com
Have a question? Write to eleksinfo@eleks.com