How photo enhance looks today. What standart pipeline does it have. What interesting open source models and libraries you can use.

1. Photo Enhance

2. 01
Agenda
Problems and Architecture
• Pipeline
• Why Classical CV is still awesome
• Why GANs not the best solution
• HSV vs RGB vs YUV
Other steps
• Super Resolution
• Style Transfer
• Noise & aberration reduction. Why JPEG
is a problem
02
03
Color enhancing
• AutoHDR
• Real-Time photo color enhance
with Bilateral Filter

3. AGENDA:
Problems and
Architecture
01

4. Pipeline
Raw to 16-32bit Image Noise & aberration reduction
Color EnhanceStyle Transfer
Super ResolutionSkin editing / Painting

5. Why Classical CV is still awesome
● White box solution
● Many tasks are based on optical physics and clean mathematics so no NN
needed
● Controlled result which is very important for professional photo editing
● Very fast and can be used almost on any device
● No training and datasets needed
- Some very simple examples of color enhancing with polynomial in my article
- Building shape descriptor of image
- Unwarping example

6. Why GANs not the best solution
● To many uncontrolled artifacts
● Require a lot of computational resources
● Hard to train (Mode collapse, dying gradient, etc)
● Need big datasets
So it’s Money x Money x Money = Money3
And bad for data conversion

7. Why GANs not the best solution
BigGan - 1 training cycle: 15 days, 8xGPUs, ~$8k-$12k

8. RGB vs YUV vs HSV
RGB
- Good for NN
- Bad for classic CV
- Not human readable
YUV
- Good for NN
- So-so for classic CV
- Not human readable
- Efficient coding,
reduce bandwidth
HSV
- Bad for NN
- Good for classic CV
- Human readable
- Based on RGB (not
real colorspace)

9. 02AGENDA:
Color enhancing

10. 10
Auto HDR

11. Auto HDR
Raw - Sensors has different physical grid,
Formats has different strategy for saving data,
Sensors data has to be processed differently.
PAIN
Use 3d party software: pip install rawkit

12. Auto HDR
Overexposed better then Underexposed
That how sensors built

13. Auto HDR
White Balance - Photographers are wrong, classical CV not always right
Train NN to learn objects that should be
white and use this for your WB settings

14. Auto HDR
Camera Response Function (CRF) - relates scene irradiance to image intensities.
It needed for most HDR transformations
God save OpenCV developers

15. Auto HDR
CLAHE - Adaptive histogram equalisation. Useful algorithm to solve histogram
imbalance of the image. Visually similar to what HDR trying to do
OpenCV, scikit-image and many others

16. Auto HDR
HDR - Basically we fuse few images with different parameters to get good
exposure of every object in the image. Tone Mapping technique used to map 32bit
image to 8bit image(for most devices)
OpenCV

17. Auto HDR
And yes we can do same with one image

18. Auto HDR
Now merge all things together and we will get something like this

19. Auto HDR
Now merge all things together and we will get something like this

20. 10
HDRNet
Real-Time photo color enhance with
Bilateral Filter

21. Bilateral Filter
Main idea of it that it use not only spatial distance,
but also intensity distance.

22. Bilateral Filter

23. Bilateral Grid

24. HDRNet

25. HDRNet
Input Result Ground Truth

26. HDRNet
Input Result Ground Truth

27. HDRNet
https://github.com/creotiv/hdrnet/
https://github.com/creotiv/hdrnet-pytorch/ - under development
- Only color enhance
- Uses some sort of segmentation with Bilateral Grid
- Supervised learning
- Cant handle edge noise
- Small network cant handle vast input image params (should be unified)

28. HDRNet
Other interesting solutions
https://github.com/nothinglo/Deep-Photo-Enhancer
https://github.com/yuanming-hu/exposure

29. 03
AGENDA:
Other steps

30. Super Resolution

31. Super Resolution - SRGAN

32. Super Resolution - ESRGAN

33. Super Resolution - SFTGAN

34. Super Resolution - SFTGAN
Input Bicubic ESRGAN

35. Super Resolution - SFTGAN

36. Super Resolution - ESRGAN
Changes from SRGAN
- Deeper model using Residual-in-Residual Dense Block (RRDB)
- Use Relativistic average GAN instead of the vanilla GAN.
- Improve the perceptual loss by using the features before activation.
- Pixel Shuffle replaced by Nearest Interpolation
- Pretrain Generator model before use in GAN
- Removed Batch Normalization to remove artifacts
https://arxiv.org/abs/1809.00219
https://arxiv.org/abs/1804.02815
https://github.com/xinntao/ESRGAN
https://github.com/xinntao/SFTGAN

37. Super Resolution
What we can see from this improvements?
- Context and Data is what matter
- All things like BN, Max Pooling which destroy data making worse for generator
- Understanding materials behind the image can greatly improve quality
- Skip connection passing context to the end of network also improve results

38. Style Transfer

39. Style Transfer

40. Style Transfer

41. Style Transfer

42. Style Transfer
What matter
- No MaxPool on Encoder network use strides
- Style poisoning works not so good as we want
- No BatchNorm on Transformer network

43. Adaptive Style Transfer

44. Adaptive Style Transfer

45. Adaptive Style Transfer
What matter
- GAN, for such tasks when Loss hard to build
- Multiple Artworks per style
- Additional loss - maybe, hard to tell
https://compvis.github.io/adaptive-style-transfer/

46. Noise & Aberrations

47. Jpeg Noise
https://csfieldguide.org.nz/en/chapters/coding-
compression/image-compression-using-jpeg/
Low frequencies
High frequencies

48. Jpeg Noise
- Waste context loss
- Noise position static
- Noise patterns repetitive
- Transform function is not
bijective
F (X1)
F (X2)

49. Jpeg Noise
UnBlock http://www.johncostella.com/unblock/
Wavelet transforms https://sci-hub.tw/10.1109/ICIEA.2006.257369
+
Detalisation Gan Network ~ SuperRes Network
https://pywavelets.readthedocs.io/en/latest/

50. Hot Noise aka Grain Noise
God Bless AutoEncoders
Simple MNIST Example http://bit.ly/2TDQaFr
Noise2Noise (Unsupervised) https://github.com/NVlabs/noise2noise
(https://arxiv.org/abs/1803.04189)

51. Chromatic Aberration

52. Chromatic Aberration
TCA_Correct + Hugin(Fulla) https://wiki.panotools.org/Tca_correct
https://wiki.panotools.org/Lens_correction_model

53. Thanks
Help to save homeless animals
Donate to animal shelter
BIT.LY / MLCATS
fb.me / anikishaev t.me / ml_world