Displacement, Velocity, Acceleration, and Second Derivatives
Pratik ibm-open power-ppt
1. Enhancing Computer Vision in
Low Visibility Conditions
Pushing the limits for machine learning and computer vision
Dr. Pratik Narang
Assistant Professor,
Department of Computer Science and Information Systems
BITS Pilani
2. What is Machine Learning?
● The science of getting computers to act without being explicitly programmed.
● Study of computer algorithms that improve automatically through experience
● Composition of non-linear transformation of data
Goal : Learn useful representation
● These representations, known as features, are derived directly from data
● There can be several varieties to achieve this goal - supervised, unsupervised,
reinforcement etc.
4. And, what is Computer Vision ?
“ Learning feature representations” from
visual data is not enough.
The classical 3 R’s of computational vision :
Reconstruction
Recognition
Re-organization
- Jitendra Malik, UC Berkeley
5. Real World
Images, videos, 3D scans,
sensor data, Structured Visual
Sets etc.
3
Model of the
Visual World
Image filtering, Interest points,
Feature Encoding, Wavelet
descriptors ...
2
Information
Object Recognition, Acton
Recognition, Video
Summarization, Large-Scale
Scene Understanding etc.
1
8. LIMITATIONS… ??
● Would you use an autonomous vehicle on a dark night, in a
severe snowstorm / sand storm, or in dense fog?
● Most approaches are designed to work on images/videos
captured in clear lighting conditions and good resolution.
● The performance deteriorates in low visibility conditions.
● Conditions of low-visibility are a frequent occurrence in real-life.
9. Computer Vision in Low Visibility?
● Indonesia's airlines face losses of flight delays and
cancellations caused by smoke from wildfires. From Sep 3 to
Sep 21, 750 flights have been cancelled.
● Delhi air pollution: 19 flights cancelled, over 550 delayed
and 37 diverted.
● Malaysia’s Penang International Airport chooses to shut
down as visibility drops below 800m.
11. “What I cannot create, I do not understand.”
● Computer Vision and Deep Learning
are great at classification models
● The tricky part is to develop models
and algorithms that can analyze and
understand treasure of data we have
today.
● Generative models are one of the
most promising approaches towards
this goal.
13. Generative Adversarial Networks
● Train two algorithms simultaneously (Goodfellow et al 2014)
“Competition is always a good thing. It forces us to do our best. A monopoly
renders people complacent and satisfied with mediocrity.” – Nancy Pearcy.
14. Generative Adversarial Net framework
● A game between two players
a. Discriminator D
b. Generator G
● D tries to discriminate between
a. A sample from data distribution
b. And a sample from the generator G
● G tries to trick D by generating samples that are hard for D to distinguish
from data
● Choose actions, pivoting on mixed strategy equilibrium
19. ● Given enough data of
○ Clear images
○ Images in low visibility
Can we learn to generate clear images from the images captured in low
visibility?
Note: the data need not be paired
Goal
21. Our Hypothesis
Clean Image
Low Visibility Images
Hyperspectral
Translation
Post Processing
HIDeGAN: A Hyperspectral-guided Image Dehazing GAN
Aditya Mehta, Harsh Sinha, Murari Mandal and Pratik Narang
CVPRW 2020
22. Image-to-Hyperspectral Translation
● Key Motivation: extending
the input image domain to
incorporate the complete
visual spectrum
HIDeGAN: A Hyperspectral-guided Image Dehazing GAN
Aditya Mehta, Harsh Sinha, Murari Mandal and Pratik Narang
CVPRW 2020
23. Visualizing the visual-spectrum
Comparison of reconstructed luminance across several spectral
bands for a hazy input image
HIDeGAN: A Hyperspectral-guided Image Dehazing GAN
Aditya Mehta, Harsh Sinha, Murari Mandal and Pratik Narang
CVPRW 2020
24. Visualizing the visual-spectrum
Comparison of reconstructed luminance across several spectral
bands for a hazy input image
HIDeGAN: A Hyperspectral-guided Image Dehazing GAN
Aditya Mehta, Harsh Sinha, Murari Mandal and Pratik Narang
CVPRW 2020
25. Visualizing the visual-spectrum
Comparison of reconstructed luminance across several spectral
bands for a hazy input image
HIDeGAN: A Hyperspectral-guided Image Dehazing GAN
Aditya Mehta, Harsh Sinha, Murari Mandal and Pratik Narang
CVPRW 2020
26. Hyperspectral Imagery?
● Obtain the spectrum for each pixel in the image of a scene, with the
purpose of finding objects, identifying materials.
● Special spectral cameras are required.
● Hyperspectral images consist of rich information from visual bands
● However, existing techniques for primary vision tasks such as
segmentation and classification are based on RGB images.
35. Low-Light Image Enhancement
Learning to Enhance Visual Quality via Hyperspectral Domain Mapping
Harsh Sinha, Aditya Mehta, Murari Mandal and Pratik Narang
AAAI 2021
36. Improving Image-to-Hyperspectral translation
● GAN based image-translation
have a common basic structure.
● The loss functions used are
focused entirely on the output
image space.
● Although the results are visually
pleasing, but they often cause
spectral aberrations. Durall et al. CVPR 2020
37. Spectral profile Optimization
Learning to Enhance Visual Quality via Hyperspectral Domain Mapping
Harsh Sinha, Aditya Mehta, Murari Mandal and Pratik Narang
AAAI 2021
38.
39. Comparative Results
Learning to Enhance Visual Quality via Hyperspectral Domain Mapping
Harsh Sinha, Aditya Mehta, Murari Mandal and Pratik Narang
AAAI 2021
40. Domain Aware Unsupervised Hyperspectral Reconstruction for Aerial Image Dehazing
Aditya Mehta, Harsh Sinha, Murari Mandal and Pratik Narang
WACV 2021
● RGB-HSI Unsupervised Domain Adaptation
● A novel GAN framework to incorporate HSI guidance in image-to-image
translation tasks
○ Distributional Discrepancy between RGB and hyperspectral images
○ Non-existence of hazy hyperspectral datasets
○ Robust to distortions and visual degradation
○ Combines adversarial distribution discrepancy alignment and cycle-consistency
constraint
Key Contributions
41. ● A novel neuro-symbolic approach to image-translation task using multi-
channel spectral-profile optimization
● Extending the viability of using hyperspectral images in image enhancement.
● Proposed a large-scale Hazy Aerial Image Dataset (HAI) with 65000 images
Learning to Enhance Visual Quality via Hyperspectral Domain Mapping
Harsh Sinha, Aditya Mehta, Murari Mandal and Pratik Narang
AAAI 2021
Key Contributions
42. References
● Harsh Sinha, Aditya Mehta, Murari Mandal and Pratik Narang.Learning to Enhance Visual Quality via Hyperspectral Domain
Mapping. Proceedings of the AAAI Conference on Artificial Intelligence 2021
● Aditya Mehta, Harsh Sinha, Murari Mandal and Pratik Narang. Domain-Aware Un-supervised Hyperspectral Reconstruction
for Aerial Image Dehazing. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision 2021
● Aditya Mehta, Harsh Sinha, Pratik Narang and Murari Mandal. HIDeGAN: A Hyperspectral-guided Image Dehazing GAN.
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops 2020
