Recommended
Recommended
More Related Content
Viewers also liked
Viewers also liked (20)
Similar to Applying Computer Vision Techniques to Web Public Images for the Environmental Monitoring Purposes
Similar to Applying Computer Vision Techniques to Web Public Images for the Environmental Monitoring Purposes (20)
Recently uploaded
Recently uploaded (20)
Applying Computer Vision Techniques to Web Public Images for the Environmental Monitoring Purposes
- 1. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA SnowWatch Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring Roman Fedorov: roman.fedorov@polimi.it
- 2. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA UGC 2 Media User Generated Content (UGC) publicly available on the Web is reaching unprecedented size nowadays: • Facebook ~ 350 MM daily photo uploads • Flickr ~ 8.000 MM photographs with 3.5 MM daily uploads • Panoramio reached ~ 75 MM geo-tagged photograph It contains an enormous amount of latent implicit knowledge, the challenge is to extract it. Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
- 3. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA Introduction Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring 3 • Challenge: transform geolocated and timestamped public images depicting natural scenarios into virtual environmental monitoring stations. • Complement existing measurements.
- 4. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA Project 4 SnowWatch aims at: • improving snow analysis and water management through the acquisition and analysis of public media content depicting alpine mountains • Creating alpine snow media dataset with open contribution of new media and access to processed media and its results for any environmental researchers Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
- 5. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA Content Providers SnowWatch Engine SnowWatch Architecture User Photos 5 Flickr Photos Content Consumers Webcams API Media Storage Get Image Orientation, Identify Mountain Peaks Get Image Snow Pixel- level classification Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
- 6. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA Flickr Photographs Crawling 6 Does an image contain a clear a mountain slope view? - Heavily depends on the photograph shooting point altitude Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
- 7. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA Flickr Crawling Implementation 7 SVM Image Content-based Classifier: Bag of Visual Words / Fisher Vector SIFT/SSIM/GIST/HOG2x2 Descriptors Spatial Histogram, Vertical Bands Manually annotated 1658 samples (~70%) for training and validation and 710 samples (~30%) for testing. Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
- 8. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA Webcam Images Aggregation 8 Identified ~2.000 webcams depicting mountains in Alpine regions. Each webcam produces an image every 1’ – 15’. • 67% of the images are not suitable due to bad weather • More than 50.000 images and 6 GB of data acquired daily Through the extraction of the edges we can compute the percentage of the skyline clearly visible, and discard all images with visibility lower than a certain threshold, obtaining the following ROC curve varying the threshold: Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
- 9. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA Webcam Images Aggregation 9 Remaining small moving obstacles (such as low clouds, persons, vehicles) and varying illumination conditions are managed through the median aggregation of all daily images Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
- 10. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA Photo Peak Identification 10 An orientation of a geo-tagged photograph is needed. Given a Digital Elevation Model (a 3D terrain model) we can simulate the terrain silhouettes that should be seen from the photographer point of view. Steps Panorama Scale Edges Filter Match Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
- 11. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA Photo Panorama 11 Steps Panorama Scale Edges Filter Match Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
- 12. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA Photo Scaling 12 Photo must be scaled properly, so the mountains have the same pixel dimension on the photograph and on the panorama. This can be obtained by calculating the Field Of View (FOV) of the photo: Steps Panorama Scale Edges Filter Match Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
- 13. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA Photo Edge Extraction 13 Steps Panorama Scale Edges Filter Match Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
- 14. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA Photo Edge Filtering 14 Too many edges on the image that do not belong to a terrain contour! • Skyline detection algorithm is employed, and all the edge pixels above the skyline are removed, being considered obstacles or clouds. • Weighting mechanism is applied, which assigns decreasing weights to the edge pixels as the distance from the skyline increases. Steps Panorama Scale Edges Filter Match Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
- 15. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA Photo Alignment 15 Find the best overlap maximizing the vector cross correlation score: Steps Panorama Scale Edges Filter Match Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
- 16. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA Photo Peak Alignment 16 A global overlap between the photo and the panorama is often not enough, a neighborhood vector cross correlation adjustment is applied to each identified peak: Steps Panorama Scale Edges Filter Match Local Match Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
- 17. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA Photo Peak Identification 17 Fraction of photographs with orientation correctly identified Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
- 18. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA Snow Pixels Classification 18 Classifying each pixel as snow covered/terrain is much more than simply checking bright/dark pixels: Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
- 19. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA Snow Pixels Classification 19 SnowGround GroundGroundSnow Snow Snow Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
- 20. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA Snow Pixels Classification 20 Random Forest Classifier that considers pixel and pixel neighbors color, global and local intensity of the image. Trained on 59 manually labeled images (~ 7 Million pixels), outperform current methods: Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
- 21. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA Environmental Impact Evaluation 21 Environmental consistency evaluation of extracted snow cover indexes. No direct ground truth, how do we evaluate it? Practical utility w.r.t. an environmental model Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
- 22. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA Environmental Impact Evaluation 22 Use case: Como Lake water management Flooding vs Irrigation trend, what is the best solution? Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
- 23. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA SnowWatch 2°°°° year 23 If the physical snow estimations provided by ARPA (Italian EPA) are not available, Adding Virtual Snow Measures improves the performance by ~100%! Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
- 24. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA SnowWatch 2°°°° year 24 With some irrigation <-> flooding target solutions, using SnowWatch virtual measurements is even better than using ARPA snow estimation! Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring
- 25. DIPARTIMENTO DI ELETTRONICA, INFORMAZIONE E BIOINGEGNERIA Current and Future Work 25 Current: • MountainWatch: real time augmented reality mobile application • Expand tests on environmental impact Future: • Expand area of interest and content providers • Orthophotos and new snow indexes Applying Computer Vision Techniques to Public Web Images for Environmental Monitoring