LocWeb2018 | WWW 2018 Workshop

1. Firma convenzione
Politecnico di Milano e Veneranda Fabbrica
del Duomo di Milano
Aula Magna – Rettorato
Mercoledì 27 maggio 2015
A Location-Based Virtual Reality
Application for Mountain Peak Detection
Antonio La Salandra - Piero Fraternali - Darian Frajberg
LocWeb2018 | WWW 2018 Workshop
Lyon, April 24, 2018

2. Department of Electronics, Information and Bioengineering
Introduction and motivation
User Generated Content publicly available on the web is a
valuable resource for the extraction of data useful for scientific
and social purposes
• Facebook: ~300M daily photo uploads
• Twitter: ~500M tweets sent per day
• Instagram: ~95M photos and videos shared per day
Smartphones are the most used means of acquiring and sharing
data

3. Department of Electronics, Information and Bioengineering
Introduction and motivation
Citizen Science can be described as the process of active
engagement of citizens and communities to collaborate to solve
common social issues
Mobile Outdoor Applications are powerful means of promoting
the collection of geo-referenced data useful to support scientific
researches
Several mobile applications have been released for environmental
monitoring purposes by following the citizen science paradigm:
plant study, climate change, noise pollution, bird migration…

4. Department of Electronics, Information and Bioengineering
The SnowWatch Project
Our contribution: monitoring of mountain snow coverage, and
hence of water availability, through the analysis of low-cost
multimedia content

5. Department of Electronics, Information and Bioengineering
From 2D to 3D visualization of geo-located data
Augmented Reality (AR) and
Virtual Reality (VR) enable the
visualization of geo-located
data from the classic 2D planar
representation to 3D immersive
environments

6. Department of Electronics, Information and Bioengineering
The PeakLens Application
PeakLens is an AR location-based mobile app developed for
collecting photos of mountains for the SnowWatch data set

7. Department of Electronics, Information and Bioengineering
The PeakLens Application
The application is used as a
means of engaging users in the
collection of data by providing
them with a peak identification
service in real-time
It was released for Android in
February 2017 and it has already
achieved about 200k installs, 4
stars’ ratings and most positive
feedbacks

8. Department of Electronics, Information and Bioengineering
PeakLensVR
PeakLensVR aims at boosting the community of PeakLens users
and SnowWatch contributors
• It enables the acquisition of geo-located, geo-referenced,
timestamped panoramic photos of mountain landscapes by
using a mobile phone
• It extends the peak identification service to cope with such
images
• It enables the visualization of captured panoramas enriched with
information about peaks in view in a virtual reality environment

9. Department of Electronics, Information and Bioengineering
PeakLensVR – Processing Pipeline
Overview of PeakLensVR main processing steps and modules

10. Department of Electronics, Information and Bioengineering
PeakLensVR – Scene acquisition
A sequence of photos is taken with the smartphone camera

11. Department of Electronics, Information and Bioengineering
Data about user geographical position, smartphone spatial
orientation and camera parameters are saved as photo
metadata to be used in the next phases of the processing
pipeline
• Geographical coordinates: latitude and longitude
• Spatial orientation: azimuth, pitch and roll
• Camera parameters: horizontal and vertical field of views,
frame width and height
PeakLensVR – Scene acquisition

12. Department of Electronics, Information and Bioengineering
PeakLensVR – Panorama composition
Image sequence and sensor data are used to compose the
panoramic image

13. Department of Electronics, Information and Bioengineering
PeakLensVR – Panoramic image enrichment with peaks
PeakLens integrate geo-data from different sources and expose
them through an API
Main data sources:
• OpenStreetMap
• NASA SRTM DEM
Server-side

14. Department of Electronics, Information and Bioengineering
PeakLensVR – Panoramic image enrichment with peaks
Different DEM and POI data representations are accessible
through the PeakLens API

15. Department of Electronics, Information and Bioengineering
PeakLensVR – Panoramic image enrichment with peaks
Sensor-based peak positioning
GPS Coordinates
Peaks + Skyline JSON
Orientation data + Camera parameters
Panorama augmented with peak data

16. Department of Electronics, Information and Bioengineering
PeakLensVR – Panoramic image enrichment with peaks
Content-based position refinement
Skyline extraction
+
Skyline matching

17. Department of Electronics, Information and Bioengineering
PeakLensVR – Panoramic image enrichment with peaks
Sensor-based vs Content-based image alignment

18. Department of Electronics, Information and Bioengineering
PeakLensVR – Visualization and sharing
VR mode visualization by using a Google Cardboard VR headset

19. Department of Electronics, Information and Bioengineering
PeakLensVR – Visualization and sharing
Fullscreen immersive mode visualization on smartphone

20. Department of Electronics, Information and Bioengineering
PeakLensVR – Visualization and sharing
Sharing and visualization on Facebook 360 platform

21. Department of Electronics, Information and Bioengineering
Experimental Study: Goal
• Sensor-based peak positioning errors
• Accuracy of peak positions identified by using sensors
• Smartphone sensor precision
• Content-based position refinement errors
• Improvements introduced with respect to the sensor-
based approach
• Precision of the final peak positioning

22. Department of Electronics, Information and Bioengineering
Experimental Study: Dataset
• N. Panoramic Photos: 10
• Location: Como lake area
in North Italy and Pollino
National Park in South Italy
• Horizontal Field of View:
96 – 360 degree
• N. Peaks: 1 – 11
• Photos have been taken in
various weather conditions,
with different exposures
and, in some of them, with
objects partially occluding
the skyline

23. Department of Electronics, Information and Bioengineering
Experimental Study: Ground truth
Manually computed with the support of an online service we developed, that given
the geographical coordinates of the view point, returns a digital representation of
the panorama visible from that point, enriched with the labels of each visible peak
placed in the correct position

24. Department of Electronics, Information and Bioengineering
Experimental Study: Results
Sensor-based and Content-based Euclidean Degree Error (EDE) comparison

25. Department of Electronics, Information and Bioengineering
Experimental Study: Results
Classification of the frequency distribution of peaks with respect to their
EDE after the sensor-based and the content-based matching approaches

26. Department of Electronics, Information and Bioengineering
Experimental Study: Results
Sensor-based: 35% of peaks have EDE < 3° and 17,5% have EDE > 6°

27. Department of Electronics, Information and Bioengineering
Experimental Study: Results
Content-based: 85% of peaks have EDE < 3° and 0% have EDE > 6°

28. Department of Electronics, Information and Bioengineering
Conclusions
• The application prototyped during this work may become a
new channel for the acquisition of geo-located, geo-
referenced and timestamped panoramic photos useful to
environmental monitoring purposes
• The content-based alignment technique developed during the
SnowWatch project has been proved to work with panoramic
images involving an average 54% improvement compared
with the sensor-based peak positioning approach

29. Department of Electronics, Information and Bioengineering
Future work
• The introduction of the crowdsourcing module and the
refinement of the sharing mechanism, to allow users to
contribute by sharing their own photos with the community
• The rendition in VR mode of other objects of interest, such as
trails and huts, besides peaks
• The creation of an online platform that aggregates the user-
generated panoramas to automatically produce maps and
virtual tours in mountain areas

30. Department of Electronics, Information and Bioengineering
Q&A
Thank you for your attention
Questions?
A Location-Based Virtual Reality Application
for Mountain Peak Detection
Antonio La Salandra - Piero Fraternali - Darian Frajberg
antonio.lasalandra | piero.fraternali | darian.frajberg
@polimi.it