DSAA2017 4th IEEE International Conference on Data Science and Advanced Analytics Tokyo, Japan October 2017

1. Heterogeneous information integration for
mountain augmented reality mobile apps
Darian Frajberg
Piero Fraternali
Rocio Nahime Torres
Department of Electronics, Information and Bioengineering, Politecnico di Milano
October 19, 2017

2. 2
Introduction and motivation
Mobile outdoor Augmented Reality (AR) applications are
an emerging category of solutions that hold the promise
to design engaging user experiences around geo-
referenced data
They exploit the sensor readings of the device:
- GPS position
- Orientation
- Motion
- Camera frame content
Their goal is to understand what the user is watching at
and enrich the view captured by the device with
contextual information

3. 3
Introduction and motivation
Mobile outdoor applications offer a
new way to promote the collection
of geo-referenced information
through citizen science
Citizen science engages common
citizens to help to solve a scientific
problem, such as gathering
environmental data (e.g., images of
plant species and mountain snow
coverage)

4. 4
Related work
Some mobile applications for crowdsourcing environment purposes:
- Plants
- Climate change
- Birds
- Water
- Noise pollution
- Meteorology
Our contribution:
- Mountains

5. The realization of an outdoor mobile AR application relies
on a multi-stage and multi-source processing pipeline
5
Outdoor Augmented Reality for mountain data collection

6. 6
Outdoor Augmented Reality for mountain data collection

7. 7
Outdoor Augmented Reality for mountain data collection
Geo object acquisition
{"lat":0.1707444, "lng":6.6452082,
"elevation":861, "id":"2331692496",
"name":"Pico Maria Fernandes"}
<node id= "57697260” visible="true”
lat="35.3631518" lon="138.7302707">
<tag k="name:en" v="Mount Fuji"/>
<tag k="ele" v="3776"/>
</node>

8. 8
Outdoor Augmented Reality for mountain data collection

9. 9
Outdoor Augmented Reality for mountain data collection
Model integration
(ASTER GDEM) (NASA SRTM)

10. 10
Outdoor Augmented Reality for mountain data collection

11. Objects to model registration + Data decompression
11
Outdoor Augmented Reality for mountain data collection

12. 12
Outdoor Augmented Reality for mountain data collection

13. 13
Outdoor Augmented Reality for mountain data collection
Objects identification

14. 14
Outdoor Augmented Reality for mountain data collection

15. 15
Outdoor Augmented Reality for mountain data collection
Objects ranking
User-defined criteria:
• Distance
• Height
• Popularity
• Category
• Etc. Filter Rank

16. 16
Outdoor Augmented Reality for mountain data collection

17. Objects projection + Windowing
17
Outdoor Augmented Reality for mountain data collection

18. 18
Outdoor Augmented Reality for mountain data collection

19. 19
Outdoor Augmented Reality for mountain data collection
Rendition

20. Satisfactory user experience must face several challenges:
• High object positioning accuracy
• Fast response time
• Low memory and battery consumption
Applications for outdoor usage must cope with:
• Uncontrolled viewing conditions
• Uncertain Internet connection
• Uncertain positioning
20
Outdoor Augmented Reality for mountain data collection

21. 21
Outdoor Augmented Reality for mountain data collection
Computer Vision and AI can enhance positioning accuracy
(Sensor-based) (CV-based)

22. 22
Outdoor Augmented Reality for mountain data collection
Image to terrain alignment on board (CV-based)

23. 23
Dimensions of heterogeneous information integration
Dimensions:
- Object semantics: What is it?
- Objects provenance: Where does it come from?
- Object storage and availability: How is it stored?
Online/Offline?
- Object data compression: Is it compressed?
- Object media type: What are the types of their related files?
- Object visualization: How to render?

24. 24
Dimensions of heterogeneous information integration
Semantic Provenance Storage/
Availability
Compression Media type Visualization
DEM
NASA SRTM Online
Offline
Yes Point cloud Virtual
panorama
ASTER GDEM Online
Offline
Yes Point cloud Virtual
panorama
Peaks
Open Street
Map
Online
Offline
No Text, image,
link
Point
Peakware Online
Offline
No Text Point
PeakBagger Online
Offline
No Text, image,
link
Point

25. 25
Dimensions of heterogeneous information integration
Semantic Provenance Storage/
Availability
Compression Media type Visualization
Waterbody
Waterways
guide
Online
Offline
Yes Text, image,
link
2D area
Geonames Online
Offline
Yes Text, link 2D area
Wikidata Online
Offline
No Text, link Point
Alpine huts
Norwegian
trekking
association
Online
Offline
No Text Point
Mountinhuts Online
Offline
No Text Point

26. 26
Dimensions of heterogeneous information integration
Semantic Provenance Storage/
Availability
Compression Media type Visualization
Castles
GPS Data Team Online
Offline
No Text Point
Wiki Data Online
Offline
No Text, image,
link
Point
Trails
Norwegian
trekking
association
Online
Offline
Yes Text Polyline
Wikiloc Online
Offline
Yes Text, image,
link
Polyline
Trailforks Online
Offline
Yes Text, image,
link
Polyline

27. 27
Dimensions of heterogeneous information integration
Semantic Provenance Storage/
Availability
Compression Media type Visualization
Towns
Open Street
Map
Online
Offline
Yes Text 2D area
GeoNames Online
Offline
No Text, link Point
Events
Get Events Online No Text, image,
link
2D area
Facebook Online No Text, link 2D area

28. PeakLens is an outdoor AR mobile application that
identifies mountain peaks and overlays them in
real-time on the view
It was released for Android in February 2017 and it has already
achieved more than 100k installs, with a rating of 4/5 and
mostly positive feedback
28
Usage experience

29. Negative reviews and change requests correspond to:
- Data quality:
- Issue: Some absent or erroneous peak meta-data
- Solution: Fusion of more data sources and crowdsourcing to annotate missing metadata (IN
PROCESS)
- Data management:
- Issue: Missing option to store offline data in the SD card of the device
- Solution: Implementation of the requested feature (DONE)
- Usability:
- Issue: Scroll button to visualize multiple subsets of peaks not understood
- Solution: New design of pagination mechanism, also with other parameters such as altitude
and distance range of objects to display (IN PROCESS)
- Precision:
- Issue: Misplacement of peaks due to erroneous device’s sensors
- Solution: Optimization of the algorithm to perform even better alignments (IN PROCESS)
30
Usage experience

30. 31
Environment application
Geo-referenced and time-stamped images of mountain ranges can be used for
environmental purposes such as monitoring:
- Snow coverage
- Plant diseases
- Vegetation recovery from bush fires
PeakLens is developed as a complement to the SnowWatch project, which aims at
creating novel and low cost tools to monitor and predict the water availability in the
dry season in mountain regions, where water supply is mostly conditioned by the
snow cover in the mountain ranges

31. Concept
– Ongoing experience of the development of a real world mobile outdoor AR
app that integrates DEM and GIS data to enrich the view of mountain
panoramas
– Different classes of data that can be exploited
– Technical challenges to face
Future work
– Harvest all of the mentioned classes of data, their encoding and compression
for offline and online usage, and their representation on the screen
– Continue soliciting feedback to improve the user experience
– Introduce 2 crowdsourcing modules to allow users to contribute by uploading
images and also by improving the correctness and completeness of the
different points of interest
32
Conclusions

32. 33
Thanks For Your Attention!
Heterogeneous information
integration for mountain
augmented reality mobile apps
Darian Frajberg
Piero Fraternali
Rocio Nahime Torres
darian.frajberg | piero.fraternali | rocionahime.torres
@polimi.it