Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Multimedia on the mountaintop: presentation at ACM MM2016

443 views

Published on

This paper merges multimedia and environmental research
to verify the utility of public web images for improving water
management in periods of water scarcity, an increasingly
critical event due to climate change. A multimedia processing
pipeline fetches mountain images from multiple sources
and extracts virtual snow indexes correlated to the amount
of water accumulated in the snow pack. Such indexes are
used to predict water availability and design the operating
policy of Lake Como, Italy. The performance of this informed
policy is contrasted, via simulation, with the current
operation, which depends only on lake water level and day of
the year, and with a policy that exploits ocial Snow Water
Equivalent (SWE) estimated from ground stations data and
satellite imagery. Virtual snow indexes allow improving the
system performance by 11.6% w.r.t. the baseline operation,
and yield further improvement when coupled with ocial
SWE information, showing that the two data sources are
complementary. The proposed approach exempli es the opportunities
and challenges of applying multimedia content
analysis methods to complex environmental problems.

Published in: Environment
  • Be the first to comment

Multimedia on the mountaintop: presentation at ACM MM2016

  1. 1. Multimedia on the Mountaintop: Using Public Snow Images to Improve Water Systems Operation A. Castelletti, R. Fedorov, P. Fraternali, M. Giuliani Politecnico di Milano, Italy ACM MM 2016, Amsterdam BNI session
  2. 2. The (hopefully brave new) idea • There is a lot of multimedia content out there, produced by – People – Ground sensors • There are many environmental problems that lack affordable and accessible input data • Question: is public web visual content good enough to help in such environmental problems?
  3. 3. Observing the earth • Not everything can be done from above • There is not a single satellite product good for all • (Useful) satellite products are costly • Clouds may be a problem
  4. 4. The grand challenge: water scarcity • Climate change, urban concentration and agriculture put water resources under stress • Predicting future availability is key • When you have mountains, water is stored as snow UK_WATER SUPPLY UTILITY 15 million customers 2.6 Gl/day drinking water 3 billion $ revenue (2013-14)
  5. 5. The content Input • User generated – 700.000 Flickr images crawled so far within 300x160 km • Sensor generated – 2000 webcams queried every minute (10 – to 1500 images per web cam per day) – More than 10M images crawled so far Output • Virtual Snow Indexes: numerical time series that are a proxy of the quantity of water stored in the snow pack (Snow Water Equivalent – SWE)
  6. 6. The multimedia pipelines • Differences – Web cam images have high temporal density, UG images have broader spatial coverage – UG photos searched by keywords may be irrelevant, webcam images always portrait mountains – UG photo mountain classifier already discards bad weather images
  7. 7. UG Image relevance • 7000 images randomly sampled and used for a crowdsourcing experiment: “Do you see a mountain in this picture?” • Classifier trained (94% precision, 96.3% recall)
  8. 8. Webcam image enhancement Remove/attenuate: • Variability of illumination • Shadows • People & irrelevant objects Daily median image
  9. 9. Mountain peak identification orginal image edge maps skyline estimation DEM generated virtual panoram VCC best matching
  10. 10. Snow mask extraction Snow classification at the pixel level Snow mask extraction
  11. 11. Snow Virtual Indexes
  12. 12. The case study • Regulation of mountain inflow dependent lakes Lake Como Hydropower reservoir Power plant Como city Penstock River Adda River Adda Legend Lario Lario catchment River Irrigated area 0 10 20 30 40 505 Kilometers Catchment area Lake Como 4500 km2 Reservoirs Lake Como 247 Mm3 Alpine HP 545 Mm3 Stakeholders Farmers: irrigated area 1400 km2 Floods: lake and downstream ….
  13. 13. Local folklore
  14. 14. Formalization: 2 objectives optimization • Decide the daily lake outflow ( lake level) • So to – Maximize water for downstream irrigation – Minimize # of flood days • Respecting – Minimum outflow requirement for ecological preservation of effluents • Based on – Policy input (X) • Regulator's policies – Baseline: regulator only considers lake level and day of year – Upper bound: regulator knows the water that will be available (lake inflow) in the future – P_x: regulator knows partial information (x) on the water that will be available (lake inflow) in the future • What is X? – P1: Official snow water equivalent data estimated from Region Lombardy – P2: virtual snow indexes from nearby mountain images – P3: official SWE data + virtual snow indexes PS: Upper bound policy can be calculated retrospectively for the past, where you know how much water you actually got day by day
  15. 15. Assessment method Select information based on its expected value (Iterative Input Selection) Design control policy based on selected input information Quantify performance of policy + selected information Quantify value of perfect information Expected Value of Perfect Information (EVPI) Inflow data series Outflow data series Baseline policy Upper bound policy Input data series (exogenous variables) Most Valuable Information (X) X_informed control policy (P_x) J(P_x) Performance of P_x Performance metrics Hyper Volume Indicator (HV) Performance improvement over baseline (ΔHV)
  16. 16. Assessment results
  17. 17. Thank you & … see you soon in the PlayStore
  18. 18. Content processing pipeline • Photo contains/does not contain mountain landscape binary classifier – SVM with Dense SIFT, Spatial Histograms. 7k annotated images (majority of 3 votes). 95.1% Accuracy on balanced dataset. • Peak identification / Photo orientation estimation – Ad-hoc algorithm with edge extraction and vector cross- correlation. 160 images manually aligned w.r.t. Digital Elevation Model. 75-81% of images correctly aligned (depending on weather conditions). • Pixel-wise snow/non snow classifier Random Forest, trained/evaluated on 60 manually segmented images (single annotator) for a total of 7M of labeled pixels. 91% accuracy.
  19. 19. Iterative input selection Select information based on its expected value (Iterative Input Selection) Design control policy based on selected input information Quantify performance of policy + selected information Quantify value of perfect information Expected Value of Perfect Information (EVPI) Inflow data series Outflow data series Baseline policy Upper bound policy Input data series (exogenous variables) Most Valuable Information (X) X_informed control policy (P_x) J(P_x) Performance of P_x Performance metrics Hyper Volume Indicator (HV) Performance improvement over baseline (ΔHV) D=distance metric
  20. 20. Policy search Select information based on its expected value (Iterative Input Selection) Design control policy based on selected input information Quantify performance of policy + selected information Quantify value of perfect information Expected Value of Perfect Information (EVPI) Inflow data series Outflow data series Baseline policy Upper bound policy Input data series (exogenous variables) Most Valuable Information (X) X_informed control policy (P_x) J(P_x) Performance of P_x Performance metrics Hyper Volume Indicator (HV) Performance improvement over baseline (ΔHV)
  21. 21. Good decisions matter WATER DEFICIT FLOOD THRESHOLD EFFECT OF REGULATION
  22. 22. For more info • A. Castelletti, R. Fedorov, P. Fraternali, M. Giuliani: name.surname@polimi.it • http://snowwatch.polimi.it/

×