4B_3_Automatically generating keywods for georeferenced imaged


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

4B_3_Automatically generating keywods for georeferenced imaged

  1. 1. Automatically generating keywords for georeferenced images<br />Ross Purves1, Alistair Edwardes1, Xin Fan2, Mark Hall2 and Martin Tomko1<br />1University of Zurich<br />2University of Sheffield<br />3Cardiff University<br />
  2. 2. What I want to talk about…<br />How are images indexed?<br />How do we describe images?<br />How can we (semi?)-automaticallygenerate (geographically related) keywords that describe image content?<br />
  3. 3. Image indexing<br />Image indexing is crucial, since image search is almost always based on keywords<br />Keywords basically assigned in four ways:<br />Manual annotation<br />Terms freely chosen by indexers (these might include tags nowadays)<br />Terms selected from a controlled vocabulary<br />Automatic annotation<br />Terms extracted from text thought to be related to an image<br />Terms assigned to an image on the basis of content-based techniques<br />
  4. 4. …but…<br />Image search (based on such<br />annotation) often does not meet user<br />expectations<br />
  5. 5. The semantic gap…<br />“…the lack of coincidence between the information that one can extract from the visual data and the interpretation that the same data have for a user in a given situation.” <br />(Smeulders et al., 2000)<br />
  6. 6. …in other words<br />Search fails because annotations are not the same as search terms…<br />…unlike text search – we are using a proxy for content<br />The challenge is, therefore, to develop methods that better match user expectations -> and are therefore more universal<br />Taken up by the ESP game/ Google’s image labeller<br />
  7. 7. How should we describe these pictures?<br />Image Javier Corripio <br />Image Google Streetview<br />
  8. 8. Theory of image description<br />Panofsky-Shatford facet matrix – Shatford (1986) <br />
  9. 9. The where facet<br />In the Tripod project, we are especially interested in describing images based on where they were taken<br />We suppose, that in the medium-term all images will be georeferenced<br />Panofsky-Shatfordmatrix suggests some ways we might describe images<br />I’m going to concentrate on where/generic of<br />(Martin will talk tomorrow about one element of<br />the where/ specific of)<br />
  10. 10. Where/ generic of<br />The generic of represents a kind of place<br />Kinds may relate to basic levels<br />Basic levels (e.g. Rosch, 1977) are terms used in natural language which are informative and summative (e.g. table vs. furniture or square table) <br />Basic levels are probably very good indexing terms<br />For example:<br />Mountains, valleys, desert, ravines<br />Street, pavement, house<br />Empirical research has explored what these basic levels are in human subject experiments and we explored them in UGC/VGI<br />How can we generate terms which relate to these kinds automatically?<br />
  11. 11. Basic process<br />Start with an image associated with a coordinates and (sometimes) direction<br />Basic process<br />Identify potential visible area<br />Query spatial data within visible area for candidate keywords<br />Rank and filter candidate keywords to generate final keyword list<br />
  12. 12. Identifying visible area<br />Camera parameters extracted from EXIF<br />Content-based check for building combined with landcover data to determine urban or rural case – controls range of viewshed<br />If no direction information, 360° viewshedgenerated, otherwise sector defined on basis of camera parameters<br />
  13. 13. Data: OpenStreetMap, ©SwissTopo<br />and ©Ordnance Survey<br />
  14. 14. Basic keywording process<br />Identify available data in region<br />Query data using viewshed for data classes<br />Map data classes to (potentially multiple) concepts – remove duplicates<br />Concepts expanded to multiple (potentially multi-lingual) candidate keywords<br />Rank candidate keywords according to area, probabilistic salience and web salience<br />
  15. 15. Initial result<br /><ul><li>Datasets: Corine, OS Strategi, OS VectorMap
  16. 16. Spotted feature types:
  17. 17. Corine:0_wcsNodata, 312_coniferousForest, 322_moorsAndHeathland, 512_waterBodies
  18. 18. Strategi:5610, 5250, 5385
  19. 19. VectorMap:PrivateRoadRestrictedAli, MinorRoadAlignment, UnimprovedGrass, HeathlandAndUnimprovedGrass, ConiferousWoodland, WaterBodies, RiverWaterPolygon, Lakesselect, MixedWoodland, BroadleafedWoodland, WaterFeature, BuildingPolygon</li></li></ul><li>Linking spatial data to concepts<br />Data: ©SwissTopo<br />
  20. 20. Initial filtering <br /><ul><li>Candidate keywords expanded from concept ontology mappings
  21. 21. Duplicates removed
  22. 22. Returned ranked by area in viewshed</li></ul>Woodland, Lakes, NaturalGrassland,<br />Reservoir, SingleBuiltWorksByFunction,<br />Tracks, Moor, Streets, Stream, Pond,<br />Forest, ConiferousTrees, Tarn, Heath,<br />Byways<br />
  23. 23. Final filtering and ranking<br />Ranking and filtering based on spatial extent, descriptive and web salience<br />Spatial extent: concepts covering large area typically important (but favours landcover related concepts)<br />Descriptive salience: weight rare concepts with respect to surroundings higher than common (e.g. a village shop is more salient than one on a high street)<br />Web salience: Query web with keywordsand local toponymsto find common combinations <br />
  24. 24. Keywords:Reservoir,<br />Forest, Moors, Woodland<br />
  25. 25. Keywords: Lake,<br />Meadows, Settlement,<br />Reservoir, Forest<br />
  26. 26. Keywords:Street,<br />Footpath, Gallery, Steps,<br />Parkland, Broadway<br />
  27. 27. Are we any good overall?<br />Initial results (with an interim solution) – about 40% of keywords are good or very good for a set of 20 images, assessed by ~70 users<br />
  28. 28. Where are we?<br />We can (semi) automatically annotate images based on their location, with keywords related to geography<br />Concept ontology allows us to switch data providers in and out easily<br />Filtering and ranking reduces initial long list of candidate concepts to set of keywords<br />~40% of keywords are good/very good<br />
  29. 29. Some lessons learned<br />Many devices still relatively error prone (particularly with respect to direction)<br />Open Street Map has very rich (but sometimes esoteric) attribution – but richness in urban areas very beneficial<br />National Mapping Agency data essential in rural areas<br />General classes (e.g. Building) difficult to use (either too many or too general keywords) <br />
  30. 30. 25<br />Acknowledgements<br />I’d like to gratefully acknowledge contributors to Geograph British Isles, see http://www.geograph.org.uk/credits/2007-02-24, whose work is made available under the following Creative Commons Attribution-ShareAlike 2.5 Licence (http://creativecommons.org/licenses/by-sa/2.5/).<br />Much of the research reported here was part of the project TRIPOD supported by the European Commission under contract 045335. <br />
  31. 31. Questions?<br />
  32. 32. Example problems<br />
  33. 33. zurich_20070709_45.jpg <br />Kreuzplatz photographed at 3.44 pm at the corner of Utoquai and Schoeckstrasse west of Bellevue in the Zürich (Region) <br />Keywords: Residence, Settlement, Building, Byway, Lane, Street, Lake, Road, Pool<br />Datasets: SwisstopoVec 25, OSM<br />Spotted feature types:<br />Vec25: 4_klass, 1_klass, Str_bahn, Z_innenhof, Gedbrue, Z_gebaeude, Z_station, Z_kirche, Ww50, Z_bharea, Z_see, Obstbaum, See, 2_klass, Obreihe, Kiturm, Z_siedl<br />OSM: Living_street, Rail_station_poly, Service, Wiese, Museum, Pedestrian, Cycleway, Rail, Retail, Pear_a, Theatre, University_complet, Primary, Residential, Footway, Secundary, Path, Water, School_a, Spielplatz, Steps, Place_of_worship, Cemetery, Buildings, Platform, Tertiary, Wohngebiet,<br />Concepts:<br />Trees, Turnpikes, SingleBuiltWorksByFunction, Towpaths, Streets, Footpaths, Cathedral, MotorwayFlyover, Byways, Waterbodies, Bridges, Trains, Steps, Motorways, Roads, MotorwayExtension, Museums, Reservoir, Lanes, ChurchTower, Bypasses, Meadow, RailwayTracks, Highways, Trackbed, Theatres, Chapels, Graveyard, Orchards, Churchyard, Colleges, Pond, Housing, CyclePaths, Tramway, Mainline, Broadways, Churches, Shops, Temples, Lakes, Abbey, RailwayStations, RailwaySidings, Platforms, ReligiousStructures, Courtyard, Cemeteries, Settlements, Paths, Greenways, Tarn, University, Playground, Schools,<br />
  34. 34. rimg0100.jpg (Peak District) <br /><ul><li>Riley Graves photographed in the early afternoon near Church Street in Eyam (Region)
  35. 35. Keywords:Meadow Downs Fields Beech Wood Villages
  36. 36. Datasets: Corine, OS Strategi, OS VectorMap
  37. 37. Spotted feature types:
  38. 38. Corine:0_wcsNodata, 23_231_pastures, 311_broadleavedForest
  39. 39. Strategi:5325, 5345, 5351
  40. 40. VectorMap:PrivateRoadRestrictedAlignment, Orchard, MinorRoadAlignment, BroadleafedWoodlandAndShrub, UnimprovedGrass, ImportantBuilding, Marsh, ConiferousWoodland, LocalStreetAlignment, WaterBodies, Shrub, MixedWoodland, BRoadAlignment, MixedWoodlandAndShrub, UrbanExtent, Heathland, BroadleafedWoodland, WaterFeature, BuildingPolygon, ARoadPrimaryAlignment, ShrubAndUnimprovedGrass, GlasshousePolygon
  41. 41. Concepts:
  42. 42. Hamlets, NaturalGrassland, Undergrowth, SingleBuiltWorksByFunction, Villages, Streets, Orchards, Bushes, AgriculturalLand, Heath, Dairy, Byways, ShrubAssociations, Copse, TransitionalWoodland, Woodland, DeciduousTrees, Marshes, Hedgerows, City, Pasture, Meadow, Settlements, Stream, Forest, Towns, Grass</li>