Osmose, quality assurance tool

889 views
788 views

Published on

Sotm Birmingham Osmose quality assurance.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
889
On SlideShare
0
From Embeds
0
Number of Embeds
103
Actions
Shares
0
Downloads
8
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Osmose, quality assurance tool

  1. 1. SoTM 2013 Osmose quality assurance tool Birmingham 2013-09-06 Frédéric Rodrigo <frederic@carte-libre.fr> (c)left 2013 - CC-BY-SA v3.0 http://osmose.openstreetmap.fr
  2. 2. Context QA tool – Since 2008 by Etienne Chové – Maintained by Jocelyn Jaubert and Frédéric Rodrigo Sources available on gitorious – https://gitorious.org/osmose/
  3. 3. Ecosystem No duplicate analysis from KeepRight and Osm Inspector But integrate good idea from “minor” other tools Idea from Osmose integrated into JOSM validator by don_vip (building intersection, powerline...)
  4. 4. Support France mainland and oversea HOT activated countries All full French speaking countries and Quebec Switzerland, Belgium, Luxembourg and Iceland Hiring for community to support local server for new countries
  5. 5. The Osmose Stack Frontend BackendsAPI
  6. 6. Backends Input XML or PBF export Not all supported country are available at Geofabrik OSM-FR generate also export – http://download.openstreetmap.fr/extracts/ – Diff available – Need for Osmose
  7. 7. Analyser - SAX Process one primitive at time Each analyser is a plugin – Check node, way and relation – Can report errors Use external data – Wikipedia tree species Running filter on language and country
  8. 8. Analyser - SAX Format: number, date, postcode, wikipedia... Naming: abbreviation, too many space, uppercase... Bag tag: ref in name, typo, deprecated... Ensure multiple tags coherence from wiki rules: multipolygon, parking lanes... Inconsistency between tags: toll and fee, opposite cycleway without oneway...
  9. 9. class Structural_UnclosedArea(Plugin): def init(self, logger): self.errors[1100] = { "item": 1100, "level": 3, "tag": ["geom", "fix:imagery"], "desc": T_(u"Unclosed area") } def way(self, data, tags, nds): if "area" not in tags or tags["area"] == "no": return if nds[0] != nds[-1]: return [(1100, 0, {})]
  10. 10. Analyser - Osmosis Osmosis, a lossless schema Multiple objects checking at once Geospatial analysis DB – Lot of indices – create/drop DB: at each Osmose run, lake of disk sppace – Keep DB: Quebec (diff mode)
  11. 11. Analyser – Osmosis – Non spatial Done by topology Roundabout analyser – Wrong highway level on roundabout – Missing oneway on link highway – Roundabout shortcut – Roundabout crossing
  12. 12. Analyser – Osmosis - Geospatial Roundabout – Missing junction=roundabout – Reverse roundabout Deadend by highway level – Small gap in highway level continuity Power line – Isolated tower – Unconnected powerline – Missing tag power=tower – Predictive position of unmapped power=tower
  13. 13. Analyser – Osmosis - Convergent Soundex – Phonetic index – Index all name, way kind + way name – Compare numerous name again similar with low frequency Automatic typo on tags detection – Not work
  14. 14. Analyser - Merge OpenData Integration – Integration vs Importation – Help contributor to add external data Row external data set (csv) + Osmosis DB Mapping – Column to tag – Complex mapping Analyser code is just configuration – Easy to add new data set
  15. 15. Analyser - Merge Conflation by ref Conflation by distance Detect – OSM object without ref – OSM object not found in OpenData set – OpenData object not found in OSM “OpenData by OSM” as sub product
  16. 16. Analyser – Merge French school Wikipedia Level crossing
  17. 17. self.csv_encoding = "ISO-8859-15" self.osmTags = { "amenity": "post_office" } self.osmRef = "ref:FR:LaPoste" sself.osmTypes = ["nodes", "ways"] self.sourceSRID = "4326" self.defaultTag = { "amenity": "post_office", "operator": "La Poste", "source": "data.gouv.fr:LaPoste – 01/2013" } self.defaultTagMapping = { "ref:FR:LaPoste": "identifiant", "name": "nom", "wheelchair": lambda res: "yes" if self.bool[res["autonome_fauteuil_roulant"]] else "limited" if self.bool[res["pas_ressaut"]] else "no" } self.conflationDistance = 1000
  18. 18. Analyser – OSMBin OSMBin: Osmose proprietary DB format – Flat storage, access in O(1) Multipolygon / boundary closed fast checking
  19. 19. Analyser – External tools OpenStreetBug – Close OSB entries when close it on Osmose Need support OSM Notes
  20. 20. Configuration Configuration by extract (country and smaller part) List of analysers – Language: mainly french specific – Local way of mapping – Source available: Bing hires, French cadastre... Options – Projection – Driving side – Language
  21. 21. Boundary clipping Extract have buffer on boundary – Duplicate error – False error (e.g. unconnected) Clip on exact boundary polygon after analyse
  22. 22. About Time Full run every 2 days Take 80% of availible CPU 2 50 35 43 67 ïle-de-France - Total time 3h17 Download Osmosis Import Indexing SAX Analyser Osmosis Analyser
  23. 23. Diff mode Speedup by running analyser only on change Sax Osmosis – “changed” tables : edited nodes, ways, relations and transitive dependencies – Keep DB between runs Not all can be run in diff mode – Error based on other OSM data (clustering, connection...) Full run every month to keep safe
  24. 24. The Osmose Stack Frontend BackendsAPI
  25. 25. Errors Backends send errors to centralized Frontend Errors report are by Analyser and Country (aka source) Error file replace all errors of his kind on frontend – Except on diff mode
  26. 26. Error Source (country, analyser) Identity – Class – Subclass Metainfo – Level: 1, 2, 3 as severity – Item number: display category and subcategory Tags – Other classificatio:, by nature of OSM object, highway, waterway, name... – What is require for fix: chair, imagery, survey
  27. 27. Error fixes Analyser can propose some error fixes Add, delete or modify many tag at once Create a new node with tags
  28. 28. The Osmose Stack Frontend BackendsAPI
  29. 29. Frontend - Map Selector – Levels of severity – Items – Other options available on URL (see API) Markers – Status: open, close, false-positive – JOSM remote – JOSM fix – Wiki help Layers – Heatmap – Debug map
  30. 30. Stats Errors listing History graph By user – RSS Update Matrix
  31. 31. France - Level 1 Add new analyser Natural fix over the time Heavy fix
  32. 32. Haiti – All errors Correct error with locals
  33. 33. France – Cadastre building overlap New gap detection Heavy fdix
  34. 34. French OpenData School Bugs OpenData Update
  35. 35. The Osmose Stack Frontend BackendsAPI
  36. 36. API External fixing tools Stats JOSM – confidential plugin – “Quality Assurance Tools script” API doc on Wiki – URL parameters apply any where in the frontend
  37. 37. Hardware Backends – osm3.openstreetmap.fr HOT and Quebec – osm5.openstreetmap.fr France – osm6.openstreetmap.fr France – Iceland server: Iceland Frontend – osm103.openstreetmap.fr ← osmose.openstreetmap.fr
  38. 38. Conclusion osmose.openstreetmap.fr → o.osmfr.org https://gitorious.org/osmose/ We need you – Translation: frontend UI and backend errors – Run your own Osmose local backend server – Errors correction! Long list of analysers request on french Trac Localise language or country specific analyser – Main st → Main street

×