John Fagan: The Black Art of Geocoding

The Black Art of GeocodingFinding that elusive lat/lon
John Fagan, Microsoft

The Black Art of GeocodingFinding that elusive lat/lon
John Fagan
Program Manager
Microsoft Corporation
@johnbfagan

We been making maps for 1000’s of years

Well known and established standards/principles

Lots of experience in building software to create bitmaps from vector and raster data

Data availability & Simple data model

Mapping easy to scale

...and so is routing
1000’s years experience in wayfinding
Over 50 years experience in routing algorithms
Dijkstra's shortest path algorithm (1959)

Data availability & Simple data model

Routing, easy to scale

Geocoding not so easy
20 years experience
10 years of global Geocoding
5 years exposing geocoding to the mass consumer
No standard algorithms
Very few databases purpose built (maybe GNAF)
Very hard to scale

Geocoding is fundamental
Cant get a map without a geocode
Cant get a route without a geocode
Cant view your data without a geocode
80% of all information contains a geographic element.

It used to be easier

Now its hard

User expectations change with unstructured input
67 hill veiw road, s61 2bn in the 1850's
1.5 hours from Nice
exact directions from Bangkok Patana School to Suvanapumi Airport in Bangkok.
10 mile radius from se20 7ua
how long would it take me to walk around cancun
how to get to m13 gb from g83 9le by car
do bearded dragons bite?

But ......Geocoding NOT about Search

52.19157,-1.70415

The reason it's called 'I'm Feeling Lucky,' is of course that's a pretty damn ambitious goal. I mean to get the exact right one thing without even giving you a list of choices, and so you have to feel a little bit lucky if you're going to try that with one go," tried to explain Sergey Brin.

Why is it hard (2 reasons)

Parsing: Hard to understand unstructured input

Finding Stratford-upon-Avon
stratford
stratford upon avon
Stratford upon haven
StratfordUponAvon
Stratford-Upon-Avon
stratford on avon
stratford-on-avon
stratford 0n avon
stratford - upon-avon
stratford on avaon
stratfordaponavon
stratford upon aavon
stratfordupponavon

Parsing
In computer science and linguistics, parsing, or, more formally, syntactic analysis, is the process of analyzing a text, made of a sequence of tokens (for example, words), to determine its grammatical structure with respect to a given (more or less) formal grammar.
http://en.wikipedia.org/wiki/Parsing

Old way of Parsing – Rules based
A rules based approach (mainly done with regular expressions)

Probabilistic approach
Machine learned
Requires you to “train” the engine
Requires truth sets of training data
http://en.wikipedia.org/wiki/Hidden_Markov_model

Probabilistic approach: Hidden Markov Model
input --> 165 fleet street london EC4A 2DY
output -->
address {
street number : 165
street : fleet street
city : london
postcode : EC4A 2DY
}

Multimap stats

Parsing has its limitations
Parsing failures
Multimap/Bing Maps (standrewsscotland)
Google (uk near Boston, MA, USA)
All fail - House number plus postcode (165, EC4A 2DY)

Parsing using a Spatial Engine
http://research.microsoft.com/en-us/people/josephj/acm_gis_2007_robust_location_search.pdf

Why is it hard (Data)

Hard to match input with reference database

[OSM-talk] Baghdad maps
I am informed that any road may have up to 4 names (which may be the same or different):
The pre-Saddam name
The Saddam-era name.
The "public" name - What the people who live there call it.
The "Official" name - What the new Government calls it.
This situation is further complicated by language and social issues: Language
The roads are names in Arabic.
There is no fixed translation between the Arabic and Latin alphabets.
Social Issues:
1) Sunnis tend to use the Saddam-era names
Shia tend to rename streets and won't acknowledge Saddam-era names.
Ethnic cleansing is changing the neighbourhoods and hence the names.
Names (such as 14th July Bridge) will change later.
My translator's opinion is that street names are going to take at least 2-3 years to settle down.
http://lists.openstreetmap.org/pipermail/talk/2007-February/011273.html

Don't throw away your data
Multimap have always kept old postcodes
10% of Multimap’s postcode database is of “dead” postcodes
This might not work for routing and mapping, but very valuable for Geocoding

EC4A 1HE – Postcode of vintage 2002

Lash data and enrich
Stratford-upon-Avon

Future = Real time Geocoding?

Summary
Mapping and Routing – FIXED
Geocoding – Must Try Harder
Parsing
Data

thanksjohn faganubergeo.com@johnbfagan

http://www.ubergeo.com	1076
http://www.slideshare.net	4
http://search.mywebsearch.com	1
http://webcache.googleusercontent.com	1

John Fagan: The Black Art of Geocoding

by AGI Geocommunity on Sep 21, 2009

Accessibility

Categories

Tags

Upload Details

Usage Rights

4 Embeds 1,082

Statistics

John Fagan: The Black Art of Geocoding — Presentation Transcript