Relative input data are textual location descriptions which, by themselves, are not sufficient to produce an output geographic location. These produce relative geocodes that are geographiclocations relative to some other reference geographic locations Examples of these types of data include “Across the street from Togo’s” and “The northeast corner of Vermont Avenue and 36th Place.” Absolute input data are textual location descriptions which, by themselves, are sufficient to produce an output geographic location. These input data produce an absolute geocode in the form of an absolute known location or an offset from an absolute known location. Example House no-xxxx, ABC Street, JJJJJ Locality, XYZ City, QQ State, DEF Country, ###### Postal Code
LT Once in a year in Tab format, TT Quarterly in Shape,OSL and some other format
Linear-Based Reference Datasets A linear-based (line-based) reference dataset is composed of linear-based data, which can either be simple-line or polyline vectors. The type of line vector contained typically can be used as a first-order estimate of the descriptive quality of the reference data sourcePolygon-Based Reference datasets A polygon-based reference dataset is composed of polygon-based data. These datasets are interesting because they can represent both the most accurate and inaccurate forms of reference data.Point-Based Reference Datasets A point-based reference dataset is composed of point-based data. These are the least commonly encountered partly because of their usability, and partly because of the wide ranges in cost and accuracy.
Feature matching,The process of identifying a geographic feature in the reference dataset corresponding to the input data to be used to derive the final geocode output for an input. A feature-matching algorithm is an implementation of a particular form of feature matching. These algorithms are highly dependent on both the type of reference dataset utilized and the attributes it maintains about its geographic features. Feature interpolation,The process of deriving a geographic output from a reference feature selected by feature matching. A feature interpolation algorithm is an implementation of a particular form of feature interpolation. These algorithms also are highly dependent on the reference dataset in terms of the type of data it contains and the attributes it maintains about these features.
The matching algorithms are Noninteractive matching algorithms (i.e., they are automated and the user is not directly involved). Interactive matching algorithms involve the user in making choices when the algorithm fails to produce an exact match by either having the user correct/refine the input data or make a subjective, informed decision between two equally likely optionsClassifications of matching algorithms Two main categories: Deterministic - A deterministic matching method is based on a series of rules that are processed in a specific sequence. These can be thought of as binary operations; a feature is either matched or it is notProbabilistic. A probabilistic matching method uses a computational scheme to determine the likelihood, or probability, that a feature matches and returns this value for each feature in the reference set.
The main benefit of deterministic matching is the ease of implementation. These algorithms are created by defining a series of rules and a sequential order in which they should be applied. The simplest possible matching rule is the following: “Match all attributes of the input address to the corresponding attributes of the reference feature.”Attribute relaxation Attribute relaxation, the process of easing the requirement that all street address attributes must exactly match a feature in the reference data source to obtain a matching street feature, often is applied to create these less restrictive rules.
Any feature-matching algorithm requires the comparison of strings of character data to determine matches and non-matches. Character-level equivalence, enforces that each character of two strings must be exactly the same. Essence-level equivalence uses metrics capable of determining if two strings are “essentially” the same. This allows for minor misspellings in the input address to be handled, returning reference features that “closely match” what the input may have “intended.”. Word stemming is the simplest version of an essence-level equivalence technique. These algorithms reduce a word to its root (stem), which then is used for essence-level equivalence testing. The Porter Stemmer (Porter 1980) is the most famous of these. It starts by removing common suffixes (e.g., “-ed,” “-ing,”) and additionally applies more complex rules for specific substitutions such as “-sses” being replaced with “-ss.” Phonetic algorithms or the Soundex Algorithm provide an alternative method for encoding the essence of a word. These algorithms enable essence-level equivalence testing by representing a word in terms of how it sounds when it is pronounced (i.e., phonetically). The goal of these types of algorithms is to produce common representations for words that are spelled differently, yet sound the same.
Discussion • Introduction to Geocoding
• All about spatial data and real world data • Addressing system of developing nation • Addressing system in India • Data sources and standardization • Data arrangement • Geocoding • Challenges • Steps to overcome challenges • Q & A
What is Geocoding ? Geocoding
is the process of transforming a description of a location (such as a pair of coordinates, an address, or a name of a place) to a location on the earth's surface Source Definition Possible Problems Environmental Sciences Research Institute (1999) The process of matching tabular data that contains location information such as street addresses with realworld coordinates. Limited to coordinate output only. Harvard University (2008) The assignment of a numeric code to a geographical location. Limited to numeric code output only. Statistics Canada (2008) The process of assigning geographic identifiers (codes) to map features and data records. Limited input range. U.S. Environmental Protection Agency (2008) The process of assigning latitude and longitude to a point, based on street addresses, city, state and USPS ZIP Code. Limited to coordinate output only.
What is Geocoding ? Geocoding
(verb) is the act of transforming aspatial locationally descriptive text into a valid spatial representation using a predefined process. A geocoder (noun) is a set of inter-related components in the form of operations, algorithms, and data sources that work together to produce a spatial representation for descriptive locational references. A geocode (noun) is a spatial representation of a descriptive locational reference. To geocode (verb) is to perform the process of geocoding.
Geocoding ……… Some points to
ponder……. • Does geocoding refer to a specific computational process of transforming something into something else, or simply the concept of a transformation? • Is a geocode a real-world object, simply an attribute of something else, or the process itself? • Is a geocoder the computer program that performs calculations, a single component of the process, or the human who makes the decisions?
What is there in Spatial
data…. Location information • The fundamental primitive is the point, a 0-dimensional (0-D) object that has a position in space but no length. • A line is a 1-D geographic object having a length and is composed of two or more 0-D point objects. • A polygon is a geographic object bounded by at least three 1-D line objects or segments with the requirement that they must start and end at the same location (i.e., node)
Spatial Data • Spatial data
often referred to as layers • Layers represent features on, above, or below the surface of the earth • Data layers are of 2 major types • Vector data represent features as discrete points, lines, and polygons. • Raster data represent the landscape as a rectangular matrix of square cells. The Real World Vector Data Raster Data
Geocoding and Spatial Data •
Most geocoding applications work with vector-based GIS data. • The key aspects from a geocoding perspective : • Determine and record the locations of these objects on the surface of the Earth, and • Calculate distance because many geocoding algorithms rely on one or more forms of linear interpolation.
Addressing System • Addresses are
one of the fundamental means by which people conceptualize location in the modern world • Addresses are of 2 Types • Relative input data • Examples of these types of data include “Across the street from Togo’s” and “The northeast corner of Vermont Avenue and 36th Place.” • Absolute input data • Example House no-xxxx, ABC Street, JJJJJ Locality, XYZ City, QQ State, DEF Country, ###### Postal Code
Addresses • Come in a
variety of formats • Address Components • Address (house or building) number • Prefix direction • Prefix type • Street name • Street type • Suffix direction • Zone
What is street addressing? •
Street addressing is an exercise that makes it possible to identify the location of a plot or dwelling on the ground, that is, to “assign an address” using a system of maps and signs that give the numbers or names of streets and buildings • Street addressing provides an opportunity to • Create a map of the city that can be used by different municipal units • Conduct a systematic survey that collects a significant amount of information about the city and its population, and • Set up a database on the built environment
History of street addressing across
some countries • Before 1728, no street names were indicated in Paris, except in very rare cases, such as “Rue Saint- Dominique, formerly des Vaches” (1643) • In Belgrade, changes to street names are frequent • The name China itself (country of the center), which makes reference to this principle, and in the name of the capitals, Beijing (capital of the North) and Nanjing (capital of the South). Names of the provinces are also strongly influenced by these references to geographical direction • Aside from a few main thoroughfares, streets in Japan do not have names. In fact, the city districts (ku) are divided into neighborhoods (chome) that group together several dozen houses and thus form a block.
Street Addressing System Some common
Practice • Sequential alternating numbering systems • Decametric numbering system • Codification of intersections • Combined addressing system Segment Distance (m) No. left side No. right side 1 0–10 1 2 2 10–20 3 4 3 20–30 5 6 4 30–40 7 8 Sequential alternating numbering systems Codification of intersections Combined addressing system
Addressing System in India •
Spatial data capturing based on demand rather than homogeneous capture • Streets with no names or unstructured addresses • Absence of consistent and accurate dataset throughout the area being geocoded • Presence of slum-like areas that change frequently and are not street addressable • Non-existence of reference datasets or GIS data infrastructure • Lack of hierarchical data structure beyond tehsil • Absence of standardized geocoding algorithm • Non-existence of approach to validate assumptions made in the geocoding algorithm
Sample Addresses varieties of India
Address with Person Name c/o Yashwant S.Prabhu , 318, C - Wing, Suyog Co.Housing Society Ltd, T. P.S. Road & III Link Road, Vazira, Borivali, West Mumbai, Maharashtra, 400092 c/o Late Esmail Bagani, Y/2/122, Satghara Road, PO- Badartala, PS – Nadial, Kolkata, West Bengal, 700044 Address with Building names White C/403, Aamrpali Appt, opp. GHB complex, Ankur Road, Ahmedabad, Gujarat, 380013 13/9, Daksha Bldg, Vallabh Baug Lane, Ghatkopar, Mumbai, Maharashtra, 400077 Address with House No 299/15, Padmavati Vikar Mandal, Shahibaug, Ahmedabad, Gujarat, 380001 NO88, Srinivasa Nagar, 2NS Main Road, Kolathur, Chennai, Tamil Nadu 600099 Address with Street Name 1304, Cornation Road, Bargarpet, Kolar, Bangalore, Karnataka, 560000 20K, Dhakuria Station Road, Dhakuria PS: Jadavpur, Kolkata, West Bengal, 700031 Address with POI BMC Software, Next Muttha Chamber, Senapati Bapat Road, Pune, Maharashtra, 411016 Life Style International Pvt. Ltd., Near Payal Cinema Complex, Gurgaon, Haryana, 122001
India Dynamics • 35 States
& UTs, ~650 Districts , ~6000 Tehsil/Town, more than 30k postcodes • 21 regional , two official (Hindi & English) languages • 3.2m sq KM Area, 1.2 billion population • Administrative Hierarchy • State >>District>>Tehsil/Town>> Ward>>Locality/Village>>Sub locality>>Block/Pocket • Addressing Pattern • Near Govt. Hopital, Zirapur , District Rajgarh (MP) • 351 Ground Floor , Shakti khand 3 , Near St. Teresa School , Indirapuram, Ghaziabad 201010 U.P. • 9 Mansarovar Colony Opp. 3/686, Kala kuan Housing Board Alwar 301001 • 9/19/98/19-D Flat No. # 303 Hitech City Madhapur Hyderabad • 176 Devi Nagar New Sanganer Road Sodala , Jaipur Rajasthan Indian Postal Code
The History of Postal System
in India • Long colonial realm – British, Mughal, Portuguese……. • Britain's involvement in the postal services of India began in the eighteenth century • Warren Hastings (Governor General of British India from 1773-1784) opened the posts to the public in March 1774 • Main purpose of the postal system had been to serve the commercial interests of the East India Company and to serve Govt. orders
India Geocoding Data 27 Streets
• NH • SH • Local roads Place of Interest • Banks • Retails • Hospital • Other landmarks Administrative • State • District • Town • Locality • Sub locality Geography • Block • Locality • Town • Postcode Address Point • House no • Building name
Reference datasets • The reference
dataset is the underlying geographic database containing geographic features that the geocoder can use to generate a geographic output. • This dataset stores all of the information the geocoder knows about the world and provides the base data from which the geocoder calculates, derives, or obtains geocodes. Interpolation algorithms Type Example Vector line file U.S. Census Bureau’s TIGER/Line (United States Census Bureau 2008c) Vector polygon file Los Angeles (LA) County Assessor Parcel Data (Los Angeles County Assessor 2008) Vector point file Australian Geocoded National Address File (G-NAF) (Paull 2003)
Reference dataset types • Linear-Based
Reference Datasets • Roads, Ferries • Polygon-Based Reference datasets • Administrative Boundaries, Postal Codes • Point-Based Reference Datasets • POIs Source Description Coverage Cost Tele Atlas (2008c), NAVTEQ (2008) Building footprints, parcel footprints Worldwide, but sparse Expensive County or municipal Assessors Building footprints, parcel footprints U.S., but sparse Relatively inexpensive but varies U.S. Census Bureau Census Block Groups, Census Tracts, ZCTA, MCD, MSA, Counties, States U.S. Free Name Description Coverage U.S. Census Bureau’s TIGER/Line files (United States Census Bureau 2008c) Street centerlines U.S. NAVTEQ Streets (NAVTEQ 2008) Street centerlines Worldwide Tele Atlas Dynamap, MultiNet (Tele Atlas 2008a, c) Street centerlines Worldwide Supplier Product Description Coverage Government GeoNames (United States National Geospatial-Intelligence Agency 2008) Gazetteer of geographic features World, excepting U.S. Academia Alexandria Digital Library (2008) Gazetteer of geographic features World
Input data processing Address normalization
• Address normalization organizes and cleans input data to increase its efficiency for use and sharing Address standardization • Address standardization converts an address from one normalized format into another. It is closely linked to normalization and is heavily influenced by the performance of the normalization process. Sample Address 3620 South Vermont Avenue, Unit 444, Los Angeles, CA 90089-0255 3620 S Vermont Ave, #444, Los Angeles, CA 90089-0255 3620 S Vermont Ave, 444, Los Angeles, 90089-0255 3620 Vermont, Los Angeles, CA 90089
Output data • The last
component of the geocoder is the actual output data, which are the valid spatial representations derived from features in the reference dataset. • Data can have many different forms and formats, but each must contain some type of valid spatial attribute. • The most common format of output is points described with geographic coordinates (latitude, longitude). • Alternate forms can include multi-point representations such as polylines or polygons. • These geocoder outputs, while in the same format and produced through the same process, do not represent data at the same geographic resolution and must be differentiated.
Feature matching - Algorithm •
The matching algorithms are • Noninteractive matching algorithms (i.e., they are automated and the user is not directly involved). • Interactive matching algorithms • Classifications of matching algorithms • Two main categories: • Deterministic • Probabilistic
Deterministic matching • Ease of
implementation • These algorithms are created by defining a series of rules and a sequential order in which they should be applied. Like- “Match all attributes of the input address to the corresponding attributes of the reference feature.” • Attribute relaxation • Attribute relaxation, the process of easing the requirement that all street address attributes must exactly match a feature in the reference data source to obtain a matching street feature, often is applied to create these less restrictive rules. Preferred attribute relaxation order with resulting ambiguity, relative magnitudes of ambiguity and spatial error, and worst-case resolution, passes 1 – 4 Relaxed Attribute Ambiguity Relative Exponent and Magnitude of Ambiguity Relative Magnitude of Spatial Error Worst- Case Resolutio n none none (0) none certainty of address location single address location number multiple houses on single street (0) # houses on street length of street single street pre single house on multiple streets (1) # streets with same name and different pre bounding area of locations containing same number house on all streets with the same name USPS ZIP Codepost (1) # streets with same name and different post type (1) # streets with same name and different type number, pre multiple houses on multiple streets (2) # houses on street * # streets with same name and different pre bounding area of all streets with the same name number, type (2) # houses on street * # streets with same name and different type number, post (2) # houses on street * # streets with same name and different post
Probabilistic matching • Probabilistic matching
has its roots in the fields of probability and decision theory • Employed in geocoding processes since the outset (e.g., O’Reagan and Saalfeld 1987, Jaro 1989). • The exact implementation details can be quite messy and mathematically complicated, but the concept in general is quite simple.
Attribute weighting • Attribute weighting
is a form of probabilistic feature matching in which probability based values are associated with each attribute, and either subtract from or add to the composite score for the feature as a whole.
String comparison algorithms • Character-level
equivalence, • Essence-level equivalence • This allows for minor misspellings in the input address to be handled, returning reference features that “closely match” what the input may have “intended.”. • Word stemming • Phonetic algorithms or the Soundex Algorithm
Soundex Algorithm • It has
existed since the late 1800s and originally was used by the U.S. Census Bureau. • The algorithm is very simple and consists of the following steps: • Keep the first letter of the string • Remove all vowels and the letters y, h, and w, unless they are the first letter • Replace all letters after the first with numbers based on a known table • Remove any numbers which are repeated in a row • Return the first four characters, padded on the right with zeros if there are less than four Original Porter Stemmed Soundex Running Ridge run ridg R552 R320 Runs Ridge run ridg R520 R320 Hawthorne Street hawthorn street H650 S363 Heatherann Street heatherann street H650 S363
Geocoding Challenges • Unavailability of
geospatial data • Data if available is • Unstructured • Incomplete • Inaccurate • Lack precision • Does not have official name or region developed because of anthropogenic pressure • “General" geocode no longer sufficient; the "most accurate" geocode required • Inconsistent use of base maps and geocoding services within and across programs and agencies Data Inconsistency Erroneous data Frequent data changes requests Coverage New data and modified data in every vintages
Classes of geocoding failures with
an example of true address “Maulsari BnB, 142 Sunder Nagar, Near Delhi Public School, New Delhi, Delhi 110003, India” Class Geocoded Problem Example 1 No Failed to geocode because the input data are incorrect. Maulsari BnB, 142 Sunder Nagar, New Delhi, 110003, India 2 No Failed to geocode because the input data are incomplete. Maulsari BnB, Sunder Nagar, Near Delhi Public School, New Delhi, 110003, India 3 No Failed to geocode because the reference data are incorrect. 140 Sunder Nagar, New Delhi, Delhi 110003, India 4 No Failed to geocode because the reference data are incomplete. Street segment does not exist in reference data 5 No Failed to geocode because the reference data are temporally incompatible. Street segment name has not been updated in the reference data 6 No Failed to geocode because of combination of one or more of 1-5. Maulsari BnB, 142 Sunder Nagar, Near Delhi Public School, New Delhi, Delhi 110003, India where the reference data has not been updated to include Near Delhi Public School, New Delhi, Delhi address segment 7 Yes Geocoded to incorrect location because the input data are incorrect. Maulsari BnB, 142 Sunder Nagar, Near Delhi Public School, New Delhi, Delhi 110003, India was (incorrectly) relaxed and matched to Delhi Public School, New Delhi, 110003, India 8 Yes Geocoded to incorrect location because the input data are incomplete. Maulsari BnB, 142 Sunder Nagar, Near Delhi Public School, New Delhi, Delhi 110003, India was arbitrarily (incorrectly) assigned to 145 Sunder Nagar, New Delhi, Delhi 110003, India 9 Yes Geocoded to incorrect location because the reference data are incorrect. Sunder Nagar, New Delhi, Delhi 110003, India 10 Yes Geocoded to incorrect location because the reference data are incomplete. Street segment geometry is generalized straight line when the real street is extremely curvy 11 Yes Geocoded to incorrect location because of interpolation error. Interpolation (incorrectly) assumes equal distribution of properties along street segment 12 Yes Geocoded to incorrect location because of drop back error. Drop back placement (incorrectly) assumes a constant distance and direction 13 Yes Geocoded to incorrect location because of combination of one or more of 7- 12. The address range for 140-150 is reversed to 150-140 and dropback of length 0 is used
Overcome the challenge • Standardization
of addresses by governing bodies • Integrating addressing data models from variety of addressing system to develop region specific data models • Utilizing multiple data sources for data completeness • Inclusion of local landmarks in geocode process to they form integral part of Indian address system • Embed locational awareness and intelligence within geocoding data models • Multi Lingual phonetic support • Large test set to address different kinds of geocoder irregularities • Validation methodology to confirm geocode results • Changes in map policies for consistent capturing • Homogeneous information dissemination by governing bodies in regard to spatial data • Standards protocols developed for input/reference data correction • Address normalization • Address standardization
Features to address challenges •
Sub locality Features • Inclusion of Local Landmarks • POI Level Geocoding • Bank Dictionaries • Focus on Local Geography and Social Settings