Using in-house built translator & Open Source Technologies: A Technical Perspective By Faizan Tayyab Senior GIS Developer
What will be covered: Introduction to UK biggest mapping agency. Introduction to Master Map Topography Layer. Motivation behind the project. Explanation of methodology. What this presentation is not: Tutorial on XML or GML. (Read a good book for that) Designing & Implementing Application schema(s) training. In detail coding for parsing GML data.
If you know how to parse XML then you will have no problem in understanding this presentation. If you haven't worked with XML or haven’t parsed it using different languages then I would recommended you go and have a look what it is all about. Being a future GIS Professional, it is a good idea to equip yourself with XML, Open source and GML skills. Xml is here to stay. It is powerful & Flexible. If you learn XML and Schema’s then GML becomes easier to understand.
Ordnance Survey is Great Britain's national mapping agency, providing the most accurate and up-to-date geographic data, relied on by government, business and individuals. It is one of the world’s largest map producer and first came into existence in 1774. Primarily, mapping produced by Ordnance Survey was used for military purposes only. Today Ordnance Survey employs over 1500 people and is the front runner in producing world’s best paper and digital maps in the UK.
A seamless database reflecting the real world Unique Identifiers (TOIDs) Conforms to Open GIS standards (e.g., GML) Consistent national coverage
Topography Layer Administrative Boundaries Buildings Heritage and Antiquities Water Land Rail Roads, Tracks and Paths Structures Terrain and Height Address Layer Address ITN Layer Roads Road Routing Information Imagery Layer Imagery
Each feature in OS MasterMap Topography will be one of the following: Topographic Area Topographic Line Boundary Line Topographic Point Cartographic Symbol Cartographic Text In addition to the general attribute sets (TOID, Version etc.) Topographic Features all have the following attributes: Physical Presence Physical Level Make Feature Code
Descriptive Group – the primary descriptive attribute e.g. Landform, Roadside, Structure, Path, Rail, Built Environment, Height Control. Descriptive Term (Optional) – provides further description e.g. Archway, Boulder, Cliff, County, District, Pylon, Multi Surface, Rough Grassland. Physical Level – describes relative cartographic relationships between features. e.g. – underground (49), ground level (50), a level above ground (51) Physical Presence (optional) – e.g. Boundary, Closing, Extent, Obstructing, Overhead, Edge/Limit, Indicator, Minor Detail and Moveable Make (optional) – e.g. Manmade, Natural, Multiple, Unknown, Unclassified.
Customer is provided OS Master map data in GML format & CSV format (if stated by customer)
XML can be thought of as “smart” text that contains tags that have meaning to people and software! XML describes data. Different from HTML. <title> My Book </title> <author> James McGill </author>
<Farm> <Building> <Name>John Brown's House</Name> <Description>Colonial Farm House</Description> <Address>1134 15th eastline</Address> </Building> <Field /> <Field /> <Field /> </Farm> *galdos inc Field Field Field Field 15 line east East Hope Rd
The Geography Mark-up Language (GML) is an XML (extensible Mark-up Language) encoding for the modelling, transport and storage of geographic information including both spatial and non-spatial properties of geographic features. The integral part of any GML data model is the GML application schema (s), which describe the aspects of geo-spatial domain it addresses. GML application schema (s) are extended over and compliant with the standard OGC defined GML base schemas that are based on W3C schema recommendation specification. Software applications intending to support GML data models should be aware of these concepts. It is feasible to recognize these concepts by understanding the GML application schema(s). Understanding, processing and representing the GML application schema(s) enable the scope of supporting varied GML data models. Moved to GML 3.0 & later from GML 2.0. GML 3.0 far more complex.
Possibilities with GML are endless. Need GML enabled infrastructure. Avoid conflict between different formats for same features. Internet Mapping. Disaster Response System. Natural resource Management. And many other. GML is becoming ever popular especially with a shift of industry towards open standards. The Open GIS Consortium (OGC) is developing a set of standard geospatial web services (WFS,WMS,WCS).
Comprises of a Web Map Server (WMS) this serves raster versions of maps to clients. WMS datasets consist of raster images of data. The WMS Service generates the raster image and sends it to the clients browser. The original data can be vector or raster; but it is served and rendered as raster on the client side.
Db 1 Db 2 Db 3 Web Server 1 Web Server 2 Web Server 3 Internet Web Client Raster image HTTP Request Response (Raster Image)
The WMS specification standardises three operations: GetCapabilities (required) URL request Returns an XML file with metadata about the service itself and specific information about the available services GetMap (required) URL request with parameters Returns a map image whose geospatial parameters are well defined according to client request specification GetFeatureInfo (optional) URL request with parameters Returns information about particular features shown on a map – the raster image becomes ‘queryable’
Comprises of a Web Feature Server (WFS) this serves vector versions of maps to clients. The WFS Service supplies the geometry and attribute definitions that the client browser requests, and sends it in GML form. Supports INSERT, UPDATE, DELETE, QUERY and DISCOVERY operations
Db 1 Db 2 Db 3 Web Server 1 Web Server 2 Web Server 3 Internet Web Client Vector image HTTP Request Response (GML)
The WFS specification defines the following operations: GetCapabilities – As with WFS DescribeFeatureType – must be able to describe the structure of any feature type it can service GetFeature – services requests to retrieve feature instances and their properties Transaction – services transaction requests which modify geographic features LockFeature – allows lock requests on one or ore instances of a feature type for the duration of the transaction enabling serialisation
Sample Request HTTP GET http://18.104.22.168/magnesium/wfs/OS? REQUEST=GETFEATURE& TYPENAME=TopographicAreaType& SRS=OSGB:BNG& BBOX=292403,710274,294739,712607& USEBOX=TRUE& MAXFEATURES=10 HTTP POST <?xml version=&quot;1.0.0&quot; encoding=&quot;UTF-8&quot; ?> <!-- template for WFS get capabilities through a POST --> <GetFeature version=&quot;1.0.0&quot; maxFeatures=&quot;20&quot; service=&quot;WFS&quot;> <Query typeName=&quot;TopographicAreaType&quot;> </Query> </GetFeature>
Database Web Map Server Internet Connection Browser or GIS client with WMS connector IIS or equivalent
A styling language that the WMS client and WMS server can both understand. Applies explicit styling information provided by the user in order to render a map. Allows the user to create custom styles on traditional WMS layers Some of the current SLD limitations are: 1) there is no elegant way to specify a thematic or chloropleth map. 2) the ability to create styles lacks a style library service.
An SLD WMS adds the following additional operations that are not available on a basic WMS: 1. DescribeLayer – asks for an XML description of a map layer. The result is the URL of the Web Feature Server (WFS) containing the data and the feature type names included in the layer. 2. GetLegendGraphic – provides a general mechanism for acquiring legend symbols, beyond the LegendURL reference of WMS Capabilities. 3. GetStyles – used to retrieve user-defined styles from a WMS. 4. PutStyles – used to store user-defined styles into a WMS
The need to manipulate and utilize GML data has resulted in development of many commercial products in the recent years, most of which are very expensive. Ordnance Survey GML data requires appropriate technical expertise and value added product or services for loading and using the data.
The project definition was to develop an in-house GML translator capable of translating and storing GML data into a MYSQL relational database. Once stored this data was then visualized using available Open Source technologies. Achieving the above will present the possibility to manipulate Ordnance Survey GML data in variety of ways and for different applications areas. The findings were compiled and presented as a white paper to be distributed to the customer with OS Master Map topography layer.
As mentioned in previous section, number of expensive commercial Translator are available that can convert GML data and store them in spatially enabled databases for further use. There are also numbers of open source technologies that are capable of displaying GML data directly on the web but these technologies are still in their formative years and need to be improved further.
The application developed ignored the GML application schema(s) and embedded the entire processing logic into the application. The need to manipulate the application schema didn’t arise as we were only dealing with a single data model.
GML is a XML based language which means that it can be manipulated in the same way as we can manipulate XML. A number of parsers are available which allow one to manipulate XML documents. These include DOM, SAX and few more. Although DOM itself is easy to work with, it has its limitation in terms of amount of data it can handle. On the other hand SAX which is event based and does not create a document in memory, is capable of handling huge amount of data, something which looks promising for handling huge amounts of GML data.
Java programming language was utilized for developing the application. Object oriented approach. Visual Interface. Ability to extend memory available to java for processing huge GML files while maintaining performance. Directly manipulate compressed .gz files
Prompt the user to select the compressed GML file. Unzip the GML file and provide to parser Initialize the XML reader Parse file using the reader For start Document event initialize temporary data structures For Each Start Element Retrieve the element attributes Initialize Objects for respective classes For Each End Element Store data for each element Re-initialize each temporary data structure For All Characters Store characters in temporary variables For Document end Re-initialize variables.
MYSQL Database + Spatial Extension * http://dev.mysql.com/tech-resources/articles/4.1/gis-with-mysql.html GIS Data Types* More Information http://dev.mysql.com/doc/refman/6.0/en/spatial-extensions.html
My SQL spatial extensions need to be installed separately. Good resource is the MYSQL website. Establish connection from within java. Once established transfer data stored in temporary data structures to database tables. In this case tables were created before hand. However user can be provided more control by allowing them to create tables when needed from within the main application.
stmt.addBatch (&quot;INSERT INTO topo VALUES('&quot;+ ln.toid +&quot;',&quot;+ ln.version +&quot;,'&quot;+ ln.date +&quot;','&quot;+ ln.theme +&quot;','&quot;+ ln.featureCode +&quot;','&quot;+ ln.descriptiveGroup +&quot;','&quot;+ ln.descriptiveTerm +&quot;','&quot;+ ln.make +&quot;',&quot;+ ln.physicalLevel +&quot;,'&quot;+ ln.physicalPresence +&quot;',&quot;+ null +&quot;,&quot;+ null +&quot;,&quot;+ null +&quot;,&quot;+ &quot;GeomFromText('LINESTRING(&quot; + sb + &quot;)')&quot; +&quot;,'&quot;+ ln.accuracyOfPosition +&quot;','&quot;+ ch.lastChangeDate +&quot;','&quot;+ ch.lastReasonForChange +&quot;',&quot;+ null +&quot;,&quot;+ null +&quot;,&quot;+ null +&quot;,&quot;+ null +&quot;)&quot;); Variable sb holds the geometry (Line). Will need to format sb accordingly so that it is accepted by MYSQL. Use String functions. Example in MySQL: SET @g = 'LINESTRING(0 0,1 1,2 2)'; INSERT INTO geom VALUES (LineStringFromText(@g));
Java is not the only option. You can use other languages as well, Python, .NET languages etc. Extend memory available to Java. Failure to do so may give a heap size error.
Download and install geo server 1.5.4 Download and install geo server MYSQL extension. Read documentation for further configuration if required. Type: http://localhost:8080/geoserver to enter geo server administration section. Add Database >> Add Namespace >> Add Feature Type. Read Documentation for more information. Geoserver uses OpenLayers for visualizing Geographic information. More information on geoserver.org and OpenLayers.org
Ability to warp the application around different data models. This would result in more powerful application which will capable of translating any GML data. (Utilize Application Schemas). Use Postgre SQLDatabase+PostGIS+ProJ instead of MYSQL. Database Control from with the Application.
Thank you for attending this presentation. Any Questions / General Feedback? Faizan Tayyab