FOSS4G ETL tools for automatically translate from geospatial data structures to data models. Take in account that the reference in the title "How Cool .." is a joke because this operations is really painstaking.

1. How Cool Can Be The Migration of an Old GIS
Corporate Dataset to a New System
From Coverages to Geospatial Databases
Gabriel Asato
Geological and Mining Survey of Argentina
g_asato2000@yahoo.com, gabriel.asato@segemar.gov.ar
GEOLOGICAL CONGRESS
27 AUGUST - 4 SEPTEMBER 2016 | CAPE TOWN, SOUTH AFRICA

2. Converting Hundred of Datasets in
Geological Surveys
What we need
● Do it quickly
● Do Not loose information
● Do it consistently
What is happend
● Really boring process
● Error prone activity if is
made by hand.
● No integrated
consistency tests
● In the context of new
geospatial paradigms
this process is much
complex
Worries
● Topology in geospatial systems
is always a real concern!!

3. What we are Trying to Migrate?
A Geospatial Data Structure
(ESRI ARC/Info Coverages)

4. What we are Trying to Migrate?
To an Integrated
Geospatial
Complex Data Model
(a geospatial database,
spatialite format)

5. System Migration for geospatial data
doesn’t means Import/Export
Means a change of information paradigm
from Data Structures to Data Model

6. Extract, Transform, Load (ETL)
Free and Open Source Software Tools
● Data Extraction
○ Extract data from homogeneous or heterogeneous data sources
● Data Transformation
○ Transforms the data for storing it in the proper format or structure for the purposes of
querying and analysis
● Data Loading
○ Loads it into the final target
GDAL/OGR + Python + SQLite

7. GeoKettle

8. GDAL/OGR,
Python and Spatial SQLite
● GDAL/OGR Geospatial Data Abstraction Library
○ Universal translator for raster and vector using a single abstract
raster and vector library
● Python Programming Language
● SpatiaLite format
○ SQL Vector geodatabase functionality
○ Simple and powerful
○ Portable
○ Platform independent
○ Supports OGC SFA
○ OGC Geopackage is based on SpatiaLite format
○ Topology support
○ One of the natural replacement of ESRI shapefiles

9. Extract, Transform and Load Process
In Detail
● Initialize by loading corresponding drivers
● Check the existence of data in the file system
● Check data structure and map it to the new system
● Filters undesired data
● Load data

10. The Program cov2sqlite
Initialize Drivers
….
# INPUT
driver = ogr.GetDriverByName("AVCBin")
dataSource = driver.Open(cov_name,0) #0 read only, 1 write
#open layer
lyr = dataSource.GetLayerByName(lyr_type)
# lyr_type can be [‘ARC’, ‘PAL’, ‘LAB’]
….
# OUTPUT
sqlDrv = ogr.GetDriverByName("SQLite")
dataOut = sqlDrv.Open(sqlDB,1)
….

11. The Program cov2sqlite
Checking Data Structure and Translation
#Check Data Structure
dataStructureDicc = {
'geo':{'PAL':['AREA', 'PERIMETER', 'ID','SID'], 'ARC':['LENGTH', 'ID']},
'meas':{'LAB':['ID', 'AZIMUT', 'BUZAMIENTO','RUMBO','CUADRANTE']},
'folds':{'ARC':['LENGTH', 'id']}
}
#itemOut, itemIN Translation
ItemTranslationDicc={
'geo':{'PAL':{'area':'AREA', 'perimetro': 'PERIMETER' , 'id':'ID',
'sid':'SID'},'ARC':{'length':'LENGTH', 'id':'ID'}},
'meas':{'LAB':{'id':'ID','buzamiento':
'BUZAMIENTO','azimut':'AZIMUT','rumbo':'RUMBO','cuadrante':'CUADRANTE'}},
'folds':{'ARC':{'LENGTH':'length', 'id':'id'}}
}

12. Look Up Table to SLD
LUT2SLD
INPUT / OUTPUT
csvFilePath = 'nqn250colorb.csv'
#INPUT
SLDFileName = 'nqn250.sld' # OUTPUT
CSV Structure
ID
Color ( HTML Format )
Description ( alfanumeric )
Creating SLD
def GetSLDXML(gid, color, desc):
dataSLD = ( '<Rule>',
'<Name>' + desc + '</Name>',
'<ogc:Filter>',
' <ogc:PropertyIsEqualTo>',
' <ogc:PropertyName>sid</ogc:PropertyName>',
' <ogc:Literal>'+ gid + '</ogc:Literal>',
' </ogc:PropertyIsEqualTo>',
'</ogc:Filter>',
'<PolygonSymbolizer>',
' <Fill>',
' <CssParameter name="fill">'+color+'</CssParameter>',
' <CssParameter name="fill-opacity">100</CssParameter>',
' </Fill>',
'</PolygonSymbolizer>',
'</Rule>')
return dataSLD

13. Other Additional Operations Must be Done
● Set SRID
● Upload data dictionaries as tables
● Create views with SQL
● Upload palette as OGC SLD format and stored in SQLite
● Predefined Queries construction
● Check visualization and queries with other GIS softwares like QGIS.

14. Result
A Simple Solution for a Complex Task
QGIS Visualization of SQLite Geospatial Database
ETL FOSS4G Tools in the form of
GDAL/OGR + Python + SQLite
improve the process of geospatial
system migration, that means from
data structures to data models.
New geospatial technologies allows
the management of information in
unified, integrated system and easy
way, store information as complex
data models is possible.
SQLite format is open, consistent,
portable, multi platform, supports
OGC SF, it is a good replacement
of ESRI shapefiles, and it can be
upgrade to much complex systems
like PostGIS. Good for making
experiments.

15. Future
(still in development)
● Metadata
● Satellite Image Integration (mosaics)
● Topology
● Triggers
● Complex Geospatial Constraints
● OGC GeoPackage