developing and implementing spatial etl processes with open source tools

1. Developing and Implementing
Spatial ETL Processes
with Open Source Tools
Matthew Baker
Denver Public Schools
Dep't of Planning and Analysis

2. Quick FOSS4G Update
● Replaced ArcSDE with PostGIS
– Dev, QA (Prod TBA...)
● 'Non-GIS' users using QGIS
● Editable PostGIS Views!
● Data-driven cartography
– OSM data with styles saved in PostGIS
● Manager now using QGIS

3. SQL Server
Dev QA Prod
PostGIS
ArcGIS
Enterprise
maps.dpsk12.org
Database Structure

4. ETL:
Extract
Transform
Load

5. ETL Process Needs
● PostGIS read/write
● SQL Server Spatial read/write
● Daily Updating of Tables
● Daily Building of Datasets
● Daily Delivery to DPS Enterprise

6. Goals of ETL Development
● Break dependency on GUI-based tools
● Overcome 'other' FOSS ETL Tools
– Geokettle
– GDAL
● Avoid commercial ETL tools
– SSIS
– FME

7. Creating New Tables
● Import Shapefiles to Dev
– QGIS DB Manager
● Import Non-Spatial Tables
– CSVKit - Python via command line
– Read CSV Schema
● Generate SQL ‘Create Table’

9. PostGIS and Spatial “Text”
select
geom
from dpsdata."Schools_Current"

10. PostGIS and Spatial “Text”
select
cast(geom as varchar)
from dpsdata."Schools_Current"

11. PostGIS and Spatial “Text”
select
ST_AsText(geom)
from dpsdata."Schools_Current"

12. PostGIS Spatial Transformation
select
ST_Transform(geom, 2877)
from dpsdata."Schools_Current"
select
ST_AsText(ST_Transform(geom, 2877))
from dpsdata."Schools_Current"

13. Python for Databases
● Pypyodbc
– Pure python implementation of pyodbc
– Connect to databases using ODBC
● MS SQL Server
● Psycopg2
– PostgreSQL adapter (libraries) for Python

14. SQL Inserts
insert
into tablename (column1, column2)
values ('value1', 'value2')

15. SQL Inserts with Parameters
INSERT INTO "Schools_Current"
(school_name, abbreviation, elem,
mid, high, schnum, geom
classification)
VALUES (?, ?, ?, ?, ?, ?, ?);*
* postgresql syntax

16. Python ETL Pattern
Connect to databases
Truncate Destination
Insert into Destination
Select from Source

17. Python ETL Pattern
● Connect to databases
● Source
● Destination
● Set Up Cursors
● Select from Source
● Use SQL Expression (with spatial function)
● Assign data to rows (in memory)
● Insert into Destination
● Create insert statement with parameters
● Iterate through rows (data)
● Assign row values to variables
● Commit data with Insert
● Truncate Destination

18. Example: PostGIS to PostGIS
import psycopg2
connSource = psycopg2.connect("host=arcgisdev01 dbname=dpspgisdev user=dpsdata
password=*** ")
curSource = connSource.cursor()
connDest = psycopg2.connect("host=FOSS4GLin01 dbname=dpspgisqa user=dpsdata
password=*** ")
curDest = connDest.cursor()
curSource.execute('''
select addressid, cast(geom as varchar) from public."Address_Master"
''')
sql = ('''
insert into dqmt.Address_Master (addressid, geom) values (%s, %s)
''')
data = []
rows = curSource.fetchall()
for row in rows:
data = [row[0], row[1]]
curDest.execute (sql, data)
connDest.commit()
connSource.close()
connDest.close()

19. Deployed Processes
● Daily Active Students
– Extract from MSSQL View joining geometry to students
– Deliver to PostGIS and MSSQL
● Refresh Boundaries
– PostGIS Materialized Views
● Geocoding
● Enterprise Delivery
– Schools and Boundaries
– Shared Enrollment Zone Info
– Current Addresses and Boundary Information (spatial join)

20. Deployment
● Microsoft Windows Server
– Task Scheduler
– (still doesn't run FME / ArcPY scripts)

21. Ubuntu Server Deployment
● Cron Task Scheduler
0 3 * * * python /home/dpspgisqa/scripts/SchoolBoundaries_All.py
0 3 * * * python /home/dpspgisqa/scripts/SchoolBoundaries...
0 3 * * * python /home/dpspgisqa/scripts/Schools_Current.py
0 3 * * * python /home/dpspgisqa/scripts/Schools_Projected.py
* * * * * /folder/runThisFile.py
| | | | |
| | | | ----- Day of week (0 - 7) (Sunday=0 or 7)
| | | ------- Month (1 - 12)
| | --------- Day of month (1 - 31)
| ----------- Hour (0 - 23)
------------- Minute (0 - 59)

22. Other Python Tricks
● Error Handling
– On script fail
● Send Email
● Insert message to database
● Run single SQL Script
– Within 1 database
● Bulk Inserts

23. Next Steps
● Implement PostGIS Prod server
– CentOS (new IT staff!!!)
● Document Internally
● Share Externally
– github.com/DPSSpatial
● Web maps
– Internal
– External

24. THANK YOU!
planning@dpsk12.org
github.com/dpsspatial