Automating Regional Data Integration with Python & ArcPy

1. AUTOMATING REGIONAL DATA
INTEGRATION WITH PYTHON & ARCPY
Heather Widlund
GIS Coordinator, San Miguel County
West Region Data Integration Group

2. Integrated data
Standardized schemas
oStreet centerlines
oAddress/structure points
oAddressed places/landmarks
Other: ESNs, PSAP boundaries, Parcels
Emergency preparedness & response focus
oVerify emergency notification addresses (CodeRED)
oVisual mapping in multi-agency dispatch
o911 call address verification (Spillman CAD)
4/29/2016GIS COLORADO | SPRING 2016 MEETING | BRECKENRIDGE, COLORADO

3. 4/29/2016GIS COLORADO | SPRING 2016 MEETING | BRECKENRIDGE, COLORADO
Data updates are arduous.
What is the problem?
Why is this a problem?
Data updates are infrequent.
What are the consequences?

4. What is the solution?
4/29/2016GIS COLORADO | SPRING 2016 MEETING | BRECKENRIDGE, COLORADO
>>Make a machine do the work:
. . .

5. Automation Project
Phases
1) Source data with valid schema
2) Ability to integrate easily & automatically
3) Data updates pushed to Spillman CAD
4/29/2016GIS COLORADO | SPRING 2016 MEETING | BRECKENRIDGE, COLORADO
Script tools, Python & arcpy library
oFamiliar, available
oNo extra installs/libraries
Ground-up: hosted at each agency
oPros: data ownership & responsibility, specific needs
oCons: keeping scripts in sync, specific needs

6. Implementation
Set up environment
Run script tools to create
configuration files
oEsri’s Community Addresses Solution
Configure batch file
Run batch file
oManually
oWindows Task Scheduler
4/29/2016GIS COLORADO | SPRING 2016 MEETING | BRECKENRIDGE, COLORADO

7. 4/29/2016GIS COLORADO | SPRING 2016 MEETING | BRECKENRIDGE, COLORADO
replacedata.pybatch.bat runupdate.py
Validate fields
Map fields
Extract data
Drop old data
Append new data
Unique IDs
Calc Lat/Long
Strip metadata
Schematic

8. Demo
4/29/2016GIS COLORADO | SPRING 2016 MEETING | BRECKENRIDGE, COLORADO

9. Lessons learned (so far)
Test, test, test: real data & real environments
Customized agency scripts
Functions
oGenerate a result (e.g. T/F)
oKeep them short & re-useable
Tools
oIDE: PyScripter
oLogging module
oThe internet – document your sources
4/29/2016GIS COLORADO | SPRING 2016 MEETING | BRECKENRIDGE, COLORADO

10. Progress
Installations to date
Next steps
oScript integration
oScript data push to
Spillman CAD
GIS in the Rockies
4/29/2016GIS COLORADO | SPRING 2016 MEETING | BRECKENRIDGE, COLORADO

11. Heather Widlund
heatherw@sanmiguelcountyco.gov
4/29/2016GIS COLORADO | SPRING 2016 MEETING | BRECKENRIDGE, COLORADO
Special thanks to:
West Region Data Integration Group
Questions?
https://github.com/hwidlund/integration.git