2013 GISCO Track, Converting a Transportation Data Model into Geodatabase by Bruce Reagan


Published on

The City and County of Denver began a three year project to convert our city streets from a segment based system to a linear reference system. As part of the project, we delved into a deep investigation into all the data sources that conceivably could be impacted by this change. We discovered over a hundred data objects that could possibly be involved.

To manage this long project, we created a database that documented all the current objects and their associated fields, domains and relationships. Simultaneously, we developed a conceptual model for storing this information, initially based on the Transportation model put forth by Al Butler in his book. We developed a cross walk that helped to ensure that all existing information had a place in the new model and also to ensure that each item in the model had a source or at least a potential source.

During this process it occurred to us that we could use python scripts and ArcGIS tools to read the destination database object descriptions and create an empty shell that met the specification laid out in these tables. The advantage was that we could make small adjustments (such as adding values to domains) and re-create the entire database with the touch of a button and sufficient processing time.

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

2013 GISCO Track, Converting a Transportation Data Model into Geodatabase by Bruce Reagan

  1. 1. City and County of Denver GIS in the Rockies 2013
  2. 2. Conversion to a Linear Model  In 2010 the City and County of Denver embarked on a plan to convert our existing Feature Based transportation features into a linear model.  Part of the idea was to take advantage of the Roads and Highways application that was being developed.
  3. 3. Linear Referencing 2007 ESRI UC
  4. 4. Existing GIS Data  We maintained a street layer that we published into various editions primarily for Safety and for HUTF.  We also maintained some event based layers such as Pavement and Snow Routes.  Resulted in a layer with 60 fields.  Arc-Node model supported Intersections and Routes.
  5. 5. Deep Dive As part of this we did a deep dive into as many agencies and their systems as we could.  Usual customers:  Pavement management  Traffic Engineering  HUTF  Safety  Parks (trails)  City GIS users  Permitting systems  Addressing
  6. 6. Some new ones  We also reached out to Networking for fiber routing … but this is not that story.
  7. 7. A lot of data  One way we found to get our hands around all this data was to dump all of the necessary data elements into a data dictionary.
  8. 8. Processing the results  We began a process of model creation to try to optimize the overall structure of transportation features as stored in the GIS.
  9. 9. Data Modeling  It was a big job and we hired a professional. (as you can see it was too big to fit on a slide)
  10. 10. Groups of tables  We wound up with 3 main groups of tables:  Those specific to the ESRI ALRS model  Those specific to our data  The data dictionary  This presentation is about the data dictionary and how we used it to generate our schemas
  11. 11. Source Tables  Documenting the existing system
  12. 12. Source Features Included things like:  Bridges  Permits  Traffic signals  Traffic Counts  Parking meters  BCycle Stations  City projects All to be turned into Events
  13. 13. How do we examine all this?  We documented every existing feature even remotely connected to transportation  For every feature we documented every attribute, data type and every domain and their values
  14. 14. Problems with the data  A by-product of this was that we found things like the same data being stored under slightly different field names, redundant domains, names left over from coverages etc.
  15. 15. Crosswalk  In the process we uncovered a need to be thorough and to create a crosswalk table by which we mapped a source field to a target field.
  16. 16. Crosswalk (cont.)  Having this as a table allowed us to quickly and easily identify items with no match in either the source or the target.  Mismatches were part of an iterative process that forced us to think about why we needed certain data elements.
  17. 17. Creating a geodatabase  Several times early on we would create a geodatabase schema of parts of the model.  Because it was so large, we would often just use parts of it.
  18. 18. Schema Creation  It occurred to me we could use this crosswalk table to create a schema.  Using what we had I wrote some python scripts to do that.
  19. 19. Needed to be able to repeat this process  We needed to be able to do this iteratively. The process of writing the scripts helped us identify some inconsistencies between the data model and the physical implementation.  It also allowed us to keep up with the changing nature of the Roads and Highways software which was very beta at this point.
  20. 20. Script  I used Python and standard geoprocessing tools such as Create Domain, Add Value to Domain, Create table, Add Field, etc.  That made the process independent of file geodatabase vs SDE
  21. 21. Process Steps  SDE or File GeoDatabase?  Determined the connections
  22. 22. Usage Table  We had a Usage table that for each table listed the fields to be created.  There was a field for the order in which the field would be added to the table.
  23. 23. FieldName tables  Standardized field names listed data types and domains
  24. 24. Process steps  Created the domains first.  Went through field names and found unique list.  Used the CodedValue table to find the standardized list of acceptable values for the domain.
  25. 25. Process Steps  Created features using arcpy.CreateFeatureClass.  Event tables were created using arcpy.CreateTable_management  Grabbed a set of records for the fields, put them in an array so they could be sorted based on field order (haven’t we all run into the irritation of trying to rearrange the order of fields in an existing feature class?)
  26. 26. Process Steps  Last came the relationship classes.  Features have to exist
  27. 27. Load data  Loading data was another problem, required several other scripts and may be another presentation.  Most of the attributes in the street centerline became linear events  Dissolved continuous events, eliminating existing segmentation (though we preserved it as a separate event)
  28. 28. Dual purpose  The beauty of it was we could run it, load data into it, test it, determine the problems and then run it again.  It allowed the data modeler to just add values to the crosswalk table and those changes would be implemented.  We were still able to do our data dictionary comparison tasks.
  29. 29. Results  It worked!!!  It leveraged something we were doing anyway (the data dictionary) and we didn’t have to learn anything new  It was flexible enough to allow us to keep up with changes in the Roads and Highways design  Provided a non technical way of changing the structure and to QA those changes  It wasn’t fast – in the future we would probably use something else like FME (which we didn’t have at the time)
  30. 30. Contact Information Bruce Reagan Associate GIS Developer City and County of Denver Bruce.reagan@denvergov.org (720) 913 - 5883