Improving Data Compliancy Using FME


Published on

Presented by Dave van Riel, City of Kitchener
See more FME World Tour 2014 presentations at

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Just to give you some information on our organization. We are a city of 200,000 west of Toronto.We have a centralized GIS in an ESRISDE environment and in this environment we have many layers that are used, use, or are linked to information from many systems. AMANDA, Cityworks, SAP, Stormwater Rate, and CIS to name a few.We have been using FME for a while and each year we increase our use of this tool and the complexity of the work that it does.
  • Over the last few years our organization has experienced an increase in the interest and scrutiny of GIS data. While there has always been an effort to have correct and complete data there are always layers, geographic areas, or attribute fields that have slipped through the cracks for various reasons.There are many reasons why the GIS data at our city needs to be complete and correct as you can see here. But the reason in general terms is that there is a continual increase of GIS data being integrated into everyone’s everyday work. More than ever non-GIS data needs to be linked to GIS features and shown together, reports are being generated from GIS data, and decisions are being made based on these reports.
  • The types of features most viewed, most reported, and most decisions made based on those views and reports has been engineering infrastructure features – sanitary sewer, storm water sewer and water utilities. All these features are submitted together in drawings by consultants, in the case of new subdivisions, or by internal city staff, for reconstruction projects.To give you a little background on how layers, geographic areas, or attribute fields could have slipped through the cracks before the implementation of the digital submission process you’d need to understand how the data was previously captured. Scanned tifs or digitally produced pdf files were supplied to GIS staff. It was expected that most of the information needed in the GIS layers would be provided in the drawings.NEXT POINTThe first thing was to capture the location of the features and that meant georeferencing the plan and doing some heads up digitizing. Rarely was the AutoCAD file provided to GIS, it was never georeferenced, and most features would need major editing to be usable anyway. For example pipes would be snapped to the edge of manhole blocks, and not the center of the block where the point would be captured in GIS.
  • Next would be to capture the attribute information from those drawings.We can see from the earlier drawing example screenshot on the left that not all information was provided in the past. The diameter and slope are all that is supplied in terms of attribute data for this example.However, the screenshot on the right shows a more recent drawing, a digitally produced pdf file, that provides more information. One question is: are all the pieces of information that are needed there? They should be…
  • …and most of the time they were. If all the pieces are provided in the drawings, the next question was: where are they?This is a screenshot of one of the pages of documentation I was handed when I first started with the city back in 2005. It shows where to typically get the needed information for a storm water pipe off of a plan and profile drawing. Not all plan and profile drawings were the same though. The location of the information could be anywhere. It could depend on the submitting consultant; the amount of area being covered in a drawing or project; or if there was room for the information in the main part of the drawing or if the information was put in the surround notes. Sometimes information would have to be gotten off of different drawings. For example a pipe invert might be only shown on a manhole detail drawing and not on the plan and profile drawing.Basically look at all the drawings provided and find the information you need.
  • So the difficulties associated with the old method of capturing the infrastructure data were a hindrance for the smooth incorporation of GIS data in the viewing, reporting, decision making, and daily work flow processes for others in the city.NEXT POINTThe old method was slow. City staff were needing a faster turn around time. They needed that data. A typical sized subdivision would take a week of staff time to capture in GIS.NEXT POINTThe old method was prone to errors. Either the correct drawings with the information were not provided, the information was not provided in any drawings, or GIS staff couldn’t find or interpret it.NEXT POINTThe old method wasted time. Why was this information being input again in a different format? Shouldn’t this informationbe entered once by the ones making the drawings? The job of GIS staff should be to facilitate the importing of the data, but not the searching for or interpretation of that data.NEXT POINTNo one was happy. The data in GIS wasn’t as reliable as it should have been and no one used it if they didn’t have to. Engineering staff had their own repository of drawings and data. Operations staff relied on hard copy inspection and maintenance forms for data that was missing in GIS, or for city wide reports. Consultants needed the GIS data for new construction planning or for any consulting work and they couldn’t get it. And worst of all, from GIS staff’s point of view, it felt like the GIS section was getting the majority of the blame for the missing data.
  • In the GIS section at the City of Kitchener the idea had always been discussed that wouldn’t it be nice to be able to automatically check and load the data into the database. Certainly there must be technologies to enable this? We knew there were, but never have the right conditions come along: namely strong non-GIS staff support and need for the benefits of digital submission, and a tool like FME to allow for data compliancy checks.NEXT POINTCAD standards could be checked. For many years Consultants were required to use an AutoCAD template file for these standards and they had been following them properly. But the ability to check the colour, line type, and line weight of AutoCAD entities on a layer – the properties that allow for correct looking digital plans to be produced – would ensure proper looking pdf files.NEXT POINTAttribute data could be inspected and verified. This would be something entirely new for us. Are all the necessary attribute fields provided, and are the values that are provided in those fields valid.NEXT POINTFinally, the placement of the submitted AutoCAD entities can be checked with respect to other submitted features.NEXT POINTIt must be noted that the attribute and placement correctness are what I call relative. This means that it is easy to check if the attribute field or placement of an AutoCAD entity follows a rule, but not if the value or placement is necessarily correct.
  • As I said, the right conditions came into place to allow for digital submission implementation and therefore data compliancy checks. However, the city lacked the staff resources to make this a reality.NEXT POINTIt was around this time that we saw a presentation by Consortech concerning the use of FME for checking CAD standards. We approached this consultant company and asked if it was possible to create the kinds of data compliancy checks we were thinking of. They assured us it was, and out of this grew what we refer to as the CAD Standards, Attribute, and Topology Checkers.NEXT POINTThese checkers use FME scripts to read consultant submitted AutoCAD Map 3D and checker settings files to produce error outputs that can be used by these consultants to rectify any issues that are reported.NEXT POINTThere were 2 major requirements the city askedfor in all the checkers.One was that they had to produce some form of GIS or CAD output for visual purposes, and a text report. Either of these should enable the submitter of the data to easily find the issue AutoCAD entities.NEXT POINTThe other requirement was that because of the constant changes concerning the features being captured and the attributes associated with them, the basic settings for the checkers had to be customizable.
  • Here are some screenshots for the CAD Standards Checker. Consortech had already developed this tool for similar purposes for other clients.On the left is an example of the AutoCAD information for an AutoCAD entity on the water main layer: it’s colour and line type.On the right is the CAD Standards Checker Manager Excel file that contains what the properties for AutoCAD entities on specific AutoCAD layers should be. Colour, line type and line weight properties can be checked.NEXT POINTThe FME CAD Standards Checker is run and uses these two inputs. Any issues are reported out to a summary and a detailed error text file.NEXT POINTThe checker also produces a CAD file that contains AutoCAD geometric shape entities grouped by a combination of input feature entity type and error issue at the center point of each issue entity. The properties for these geometric shape entities are set in the manager file. This CAD error file is georeferenced and can be imported into the original file.NEXT POINTIf no entities are returned as issues then the correct looking pdf files can be produced. These are important to our engineering and operations staff as not all information is captured in GIS and the drawings are often used to provide a quick overall view of the infrastructure in an area.
  • Here are some screenshots for the Attribute Checker.In AutoCAD Map 3D there is the ability to attach object data, which are basic attribute tables, to AutoCAD entities. This screenshot shows the structure of the WAT_MAIN object data table on the left and it attached to an entity on the water main layer on the right with the attributes filled in.NEXT POINTThe Attribute Checker uses a csv file produced in Excel that identifies what object data field names should be attached to the entities on specific AutoCAD layers. Additional checks on each field can be performed – minimum and maximum character lengths, field data types, minimum and maximum number values, pick list restrictions, and whether the field is required to have a value or not.NEXT POINTThe FME Attribute Checker is run and uses these two inputs and, as with the CAD Standards Checker, any issues found by the Attribute Checker are reported out to a summary and a detailed error text file.NEXT POINTA shapefile is also produced that shows the location of the issue entities. This shapefile contains the same information that is present in the detailed error file and can be brought into AutoCAD Map 3D if needed.
  • And finally the Topology Checker.There are 12 topology rules that the Topology Checker tests for and the layers that each rule tests on can be customized in an Excel file. Using this topology manager file the FME Topology Checker tests lines and points in the submitted file.NEXT POINTA detailed error Excel file and a shapefile are produced. Again, this shapefile contains the same information that is present in the detailed error file and can be brought into AutoCAD Map 3D if needed.
  • If the submitted file passes all checkers without issues the drawing is approved and the data can then be loaded into the GIS database.NEXT POINTCurrently a combination of AutoCAD export profile files and ArcGIS’s Model Builder are used to extract and load the data. These are run manually but we are looking into ways to increase the automation of this step.
  • After the data is loaded into the database there are always final edits and checks to perform.NEXT POINTChecking for logical connections with existing features being the most important of them. It is always assumed that the new features being submitted are in the proper location and the existing features are snapped to them.NEXT POINTThe digitally produced pdf drawing files are registered into an in-house document management system and extent boundaries are created in GIS for each of the drawings. These boundaries are then linked to the registered files so that city staff can easily open drawings based on the boundary extent chosen.NEXT POINTOther fields that are required in GIS but not provided in the digitally submitted AutoCAD files are filled out by city GIS staff or other automation processes.
  • The benefits of the digital submission process and data compliancy checks address the main issues with the old method of capturing infrastructure data.NEXT POINTThe turn around time for GIS to actually capture the data has been drastically reduced. What might have taken a week to do has been reduced to a few hours.NEXT POINTAnd with the checkers we can ensure that the data being provided and loaded is valid and complete. No more multiple data entry. No more missing data. The data currently coming in is better and more complete than ever before.NEXT POINTAll parties understand what they need to do. The digitally submitted files must contain the needed information in the format specified. Data is checked based on the standards and rules specified, and issues are returned to the submitter for correction and resubmission. Approved data is loaded and additional data not required from the drawings are captured by GIS staff. GIS staff also remove any replaced features.NEXT POINTEveryone is happy. All data is available to city staff to view, report, and work on. GIS staff are not guestimating to fill in missing data. And sure, initially there was some grumbling from consultants about the additional work, but they are starting to realize that the amount of work is not that great, and they are benefitting as they in turn request this GIS data when planning new construction projects or performing any consulting work.
  • What’s next? Continue improving.NEXT POINTThere are constant changes taking place in the GIS database because of new inspections, maintenance programs, and work order templates in Cityworks, not to mention just additional features or attributes that were not previously captured. With all these changes it has been a challenge to keep the AutoCAD template that consultants download current, as well as knowing which versions of the checkers settings to use to check the submitted drawings.NEXT POINTOf course, keep finding uses for FME. Using it to assist in the loading of the data is the next logical step. Maybe even assist in deleting the features that need to be removed.NEXT POINTCurrently the infrastructureprojects done through and for the Region of Waterloo inside the City of Kitchener are not in any digital submission format and therefore can’t be checked for data compliancy. However, Regional requests for city data has increased over the last few years for various reasons. Just like the consultants using city GIS data for their construction projects and analysis consulting, it is hoped that these projects and others like them will eventually encourage the Region to implement a similar digital submission process that would allow for data compliancy checks.NEXT POINTThe ability to customize the basic settings of the checkers allows for data compliancy checks in other areas of the city. For example our Gas Utility staff have recently moved over to AutoCAD to leverage the use of the digital submission process and these data compliancy checkers on their drawings. We are starting to received drawings from them and expect that by the end of the upcoming construction season most drawings will be submitted in the required format.NEXT POINTAnd lastly using FME Server. Currently we have only one FME desktop license and digital submission files are sent to GIS staff for running through the checkers. Having FME Server could give Engineering staff, or the consultants themselves, access to submit the files directly to the checkers, which would therefore reduce the time taken to check for data compliancy.
  • Improving Data Compliancy Using FME

    1. 1. CONNECT. TRANSFORM. AUTOMATE. Improving Data Compliancy Using FME David van Riel GIS Technologist, City of Kitchener
    2. 2. City of Kitchener Information  1 hour west of Toronto.  200,000+ people.  Centralized GIS in ESRISDE environment.  Maintain ~400 GIS layers, many linked and/or contribute to other systems.  User of FME for last ~8 years. Initially for data exporting and transforming, but increasingly more for data loading and manipulation.
    3. 3. Need for Complete and Correct Data  Asset management:  Public Sector Accounting Board (PSAB) 3150 for reporting of tangible capital assets.  SAP for city financial and accounting needs.  For work order management, inspection programs, and reporting being done through, or housed in, Cityworks.  As usual any and all other mapping, analysis, reporting, and exporting done on city’s GIS asset information.
    4. 4. Infrastructure Features: Former Data Capture Process Majority of:  Sanitary sewer  Storm water sewer  Water  Geographic capture = retrace features
    5. 5. Infrastructure Features: Former Data Capture Process Is information present?
    6. 6. Infrastructure Features: Former Data Capture Process Where is the information?
    7. 7. Infrastructure Features: Former Data Capture Process  Slow.  Prone to errors.  Data entered multiple times.  Unhappy customers.
    8. 8. Digital Submission Capability/Ability  CAD characteristics (CAD Standards Checker)  Complete attribute data (Attribute Checker)  Correct attribute data (Attribute Checker)  Correct geographic placement (Topology Checker) * * * Relative, not absolute
    9. 9. Digital Submission Compliancy Checkers  CAD Standards, Attribute, and Topology Checkers developed by Consortech.  FME reads consultant submitted AutoCAD Map 3D files (with object data) and Excel files containing basic checker settings.  Requirements for all checkers: 1. GIS or AutoCAD, and text or Excel outputs. 2. Easily customizable settings for changing needs.
    10. 10. CAD Standards Checker
    11. 11. Attribute Checker
    12. 12. Topology Checker 12 rules:  node edge snap  node end snap  floating node  duplicate node  line end snap  line edge snap  crossed line  line end node  duplicate line  floating service  one line two nodes  both line ends snap
    13. 13. Loading Data into CoK Database  After drawing has passed all checkers the data is ready to be exported from AutoCAD and loaded into the GIS database.  Use AutoCAD epf (export profile) files to export to shapefiles, then using ArcGIS Model Builder to load data into database.
    14. 14. Manual Checks and Procedures  Check connectivity of new features to existing features.  Register .pdf files in EDRA (in house Electronic Document Registration Application) and create linked boundaries in GIS.  Fill in fields not provided by consultants (SOURCE, SOURCE_DATE, ROADSEGMENTID, QUARTER_GRID_ID, UP_MA NHOLEID, etc…).
    15. 15. Digital Submission / Data Compliancy Benefits  Time saving (solves slow method).  Accuracy/completeness of data (solves prone to error issue).  Defined deliverables with respect all parties (solves multiple data entry issue).  Everyone is happy!
    16. 16. What’s Next?  Synchronize AutoCAD template to GIS database (constantly changing).  More/improved database loading automation and automate manual GIS fields where possible (using FME for complete process?).  Incorporate Region of Waterloo projects.  Apply checkers to other processes (gas utility – AutoCAD training late fall 2013 and first digital submissions spring 2014).  FME Server.
    17. 17. Thank You!  Questions?  For more information: City of Kitchener  AutoCAD template at sp (or search for “development manual”)     
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.