Data Validation using FME Server Clay Wise, OCP, GISP Enterprise Architect Hampton Roads Sanitation District
Abstract (Hidden Slide) The Hampton Roads Sanitation District (HRSD), which serves 17 localities in southeastern Virginia and a population of 1.7 million, has embarked on a Virginia Department of Environmental Quality (VDEQ) driven effort to upgrade its utility management systems in support of a regional hydraulic wastewater model. The hydraulic model must include the entire interceptor infrastructure as well as input from the 14 separate models of the participating localities that feed into the collection system. With data inputs coming from so many different sources, each with their own data schemas, a solution needed to be developed to manage the continuous transfer of data from the localities to the regional hydraulic model, as well as from HRSD back to the localities. Topics covered in this presentation will include how FME Server was customized and implemented to streamline the input and validation of the various data sources, how the solution provided tremendous time saving and data quality insurance, and how FME Server helped resolve complex data sharing challenges.
HRSD (Hampton Roads Sanitation District Who? What you should come away with
Whistler, BC to Virginia Beach, Virginia = 5,107 km or 3,173 miles
HRSD Mission Statement Mission Prevent pollution, protect public health, and support community development by providing wastewater collection and treatment
Quick Facts Treatment of Sanitary Sewer for over 1.7 million people. More than 500 miles of pipes, 6-66 inches in diameter 102 Pump Stations 9 Major Treatment plants 231 million gallons per day is treated Serving 17 cities/counties
Regulatory Requirements Context: Sanitary Sewer Overflows Task: Regional Hydraulic Model Mostly Pressurized System 14 Jurisdictional Data Inputs 14 Jurisdictional GIS Database Designs Numerous Data Inputs Over 1,800 jurisdictional pump stations Over 6,500 miles of jurisdictional sewer mains Over 120,000 sewer structures (manholes, vaults, etc…) Challenge: Data Sharing Jurisdictional Hydraulic Model and general GIS Data Inputs to HRSD HRSD Regional Hydraulic Model Data to Jurisdictions Data Validation & Standardization, Data Updates & Corrections
Data Validation Tool
What are we validating?
Every attribute (Field)
Specific values (material, locality code)
How many were correct out of the total
Is the data spatially correct?
Is the geometry valid?
Are their connection points valid?
Validation Checks Connection Points Where a jurisdiction connects to HRSD Force Main system. Checking to see if the IDs match between their submittal and the connection point table we maintain. Spatially correct Zero length lines Self intersecting polygons **With 14 jurisdictions we have run into all levels of GIS users. This check should be in your validation process regardless of how simply or complex this process is.
How do you upload and download to FME Server? FME Workbench Desktop product to develop and publish data validation workspaces to FME Server Once published the workspace can be run through a web browser End users do not need an FME license to run the translation Can also download the posted workspace to make changes
How are we validating data? FME Workbench Desktop product to construct/develop the validation process Simple upload to publish process to FME Server Upload & Download FME Server
The “nuts and bolts” of data validation
Custom Transformers (Green)
Specific Values(Locality Code)
Showing the results Using the drop down menu you should set this parameter to an attribute that contains unique ids. Even if you don’t want to show these ids on the report it is still required. cfg01PumpStations: Calls out a specific variable in the report web page.
Showing the results (continued..) <li class="error_message">cfgFailCount of cfgFeatCount features in your dataset have blank PSID attribute values.<ul> <li>The ids of the failed features are: cfgFailedIds.</li></ul></li><br/> cfg01PumpStations: Calls out a specific variable in the report web page.
Web page editing cfg01PumpStations: This is the same variable we called out in the previous slide. After the information goes through the validation process it is placed in this variable. You can change them to whatever name suits you as long as they match the between the workbench file, and the webpage. I use Dreamweaver (CS3), but you can use cheaper software such as UltraEdit to edit the HTML.
Web page editing Color coding the message displayed. Here I simply have red for an error, and green if all the features passed. cfg01Nodes: This is the same type of variable I have spoken about in the previous slides.
Results are grouped according to the needs of the modelers
Green for passed, Red for failure
Connection points in blue are included with the summary checks
cfgPipes02 cfgPipes03 cfgPipes04
Once all the data is correct, a data download page will appear
The jurisdiction will then be able to upload their data through our firewall
Identity Management Giving the jurisdictions access to our data Oracle Identity Management Granular protection inside the HRSD firewall Single Sign on Reduce Risk Used for many other processes than just validation Virginia Beach Suffolk Chesapeake Hampton Norfolk Poquoson Portsmouth Williamsburg Gloucester Isle of Wight James City King and Queen King William Mathews Middlesex York
Suffolk FME Server Example Suffolk will log in using OIM They will then find the GIS Parcels, and Water Meters that they maintain They will then run FME Server FME Server will join their GIS data to the water consumption data, and output a zip file for their use
Good first run of the Validation Tool in winter of 2008
Modelers provided feedback for the second run
Geometry Check – Complete
Some jurisdictions didn’t have very clean data (i.e. Self intersecting polygons, zero length lines)
Brief summary at the beginning of each feature class – Complete
Summary of linear feet of pipe- Complete
Connection point validation – Complete
Positive: HRSD has control over all changes.
Benefits HRSD has control over all changes HRSD is able to quickly see where jurisdictions are failing to meet the requirements Instead of weeks of manually checks on each feature class, the data can be processed in minutes. Sharing the reports with the jurisdictions Efficient process for keeping data current, and valid. Simple form to fill out. Non-GIS personnel can run the application over the web. Get to come to cool conferences like this one and talk about it!
Questions Questions? For more information: Clay Wise: firstname.lastname@example.org HRSD: www.hrsd.com Data validation: fmepedia.com FME Server > F > FME Server Examples>D > Data Validation with FME Server