Who is HRSD? What is our purpose?What should come away with in this presentation?
HRSD Mission Statement: to prevent pollution, protect public health, and support community development by providing wastewater collection, treatment, and related services.
Quick Facts about HRSD.Just try to get the audience to understand the amount of sanitary sewer being treated.
What are the regulatory requirements?How did HRSD and the Jurisdictions get to this point?How do we comply (meet deadlines) with the state orders/standards?What happens if we do not comply?CourtFines
The Validation ToolWe are validating against a set of standards developed by HRSD and the Jurisdictions.They didn’t expect HRSD to validate against all of their data.Complete reporting of their modeling data.Explain connection points and why they are important.
Explain the process before going into data validation with both feet. Try to at least show the FME Desktop process to upload during the demo.
Very simple blow up of the beginning of one feature class being validated. Try to explain how I took the example workspace and started trying to conform it to what we needed. It was a great start. Had a lot of help from Safe Software, especially Aaron Koning.
Common tester example.Validating against all the locality codes. They have to populate these so we know which jurisdiction’s data it is.
Explaining briefly the group tester transformer that points to the html page. The results are shown based on the identifiers called out here.
Quick shot of the HTML page with the variable name called out.
Code example of the same html page. Just trying to show you can change the color, and other properties of the report to make it look any way you want.
Example of the output. I want to show how the data is grouped based off of the modelers needs, not the jurisdictions. I also am trying to show how the cfg call outs are related to prior html page.
Oracle Identity management. Currently we run the reports and provide them back to the jurisdictions. This will end in the near future with the implementation of OIM. We also have some other projects using FME Server and ArcGIS Server that need OIM to succeed. A good example is the City of Suffolk, and their need for billing data.
City of Suffolk example of how we are trying to use FME Server to push a SOA architecture. Let the jurisdiction manage when they want the data.
Feel very proud about the success we have accomplished during this project. We made sure to get a lot of feedback from the modelers. People are starting to ask for more FME Server apps to take care of processes.
My contact information.
FME User Conference Presentation
Data Validation using FME Server<br />Clay Wise, OCP, GISP<br />Enterprise Architect<br />Hampton Roads Sanitation District<br />
Abstract (Hidden Slide)<br />The Hampton Roads Sanitation District (HRSD), which serves 17 localities in southeastern Virginia and a population of 1.7 million, has embarked on a Virginia Department of Environmental Quality (VDEQ) driven effort to upgrade its utility management systems in support of a regional hydraulic wastewater model. The hydraulic model must include the entire interceptor infrastructure as well as input from the 14 separate models of the participating localities that feed into the collection system. With data inputs coming from so many different sources, each with their own data schemas, a solution needed to be developed to manage the continuous transfer of data from the localities to the regional hydraulic model, as well as from HRSD back to the localities. Topics covered in this presentation will include how FME Server was customized and implemented to streamline the input and validation of the various data sources, how the solution provided tremendous time saving and data quality insurance, and how FME Server helped resolve complex data sharing challenges.<br />
HRSD (Hampton Roads Sanitation District<br />Who?<br />What you should come away with<br /><ul><li>Whistler, BC to Virginia Beach, Virginia = 5,107 km or 3,173 miles</li></li></ul><li>HRSD Mission Statement<br />Mission<br />Prevent pollution, protect public health, and support community development by providing wastewater collection and treatment<br />
Quick Facts<br />Treatment of Sanitary Sewer for over 1.7 million people.<br />More than 500 miles of pipes, 6-66 inches in diameter<br />102 Pump Stations<br />9 Major Treatment plants<br />231 million gallons per day is treated<br />Serving 17 cities/counties<br />
Regulatory Requirements<br />Context: Sanitary Sewer Overflows<br />Task: Regional Hydraulic Model<br />Mostly Pressurized System<br />14 Jurisdictional Data Inputs<br />14 Jurisdictional GIS Database Designs<br />Numerous Data Inputs<br />Over 1,800 jurisdictional pump stations<br />Over 6,500 miles of jurisdictional sewer mains<br />Over 120,000 sewer structures (manholes, vaults, etc…)<br />Challenge: Data Sharing<br />Jurisdictional Hydraulic Model and general GIS Data Inputs to HRSD<br />HRSD Regional Hydraulic Model Data to Jurisdictions<br />Data Validation & Standardization, Data Updates & Corrections<br />
Data Validation Tool<br /><ul><li>What are we validating?
Are their connection points valid?</li></li></ul><li>Validation Checks <br />Connection Points<br />Where a jurisdiction connects to HRSD Force Main system. Checking to see if the IDs match between their submittal and the connection point table we maintain.<br />Spatially correct<br />Zero length lines<br />Self intersecting polygons<br />**With 14 jurisdictions we have run into all levels of GIS users. This check should be in your validation process regardless of how simply or complex this process is.<br />
How do you upload and download to FME Server?<br />FME Workbench<br />Desktop product to develop and publish data validation workspaces to FME Server<br />Once published the workspace can be run through a web browser<br />End users do not need an FME license to run the translation<br />Can also download the posted workspace to make changes<br />
How are we validating data?<br />FME Workbench<br />Desktop product to construct/develop the validation process<br />Simple upload to publish process to FME Server<br />Upload &<br />Download<br />FME Server<br />
The “nuts and bolts” of data validation<br /><ul><li>Example Workspace
Range (Elevations)</li></li></ul><li>Showing the results<br />Using the drop down menu you should<br />set this parameter to an attribute that <br />contains unique ids. Even if you don’t <br />want to show these ids on the report it is <br />still required.<br />cfg01PumpStations:<br />Calls out a specific variable in the report web page.<br />
Showing the results (continued..)<br /><li class="error_message">cfgFailCount of cfgFeatCount features in <br />your dataset have blank PSID attribute values.<ul><br /><li>The ids of the failed features are: cfgFailedIds.</li></ul></li><br/><br />cfg01PumpStations:<br />Calls out a specific variable in the report web page.<br />
Web page editing<br />cfg01PumpStations: This is the same variable <br />we called out in the previous slide. After the information <br />goes through the validation process it is placed in this variable. <br />You can change them to whatever name suits you as long as <br />they match the between the workbench file, and the webpage.<br />I use Dreamweaver (CS3), but you can use <br />cheaper software such as UltraEdit to edit the HTML.<br />
Web page editing<br />Color coding the message displayed. Here I simply have red for an error, and green if all the features passed.<br />cfg01Nodes: This is the same type of variable I have spoken about in the previous slides.<br />
Results<br />cfgPipes01<br /><ul><li>Results are grouped according to the needs of the modelers
Connection points in blue are included with the summary checks</li></ul>cfgPipes02<br />cfgPipes03<br />cfgPipes04<br />
Data Upload<br /><ul><li>Once all the data is correct, a data download page will appear
The jurisdiction will then be able to upload their data through our firewall</li></li></ul><li>Identity Management<br />Giving the jurisdictions access to our data<br />Oracle Identity Management<br />Granular protection inside the HRSD firewall<br />Single Sign on<br />Reduce Risk<br />Used for many other processes than just validation<br />Virginia Beach<br />Suffolk<br />Chesapeake<br />Hampton<br />Norfolk<br />Poquoson<br />Portsmouth<br />Williamsburg<br />Gloucester<br />Isle of Wight<br />James City<br />King and Queen<br />King William<br />Mathews<br />Middlesex<br />York<br />
Suffolk FME Server Example<br />Suffolk will log in using OIM<br />They will then find the GIS Parcels, and Water Meters that they maintain<br />They will then run FME Server<br />FME Server will join their GIS data to the water consumption data, and output a zip file for their use<br />
Lessons learned<br /><ul><li>Good first run of the Validation Tool in winter of 2008
Positive: HRSD has control over all changes.</li></li></ul><li>Benefits<br />HRSD has control over all changes<br />HRSD is able to quickly see where jurisdictions are failing to meet the requirements<br />Instead of weeks of manually checks on each feature class, the data can be processed in minutes.<br />Sharing the reports with the jurisdictions<br />Efficient process for keeping data current, and valid.<br />Simple form to fill out. Non-GIS personnel can run the application over the web.<br />Get to come to cool conferences like this one and talk about it!<br />
Questions<br />Questions?<br />For more information:<br />Clay Wise: email@example.com<br />HRSD: www.hrsd.com<br />Data validation: fmepedia.com<br />FME Server > F > FME Server Examples>D > Data Validation with FME Server<br />