Applying Electronic Data Checking Tools To Environmental Data
Applying Electronic Data Checking Tools to Environmental Data
Thursday, April 15, 2004 9:30 AM—10:00 AM
23rd Annual National Conference on Managing Environmental Quality Systems
Tampa Marriott Waterside Hotel and Marina, Tampa, Florida
The rate at which environmental data is being generated today and the importance of
managing that data in real time with reduced resources is driving a renewed look by all
participants in the value of using electronic tools in the collection of environmental data.
This presentation will introduce the concepts of electronic data download (EDD) file
definitions, data checking, conditional processing, and automated data transmittals. A leading
edge data PC processing tool will be introduced and opportunity for hands on experience will be
provided. Further automation of the data path for data importing and subsequent actions on
receipt such as action level criteria checking; push reporting; and automated emails will also be
explored. A new generation data verification tool will also be presented that assists data
validators in finding the “needle in the haystack” errors that require attention to ensure the data
received meets the project’s intended data quality objectives.
Additional opportunity to review the products and procedures presented will be afforded
following the presentation, or upon request at www.earthsoft.com.
Key Concepts Presented:
Importance of Data Quality
o First address Data Quality, then Data Usability
o Data Usability without Data Quality is ‘Garbage In - Garbage Out’.
o Bad data can make beautiful graphs! But, Good data leads to Good Decisions.
o Variations in formats
o Inconsistent values
o Missing content
o Errors in content
o Verification at end of the process leaves little ability to make corrections – TQM?
o Much Manual effort required for success
o Define data formats for each data type
o Establish consistent valid values
o Check EDDs for compliance and report errors for correction – at generation
o Verify EDD at receipt
o Define data package for automated processing
EDP – for Data Submitters
o A .Net product, using XML and easily deployed.
o Data producers can check and correct EDDs.
o Custom Checking scripts may be included with formats – special checking can be
Required fields and Reference Values
Conditional checking (if this field is X, then that field should be Y)
Other special or custom required data reviews QC recovery checking
Check multiple EDD format tables simultaneously
EQuIS Data Processor (EDP) - Additional Workflow Values
o Allows end user to use XML to create new EDDs.
o User Login/identification, and Audit trail capable.
o Generate error report including row number, error, and value.
o Screen data against required limits, with different regulatory limits per location.
o Upload EDD from user, via email, ftp, or portal.
o Web Enabled to automate EDD processing for high volume operations.
o Optionally upload data into defined database tables.
Data Entry - DATA ENTRY FORMS for manual data entry are useful for historical data
conversions and for field data entry.
o This XML tool for Laptops or Tablet PCs enables a controlled dialog to create the
import EDD files in a familiar layout for field staff.
o Labs and Data Providers can submit data online or via email.
o Pass or Fail results (with Error Logs, if any) are emailed back to data provider.
o Automation By Email or Online
Auto process Package
Maintain Audit Trail
Email EDD messages
Load accepted data
Initiate Push processing
DQM – Data Qualification Module
o Automate the process of Data Verification Checks.
o Reduce staff time checking good data.
o Find the Needle-in-a-haystack errors.
o Focus experts on resolving problems.
o Enable higher percentage checks – up to 100% for key data items.
o Review Data by SDG
Set Up Criteria by Method/Matrix
Enter Method and Matrix Specific Holding Times, Qualifiers, etc…
Select Checks and Qualifier Rules
Run Checks, Review Error Log
Commit Results to the Database
Generate Report Tables
DQM Benefits for Data Verification
o Produce better quality and more reliable data.
o Implement discovery and correction procedures at data submittal – TQM.
o Enable checking of a higher percentage of data.
o Enable automation of routine processes.
o Better, Faster, Cheaper overall process.