Download-manuals-ground water-manual-gw-volume8operationmanualdataprocessingparti

Government of India & Government of The Netherlands
DHV CONSULTANTS &
DELFT HYDRAULICS with
HALCROW, TAHAL, CES,
ORG & JPS
VOLUME 8
DATA PROCESSING AND ANALYSIS
OPERATION MANUAL – PART I
DATA COLLECTION, DATA ENTRY AND DATA
VALIDATION
WATER LEVEL DATA

Operation Manual – Data Processing and Analysis (GW) Volume 8 – Part I
Data Processing and Analysis March 2003 Page i
Table of Contents
1 INTRODUCTION 1-1
2 DATA COLLECTION 2-1
2.1 MANDATORY INFORMATION 2-1
2.2 FIELD FORMS OR FIELD NOTEBOOKS 2-2
2.3 DATA COLLECTION REGISTER 2-3
3 DATA ENTRY 3-1
3.1 DATA ENTRY BY HAND 3-1
3.2 DATA ENTRY FROM FILE 3-1
3.3 DATABASE CONSTRAINTS AND MASTERS 3-2
4 DATA VALIDATION 4-1
4.1 FIELD VALIDATION 4-1
4.2 DATA ENTRY VALIDATION 4-2
4.3 PRIMARY VALIDATION 4-2
4.3.1 GENERAL PRIMARY VALIDATION ACTIVITIES 4-2
4.3.2 PRIMARY VALIDATION OF WELL DATA 4-3
4.3.3 PRIMARY VALIDATION OF GROUNDWATER LEVEL
MEASUREMENTS 4-3
4.3.4 PRIMARY VALIDATION OF GROUNDWATER QUALITY DATA 4-5
4.4 SECONDARY VALIDATION 4-5
4.4.1 GROUNDWATER LEVEL MEASUREMENTS 4-5
4.4.2 GROUNDWATER QUALITY DATA 4-6
4.4.3 BOREHOLE AND WELL DATA 4-7
4.5 TERTIARY VALIDATION 4-7
4.5.1 ADVANCED STATISTICS 4-7
4.5.2 SPATIAL OVERLAYS 4-8
5 DATA PROCESSING 5-1
6 DATA MANAGEMENT AND TRANSFER 6-1

Data Processing and Analysis March 2003 Page 1-1
1 INTRODUCTION
The primary role of the Hydrological Information System is to provide reliable data sets for long-term
planning and design and to frame rules for management of water resource and water use systems
and for research activities in the related aspects. It is also desired that the system will function in such
a manner that it provides the information to users in time and in proper form. The scope of HIS is not
extended to provide data to users on a real-time basis for short-term forecasting or for operational
use.
To be able to provide this information the first step is to obtain the information on the temporal and
spatial characteristics of this object system by having a network of observational stations. The basic
data collected for different hydro-meteorological phenomenon through this observational network is
called the observed or field data. Such observed data have to be processed to ensure its reliability.
Both field and processed data sets have to be properly stored, i.e. processed data for dissemination
and field data to permit inspection and revalidation in response to queries from users.
The activities under HIS are broadly classified in the following categories (Volume 1):
• Assessing the needs of users
• Establishment of an observational network
• Management of historical data
• Data collection
• Data processing, analysis and reporting
• Data exchange and reporting
• Data storage and dissemination
• Institutional and human resource development
The establishment of standard procedures is important to achieve consistency in data collection and
data processing among the participating Centres. Protocols have been designed to ensure that
standard procedures are being followed in the process from data collection to data storage and
dissemination. These protocols are described in Volume 10.
In this Volume 8 the activities dealing with data collection, data entry, data processing, data analysis,
reporting and data exchange will be covered.
Most of the data processing activities are to be accomplished with the help of computers using
dedicated hydrological data processing software. Of particular importance is assuring the quality and
reliability of the data provided to users through the application of a variety of validation procedures
and the flagging of suspect data. The user must be informed of the quality of the data supplied and
whether the values are estimated or observed.
Reports are prepared to bring out the salient characteristics of the hydrological regime of the region
for each year or season. Special reports are also envisaged as and when required for attracting the
attention of the users towards unusual events, major changes in the hydrological regime or to
disseminate important revised long term statistics regularly.
All available data sets are maintained in well-defined computerised databases using an industry-
standard database management system. This is essential for the long-term sustainability of the data
sets in proper form and their dissemination to the end users. Both, field and processed data sets are
properly stored and archived to specified standards so that there is no loss of information. There is
flexibility for data owners to decide user eligibility for data. Once eligibility is decided all agencies
apply standard procedures for the dissemination of data to the users from the computerised
databases.

The type of data stored in the database include:
• Geographical and space oriented data, i.e. static or semi-static data on catchment features and
hydraulic infrastructures
• Location oriented data, including static or semi-static data of the observation stations and
hydraulic structures
• Time oriented data, covering equidistant and non-equidistant time series for all types of
meteorological, climatic, water quantity and quality data, and
• Relation oriented data on two or more variables/parameters used with respect to meteorological,
climatic, water quantity and quality data.
The HIS comprises components (see Figure 1.1) in each State, in the Central Ground Water Board
and in the Central Water Commission. A data transport/communication system provides for data
exchange within and between the states and central organisations.
Figure 1.1: HIS structure at regional and state level

In short, in the HIS of a state the following activities take place at the various levels.
• In the groundwater observation networks water level data and water quality samples are
collected. The water samples are brought to the Water Quality Laboratories. At regular intervals
(monthly/quarterly) the field data are computerised in the District/Division/Regional Data
Processing Centres.
• In the Water Quality Laboratories, besides the analysis of water quality samples, the analysis
results are entered in the computer and subjected to primary validation. At regular intervals, the
laboratory passes the information on to the nearest District/Divisional or Regional Data
Processing Centre.
• In the District Data Processing Centres all field data are entered or incorporated in the
database. Next, primary and secondary validation takes place on the data. The validated data are
passed on to the higher level Data Processing Centre immediately after carrying out regional
validations.
• In the Divisional/Regional Data Processing Centres, given their larger spatial coverage, more
advanced secondary data validation is carried out. The data are stored in temporary databases.
After validation, the ground water and surface water data are transferred to their respective State
Data Processing Centres.
• In the State Data Processing Centres, after reception of the data from its Divisions/Regions,
advanced validations are carried out and later the field data is transferred to the State Data
Storage Centre. The main activity of the State Data Processing Centre is final data validation,
completion, analysis and reporting. Here, the data are stored in temporary databases. At the end
of the hydrological year, once the data has been properly validated, the (authenticated) processed
data is transferred to the State Data Storage Centre. To improve the effectiveness of the final
validation, in the State Centres use is made of the relevant data collected by the Central
Agencies.
• The State Data Storage Centre stores and administers all field and (authenticated) processed
hydrological data collected in the State, and makes the data available to authorised Hydrological
Data Users. As a State archive, it also maintains a HIS-Catalogue of all data stored in its own
database and those stored in the databases of the other states and of the Central Agencies.
In HIS the data processing and the data storage functions are separated; data processing is done in
the Data Processing Centres, whereas the data archives are in the Data Storage Centres. Data
processing and validation is a technical task for which hydrologists/geo-hydrologists are qualified,
whereas final data storage, i.e. the library function, is the domain of database managers. This
distinction is absolutely necessary for a number of reasons:
• Processing and storage are different disciplines, which require different expertise, tools, hardware
support, activities and responsibilities,
• To guarantee discipline in building the database and its sustainability on the long term,
• To make sure that for design and decision making data are being used, which passed all steps of
validation,
• To avoid mixing of fully processed data and the data under processing,
• To register and control receipt and supply of authenticated data to and from the database in a
formalised manner,
• To ensure compatible database configuration and protocols by all agencies,
• To maintain a professional data security system under which each organisation maintains its
independence for user authorisation and data circulation, and
• for an easier upgrading/replacement of either data storage or data processing tools in case of
new developments.

2 DATA COLLECTION
The need for reliable data to be collected and stored in a computerised information system puts
specific requirements to the data collected. The dedicated hydrological data processing software is
designed to store and combine data from different origins and to realise this the data has to have the
right properties and should be complete.
2.1 MANDATORY INFORMATION
The database that is part of the information system, contains tables with mandatory fields, which must
be filled, otherwise the data will be refused. The tables contain the attribute data of the spatial
features, of which data is collected in the field. For example, the properties of a monitoring well or a
rainfall station.
The most important mandatory information connected to spatial data are the coordinates and the
identification:
- the location of spatial data is determined by the coordinates; in case the coordinates of a feature
are not available, such feature can not be presented on a map or used in the information system,
- each object should have an identification to allow querying and locating on a map.
The coordinates of a spatial feature should be collected according the coordinate system used in the
information system. In the dedicated hydrological data processing software geographical coordinates
are being used. All data and map layers should have the same coordinate projection, otherwise such
information can not be combined in the information system. The coordinate projection used in the
dedicated software is the polyconic projection.
The other mandatory information as required by the dedicated software relates to many attributes. In
general, these mandatory attributes are: toposheet no., administrative boundaries, area type,
locational details, name, type, use, project name, agency name, construction or inventory date and
elevation, basin details, irrigation command details and hydrogeological details.
Additionally, there are specific mandatory attributes for the specific hydrogeological structures such as
Wells, Artificial Recharge Structures, Hydromet Stations and Surface Water Sites.
The mandatory information related to measurements (temporal data) is:
Date and time - the date and usually also the time at which a measurement has been taken must be
known;
Reference - the reference point used should be known to allow conversion to the reference used
in the information system;
Method - the measurement method gives an indication of the accuracy and applicability of the
measurement;
Unit - the unit of measurement should be known to allow conversion to the unit used in the
information system.
The mandatory attributes of the temporal data ensure that time related measurements can be
processed for presentation or analysis.

2.2 FIELD FORMS OR FIELD NOTEBOOKS
To ensure that attribute data is properly collected, field forms or field notebooks should be available
on which the information is registered. The forms or notebooks need to be designed appropriately to
ensure that all data required is being collected. Preferably the forms or notebooks should use the
same units and references as required by the information system.
Various types of field forms are used in India. An example based on the data entry screens of the
dedicated software is presented below.
Additional information collected in the field, such as remarks on the conditions of the monitoring well
or on the conditions of the environment of the measuring point, will help in understanding the
conditions at the time of the measurement. The field form or notebooks should provide space to
enable the registration of such information. Preferably such information is entered in the information
system.
After data entry, field forms and field notebooks are kept in the archive. The availability of the field
forms for checking at a later date is important in case it is not possible to enter this information.
Well Inventory Field Form (Example)
Location
Administrative boundary Geographic location
x
x
x
x
x
State ………………………….
District ………………………….
Tahsil/Taluka ………………………….
Block/Mandal ………………………….
Village/Town ………………………….
x
x
x
Deg Min Sec Dir
Latitude ….. ….. ….. …..
Longitude ….. ….. ….. …..
Topo sheet no. ……………..
Other details
x
x
Area type …………………………………………………………………………
Locational details …………………………………………………………………………
Access details …………………………………………………………………………
Etc.
General
Well details
x Well identification Old well identification x Well name
…………………………… ..……………………… ………………………
x Well type x Well sub type x Well use
…………………………… ..……………………… ………………………
Well conditions
x
x
x
x
x
Type of Lifting Device …………………….. Installation Date …...……………..
Agency name …………………….. Horse Power (HP) ...………………..
Activity/Project Name Activity/Project Details
……………………………………………… ……………………………………….
…… Is water potable? x Inventory Date ………………..
…… Is well abandoned? Abandoned Date ………………..

x Status of well ……………………………………………………………………..
Topography
x Height of Measuring Point (magl) ……………………..
Surface Elevation of Ground level (m amsl) ……………………..
Bench Mark Elevation (m amsl) Location Distance (m) Direction (deg)
details: …………. ………………. …………. ………….
Survey Method ……………………………………………………………..
x Mandatory information
2.3 DATA COLLECTION REGISTER
The data collection activities, such as the field visits, will have to be recorded in a data collection
register to keep records of these activities and to provide information on the circumstances in which
the activities took place. The records enable to review the data collection activities whenever
background information on the data is required. A possible lay-out for a data collection register is
shown below.
Data Collection Register (Example)
EndorsedDate Name of the
Officer
Type of Data
Collection
Number of
Stations
Visited
Remarks on Conditions,
Problems Encountered,
etc. Name Date
The type of data collection may be water level measurements, water quality sampling, pump testing or
any other field activity in which data collection is involved. The remarks on the conditions during the
data collection activities are important, as these give the desired background information. A superior
should endorse every field visit registry entry to ensure that any relevant remarks have been noted.

3 DATA ENTRY
3.1 DATA ENTRY BY HAND
Data entry into the information system may be distinguished in data entry by hand and data entry by
importing data from files. Data entry by hand relates mainly to the data of spatial features, such as
wells or artificial recharge structures and to temporal data collected by hand, such as water level
measurements or water quality analysis. This type of data is usually available from field forms or from
field notebooks. To reduce the risk of errors it is advisable to design the forms or the notebooks
according the lay-out of the data-entry screens of the dedicated software. In the case of water quality
data it is important to have data entry screens with a similar lay-out as the water quality analysis
forms.
The entry of temporal data by hand or from file can be done only when all the basic information of the
observation structure is available in the information system. Especially the information related to the
measuring point and the current condition of the structure is important.
The entry of the collected data into the database should be done directly from the original field forms.
No copying of data should be done on separate data entry forms, because copying by hand may
introduce errors.
Entry of water quality data is discussed in Part II of the Operation Manual.
3.2 DATA ENTRY FROM FILE
Data entry by importing data from files relates usually to temporal data collected with an automatic
device, such as water level measurements from a Digital Water Level Recorder (DWLR). This type of
data often has to be converted or referenced with the software supplied with the DWLR before it can
be entered into the information system.
The entry of digital data into the information system requires special attention. Groundwater levels
collected with a DWLR must be accompanied with check data and must be converted once being
entered into the information system. The flagging of such information is important to indicate whether
conversion has taken place.
The DWLR-data should be offered in the right format, with the correct header and columns, as
described by the dedicated software. After import, the information system should generate a report of
the import, with the total number of records offered, the number of records imported and the number
of records rejected. The import-report gives the user an overview of the results of the import. The
checking of the imported data should be done with the standard functionalities of the information
system.
Also other data, such as quality data, must be made available in the right format in order to use the
data entry facilities of the info system.

3.3 DATABASE CONSTRAINTS AND MASTERS
In the information system constraints on the tables ensure that mandatory attributes are being entered
of hydrological features. For example, the coordinates of a bore well must be entered. If not, the data
of the bore well, describing its properties, can not be entered in the database.
Fixed data attributes are organised in masters, which specify the values that may be entered in
certain data fields. For example, for a well type, the information system allows a choice between: Bore
Well, Dug Cum Bore Well, Dug Well, Slim Hole, Spring, or Tube Well. An ordinary user will not be
allowed to edit the masters, only a database manager will be allowed to update these masters.
Generally, the masters may only be edited at the national level. Masters, with relevance to a state or
districts may only be edited at the lower level.
Numeric data attributes are constricted to realistic values by specifying validations in the information
system. For example, the diameter of a bore well is normally between 0.10 and 0.60 metre. A
hydrogeological information system will have the possibility to specify such restrictions. In the
information system the permissible values are specified through the validation functionality.

4 DATA VALIDATION
Once the data is being entered in the information system further checking of the data can be carried
out. The information system provides the functionalities to carry this out. The checking of the data can
be done on different levels with increasing complexity.
The different levels of data validation are according to the protocols:
(a) field validation;
(b) data entry validation;
(c) primary validation (first checks);
(d) secondary validations (integration checks);
(e) tertiary validations (advanced statistics).
The validation of the data does not have to be restricted to the functionalities offered by the
information system. Data may be exported from the information system and imported in dedicated
software to process the data to carry out checks or other analyses.
The results of the validation should be recorded in the database. This means that data are assigned a
code, a flag or a quality label when data validation is completed and the data is judged to be correct.
This could be done for static information, such as the properties of the hydrogeological structures, and
for the individual temporal data, such as the groundwater level measurements and groundwater
quality analysis data.
The validation code or validation flag indicates the level of validation, which the data has undergone.
The code or flag is updated by a data manager for the data, which successfully has passed certain
validation tests. The quality codes and flags are particularly useful for time-dependent data, such as
groundwater levels or water quality analysis data.
The quality label is generally intended to provide more information about the related data. Usually,
quality labels also give information on the method with which the data is collected and the method of
interpretation used. The quality label is coded and enables a quick overview of the properties of the
related data. For example, a quality label of borehole information may give information on the drilling
method, the geological interpretation of the drilling samples and whether analysis of the samples has
been carried out in the laboratory. Or a quality label of a well may give information on the date and
level of the last validation, the depth of the screen, whether a DWLR is present or whether the
elevation of the measuring point is known.
The use of flags or quality labels helps identifying the information in the database according the level
of validation reached. With this quality information it is easy to make selections of data, which has
undergone the required validations. Quality codes, flags or labels are not used at present in the
dedicated software.
4.1 FIELD VALIDATION
The field validation is to ensure that all basic information relating to the location and properties of the
hydrogeological structure is correct. The validation may yield correct and complete information on the
structure when using comprehensive field forms, which are designed to ensure that all data required
is being collected. The field validations are carried out by visiting the hydrogeological structures and
should at least include verification of coordinates, topo sheet, altitude, constructional details, local
geology and geomorphology.

In the case of observation wells the position and the height of the measuring point are critical,
because these determine the measured depth to the groundwater level. Any change in the conditions
of a measuring point should be recorded and entered into the database. Also, the depth of the well is
to be entered in the database, whenever a change in depth is noticed. Both measuring point height
and well depth are time dependent variables, to ensure that continuous time series of water level
measurements are stored in the database.
4.2 DATA ENTRY VALIDATION
The data entry validation is realised through the use of mandatory fields in the data entry sheets.
These fields ensure that all required data is entered into the system. It will not be possible to enter the
information into the information system in case not all data is available.
After the entry of time dependent data, checks should be carried out to ensure that no duplicates are
present in the database. For example, groundwater levels should not have the same date and time,
because it is not possible to take two measurements at the same time. Also groundwater quality
analysis data should not have the same sample date, because usually only one sample is taken at a
site on a day.
When dealing with large quantities of data specialised personnel is to execute the data entry. After
data entry the data must be printed and compared with the field forms, to check for errors.
4.3 PRIMARY VALIDATION
The primary validation uses the data selection and presentation techniques and some locational
check-functionalities offered by the dedicated software:
4.3.1 GENERAL PRIMARY VALIDATION ACTIVITIES
A general visual inspection of the entered data may be done on print-outs of the data made with the
dedicated software. The inspection should concentrate on a comparison of the field forms with the
print-outs. The inspection should try to discover errors in the spelling of names and the magnitude of
numerical values. Examples of print-outs to be used:
• lithological logs on a graph
• chemical data on a water quality report
• water level data on time series graphs
It is important for the visual inspection that reports or graphs are printed which contain all data
relevant to the validated feature. In the case of the graphs it is also important that the scale of the
axes can be chosen freely in order to prepare graphs with sufficient detail.
Another method for checking the data entry is by entering the data twice in different sessions in
separate data entry tables in the database. Both sets are compared automatically and only data,
which corresponds in both tables, is uploaded into the information system. Data which does not
correspond with data in the other data entry table is rejected and presented in a report for
checking and renewed data entry. This feature is not available in the dedicated software

The geographical locational properties of all sites should be validated by the constraints in the
database related to the coordinates. Alternatively, the locations of the wells or other structures may be
checked by plotting the locations on a map of the area of interest.
The administrative locational properties may be validated with the dedicated software by combining
the entered attribute data with map layers containing for example, information on administrative
boundaries, land use, soil or geology. This validation should be done with care, because correction of
the locational properties is only possible by hand through the modify-functions of the dedicated
software, once the locational properties have been updated.
It is also very important, that the attributes of the map layers are based on the same master
information as the database. If not, correction of the data will give unwanted results. For example, in
the case of administrative names, it is important to use the same spelling in the map layers as in the
database.
4.3.2 PRIMARY VALIDATION OF WELL DATA
The primary validation of the well data should include checks on the specific details of the wells.
Some of these checks may be done by plotting and selecting the well locations on a map, for example
for the validation of well type, well use, lifting device, installation date of the lifting device, well
abandonment, the height of the measuring point, the surface elevation of the well, etc.
Other checks, which may be carried out directly on the well data, are for example:
- General checks:
• Start Date not after End Data
• Start Date and End Date not in future
• Diameter blank casing less than a specified value (e.g. 6000 mm.)
• Diameter slotted casing less than a specified value (e.g. 1200 mm.)
- Drilling details:
• Drilled Depth more than or equal to Construction Depth
- Aquifer Performance Test, Step Drawdown Test, Preliminary Yield Test:
• Static Water Level less than Construction Depth
• Dynamic Water Level (= Static Water Level – Drawdown) less than Construction Depth
4.3.3 PRIMARY VALIDATION OF GROUNDWATER LEVEL MEASUREMENTS
Erroneous groundwater level measurements may be found by carrying out specific selections:
• all groundwater levels which are below the bottom of the well
• all groundwater levels which are above the top of the well plus 0.25 meter
or by visual inspection of the data, looking for:
• errors due to a wrong reading (mistakes in readings by 0.5 or 1.0 meter)
• errors due to a change in the height of the measuring point
• errors due to assigning the groundwater level measurement to the wrong well or filter.

Measurements
to be checked
12AMAD196
0
2
4
6
8
10
12
14
01/Jan/1998
02/Apr/1998
02/Jul/1998
01/Oct/1998
01/Jan/1999
02/Apr/1999
02/Jul/1999
02/Oct/1999
01/Jan/2000
01/Apr/2000
02/Jul/2000
01/Oct/2000
31/Dec/2000
01/Apr/2001
02/Jul/2001
01/Oct/2001
31/Dec/2001
Waterlevel(m)
12MULA10
4
5
6
7
8
9
10
01-Jan-1998
02-Apr-1998
02-Jul-1998
01-Oct-1998
01-Jan-1999
02-Apr-1999
02-Jul-1999
02-Oct-1999
01-Jan-2000
01-Apr-2000
02-Jul-2000
01-Oct-2000
31-Dec-2000
01-Apr-2001
02-Jul-2001
01-Oct-2001
31-Dec-2001
Waterlevel(m)
Figure 4.1: Comparison of hydrographs
The visual inspection usually identifies suspect groundwater levels which can be confirmed to be
erroneous only by comparing with other groundwater levels measured in the same area at the same
time.
An example is given in Figure 4.1, where two ‘doubtful’ measurements are indicated in well
12AMAD196.
The correction of identified doubtful measurements is not straightforward. The problem always is:
What was the correct value? The correction of measurements therefore should only be done in case
the correct value can be determined without any doubt. In all other cases the value should be coded
or flagged with the status: doubtful value.
Another example of an erroneous value is given in Figure 4.2. In this hydrograph the well has been
dry on a number of times in 1998 and 1999 and the groundwater level has been entered with the
value of the depth of the well. This is not the correct groundwater level because the actual
groundwater level will have been below the bottom of the well. How much below is not known,
therefore correction of the groundwater level is not possible. The solution here is to add a flag to the
groundwater level measurement indicating that the well was dry.

Figure 4.2: Hydrograph of well, which occasionally falls dry
4.3.4 PRIMARY VALIDATION OF GROUNDWATER QUALITY DATA
Primary validation of groundwater quality data is discussed in Part II of this Operation Manual.
4.4 SECONDARY VALIDATION
The secondary validation uses the data processing properties of the information system. The use of
these techniques indirectly will indicate deviating values. However the results of the validation should
be judged with knowledge of the hydrogeological conditions of the area to determine whether the
selected values are errors or natural extremes.
4.4.1 GROUNDWATER LEVEL MEASUREMENTS
1. Preparing contour maps of groundwater level data for a certain period or date may show deviating
values. These values will be shown by a high concentration of contours around the location of the
well with the erroneous value. Checking of the hydrograph of the well is necessary to determine
whether the value really is in error (see also Volume 8, Part III, Section 4.6).
2. Preparing time series graphs with multiple hydrographs and the visual inspection of the graphs
may show erroneous measurements. Doubtful measurements will show in a hydrograph if
combined with other hydrographs from wells, which represent similar hydrogeological conditions
(such graphs may also be on separate print-outs). Comparing hydrographs would normally show
similar trends and fluctuations and if not, may indicate erroneous groundwater level
measurements (see also Section 4.3.3 and Volume 8, Part III, Section 4.6.3).
3. Some simple statistical methods are available to check groundwater level measurements. These
methods may also be applied on the data through special dedicated software from outside the
information system.
3.1 Deviation from the mean or the median: check for values, which differ more than three times
the standard deviation from the mean or median. The median is less affected by values of
individual measurements and represents better the centre of the data set. Therefore the
median is preferred to the mean.
12-75VEER06
0
2
4
6
8
10
12
14
01-Jan-1998
02-Apr-1998
02-Jul-1998
01-Oct-1998
01-Jan-1999
02-Apr-1999
02-Jul-1999
02-Oct-1999
01-Jan-2000
01-Apr-2000
02-Jul-2000
01-Oct-2000
31-Dec-2000
01-Apr-2001
02-Jul-2001
01-Oct-2001
31-Dec-200
Waterlevel(m)
well bottom
actual groundwater level

Figure 4.3: Application of statistical tests for data validation
The first step is to calculate the median and the standard deviation. Preferably the time series
contains at least 50 measurements.
In Figure 4.3 all measurements fall within the range of the median +/- 3 * standard median.
One measurement is above the median + 2 * standard deviation, but it passes test, as it is
below the median + 3 * standard deviation.
The example also indicates that in case of a trend the mean may not be the correct reference.
In case of hydrographs with a trend to identify outliers the validation should be done using the
trend line +/- 3 * standard deviation.
The above may also be done for a specific part of a year or date in case that a sufficiently
long period of measurements is available (for example more than 10 years).
3.2 Frequency analysis: calculate the frequency of the measurements in order to identify
values, which appear more often then normally expected. Such values may result from
measurements affected by a casing joint or from overflowing or siltation of the well. The
consistent rounding of measurements to a near value also may cause such values.
4.4.2 GROUNDWATER QUALITY DATA
Preparing water quality diagrams, such as a Piper-diagram or a Stiff-diagram, may show deviating
values.
In a Piper-diagram, outlying points and in a Stiff-diagram, deviating shapes may indicate extreme
water quality conditions. However such results may also indicate erroneous values for the constituting
components, for example in the case that the concentration of a component was determined by
closing the ion balance with a residual value.
median
median + 2 * sdev
median – 2 * sdev
median – 3 * sdev
median + 3 * sdev12AMAD1960
2
4
6
8
10
12
14
16
18
20
0...
... 0...
... 0...
... 0...
... 0...
... 0...
... ... ... 0...
... ...
Waterlevel(m)
trendline

Figure 4.4:
Piper–diagram showing outliers
4.4.3 BOREHOLE AND WELL DATA
Preparing cross-sections and fence diagrams of hydrogeological conditions may show deviating data.
The cross-sections and fence diagrams are composed from the basic information obtained from
lithological logs or groundwater quality data and these will show any deviating data. Hydrogeological
knowledge of the area is necessary to determine whether such data is erroneous.
The primary and secondary validations use some of the standard functionalities of the information
system, which may be used directly. Any erroneous values found should be recorded and corrected
manually through the standard data editing functionalities of the information system.
4.5 TERTIARY VALIDATION
The tertiary validation involves advanced techniques for the analysis and validation of spatial and
temporal data. GEMS does not offer all these techniques. The advanced data processing techniques
include:
1. advanced statistics, and
2. spatial overlays
The use of the technologies of the tertiary validation requires knowledge on the tools used and will
have to be carried out by appropriately trained staff. The results of the validation are obtained through
interpretation and will have to be studied to judge, whether errors can be detected.
4.5.1 ADVANCED STATISTICS
Especially in the case of the advanced statistics it is very complicated to determine whether detected
outliers are natural extremes or erroneous outliers. Therefore, it can not be expected that such
methods yield results, which can easily be interpreted. Other evidence will be necessary to decide

whether the detected values are permissible. Too strict conditions often result in a large number of
values, which need to be checked. Too lenient conditions often result in a small number of outliers of
which the majority turns out to be a natural extreme. The time spent on this type of validation
therefore will often be in excess in relation to the results obtained.
The advanced statistics dealing with the identification of outliers involves the processing of the
groundwater level time series for trend detection and the decomposition in components. Many
parametric and non parametric statistical trend detection techniques have been developed for time
series to provide the most likely estimate of the water level changes over time and the corresponding
confidence interval. The determination of the outliers involves the comparison of these estimates with
the measured levels. Helsel and Hirsch (1995) describe a number of trend tests for use in water
resources. A drawback of regression analysis is that it makes fairly strong assumptions about the
distribution of the variable over time. Alternatively non parametric procedures can be used. A very
robust and a simple non parametric trend test is the Mann-Kendall test.
The identification of outliers using the simple Q-test is already discussed in Volume 7, Chapter 8.2.5.
The reader is referred to the approaches described there.
The identification of outliers by correlating time series for different wells is described in Volume 8,
Part II, Section 4.6.3.
4.5.2 SPATIAL OVERLAYS
The validation by spatial overlays involves the confrontation of basic data with derived data in the form
of maps. The basic data should conform to the maps. If not either the basic data or the map is in error.
An example of a validation by spatial overlay is the combination of the results of aquifer performance
tests with a map with hydrogeological units. In general the values of the transmissivity and the storage
coefficient derived from the pumping test should conform to the values expected for the
hydrogeological units displayed on the map.

5 DATA PROCESSING
Once the data is entered and validated, it may be used for analysis or presentation purposes. The
analysis will make it possible to use the data for example for the determination of groundwater
availability or groundwater contamination. The presentation will enable to prepare maps and graphs
for yearbooks or other publications describing the state of the groundwater resources.
More information on data processing is included in Part II of this Volume.

6 DATA MANAGEMENT AND TRANSFER
The transfer of data from the Data Processing Centres to the Data Storage Centres is extensively
dealt with in other parts of the manual and therefore is not described here. Only a few principles are
set out below as a reminder:
1. Make export of the relevant part of the database into an export file
2. Send the export file (only validated data, data to carry validation certification)
3. Make preliminary import (best into a temporary database)
• Check differences between old and new data for hydrogeological structures
• Check overlap between old and new data for temporal data
• Notify if any problems exist with the imported data
4. Make final import
5. Send message when import has succeeded or failed.
All data activities are to be registered in a registry. The registry will enable to review all data entry and
validation activities and give an overview of all export, import and data correction activities. A
possible simple lay-out for a data activity register is shown below. A more detailed register is of
course possible, according the requirements laid out for the data management registration.
Data Activity Register (Example)
EndorsedDate Name of the Officer Type of Data Activity Description of the Activity
Name Date
The type of data activity may be data entry manual, data entry by import, primary data validation by
inspection of hydrographs, data export of well site data, etc. The description of the activity should give
ample information on the contents of the activity, as these give the desired background information. A
superior should endorse data activity registry entry to ensure that all relevant remarks have been
noted.
References:
UN/ECE Task Force Monitoring & Assessment, 1999. State of the art on monitoring and
assessment of groundwaters
Authors: H. Uil, F.C. van Geer, J.C. Gehrels, F.H. Kloosterman
Netherlands Institute of Applied Geoscience TNO, Delft, The Netherlands.
Helsel, D.R. and Hirsch, R.M., 1995. Statistical methods in water resources. Studies in
Environmental Science 49. USGS, Elsevier, page 529.

Download-manuals-ground water-manual-gw-volume8operationmanualdataprocessingparti

Recommended

Recommended

More Related Content

What's hot

What's hot (17)

Viewers also liked

Viewers also liked (7)

Similar to Download-manuals-ground water-manual-gw-volume8operationmanualdataprocessingparti

Similar to Download-manuals-ground water-manual-gw-volume8operationmanualdataprocessingparti (20)

More from hydrologyproject001

More from hydrologyproject001 (20)

Recently uploaded

Recently uploaded (20)

Download-manuals-ground water-manual-gw-volume8operationmanualdataprocessingparti