LINUX Tag 2008: 4D Data Visualisation and Quality Control
4D Data Visualisation and Quality
Peter Löwe, Jens Klump
Intention of the talk
● Give a better understanding of the relevance of
„preview formats“ for quality control
of complex enironmental data.
● Demonstrate how preview formats can be generated with
FOSS geoinformatics applications.
● Real world example:
– A soil erosion topic, relying on the processing of
– which uses FOSS GIS for this task.
Soil erosion is the process of
soil destruction, a natural process,
which can be initiated or amplified by
human land management. ... Soil
erosion can deminish the agricultural
Soil Erosion: The Problem
In addition to human use of the
Earth's surface, climate is a key
factor. It provides a means of
transportation for soil material to be
How can theseHow can these
processes beprocesses be
modelled ?modelled ?
SOIL EROSIONWind Water
Water, as in ... rainfall
● We need a map of the rainfall distribution (simple task)
– Regular updates
– Sufficient resolution
● The changes of this map can be used to calculate the
„potential erosivity“ of the rainfall for a given area:
– Total amount: how much water comes down in total ?
– Temporal Pattern: Once only, repeated soaking ?
– Small droplets, big droplets ?
Idea and approach
Hypothesis: There are small, temporally fluctuating peaks ofThere are small, temporally fluctuating peaks of
erosiviy due to the convective weather situation. How can theseerosiviy due to the convective weather situation. How can these
erositiy peaks be charted?erositiy peaks be charted?
A sufficiently high temporal (When ?) and spatial data coverage (Where ?),
is needed, and also a measure of confidence for the data values (How
To answer „When - How much - Where“ the radar reflectivity
products must be accessed and processed.
encoding of the
We use ground-based weather radar for a test
– 5 Minutes scan rate (200 km radius, 18 km vertical)
– Pixel/Voxel resolution 1 km
– continous Reflectivity Data (not rain)
18 km Lower Atmosphere
for defined altitudes
Rainfall maps, Pluviogram
X X X
What type of
Where does erosivity potential
When does how much rain fall
3D -> 2D
of „radar rain“
A „virtual rain gauge“ [state
machine] is simulated for each
spatial cell of the radar coverage.
Erosivity values are derived
according to the cell's individual
input data stream.
For this reason, agent technology
is used, as each „gauge-agent“
must keep its own record of
previous precipitation events.
data stream Maps
Tools of the trade
● GRASS GIS
– Raster and volume data processing
– 2D Animations (flip-books)
● NVIZ (part of GRASS)
– 3D visualisation and animation
– 3D visualisation and animation
● GRASS GIS is a Geographic Information
System (GIS) used for geospatial data
management and analysis, image processing,
graphics/maps production, spatial modeling,
● Oldest and largest FOSS GIS project
● GRASS is official project of the Open Source
● „Backend use“ in QuantumGIS, PyWPS,
Erosivity-Totals display of local erosivity pulses
Elevation: Rainfall total
Color: Erosivity total
The model implementation works !
Animated Time Series
Animated displays of reflectivity, derived “radar rainfall” and the
corresponding erosivity peak pattern were created with GRASS
Reflectivity “Radar-Rain” Erosivity
The erosivity ribbons (right) follow the rainfall fields (center): The model works
● 2D animations and 2.5D images show that the
erosivity modelling „works well“
– [the erosivitiy peaks follow the rainfall fields]
● However, the model depends on input data:
● Can we trust the data ?
– Metadata appears correct.
– [are the rainfall fields correct ?]
● Weather Radar provides 3D data.
– [3D->2D transformation: Correctly done?]
● „GRASS in-house“ visualisation tool
● Pro: Works directly on the internal database
● Pro: Scriptable
● Con: small user base, bugs, missing documentation
● Parallel Visualisation Toolkit
● Frontend to VTK + QT
● Large userbase.
● GRASS-related import issues:
– Loosely coupled via file system
– Ascii-VTK-Format – currently not effective for use. Improvements
are hoped for later this year
Garbage in, Garbage out
● Can we trust the rainfall information of the
weather radar ?
● Model results are based on rainfall data.
● Errors and Biases in the rainfall data will affect
all derived products.
● What about transient biases which might
vary in time or space?
Boredom in, Boredom out
● Large data archives exist and more data are
added every day.
● How can we easily identify time intervals when
„some interesting weather“ has occurred?
● We could watch it all in 4D (3D over time):
– Takes too much time, is incredibly boring
– Problem to watch the right things at the right time.
Introducing Preview Formats
● A visual preview format provides a condensed
view with the relevant information for the
current question („determinant“).
● Humans are visualy oriented: preview formats
(shapes/volumes) are easier to comprehend
Transient Phenomena Preview
● What was the weather like
for a 24 h period ?
● Try this: 2D Animation
● Alternative: Stacking of the flip-books pages
(just the ink, really) and look at all pages at
● Howto: Creation of a volume in GRASS GIS,
visualisation by Paraview.
From single drawings
1 2 3
2D Space: Rainfall field
Ce n'est pas un nuage!
● Detailed top-view of the track of a precipitation field (yellow) and the derived erosivity pulses
(red). Note the highly localized distribution of the erosivity pulses. This information can be
used to calibrate the interaction between point-sampling rain gauge networks, weather radar
calibration and soil erosion plots.
Painting of a pipe
(+ „erosivity tracks“)
Quality Control 1.0
A precipitation field and its resulting erosivity
pulses shown in side-view.
The image does not show real world clouds but
precipitation- and erosivity tracks.
„Soaking“ The height of a rainfall track
tells us how long it did rain at
a certain location
The 2D (xy) rainfall field was „squeezed“ out of
the 3D (xyz) weather radar data, implicitly
„collapsing“ the vertical dimension.
The stacking of the time frame „flip-book“ pages
substituted the altitude (z) dimension by the
This approach can be followed further:
● In the previous example we collapsed the z-
● Now we collapse the horizontal (xy)
● The resulting diagram is a preview format
commonly used in meteorology: the
„Contoured Altitude by Frequency Diagram“
Contoured Frequency by Altitude
● CFAD can be created from 3D radar reflectivity data
(original airspace radar scan). The 3D data set is sliced
● A histogram of the reflectivities (1D) is generated for each
● Stacking the histograms gives us a 2D synopsis of the
current situation in the scanned airspace.
● This tells us a lot about the weather and potential
● In our case, this task is done in GRASS.
CFAD – An Example
● Contoured Frequency by Altitude Diagram (CFAD).
Numbers on contour lines give the number of voxels in the
observation area with a given radar reflectivity. The CFAD
gives a snapshot of weather intensity at different altitudes in
the lower atmosphere.
Leafing through the flip-book:
CFATD: One step beyond
● Contoured Frequency Altitude by Time Diagram
adds the time dimension, resulting in a volume
● The shape of the CFATD makes it easy to
● periods of high radar reflectivity, i.e. intense
● Errors in the radar or processing chain.
● Done in GRASS, displayed in Paraview
● Once again we can create a volume from the
resemble levels of
droplet counts (a
few, many, lots)
Critical threshold: If the inner
layer (many droplets) of the
„loaf“ exceeds it, then there is
heavy downpour or even hail.
Visual Quality Control
● CFATD gives a convenient and reliable quality measure
for observations not to use
● If the CFATD structure appears blocky, or „non-organic“:
discard the data
Better data, better models
● 4D previews for „Live Quality Control“ in sensor
– Weather Radar does „now-casting“
● It looks into the distance (right now)
● but not into the future
– Real-time generation of CFATD „loaves“ could be
used for radar system calibration and maintenance.
What level of quality
do we get RIGHT NOW ?
Applications in Grid/eScience
● Dimensional collapse and 3D animation of data
can be used as a preview format for very
● The computing power needed for the
generation of these preview formats can be
sourced from the Grid.
● Complex (4D) data are not easy to interpret.
● Preview formats enable identification of biases
drifting in time and space in complex data.
● Preview formats save time in the selection of
● You can do it, too:
– Ukrainian Radarsystem (MRL-5 + Linux-based
Operating Software (Titan))
– GRASS + Clips + Database + Paraview