LINUX Tag 2008: 4D Data Visualisation and Quality Control


Published on

LINUX Tag 2008 presentation by Peter Löwe and Jens Klump.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

LINUX Tag 2008: 4D Data Visualisation and Quality Control

  1. 1. 4D Data Visualisation and Quality Control Peter Löwe, Jens Klump GeoForschungsZentrum Potsdam,
  2. 2. Intention of the talk ● Give a better understanding of the relevance of „preview formats“ for quality control of complex enironmental data. ● Demonstrate how preview formats can be generated with FOSS geoinformatics applications. ● Real world example: – A soil erosion topic, relying on the processing of „complex data“ – which uses FOSS GIS for this task.
  3. 3. •Topic, theory •Hypothesis and approach •Technical frameworkTechnical framework •Modelling •Results / Lessons learnedResults / Lessons learned Overview: Soil Erosion Project
  4. 4. Soil erosion is the process of soil destruction, a natural process, which can be initiated or amplified by human land management. ... Soil erosion can deminish the agricultural yield significantly. Soil Erosion: The Problem In addition to human use of the Earth's surface, climate is a key factor. It provides a means of transportation for soil material to be carried away. How can theseHow can these processes beprocesses be modelled ?modelled ? Climate Terrain Soils Vegetation Humans SOIL EROSIONWind Water Transportation Control Enhance Start
  5. 5. Water, as in ... rainfall ● We need a map of the rainfall distribution (simple task) – Regular updates – Sufficient resolution – Reliable ● The changes of this map can be used to calculate the „potential erosivity“ of the rainfall for a given area: – Total amount: how much water comes down in total ? – Temporal Pattern: Once only, repeated soaking ? – Small droplets, big droplets ?
  6. 6. Idea and approach Hypothesis: There are small, temporally fluctuating peaks ofThere are small, temporally fluctuating peaks of erosiviy due to the convective weather situation. How can theseerosiviy due to the convective weather situation. How can these erositiy peaks be charted?erositiy peaks be charted? A sufficiently high temporal (When ?) and spatial data coverage (Where ?), is needed, and also a measure of confidence for the data values (How much ?). To answer „When - How much - Where“ the radar reflectivity products must be accessed and processed. ModellingTechnical Framework Results Analysis and encoding of the Erosivity. Geoinformatics, Information- logistics, Remote sensing, Radarmeteorology •Verification •Validation •Results
  7. 7. Practical Approach We use ground-based weather radar for a test site: – 5 Minutes scan rate (200 km radius, 18 km vertical) – Pixel/Voxel resolution 1 km – continous Reflectivity Data (not rain) 400 km 18 km Lower Atmosphere
  8. 8. X X X X X a b c d Erosivity maps Reflectivity GIS-layers for defined altitudes Rainfall maps, Pluviogram X XXX X X X X 1 4 2 Erosivity- Model What type of weather occurs when, where? Where does erosivity potential occur ? When does how much rain fall where ? - Data flow 3D -> 2D transformation 18km altitude 1km altitude
  9. 9. Precipitation- data stream of „radar rain“ Init State 1 Hibernate State 2 Store State 3 Pause P: Precipitation D: Dry D D D P P P A „virtual rain gauge“ [state machine] is simulated for each spatial cell of the radar coverage. Erosivity values are derived according to the cell's individual input data stream. For this reason, agent technology is used, as each „gauge-agent“ must keep its own record of previous precipitation events. Erosivity-Modelling Erosivity- data stream Maps Index- value Cell- Agent
  10. 10. Tools of the trade ● GRASS GIS – Raster and volume data processing – 2D Animations (flip-books) ● NVIZ (part of GRASS) – 3D visualisation and animation ● Paraview – 3D visualisation and animation
  11. 11. GRASS GIS ● GRASS GIS is a Geographic Information System (GIS) used for geospatial data management and analysis, image processing, graphics/maps production, spatial modeling, and visualization. ● Oldest and largest FOSS GIS project ● GRASS is official project of the Open Source Geospatial Foundation. ● Scriptable ● „Backend use“ in QuantumGIS, PyWPS, JGRASS
  12. 12. Calculating Totals 24h total erosive 16:18:50 Hours 16:43:30 Hours 16:59:56 Hours Erosivity Reflectivity Σ Σ Left: Reflectivity Centre: Rainfall Right: Erosivity
  13. 13. Erosivity-Totals display of local erosivity pulses Elevation: Rainfall total Color: Erosivity total Conclusion: The model implementation works !
  14. 14. Animated Time Series Animated displays of reflectivity, derived “radar rainfall” and the corresponding erosivity peak pattern were created with GRASS GIS. Reflectivity “Radar-Rain” Erosivity The erosivity ribbons (right) follow the rainfall fields (center): The model works
  15. 15. The Challenge ● 2D animations and 2.5D images show that the erosivity modelling „works well“ – [the erosivitiy peaks follow the rainfall fields] ● However, the model depends on input data: ● Can we trust the data ? – Metadata appears correct. – [are the rainfall fields correct ?] ● Weather Radar provides 3D data. – [3D->2D transformation: Correctly done?]
  16. 16. 3D: Straightforward Approach NVIZ-Animation (GRASS): „Real Clouds“ Volume [Full Information]= Reflectivity Values (30/40/50 dBz) 2.5D Surface: MaxReflecivity(Color), Radar horizon (Elevation) Flattening
  17. 17. NVIZ ● „GRASS in-house“ visualisation tool ● Pro: Works directly on the internal database ● Pro: Scriptable ● Con: small user base, bugs, missing documentation ●
  18. 18. ● Parallel Visualisation Toolkit ● Frontend to VTK + QT ● Large userbase. ● GRASS-related import issues: – Loosely coupled via file system – Ascii-VTK-Format – currently not effective for use. Improvements are hoped for later this year ●
  19. 19. Garbage in, Garbage out ● Can we trust the rainfall information of the weather radar ? ● Model results are based on rainfall data. ● Errors and Biases in the rainfall data will affect all derived products. ● What about transient biases which might vary in time or space? Flattening Trust Trust 3D data 2D information
  20. 20. Boredom in, Boredom out ● Large data archives exist and more data are added every day. ● How can we easily identify time intervals when „some interesting weather“ has occurred? ● We could watch it all in 4D (3D over time): – Takes too much time, is incredibly boring – Problem to watch the right things at the right time.
  21. 21. Introducing Preview Formats ● A visual preview format provides a condensed view with the relevant information for the current question („determinant“). ● Humans are visualy oriented: preview formats (shapes/volumes) are easier to comprehend than numbers.
  22. 22. Transient Phenomena Preview ● What was the weather like for a 24 h period ? ● Try this: 2D Animation (Flip-Book) ● Alternative: Stacking of the flip-books pages (just the ink, really) and look at all pages at once. ● Howto: Creation of a volume in GRASS GIS, visualisation by Paraview.
  23. 23. From single drawings 1 2 3 „Radar Rain“ Erosivity
  24. 24. Flip-book Volume time 1 2D Space: Rainfall field 2 3 Yellow: Rainfall Red: Erosivity Data Errors (ground targets) Rainfall field Rainfall field Erosivity Peaks Not real clouds ! 1 2 3
  25. 25. Ce n'est pas un nuage! ● Detailed top-view of the track of a precipitation field (yellow) and the derived erosivity pulses (red). Note the highly localized distribution of the erosivity pulses. This information can be used to calibrate the interaction between point-sampling rain gauge networks, weather radar calibration and soil erosion plots. Painting of a pipe Rainfall tracks of clouds (+ „erosivity tracks“)
  26. 26. Quality Control 1.0 A precipitation field and its resulting erosivity pulses shown in side-view. The image does not show real world clouds but precipitation- and erosivity tracks. „Soaking“ The height of a rainfall track tells us how long it did rain at a certain location Rods of eternal soaking: Data errors
  27. 27. Dimensional Collapse The 2D (xy) rainfall field was „squeezed“ out of the 3D (xyz) weather radar data, implicitly „collapsing“ the vertical dimension. The stacking of the time frame „flip-book“ pages substituted the altitude (z) dimension by the time dimension.
  28. 28. Collapse 2.0 This approach can be followed further: ● In the previous example we collapsed the z- dimension ● Now we collapse the horizontal (xy) dimension. ● The resulting diagram is a preview format commonly used in meteorology: the „Contoured Altitude by Frequency Diagram“ (CFAD).
  29. 29. Contoured Frequency by Altitude Diagrams (CFAD) ● CFAD can be created from 3D radar reflectivity data (original airspace radar scan). The 3D data set is sliced vertically. ● A histogram of the reflectivities (1D) is generated for each slice/layer. ● Stacking the histograms gives us a 2D synopsis of the current situation in the scanned airspace. ● This tells us a lot about the weather and potential measurement errors. ● In our case, this task is done in GRASS.
  30. 30. CFAD – An Example ● Contoured Frequency by Altitude Diagram (CFAD). Numbers on contour lines give the number of voxels in the observation area with a given radar reflectivity. The CFAD gives a snapshot of weather intensity at different altitudes in the lower atmosphere.
  31. 31. 2D Animation Leafing through the flip-book: Weather CFAD
  32. 32. CFATD: One step beyond ● Contoured Frequency Altitude by Time Diagram adds the time dimension, resulting in a volume body. ● The shape of the CFATD makes it easy to identify: ● periods of high radar reflectivity, i.e. intense weather, and ● Errors in the radar or processing chain. ● Done in GRASS, displayed in Paraview
  33. 33. Flip-book volume ● Once again we can create a volume from the flipbook. Time Altitude „Droplet Size“ Iso Surfaces resemble levels of droplet counts (a few, many, lots) Critical threshold: If the inner layer (many droplets) of the „loaf“ exceeds it, then there is heavy downpour or even hail.
  34. 34. Visual Quality Control ● CFATD gives a convenient and reliable quality measure for observations not to use ● If the CFATD structure appears blocky, or „non-organic“: discard the data Faulty data Faulty data
  35. 35. Better data, better models ● 4D previews for „Live Quality Control“ in sensor systems: – Weather Radar does „now-casting“ ● It looks into the distance (right now) ● but not into the future – Real-time generation of CFATD „loaves“ could be used for radar system calibration and maintenance. What level of quality do we get RIGHT NOW ?
  36. 36. Applications in Grid/eScience ● Dimensional collapse and 3D animation of data can be used as a preview format for very large/complex datasets. ● The computing power needed for the generation of these preview formats can be sourced from the Grid.
  37. 37. Conclusion ● Complex (4D) data are not easy to interpret. ● Preview formats enable identification of biases drifting in time and space in complex data. ● Preview formats save time in the selection of useful data. ● You can do it, too: – Ukrainian Radarsystem (MRL-5 + Linux-based Operating Software (Titan)) – GRASS + Clips + Database + Paraview
  38. 38. Thank you for your attention !