An IDL-BasedValidation Toolkit:
Extensions to use the HDF-EOS
Swath Format
Ken Stone,
Center for Lower Atmospheric Studies...
Acknowledgements
• Larry L. Gordley - GATS, Inc., Newport
News, Virginia
• The HALogen Occultation Experiment
(HALOE) Team...
Overview
• Overview of the toolkit
– History
– Motivation
– Some examples

• HDF-EOS extensions
– Problems & Limitations
–...
Motivation
• Problem: It takes considerable time and
effort to make the many multiple
comparisons, internal and external, ...
History
• LIMS - Early 80’s
–
–
–
–

Fortran routine for comparisons and plotting
Fast, low memory usage
Input: multiple f...
Toolkit Overview
• IDL procedures with Perl scripts for control
– Complex scenarios developed both with interactive use (s...
Toolkit Detail
• Prior to HDF-EOS extension the Toolkit used a single file
type for input (BSEL format)
– Similar in some ...
Toolkit Script Methodology
User Input: -f filename -I id, -o outputfile -g -h, etc

Mask File

Perl Script
(e.g., 2dlat)

...
Archive

Toolkit Data Flow: Archive and
Working Formats

D
a
t
a
Selection
S
o
u
r
c
e
“Standard” Data
Format

Analysis

B...
Toolkit Data Model (BSEL Format)
BSEL File

Event (1)

Event Header
Real(8)
Long(8)
char*60(6)

Plot Info
Range, axis Labe...
HIRDLS HDF-EOS Swath
HDF-EOS Swath File

Metadata

Dimensions:

Global Attributes

nTimes, nLevels etc.

Geolocation Field...
HDF-EOS Extension Strengths
• Fast! Ten times faster than native (Bsel)
format
• Easier to identify field names (and more
...
HDF-EOS Extension Difficulties
• Lots of different ways to store data and still
meet the standard.
– Aura standard format ...
Lessons Learned
• Developed general purpose swath reader in
IDL
– usage: data = get_sw_data(filename, swathname, varname)
...
Mapping Table
HDF-EOS - Internal
Internal to HDF-EOS
SwathName VarName
HIRDLS
H2O
HIRDLS
NO2
HIRDLS
O3
HIRDLS
CH4
HIRDLS
C...
Summary
• HDF-EOS Swath format provides a good
storage facility for atmospheric profile data.
– Current swath implementati...
Future
• Extend data base table for mapping fields.
• Develop interface for Grid format.
– Already tested in limited cases...
An IDL-Based  Validation Toolkit: Extensions to  use the HDF-EOS Swath Format
An IDL-Based  Validation Toolkit: Extensions to  use the HDF-EOS Swath Format
An IDL-Based  Validation Toolkit: Extensions to  use the HDF-EOS Swath Format
An IDL-Based  Validation Toolkit: Extensions to  use the HDF-EOS Swath Format
Upcoming SlideShare
Loading in …5
×

An IDL-Based Validation Toolkit: Extensions to use the HDF-EOS Swath Format

455 views

Published on

Source: http://hdfeos.org/workshops/ws04/presentations/Stone/SW_HDF-EOS_Talk.ppt

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
455
On SlideShare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

An IDL-Based Validation Toolkit: Extensions to use the HDF-EOS Swath Format

  1. 1. An IDL-BasedValidation Toolkit: Extensions to use the HDF-EOS Swath Format Ken Stone, Center for Lower Atmospheric Studies University of Colorado, Boulder with help from...
  2. 2. Acknowledgements • Larry L. Gordley - GATS, Inc., Newport News, Virginia • The HALogen Occultation Experiment (HALOE) Team • Cheryl Craig, NCAR • Joe McInerney, University of Colorado
  3. 3. Overview • Overview of the toolkit – History – Motivation – Some examples • HDF-EOS extensions – Problems & Limitations – Strengths – Lessons learned • Summary
  4. 4. Motivation • Problem: It takes considerable time and effort to make the many multiple comparisons, internal and external, needed during validation phase. • Goal: A simple set of tools that allow validation staff to select, line-up, compare and plot profiles and cross sections. • Latest twist: HDF-EOS files are now the archive format of choice.
  5. 5. History • LIMS - Early 80’s – – – – Fortran routine for comparisons and plotting Fast, low memory usage Input: multiple files Output: text summaries and plots • HALOE - Early 90’s – Perl Scripts and Standard I/O file format (bsel format) – Fortran routine ported to IDL – New functionality and visualization tools. • HIRDLS – HDF-EOS Swath capability – Additional Tools (dot-plots, map differences)
  6. 6. Toolkit Overview • IDL procedures with Perl scripts for control – Complex scenarios developed both with interactive use (storing results in “journal” file) and stringing scripts together with Unix shell. • Analytical Visualization tools – Mapping, cross sections, line plots. – Focus on quality and simplicity rather than complex visualizations. • Analysis tools – Coincidence finder, statistical computations. • Standardized file format(s) for I/O • “Profile” based format hinges on vectors of geo-referenced quantities.
  7. 7. Toolkit Detail • Prior to HDF-EOS extension the Toolkit used a single file type for input (BSEL format) – Similar in some ways to HDF 4 SDS but limited to sets of 1-d arrays stored as records. • User input for any given IDL scenario can be recorded in a journal (text) file for later use. • Perl “wrapper” scripts create this file as output. – Keyword substitutions in a “mask” file. – Depending on the mode (mood?) the user is in, they could modify the journal file directly, or change the inputs on the script.
  8. 8. Toolkit Script Methodology User Input: -f filename -I id, -o outputfile -g -h, etc Mask File Perl Script (e.g., 2dlat) Perl helper lib Input File Journal file creation from IDL scripts IDL procedure or C, or Fortran... Analysis routines Postscript BSEL File
  9. 9. Archive Toolkit Data Flow: Archive and Working Formats D a t a Selection S o u r c e “Standard” Data Format Analysis Bsel HDF-EOS Display “working” data format Toolkit Extension
  10. 10. Toolkit Data Model (BSEL Format) BSEL File Event (1) Event Header Real(8) Long(8) char*60(6) Plot Info Range, axis Label, legend Label, Scale, Logflag . . . Data Header Data order (=2) J (Profile count) K (levels/event) Data (j,k) Event (i)
  11. 11. HIRDLS HDF-EOS Swath HDF-EOS Swath File Metadata Dimensions: Global Attributes nTimes, nLevels etc. Geolocation Fields Data Fields nLevels Time Pressure Ozone Mixing Ratio nTimes Inst. Name, Ranges, Labels, Scales, Missing Values, Profile count
  12. 12. HDF-EOS Extension Strengths • Fast! Ten times faster than native (Bsel) format • Easier to identify field names (and more flexible) • Standard way of identifying missing data • IDL/Fortran/C I/O interfaces • Potentially more tools available
  13. 13. HDF-EOS Extension Difficulties • Lots of different ways to store data and still meet the standard. – Aura standard format guidelines helps here. • HDF-EOS Swath isn’t “packet-like” – You can’t just “cat” files together! • IDL/Fortran/C I/O interfaces – Some trouble with stability of certain I/Fs. • Point Interface (and raw Vdata) failed from the start due to lack of “id” resources.
  14. 14. Lessons Learned • Developed general purpose swath reader in IDL – usage: data = get_sw_data(filename, swathname, varname) – Generally reads any variable or attribute. – http://www.eos.ucar.edu/hirdls/get_sw_data.pro • Working out the best way to “easily select” variables from swaths is a work-in-progress. • To map HDF to the internal data structure a small external data table was needed...
  15. 15. Mapping Table HDF-EOS - Internal Internal to HDF-EOS SwathName VarName HIRDLS H2O HIRDLS NO2 HIRDLS O3 HIRDLS CH4 HIRDLS CFCl3 CH4 L2gpValue O3 L2gpValue Internal to toolkit Label H2O Mixing Ratio NO2 Mixing Ratio O3 Mixing Ratio CH4 Mixing Ratio CFCl3 Mixing Ratio CH4 Mixing Ratio O3 Mixing Ratio ID 2 3 4 6 151 6 4 Min 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Max 10.0 10.0 10.0 2.0 0.5 2.0 10.0 Log 0 0 0 0 0 0 0 Scale 1.0e-06 1.0e-09 1.0e-06 1.0e-06 1.0e-09 1.0e-06 1.0e-06 Designations for MLS Files Unit ppmv ppbv ppmv ppmv ppbv ppmv ppmv
  16. 16. Summary • HDF-EOS Swath format provides a good storage facility for atmospheric profile data. – Current swath implementation (a la Aura) provides solid groundwork for future sophisticated software development. • Interface instability can contribute to major headaches. • Ancillary tools to “view and edit” or simply compute additional fields from existing ones could prove quite useful.
  17. 17. Future • Extend data base table for mapping fields. • Develop interface for Grid format. – Already tested in limited cases. • Testing, testing, testing… • Perhaps incorporate new IDL features – Improved color contouring • Develop an easier to use interface for the scripting tools.

×