• Like

An IDL-Based Validation Toolkit: Extensions to use the HDF-EOS Swath Format

Uploaded on

Source: http://hdfeos.org/workshops/ws04/presentations/Stone/SW_HDF-EOS_Talk.ppt

Source: http://hdfeos.org/workshops/ws04/presentations/Stone/SW_HDF-EOS_Talk.ppt

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads


Total Views
On Slideshare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. An IDL-BasedValidation Toolkit: Extensions to use the HDF-EOS Swath Format Ken Stone, Center for Lower Atmospheric Studies University of Colorado, Boulder with help from...
  • 2. Acknowledgements • Larry L. Gordley - GATS, Inc., Newport News, Virginia • The HALogen Occultation Experiment (HALOE) Team • Cheryl Craig, NCAR • Joe McInerney, University of Colorado
  • 3. Overview • Overview of the toolkit – History – Motivation – Some examples • HDF-EOS extensions – Problems & Limitations – Strengths – Lessons learned • Summary
  • 4. Motivation • Problem: It takes considerable time and effort to make the many multiple comparisons, internal and external, needed during validation phase. • Goal: A simple set of tools that allow validation staff to select, line-up, compare and plot profiles and cross sections. • Latest twist: HDF-EOS files are now the archive format of choice.
  • 5. History • LIMS - Early 80’s – – – – Fortran routine for comparisons and plotting Fast, low memory usage Input: multiple files Output: text summaries and plots • HALOE - Early 90’s – Perl Scripts and Standard I/O file format (bsel format) – Fortran routine ported to IDL – New functionality and visualization tools. • HIRDLS – HDF-EOS Swath capability – Additional Tools (dot-plots, map differences)
  • 6. Toolkit Overview • IDL procedures with Perl scripts for control – Complex scenarios developed both with interactive use (storing results in “journal” file) and stringing scripts together with Unix shell. • Analytical Visualization tools – Mapping, cross sections, line plots. – Focus on quality and simplicity rather than complex visualizations. • Analysis tools – Coincidence finder, statistical computations. • Standardized file format(s) for I/O • “Profile” based format hinges on vectors of geo-referenced quantities.
  • 7. Toolkit Detail • Prior to HDF-EOS extension the Toolkit used a single file type for input (BSEL format) – Similar in some ways to HDF 4 SDS but limited to sets of 1-d arrays stored as records. • User input for any given IDL scenario can be recorded in a journal (text) file for later use. • Perl “wrapper” scripts create this file as output. – Keyword substitutions in a “mask” file. – Depending on the mode (mood?) the user is in, they could modify the journal file directly, or change the inputs on the script.
  • 8. Toolkit Script Methodology User Input: -f filename -I id, -o outputfile -g -h, etc Mask File Perl Script (e.g., 2dlat) Perl helper lib Input File Journal file creation from IDL scripts IDL procedure or C, or Fortran... Analysis routines Postscript BSEL File
  • 9. Archive Toolkit Data Flow: Archive and Working Formats D a t a Selection S o u r c e “Standard” Data Format Analysis Bsel HDF-EOS Display “working” data format Toolkit Extension
  • 10. Toolkit Data Model (BSEL Format) BSEL File Event (1) Event Header Real(8) Long(8) char*60(6) Plot Info Range, axis Label, legend Label, Scale, Logflag . . . Data Header Data order (=2) J (Profile count) K (levels/event) Data (j,k) Event (i)
  • 11. HIRDLS HDF-EOS Swath HDF-EOS Swath File Metadata Dimensions: Global Attributes nTimes, nLevels etc. Geolocation Fields Data Fields nLevels Time Pressure Ozone Mixing Ratio nTimes Inst. Name, Ranges, Labels, Scales, Missing Values, Profile count
  • 12. HDF-EOS Extension Strengths • Fast! Ten times faster than native (Bsel) format • Easier to identify field names (and more flexible) • Standard way of identifying missing data • IDL/Fortran/C I/O interfaces • Potentially more tools available
  • 13. HDF-EOS Extension Difficulties • Lots of different ways to store data and still meet the standard. – Aura standard format guidelines helps here. • HDF-EOS Swath isn’t “packet-like” – You can’t just “cat” files together! • IDL/Fortran/C I/O interfaces – Some trouble with stability of certain I/Fs. • Point Interface (and raw Vdata) failed from the start due to lack of “id” resources.
  • 14. Lessons Learned • Developed general purpose swath reader in IDL – usage: data = get_sw_data(filename, swathname, varname) – Generally reads any variable or attribute. – http://www.eos.ucar.edu/hirdls/get_sw_data.pro • Working out the best way to “easily select” variables from swaths is a work-in-progress. • To map HDF to the internal data structure a small external data table was needed...
  • 15. Mapping Table HDF-EOS - Internal Internal to HDF-EOS SwathName VarName HIRDLS H2O HIRDLS NO2 HIRDLS O3 HIRDLS CH4 HIRDLS CFCl3 CH4 L2gpValue O3 L2gpValue Internal to toolkit Label H2O Mixing Ratio NO2 Mixing Ratio O3 Mixing Ratio CH4 Mixing Ratio CFCl3 Mixing Ratio CH4 Mixing Ratio O3 Mixing Ratio ID 2 3 4 6 151 6 4 Min 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Max 10.0 10.0 10.0 2.0 0.5 2.0 10.0 Log 0 0 0 0 0 0 0 Scale 1.0e-06 1.0e-09 1.0e-06 1.0e-06 1.0e-09 1.0e-06 1.0e-06 Designations for MLS Files Unit ppmv ppbv ppmv ppmv ppbv ppmv ppmv
  • 16. Summary • HDF-EOS Swath format provides a good storage facility for atmospheric profile data. – Current swath implementation (a la Aura) provides solid groundwork for future sophisticated software development. • Interface instability can contribute to major headaches. • Ancillary tools to “view and edit” or simply compute additional fields from existing ones could prove quite useful.
  • 17. Future • Extend data base table for mapping fields. • Develop interface for Grid format. – Already tested in limited cases. • Testing, testing, testing… • Perhaps incorporate new IDL features – Improved color contouring • Develop an easier to use interface for the scripting tools.