HDF-EOS Aura File Format Guidelines

HDF-EOS Aura File Format
Guidelines
Cheryl Craig
September 20, 2000

Cheryl Craig

HD...
HDF-EOS Aura File Format Guidelines

Preliminary
• Aura Data System Working Group will
meet in October to discuss and hope...
HDF-EOS Aura File Format Guidelines

What Is Aura?
• Third in series of EOS atmospheric chemistry
satellites (1:45 PM equa...
HDF-EOS Aura File Format Guidelines

Aura Instruments
• HIRDLS (High Resolution Dynamics Limb Sounder)
– Limb infrared sou...
HDF-EOS Aura File Format Guidelines

Why Guidelines for Aura Instruments?
• HDF-EOS Swath allows a lot of flexibility
• Ea...
HDF-EOS Aura File Format Guidelines

Areas of Potential Mismatch
•
•
•
•
•
•

Organization of data fields and attributes
D...
HDF-EOS Aura File Format Guidelines

Guidelines List
• Instrument teams will use the same major
version of HDF-EOS
• Aura ...
HDF-EOS Aura File Format Guidelines

Guidelines List (continued)
• Data fields will be ordered with pressure
coordinate be...
HDF-EOS Aura File Format Guidelines

Organization of Data Fields and Attributes
Type 1

Type 2

Swathname: Instrument Name...
HDF-EOS Aura File Format Guidelines

Standard Names

•
•
•
•
•

Adhere to strict names (spacing and capitalization
require...
HDF-EOS Aura File Format Guidelines

HDF-EOS V2.x Versus V3.x
Aura will decide at future date. At January, 2000 DSWG, indi...
HDF-EOS Aura File Format Guidelines

Conclusions
• Preliminary standardization complete – meeting in
October to discuss an...
HDF-EOS Aura File Format Guidelines

Valids from Document
Swath Name:
Instrument Name
- or Measured Field Name (see Data F...
HDF-EOS Aura File Format Guidelines
Geolocation Fields:
This is data which describe the scientific measured quantities. Th...
HDF-EOS Aura File Format Guidelines

Geolocation Fields (continued):
HDF-EOS Name
HIRDLSScanElevationAtNominalAltitude
HIR...
HDF-EOS Aura File Format Guidelines
Data Field Names:
This is the actual scientific data.
IMPORTANT NOTE – Array ordering ...
HDF-EOS Aura File Format Guidelines
Data Field Names (continued):
HDF-EOS Name
HOCl
ClO
BrO
SO2
H2O2
HDO
C2H6
C2H2
HCOOH
C...
HDF-EOS Aura File Format Guidelines

Data Field Names (continued):
HDF-EOS Name
CirrusIceContent
CloudTopHeight
CloudTopPr...
HDF-EOS Aura File Format Guidelines

Data Field Names (continued):
HDF-EOS Name
ConvergenceFlag
nIterations

Dimension
(nT...
HDF-EOS Aura File Format Guidelines

F5 Note

Global Attributes:
This is information which helps to describe this particul...
Upcoming SlideShare
Loading in …5
×

HDF-EOS Aura File Format Guidelines

385 views

Published on

Source: http://hdfeos.org/workshops/ws04/presentations/Craig/HDF-EOS%20Aura%20File%20Format%20Guidelines%20Presentation.ppt

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
385
On SlideShare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

HDF-EOS Aura File Format Guidelines

  1. 1. HDF-EOS Aura File Format Guidelines HDF-EOS Aura File Format Guidelines Cheryl Craig September 20, 2000 Cheryl Craig HDF-EOS Workshop IV Sept 20, 2000 1
  2. 2. HDF-EOS Aura File Format Guidelines Preliminary • Aura Data System Working Group will meet in October to discuss and hopefully adopt • Authors from each instrument team are in agreement Cheryl Craig HDF-EOS Workshop IV Sept 20, 2000 2
  3. 3. HDF-EOS Aura File Format Guidelines What Is Aura? • Third in series of EOS atmospheric chemistry satellites (1:45 PM equator crossing time) • Formerly known as CHEM • To be launched June 2003 • Level 2 data products include: – Vertical profiles of temperature, pressure, chemical and aerosol species – Column amounts of chemical species – Cloud properties Cheryl Craig HDF-EOS Workshop IV Sept 20, 2000 3
  4. 4. HDF-EOS Aura File Format Guidelines Aura Instruments • HIRDLS (High Resolution Dynamics Limb Sounder) – Limb infrared sounder – University of CO and Oxford • MLS (Microwave Limb Sounder) – Limb microwave sounder – JPL • OMI (Ozone Monitoring Instrument) – Nadir wide-field-imaging spectrometer – Netherlands, Finland and US • TES (Tropospheric Emission Spectrometer) – Nadir and limb infrared-imaging spectrometer – JPL All instruments have world-wide co-investigators Cheryl Craig HDF-EOS Workshop IV Sept 20, 2000 4
  5. 5. HDF-EOS Aura File Format Guidelines Why Guidelines for Aura Instruments? • HDF-EOS Swath allows a lot of flexibility • Ease cross-platform use of Aura files – Ease development of software – Easier to understand • Intended to standardize Level 2 files (may apply to Level 3 and above as well) Cheryl Craig HDF-EOS Workshop IV Sept 20, 2000 5
  6. 6. HDF-EOS Aura File Format Guidelines Areas of Potential Mismatch • • • • • • Organization of data fields and attributes Dimension names Geolocation names and dimension ordering Data field names and dimension ordering Units for data fields Attribute names, values and units Cheryl Craig HDF-EOS Workshop IV Sept 20, 2000 6
  7. 7. HDF-EOS Aura File Format Guidelines Guidelines List • Instrument teams will use the same major version of HDF-EOS • Aura Level 2 files will use HDF-EOS Swath • HDF-EOS structure names will strictly adhere to Valids list • Data will be reported on a pressure grid and be ordered from ground to space Cheryl Craig HDF-EOS Workshop IV Sept 20, 2000 7
  8. 8. HDF-EOS Aura File Format Guidelines Guidelines List (continued) • Data fields will be ordered with pressure coordinate being the fastest incrementing coordinate • Data fields will be in units specified • HDF fill value and MissingValue will have the same value • Instrument data fields unique to particular instrument will be prefixed with instrument name Cheryl Craig HDF-EOS Workshop IV Sept 20, 2000 8
  9. 9. HDF-EOS Aura File Format Guidelines Organization of Data Fields and Attributes Type 1 Type 2 Swathname: Instrument Name Swathname: Meas. Species Name Dimensions: Dimensions: nTimes nLevels etc. Geolocation Fields: nTimes nLevels etc. Geolocation Fields: Time Latitude etc. Data Fields: Time Latitude etc. Data Fields: Temperature O3 etc. Global Attributes: L2gpValue L2gpPrecision Global Attributes: Instrument Name Process Level etc. Instrument Name Process Level etc. HDF-EOS V3.x Note – Two types may be merged into one using links for Data Field Names Cheryl Craig HDF-EOS Workshop IV Sept 20, 2000 9
  10. 10. HDF-EOS Aura File Format Guidelines Standard Names • • • • • Adhere to strict names (spacing and capitalization required) Dimension names Geolocation Fields – Used to hold descriptor data (includes SolarZenithAngle, OrbitAscendingFlag, etc.) Data Fields – Contains measured and derived data Global Attributes – Includes InstrumentName and ProcessLevel Data Field Attributes – only if use HDF-EOSV3 (includes MissingValue, ScaleFactor, Units, etc.) Cheryl Craig HDF-EOS Workshop IV Sept 20, 2000 10
  11. 11. HDF-EOS Aura File Format Guidelines HDF-EOS V2.x Versus V3.x Aura will decide at future date. At January, 2000 DSWG, indicated desire to used V3.x if packages like IDL will be available in V3.x HDF-EOS V2.x HDF-EOS V3.x PROS • Currently used by Terra and Aqua teams • Currently available CONS • Based on HD4 – supported by HDF group in maintenance only mode PROS • Contains new functionality – local data attributes – more than one unlimited dimension – compression with unlimited dimensions • Based on HDF5 CONS • IDL support (unknown?) Cheryl Craig HDF-EOS Workshop IV Sept 20, 2000 11
  12. 12. HDF-EOS Aura File Format Guidelines Conclusions • Preliminary standardization complete – meeting in October to discuss and adopt • HDF-EOS version decision to be decided at a future date • Some decisions postponed until prototyping efforts complete • Standard may evolve as instrument teams get better understanding of their data storage needs • URL of document: http://www.eos.ucar.edu/hirdls/HDFEOS_Aura_File_Format_Guidelines.pdf Cheryl Craig HDF-EOS Workshop IV Sept 20, 2000 12
  13. 13. HDF-EOS Aura File Format Guidelines Valids from Document Swath Name: Instrument Name - or Measured Field Name (see Data Field Names for list of names) Dimensions: These are the actual dimensions of the Geolocation and Data field quantities. nTimes nLevels nWavel nXtrack Number of times (profiles) in data set (this may be unlimited) Number of pressure levels Number of wavelengths Number of pixels in the across track direction (OMI specific) Cheryl Craig HDF-EOS Workshop IV Sept 20, 2000 13
  14. 14. HDF-EOS Aura File Format Guidelines Geolocation Fields: This is data which describe the scientific measured quantities. They provide information to aid in describing the data’s “location”. Instrument H M O X X X X X X X X X X X HDF-EOS Name Time Latitude Latitude Longitude Longitude Pressure+ Dimension (nTimes) (nTimes) (nXtrack,nTimes) (nTimes) (nXtrack,nTimes) (nLevels) Altitude SecondsInDay (nLevels,nTimes) (nTimes) Suggested units seconds from Jan 1, 1993 degrees degrees degrees (-180 to 180) degrees (-180 to 180) hPa (ordered from ground to space) meters seconds SolarZenithAngle LocalSolarTime SpacecraftLatitude SpacecraftLongitude SpacecraftAltitude OrbitAscendingFlag OrbitGeodeticAngle LineOfSightAngle SolarAzimuthAngle ViewingZenithAngle ViewingAzimuthAngle SpacecraftAzimuth SpacecraftZenith HIRDLSScienceScanMode (nTimes) (nTimes) (nTimes) (nTimes) (nTimes) (nTimes) (nTimes) (nTimes) (nXtrack,nTimes) (nXtrack,nTimes) (nXtrack,nTimes) (nTimes) (nTimes) (nTimes) degrees hours degrees degrees (-180 to 180) meters logical degrees degrees (East of North) degrees (East of North) degrees degrees (East of North) degrees (East of North) degrees short integer X X X X X X HIRDLSScanUpFlag (nTimes) logical X Cheryl Craig X X T X X Notes time in TAI units Geodetic Latitude Geodetic Latitude X X Pressure will be a superset of the UARS pressure levels X Seconds from midnight of day listed in global attributes X X X X X X X X A A A A if true then orbit is ascending X X X X X A A A X HIRDLS Science Scan Mode identifier HIRDLS Scan Up identifier HDF-EOS Workshop IV Sept 20, 2000 14
  15. 15. HDF-EOS Aura File Format Guidelines Geolocation Fields (continued): HDF-EOS Name HIRDLSScanElevationAtNominalAltitude HIRDLSScanAzimuthAtNominal Altitude HIRDLSTangentHeightAtNominalAltitude HIRDLSViewDirectionAtNominalAltitude TESDate Dimension (nTimes) (nTimes) (nTimes) (nTimes) (nTimes) Suggested units degrees degrees meters degrees (East of North) TESSequenceID TESBoresightAngle TESFiltersUsed TESPixelsUsedFlag (nTimes) (nTimes) (nTimes) (nTimes) Instrument H M O X X X X sequence within current survey degrees bit mask bit mask T Notes A TES survey may cover multiple days X A A A X – Field in standard file A – Data items which will be carried in an ancillary file that is created only once per set of standard products + nLevels for TES is currently set to 14 levels (based on UARS standard) but may be changed to 85 levels. nLevels for HIRDLS is set to 145 (1000. * 10 ** (-i/24) i=0,144) Cheryl Craig HDF-EOS Workshop IV Sept 20, 2000 15
  16. 16. HDF-EOS Aura File Format Guidelines Data Field Names: This is the actual scientific data. IMPORTANT NOTE – Array ordering is Fortran ordering, the first dimension is the most rapidly incrementing. For C, reverse the array indices. HDF-EOS Name L2gpValue Dimension (nLevels,nTimes) Suggested units varies L2gpPrecision (nLevels,nTimes) varies Temperature O3 O3 (nLevels,nTimes) (nLevels,nTimes) (nLevels,nXtrack,nT imes) (nLevels,nTimes) (nLevels,nTimes) (nLevels,nTimes) nLevels,nTimes) (nLevels,nTimes) (nLevels,nTimes) (nLevels,nTimes) (nLevels,nTimes) (nLevels,nTimes) (nLevels,nTimes) (nLevels,nTimes) (nLevels,nTimes) (nLevels,nTimes) (nLevels,nTimes) (nLevels,nTimes) K parts per volume parts per volume X X parts per volume parts per volume parts per volume parts per volume parts per volume parts per volume parts per volume parts per volume parts per volume parts per volume parts per volume parts per volume parts per volume parts per volume parts per volume X X X X X X X X X H2O ClONO2 N2O5 N2O NO2 CH4 HNO3 CFC11 CFC12 OH HO2 CO HCN NO HCl H M O X T X S S X X X S S S S S S S S Cheryl Craig Notes MLS Generic Data Field Name (species identified by SwathName) MLS Generic Precision Name (species identified by SwathName) OMI imager data X Sp Sp Sp X X X X Sp X Sp TES special product, tropospheric control TES limb only TES limb only TES limb only HDF-EOS Workshop IV Sept 20, 2000 16
  17. 17. HDF-EOS Aura File Format Guidelines Data Field Names (continued): HDF-EOS Name HOCl ClO BrO SO2 H2O2 HDO C2H6 C2H2 HCOOH CH3OH PAN CH3C(O)CH3 C2H4 H02N02 NH3 CCl4 CCl3F CCl2F2 CHCl2F CHClF2 COS H2S SF6 XX.XXMicronAerosolExtinction yyyPrecision yyyNormChiSq Dimension (nLevels,nTimes) (nLevels,nTimes) (nLevels,nTimes) (nLevels,nTimes) (nLevels,nTimes) (nLevels,nTimes) (nLevels,nTimes) (nLevels,nTimes) (nLevels,nTimes) (nLevels,nTimes) (nLevels,nTimes) (nLevels,nTimes) (nLevels,nTimes) (nLevels,nTimes) (nLevels,nTimes) (nLevels,nTimes) (nLevels,nTimes) (nLevels,nTimes) (nLevels,nTimes) (nLevels,nTimes) (nLevels,nTimes) (nLevels,nTimes) (nLevels,nTimes) (nLevels,nTimes) (nLevels,nTimes) (nTimes) Suggested units parts per volume parts per volume parts per volume parts per volume parts per volume parts per volume parts per volume parts per volume parts per volume parts per volume parts per volume parts per volume parts per volume parts per volume parts per volume parts per volume parts per volume parts per volume parts per volume parts per volume parts per volume parts per volume parts per volume extinction (1/km) H M O S S S T Sp Sp Sp Sp Sp Sp Sp Sp Sp Sp Sp Sp Sp Sp Sp Sp Sp Sp Sp Sp X X X Cheryl Craig Notes TES special product, volcanic plumes Aerosol at XX.XX Random Errors for species yyy Normalized Chi2 for species yyy HDF-EOS Workshop IV Sept 20, 2000 17
  18. 18. HDF-EOS Aura File Format Guidelines Data Field Names (continued): HDF-EOS Name CirrusIceContent CloudTopHeight CloudTopPressure CloudTopPressure ColumnAmountO3 ColumnAmountNO2 ColumnAmountSO2 ColumnAmountBrO ColumnAmountHCHO ColumnAmountOCLO CloudFraction AerosolOpticalThickness Suggested units g/m3 meters hPa hPa Dobson units molec/cm2 molec/cm2 molec/cm2 molec/cm2 molec/cm2 dimensionless dimensionless SurfaceTemperature Dimension (nLevels,nTimes) (nTimes) (nXtrack,nTimes) (nTimes) (nXtrack,nTimes) (nXtrack,nTimes) (nXtrack,nTimes) (nXtrack,nTimes) (nXtrack,nTimes) (nXtrack,nTimes) (nXtrack,nTimes) (nWavel, nXtrack,nTimes) (nTimes) H M O S T K X RandomError LandSurfaceEmissivity SystematicError TotalColumnDensity TotalColumnDensityError EstRandomErrors EstSystematicErrors EstFractionalExplainedVariance EstCorrelationLength Convergence CompConvergenceToEstimate (nLevels,nTimes) (nTimes) (nLevels,nTimes) (nTimes) (nTimes) (nLevels,nTimes) (nLevels,nTimes) (nLevels,nTimes) (nLevels,nTimes) (nTimes) (nTimes) range: 0-1 X A X X X X X X X X A Notes X X X S range: 0-1 moles/cm2 moles/cm2 range: 0-1 range: 0-1 range: 0-1 0-1000 km Cheryl Craig (OMI imager data) X X X X X X X X X (May be cloud top for limb or may be omitted for limb) TES may still split into Trop/Strat TES may still split into Trop/Strat (Chi2) final retrieval – not normalized Comparison between actual Chi2 and estimated Chi2 – not normalized HDF-EOS Workshop IV Sept 20, 2000 18
  19. 19. HDF-EOS Aura File Format Guidelines Data Field Names (continued): HDF-EOS Name ConvergenceFlag nIterations Dimension (nTimes) (nTimes) Suggested units 0-1 (Boolean) 0-8 small int. nonConvergenceFlag (nTimes) 0-1 (Boolean) OutofExpectedRange AverageRMSError MaxRMSError CloudCover ProcessingTime (nTimes) (nTimes) (nTimes) (nTimes) (nTimes) H M O T A A X X X X A A 0-1 (Boolean) Notes Final retrieval converged Number of model iterations for final retrieval Did not converge at least during an intermediate step Identified through retrieval or L1B X – Field in standard file S – Swathname (MLS will identify field with the swathname rather than the fieldname) A – Data items which will be carried in an ancillary file that is created only once per set of standard products Sp – TES special data product Cheryl Craig HDF-EOS Workshop IV Sept 20, 2000 19
  20. 20. HDF-EOS Aura File Format Guidelines F5 Note Global Attributes: This is information which helps to describe this particular data set. Attribute Name InstrumentName ProcessLevel GranuleMonth GranuleDay GranuleYear Version VerticalCoordinate Attribute Type character HDFE_NATIVE_SHORT HDFE_NATIVE_INT HDFE_NATIVE_INT HDFE_NATIVE_INT HDFE_NATIVE_CHAR HDFE_NATIVE_CHAR Attribute Values “HIRDLS”, “MLS”, “TES” or “OMI” 2, 3, etc. 1-12 1-31 “Pressure”, “Altitude”, “Potential Temperature” Data Field Attributes This is information which helps to describe the individual data fields. Data Field Attributes are a feature which can be useful in annotating plots as well as describing the data product to input routines. It is a feature which was cumbersome to implement and retrieve in HDF-EOS V2.x, but is a subroutine call in HDF-EOS V3.x. For this reason, it is proposed that we only use data field attributes if the Aura team adopts HDF-EOS V3.x as the version to use. Data Field Attributes: Attribute Name MissingValue ScaleFactor Offset Title Units Attribute type Same type as Data Field HDFE_NATIVE_FLOAT HDFE_NATIVE_FLOAT HDFE_NATIVE_CHAR HDFE_NATIVE_CHAR Attribute Description Contains the value for missing data Factor for scaling data (if applicable) Value to add to the data (if applicable) For labeling a plot or axis Labeling units (for labeling color bars, converting between units, etc). Values for Units: "m", "km", "hPa", "Pa", "mAtm-cm", "ppbv", “K”, “micron”, “degree”, “second”, “hour”, “m-1”, “dimensionless”, etc. Cheryl Craig HDF-EOS Workshop IV Sept 20, 2000 20

×