HDF-EOS Overview and Status

205 views

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
205
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

HDF-EOS Overview and Status

  1. 1. HDF-EOS Overview and Status Larry Klein Emergent Information Technologies, Inc. September 19, 2000 larry@eos.hitc.com
  2. 2. What is HDF-EOS? • An extension to HDF, built with HDF objects. • A standard for data structures for remote sensing data and in-situ measurements with: – Tightly coupled geolocation information – Access based on geolocation • Incorporates ECS metadata attributes. • Used for many EOS standard products. 2
  3. 3. HDF-EOS File Contents • Global HDF attributes – Core Metadata - searchable attributes, copied in database – Archive Metadata - non-searchable attributes • Any number of HDF objects specifying HDF-EOS objects (Grid, Point, Swath) – Structural metadata - links geolocation to science data • Additional HDF objects. 3
  4. 4. HDF-EOS Grid Structure • Usage - Data which is organized by regular geographic spacing, specified by projection parameters. • Structure – Any number of 2-D to 8-D data arrays per structure, one per data type (e.g. temperature) – Geolocation information contained in projection formula, coupled by structural metadata. – Any number of Grid structures per file allowed. 4
  5. 5. Grid Data Example 5
  6. 6. Grid Data: 1 Km. Digital Elevation Model South Pole 6
  7. 7. HDF-EOS Swath Structure • Usage - Data which is organized by time, monotonic geolocation, track parameter. Spacing can be irregular. • Structure – Geolocation information stored explictly in Geolocation Field (2-D array) – Data stored in 2-D or 3-D arrays – Time stored in 1-D or 2-D array, TAI is standard. – Geolocation/science data connected by structural 7 metadata
  8. 8. Swath Data Example 8
  9. 9. HDF-EOS Point Structure • Usage - Data which is specified temporally and/or spatially, but with no particular organization. • Structure – Tables (Vdatas) used to store science data at a particular Lat/Long/Height. – Up to eight levels of data allowed. Structural metadata specifies relationship between levels. 9
  10. 10. Point Data Example Lat 61.12 45.31 38.50 38.39 30.00 37.45 18.00 43.40 34.03 32.45 33.30 42.15 35.05 34.12 46.32 47.36 39.44 21.25 44.58 41.49 25.45 Lon -149.48 -122.41 -77.00 -90.15 -90.05 -122.26 -76.45 -79.23 -118.14 -96.48 -112.00 -71.07 -106.40 -77.56 -87.25 -122.20 -104.59 -78.00 -93.15 -87.37 -80.11 Temp(C) 15.00 17.00 24.00 27.00 22.00 25.00 27.00 30.00 25.00 32.00 30.00 28.00 30.00 28.00 30.00 32.00 31.00 28.00 32.00 28.00 19.00 10 Dewpt(C) 5.00 5.00 7.00 11.00 7.00 10.00 4.00 14.00 4.00 8.00 10.00 7.00 9.00 9.00 8.00 15.00 16.00 7.00 13.00 9.00 3.00
  11. 11. HDF-EOS Interface Schematic User Application HDF-EOS Interface Grid Point ECS Metadata Interface Swath Low Level HDF File I/O 11
  12. 12. User Interface • • • • • • • Access - Open, close objects/files Definition - Set key features of data sets Basic I/O - Read write data Index I/O - Link tables (Point only) Inquiry - Return information about data Subset - Read data from specific region Metadata access (SDP toolkit) 12
  13. 13. Current Archive Holdings • GSFC DAAC: MODIS L1, Atmos./Ocean – 50 TB, 105,000 granules, 300 GB/day – will be ~1.3 TB / day with addition of MODIS L2 and EOS Aqua data. • EDC DAAC: Landsat/ASTER/MODIS Land – 20 TB, 106,000 granules, 274 GB/day • Langely DAAC: MISR – 9 TB, 39,000 granules, 88 GB/day 13
  14. 14. Availability • HDF library - ftp hdf.ncsa.uiuc.edu • HDF-EOS V2.6 library (HDF4 based) – http://newsroom.gsfc.nasa.gov/sdptoolkit/toolkit.html • SDP Toolkit library - same location as HDF-EOS • Platform Support – DEC, HP, IBM, Sun, SGI, Windows, Linux (soon) • Language Support - C, C++, F77, F90 • HDF4-based library will be supported indefinitely 14
  15. 15. HDF5 - Based Development • NCSA has introduced HDF5 – A complete rewrite of HDF4, a new user interface. – Simplified data model – Breaks 2 GB file size limit – Potential for parallel I/O, multi-threading 15
  16. 16. HDF5 - Based Development HDF-EOS 3.0 • Development overview – Maintain current HDF-EOS interface to maximum extent – Allow all HDF5 features through interface – Build HDF-EOS (4 to 5) conversion function – Data production executable can be built with either or both HDF4 and HDF5 - based library – Support heterogeneous ECS archive (future) 16
  17. 17. Design Schematic User Application HDF-EOS Interface Grid, Point, Swath ECS Metadata Interface Common Functions Definition, Access, I/O, Inquiry, Subset *Implemented if required HDF4 Interface* HDF5 Interface Combined HDF4/5 Interface* 17 Build Options
  18. 18. Applications • H5EOSView - EOSView for HDF5 • Java Browser - Browser for both HDF4 and 5 based granules • HDF-EOS 4 to 5 converter, HDF-EOS 2.x to HDF-EOS 3.x 18
  19. 19. Schedule • Jan., 2000 - Beta Version of library – H5EOSView, Sun, SGI, C support • Oct. 2000 - Beta2 Version – Add F77, F90, DEC support – Local Attributes, Profile Interface, Chunking/Tiling – HDF-EOS 4 to 5 converter and Java Browser 19
  20. 20. Schedule (cont.) • Jan. 2001 - SCF version – add HP, IBM, Windows, Linux (maybe a few months later), C++ support • DAAC version (ECS Drop 6B) - Summer 2001. – Archives can contain both HDF4 and HDF5 based granules. – ECS subsystems don’t crack granules, but regression testing needed. – 2 GB size limit is an issue. 20

×