ESDIS Status

Richard Ullman
ESDIS Project
Richard.E.Ullman@nasa.gov

05 December, 2002

HDF & HDF-EOS Workhop VI

1
Outline
• ESDIS General Status
• HDF-EOS Plans
• Website http://hdfeos.gsfc.nasa.gov

05 December, 2002 HDF & HDF-EOS Work...
ESDIS Science Data Services Today
• EOSDIS provides support for high data volumes from Aqua, Terra,
and Landsat 7, and con...
Science Operations Timeline

Science
Operations
Supported

Heritage
Missions
T/P, UARS,
TOMS, ERBE
SeaWiFS
Radarsat

TRMM
...
EOSDIS DAAC Data delivery FY’02
October 1, 2002 through September 30, 2002
• Over 1.8 million distinct users
• 16,013,980 ...
16,013,980 Data Products Delivered
October 1, 2001 - September 30, 2002

05 December, 2002 HDF & HDF-EOS Workhop VI
6
Over 1.8 Million Distinct Users
October 1, 2001 - September 30, 2002

05 December, 2002 HDF & HDF-EOS Workhop VI
7
ESDIS Status
• After years of development, ECS is operational and
generally recognized as successful.
• Primary ECS develo...
HDF-EOS Plans
• HDF-EOS 5 development is nearly complete.
• Continue to maintain, port to newer operating systems,
bug-fix...
Data Pools Concept:
The Right Data, the Right Way, Right Now!
User-defined
Views, Presentations and Data Access Requests

...
General Capabilities
– The Right Data: Data Location
•
•
•
•
•
•

Groups, themes, bookmarks and views
Navigation and machi...
EOS ClearingHOuse (ECHO)
http://eos.nasa.gov/echo
• ECHO is a metadata clearinghouse
– A single Internet portal for Earth ...
ECHO Data Providers
• Current ECHO Data Providers
– EOSDIS Core System DAACs
(17% thus far)
• EDC Land Processes DAAC
• Go...
HDF-EOS Tools and Information Web Site
http://hdfeos.gsfc.nasa.gov
Richard.E.Ullman@nasa.gov

05 December, 2002

HDF & HDF...
http://hdfeos.gsfc.nasa.gov

05 December, 2002 HDF & HDF-EOS Workhop VI
15
Website status
• A resource for discovering about hdf-eos in particular.
– This workshop series’ presentation archive
– Li...
HDF-EOS Profile

Richard Ullman
ESDIS Project
Richard.E.Ullman@nasa.gov

05 December, 2002

HDF & HDF-EOS Workhop VI

17
History
• September 1993, HDF adopted as baseline standard for EOSDIS Core
System standard data product generation, archiv...
HDF-EOS Data Objects
•
•
•
•
•

Point
Swath
Profile (Swath subtype)
Grid
Zonal (HDF-EOS 5 only)

05 December, 2002 HDF & H...
HDF-EOS 5
• Based on HDF5, a complete rewrite of HDF4 with a
different interface.
– First released in 2000.

• Designed to...
HDF-EOS 5 Functionality
•
•
•
•
•
•
•
•

Basic File I/O
Fill Values
Compression
Chunking/Tiling
Swath Interface
Grid Inter...
Top Level of HDF-EOS 5
Root -- “/”
HDFEOS
INFORMATION

HDFEOS

STRUC. METADATA
ADDITIONAL
SWATH

GRID

POINT

Global (file...
Swath Structure
Global Attribute
<SwathName>:<AttrName>

Group Attribute
<DataFields>:<AttrName>
Local Attribute
<FieldNam...
HDF-EOS Point
• Intended use
– Discrete points in time and/or
location.
– Table of data linked to table of
geographic info...
HDF-EOS Point

Latitude
25.2645
22.3549
23.2564

Longitude
091.2564
-93.4657
-89.2546

Buoy_ID
0126
3564
1256

• Hierarchi...
HDF-EOS Swath
• Intended use
– Across track scanning
instruments.
– Sounding instruments
– Level 1: Geolocated Sensor
Unit...
HDF-EOS Swath
Data Fields

“Brightness
Temperature”

Geolocation Fields
Dimension
Dimension
Name: Scan
Name: Scan
Size: 16...
HDF-EOS Grid
• Intended use
– Variables mapped on uniform
space-time grid scales
– Level 3 - Gridded single
measurement pa...
HDF-EOS Grid

Altitude
Size 30

Xdim
Size: 2000

Projinfo

Ydim
Size: 800

05 December, 2002 HDF & HDF-EOS Workhop VI
29
HDF-EOS Grid
•
•
•
•
•
•
•
•

Projections Supported
Geographic
Transverse Mercator
Universal Transverse Mercator
Hotine Ob...
Product Levels
• Level 0 - Reconstructed,
unprocessed instrument/payload data at full resolution; any
and all communicatio...
Resources
– HDF-EOS on the web:
• http://hdfeos.gsfc.nasa.gov/
• http://newsroom.gsfc.nasa.gov/sdptoolkit/toolkit.html

– ...
Upcoming SlideShare
Loading in …5
×

ESDIS Status (2002)

362 views

Published on

Source: http://hdfeos.org/workshops/ws06/presentations/Ullman/ESDIS_Status.ppt

Published in: Technology
  • Be the first to comment

  • Be the first to like this

ESDIS Status (2002)

  1. 1. ESDIS Status Richard Ullman ESDIS Project Richard.E.Ullman@nasa.gov 05 December, 2002 HDF & HDF-EOS Workhop VI 1
  2. 2. Outline • ESDIS General Status • HDF-EOS Plans • Website http://hdfeos.gsfc.nasa.gov 05 December, 2002 HDF & HDF-EOS Workhop VI 2
  3. 3. ESDIS Science Data Services Today • EOSDIS provides support for high data volumes from Aqua, Terra, and Landsat 7, and continues to support QuikSCAT, ACRIMSat, SAGE-III, JASON and pre-EOS-era data including TRMM, UARS, TOPEX/Poseidon, RADARSat, and others. – EOSDIS finalizing preparations to support ICESat and SORCE – In all, EOSDIS is providing data processing, archival, and/or distribution for over 15 Earth science satellite missions. • EOSDIS has set a new benchmark for data management. The total volume of the science data in our archives totals over 2 Petabytes. Since 1998, the science data volume managed by the EOSDIS has increased eight-fold, and continues to grow at a rate of over 2 Terabytes per day • In FY02, EOSDIS provided more than 16 million data and information products to over 1.8 million individuals. 05 December, 2002 HDF & HDF-EOS Workhop VI 3
  4. 4. Science Operations Timeline Science Operations Supported Heritage Missions T/P, UARS, TOMS, ERBE SeaWiFS Radarsat TRMM Landsat 7 QuikSCAT Terra ACRIMSat SAGE III JASON Aqua GRACE 2 Petabytes Archive Volume has doubled each year for the past three years Archive Growth 15 Million Products Distributed FY94/95 FY96 FY97 FY98 FY99 FY00 05 December, 2002 HDF & HDF-EOS Workhop VI 4 FY01 FY02
  5. 5. EOSDIS DAAC Data delivery FY’02 October 1, 2002 through September 30, 2002 • Over 1.8 million distinct users • 16,013,980 products delivered • Notes to charts on following slides: – Distinct users includes users accessing DAAC web pages, including web-crawlers – Distinct user type is based on email addresses of users or URLs – Product delivered is defined as the smallest deliverable unit of data – Product delivery breakdown is based on email addresses of users receiving ECS and Non-ECS data – “FTP Delivery” are to URLs not mapped to specific domains – “Foreign Other” includes foreign email addresses whose country is known but domain-type could not be determined 05 December, 2002 HDF & HDF-EOS Workhop VI 5
  6. 6. 16,013,980 Data Products Delivered October 1, 2001 - September 30, 2002 05 December, 2002 HDF & HDF-EOS Workhop VI 6
  7. 7. Over 1.8 Million Distinct Users October 1, 2001 - September 30, 2002 05 December, 2002 HDF & HDF-EOS Workhop VI 7
  8. 8. ESDIS Status • After years of development, ECS is operational and generally recognized as successful. • Primary ECS development contract is essentially completed. • ECS Maintenance and Development (EMD) will emphasize maintenance more than development. • Already the majority of ESDIS budget has shifted from development to operations. • Still two major areas of new capability (see posters at AGU for more information): – Data Pools – ECHO 05 December, 2002 HDF & HDF-EOS Workhop VI 8
  9. 9. HDF-EOS Plans • HDF-EOS 5 development is nearly complete. • Continue to maintain, port to newer operating systems, bug-fix. • Need the advise of this community workshop - we will discuss this afternoon. – What tools or capabilities are now needed? • HDF-EOS 2 and HDF-EOS 5 – When is it the right time to press EOS science teams to migrate to HDF-EOS 5? – What steps should NASA take to facilitate? 05 December, 2002 HDF & HDF-EOS Workhop VI 9
  10. 10. Data Pools Concept: The Right Data, the Right Way, Right Now! User-defined Views, Presentations and Data Access Requests Data Items End Users Value-Added Providers Data Producers Other Data Pools Geo PIPE Other Data Pools Data Providers Data Tailoring Workflow Management Data Services Dynamic Web and FTP Data Views, User-specified Data Access PIPE = Personalized Information Presentation Engine 05 December, 2002 HDF & HDF-EOS Workhop VI 10 Data Service Developers
  11. 11. General Capabilities – The Right Data: Data Location • • • • • • Groups, themes, bookmarks and views Navigation and machine-based location Science views (e.g., science metadata) Applications views (e.g., OGIS coverage server) Location aids (e.g., geopolitical overlays) External location-support services – The Right Way: Data Tailoring • • • • Data reduction, manipulation and reformatting services Virtual data products Workflow management and execution monitoring External tailoring services – Right Now: Rapid Access • • • • • Low latency data transfers Secure remote file access On-the-fly data compression Automated request routing and load balancing Near real-time data 05 December, 2002 HDF & HDF-EOS Workhop VI 11
  12. 12. EOS ClearingHOuse (ECHO) http://eos.nasa.gov/echo • ECHO is a metadata clearinghouse – A single Internet portal for Earth science metadata search – Index of data provider inventory-level data holdings metadata. • .ECHO is a data order broker – Forwards orders for data discovered to the data providers to fill. – Data providers retain customer fulfillment service • ECHO is a data service broker – Registered service are associated with registered datasets – Four kinds of service association • Advertised, Context Passing, Brokered, Order Option • ECHO is an open client API for custom user clients – The EOS Data Gateway (EDG) is ESDIS’ ECHO client 05 December, 2002 HDF & HDF-EOS Workhop VI 12
  13. 13. ECHO Data Providers • Current ECHO Data Providers – EOSDIS Core System DAACs (17% thus far) • EDC Land Processes DAAC • Goddard DAAC • NSIDC DAAC – ORNL DAAC (100%) • ECHO holds the metadata for over 3 million granules, and growing ECHO Metadata Holdings 4,000,000 3,500,000 3,000,000 2,500,000 ORNL_DAAC NSIDC_ECS 2,000,000 LP_ECS GSFC_ECS 1,500,000 1,000,000 500,000 0 Jan-06 Feb- Mar-06 Apr-06 May- Jun-06 Jul-06 Aug06 06 06 Date 05 December, 2002 HDF & HDF-EOS Workhop VI 13 Sep06
  14. 14. HDF-EOS Tools and Information Web Site http://hdfeos.gsfc.nasa.gov Richard.E.Ullman@nasa.gov 05 December, 2002 HDF & HDF-EOS Workhop VI 14
  15. 15. http://hdfeos.gsfc.nasa.gov 05 December, 2002 HDF & HDF-EOS Workhop VI 15
  16. 16. Website status • A resource for discovering about hdf-eos in particular. – This workshop series’ presentation archive – Links to hdf-eos tools • Site has been revamped according to comments received at the last workshop. – Tools download page now has opportunity for user feedback. – Workshop presentations are keyword searchable • New features planned – Better introductory material. • Post and organize documentation of HDF-EOS. • Better navigation to NCSA site for HDF – Host hdf-eos “web forum” • Incorporate the eostools@eos.nasa.gov listserv 05 December, 2002 HDF & HDF-EOS Workhop VI 16
  17. 17. HDF-EOS Profile Richard Ullman ESDIS Project Richard.E.Ullman@nasa.gov 05 December, 2002 HDF & HDF-EOS Workhop VI 17
  18. 18. History • September 1993, HDF adopted as baseline standard for EOSDIS Core System standard data product generation, archival, ingest, and distribution capabilities • Dec. 94 - ECS Engineering Support Directive to create HDF-EOS • June 1996, HDF-EOS v1.0 library released • Upgrades every 6 mo., – Current version 2.8 on HDF 4 • HDF5 support (called HDF-EOS 5) beginning November 2000 – Current version 5.1.3 on HDF5 05 December, 2002 HDF & HDF-EOS Workhop VI 18
  19. 19. HDF-EOS Data Objects • • • • • Point Swath Profile (Swath subtype) Grid Zonal (HDF-EOS 5 only) 05 December, 2002 HDF & HDF-EOS Workhop VI 19
  20. 20. HDF-EOS 5 • Based on HDF5, a complete rewrite of HDF4 with a different interface. – First released in 2000. • Designed to ‘resemble’ HDF-EOS 2 to the maximum extent possible. – – – – Support same data structures Added prefix ‘HE5_’ to HDF-EOS 2 functions. Doesn’t preclude HDF5 functionality. Data Type changes, e.g. INT64 -> H5T_NATIVE_LONG 05 December, 2002 HDF & HDF-EOS Workhop VI 20
  21. 21. HDF-EOS 5 Functionality • • • • • • • • Basic File I/O Fill Values Compression Chunking/Tiling Swath Interface Grid Interface Point Interface Profile Interface • • • • • • • Global (File), Group & Local Attributes External Data Files Subsetting Unix/Linux Support Threadsafe Version FORTRAN, C, C++ General Table Interface (proposed) 05 December, 2002 HDF & HDF-EOS Workhop VI 21
  22. 22. Top Level of HDF-EOS 5 Root -- “/” HDFEOS INFORMATION HDFEOS STRUC. METADATA ADDITIONAL SWATH GRID POINT Global (file) Attributes The new ADDITIONAL Group has global (file) attributes The new functionality is added to the EH(utility) interface. 05 December, 2002 HDF & HDF-EOS Workhop VI 22
  23. 23. Swath Structure Global Attribute <SwathName>:<AttrName> Group Attribute <DataFields>:<AttrName> Local Attribute <FieldName>:<AttrName> SwathName Data Fields Data Data Field.1 Field.n Profile Fields Profile Field.1 Profile Field.n Geolocation Fields Longitude Latitude Time Each Data Field can have Attributes and/or Dimension Scales CoLatitude Shaded Objects are implemented in a fixed way. User doesn’t have direct access via the interface Group Attribute 05 December, 2002 HDF & HDF-EOS Workhop VI 23 Dataset
  24. 24. HDF-EOS Point • Intended use – Discrete points in time and/or location. – Table of data linked to table of geographic information. – 8-level Hierarchical, each level may contain indices to the level below Latitude Longitude Temperature oC Dew Point o C 61.12 -149.48 15.00 5.00 45.31 -122.41 17.00 5.00 38.50 -77.00 24.00 7.00 38.39 -90.15 27.00 11.00 M M M M 05 December, 2002 HDF & HDF-EOS Workhop VI 24
  25. 25. HDF-EOS Point Latitude 25.2645 22.3549 23.2564 Longitude 091.2564 -93.4657 -89.2546 Buoy_ID 0126 3564 1256 • Hierarchical links: – Every level in a Point data set must be linked into the hierarchy. – Before two levels can be linked, a link field must exist Buoy _ID 0126 0126 3564 1256 1256 0126 3564 Time 01:26 05:56 06:28 08:12 09:58 09:59 10:16 Wave Height(ft) 2.54 3.58 12.64 7.58 7.76 4.23 10.23 05 December, 2002 HDF & HDF-EOS Workhop VI 25 Temp (C) 18.4 18.2 16.4 17.1 17.2 20.1 17.5
  26. 26. HDF-EOS Swath • Intended use – Across track scanning instruments. – Sounding instruments – Level 1: Geolocated Sensor Units – Level 2: Geophysical Parameters Scan Lines Instrument Path Along Track Instrument Path Along Track 05 December, 2002 HDF & HDF-EOS Workhop VI 26
  27. 27. HDF-EOS Swath Data Fields “Brightness Temperature” Geolocation Fields Dimension Dimension Name: Scan Name: Scan Size: 16 Size: 16 Dimension Dimension Name: Track Name: Track Size: 42 Size: 42 Dimension Dimension Name: Geotrack Name: Geotrack Size: 21 Size: 21 “Time” “Latitude” “Longitude” Map1 Map1 DataDimension: “Track” DataDimension: “Track” Geodimension: “Geotrack” Geodimension: “Geotrack” Offset: 1 Offset: 1 Increment: 2 Increment: 2 05 December, 2002 HDF & HDF-EOS Workhop VI 27
  28. 28. HDF-EOS Grid • Intended use – Variables mapped on uniform space-time grid scales – Level 3 - Gridded single measurement parameters – Level 4 - Modeled or derived from multiple measurements 05 December, 2002 HDF & HDF-EOS Workhop VI 28
  29. 29. HDF-EOS Grid Altitude Size 30 Xdim Size: 2000 Projinfo Ydim Size: 800 05 December, 2002 HDF & HDF-EOS Workhop VI 29
  30. 30. HDF-EOS Grid • • • • • • • • Projections Supported Geographic Transverse Mercator Universal Transverse Mercator Hotine Oblique Mercator Space Oblique Mercator Polar Stereographic Lambert Azimuthal Equal Area • • • • • • • • Lambert Conformal Conic Polyconic Interrupted Goode’s Homolosine Integerized Sinusoidal Compression Methods Run-Length Encoding Adaptive Huffman Gzip 05 December, 2002 HDF & HDF-EOS Workhop VI 30
  31. 31. Product Levels • Level 0 - Reconstructed, unprocessed instrument/payload data at full resolution; any and all communications artifacts, e.g., synch. frames, communications headers, duplicate data removed. • Level 1A- Reconstructed, unprocessed instrument data at full resolution, time-referenced, and annotated with ancillary information, including radiometric and geometric calibration coefficients and georeferencing parameters, e.g., platform ephemeris, computed and appended but not applied to the Level 0 data. • • • • Level 1B - Level 1A data that have been processed to sensor units (not all instruments will have a Level 1B equivalent). Level 2 - Derived geophysical variables at the same resolution and location as the Level 1 source data. Level 3 - Variables mapped on uniform space-time grid scales, usually with some completeness and consistency. Level 4 - Model output or results from analyses of lower level data, e.g., variables derived from multiple measurements. 05 December, 2002 HDF & HDF-EOS Workhop VI 31
  32. 32. Resources – HDF-EOS on the web: • http://hdfeos.gsfc.nasa.gov/ • http://newsroom.gsfc.nasa.gov/sdptoolkit/toolkit.html – HDF and HDF5 on the web: • http://hdf.ncsa.uiuc.edu/ – HDF-EOS and HDF via email: • eostools@eos.nasa.gov • hdfhelp@ncsa.uiuc.edu 05 December, 2002 HDF & HDF-EOS Workhop VI 32

×