Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
The HDF Group

HDF-OPeNDAP Project
Update
Joe Lee and Kent Yang
The HDF Group
James Gallagher
OPeNDAP, Inc.
The 13th HDF a...
Outline

• Introduction to OPeNDAP
• What’s New?
o HDF5-OPeNDAP Handler (minor)
o HDF4-OPeNDAP Handler (major)

November 3...
Access HDF via FTP
Aqua/Aura
Users

IDL

AIRS/OMI
FTP

Local HDF
Files
November 3-5, 2009

FTP
Clients

FTP
Servers
HDF/HD...
OPeNDAP Concept
Typical Remote Access

View Data
(User)

FTP/HTTP

Remote Data
(HDF4/5)

Visualization
Tools
tools
(e.g., ...
Access HDF via OPeNDAP
Aqua/Aura

Users
IDV

HyraxHANDLER
HDF
AIRS/OMI

libnc-dap
DAP

Visualization
Tools
November 3-5, 2...
What’s New?

HDF5-OPeNDAP Handler Work

November 3-5, 2009

HDF/HDF-EOS Workshop XIII

6

www.hdfgroup.org
HDF5 Handler Update
Aura

Users
IDV

HDF5 HANDLER
Update
Hyrax
OMI Swath
Grid

libnc-dap
DAP

Visualization
Tools
November...
HDF5-OPeNDAP Handler Update

• Support Aura HDF-EOS5 OMI Swath
• Access some files that follow HDF5
Dimension Scale Specif...
Summary: Updated HDF5 Handler

• Official Release: October 6, 2009
• http://opendap.org/hdf5_handler

November 3-5, 2009

...
What’s New?

HDF4-OPeNDAP Handler Work

November 3-5, 2009

HDF/HDF-EOS Workshop XIII

10

www.hdfgroup.org
HDF4-OPeNDAP Handler Work

• Problems with the old HDF4 Handler
• Our Work
• Limitations
• Future Work

November 3-5, 2009...
Old HDF4 Handler Problem

• OPeNDAP Visualization Tools CAN
NOT display most NASA HDF-EOS2
and HDF4 data

November 3-5, 20...
Old HDF4 Handler Problem
Aqua

Users
IDV

Hyrax
Old HDF4 HANDLER
AIRS Grid

libnc-dap
DAP

Visualization
Tools
November 3-...
Our Work
• Let Visualization Clients Access HDFEOS2 and HDF4 files

November 3-5, 2009

HDF/HDF-EOS Workshop XIII

14

www...
What’s Missing?
• My house costs 1.
• Data – Computer Science
• int my_house = 1; // Use boolean to save?

• My house in H...
What’s Missing?
•

Clients expect certain data structures
o DAP Grids and Shared dimensions

• Clients expect some key att...
Solution: Metadata Parser

• Goal: Meet the Client’s Requirements
• Parse the structMetadata to
o generate the DAP Grids a...
New HDF4 Handler (parser)
Aqua

Users
IDV

libnc-dap

New HDF4 HANDLER (parser)
Hyrax
Old HDF4 HANDLER
AIRS Grid
DAP

Visu...
Demo: Parser Solution

Problem: Only a few HDF-EOS2 files work!

AIRS Grid
AIRS Swath MODIS Swath
MODIS Grid
AIRS.2008.10....
New Challenges for HDF4 Handler

• Hundreds of HDF-EOS2 and HDF4 products
o Grid Projections
o Swath Dimension Map

Novemb...
New Challenges
HDF-EOS2 Grid projections
•
•
•
•

Geographic
Sinusoidal
Polar Stereographic
etc.

HDF-EOS2 Swath Dimension...
Solution: Use HDF-EOS2 Library

• Goal: Support more HDF-EOS2 products
• Use HDF-EOS2 Library to
o handle different Grid p...
New HDF4 Handler (Parser)
New HDF4 Handler (EOS2 library)
New HDF4 HANDLER (parser)
(EOS2 library)

Aura

Users
IDV

MODIS...
Demo: HDF-EOS2 Library Solution

Swath with Dimension Map
November 3-5, 2009

HDF/HDF-EOS Workshop XIII

24

www.hdfgroup....
Two Solutions

Metadata Parser

HDF-EOS2 Library

• Pro: Simple installation

• Pro: More products

• Con: Limited product...
Two Solutions - Implementation
• Adapted the implementation of previous
software packages
Metadata Parser

HDF-EOS2 Librar...
Solution Comparison

November 3-5, 2009

HDF/HDF-EOS Workshop XIII

27

www.hdfgroup.org
Live Demo
• Tutorial Example
o http://hdfeos.org/software/idv.php
• Demo Servers
o http://hdfgroup.org/projects/opendap
o ...
Limitations

• 7 Reasons for Visualization Failures
• Sample Files for Visualization Failures
o http://hdfgroup.org/projec...
The Tip of the Iceberg
The current HDF4 handler

CAN NOT visualize most of NASA HDF-EOS2 / HDF4 files

November 3-5, 2009
...
More Challenges for HDF4 Handler

• Tackle limitations and other challenges.
• Handle NASA HDF4 (e.g.,CERES) files.
• In 2...
Summary: Enhanced HDF4 Handler
• Official Release on Oct. 6, 2009.
• http://opendap.org/hdf4_handler
• Support some HDF-EO...
The HDF Group

Thank You!

November 3-5, 2009

HDF/HDF-EOS Workshop XIII

33

www.hdfgroup.org
Acknowledgements
This work was supported by cooperative agreement
number NNX06AC83A, NNX08AO77A and
NNX06AG75A from the Na...
The HDF Group

Questions/comments?

November 3-5, 2009

HDF/HDF-EOS Workshop XIII

35

www.hdfgroup.org
Upcoming SlideShare
Loading in …5
×

HDF OPeNDAP project update and demo

391 views

Published on

In this presentation, we will give an update on the HDF OPeNDAP project. We will update the new features inside the HDF5 OPeNDAP data handler. We will also introduce the enhanced HDF4 OPeNDAP data handler and demonstrate how it can help users to view and analyze remote HDF-EOS2 data. A demo that uses OPeNDAP client tools to handle AIRS and MODIS Grid/Swath data with the enhanced handler will be presented.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

HDF OPeNDAP project update and demo

  1. 1. The HDF Group HDF-OPeNDAP Project Update Joe Lee and Kent Yang The HDF Group James Gallagher OPeNDAP, Inc. The 13th HDF and HDF-EOS Workshop November 3-5, 2009 November 3-5, 2009 HDF/HDF-EOS Workshop XIII 1 www.hdfgroup.org
  2. 2. Outline • Introduction to OPeNDAP • What’s New? o HDF5-OPeNDAP Handler (minor) o HDF4-OPeNDAP Handler (major) November 3-5, 2009 HDF/HDF-EOS Workshop XIII 2 www.hdfgroup.org
  3. 3. Access HDF via FTP Aqua/Aura Users IDL AIRS/OMI FTP Local HDF Files November 3-5, 2009 FTP Clients FTP Servers HDF/HDF-EOS Workshop XIII HDF Files 3 www.hdfgroup.org
  4. 4. OPeNDAP Concept Typical Remote Access View Data (User) FTP/HTTP Remote Data (HDF4/5) Visualization Tools tools (e.g., grads) (gradsdap) Handler (hdf4/5_handler) Client Library (libnc_dap) Server (Hyrax) Data Access Protocol (DAP) via Internet November 3-5, 2009 HDF/HDF-EOS Workshop XIII 4 www.hdfgroup.org
  5. 5. Access HDF via OPeNDAP Aqua/Aura Users IDV HyraxHANDLER HDF AIRS/OMI libnc-dap DAP Visualization Tools November 3-5, 2009 OPeNDAP Clients OPeNDAP Servers HDF/HDF-EOS Workshop XIII HDF Files 5 www.hdfgroup.org
  6. 6. What’s New? HDF5-OPeNDAP Handler Work November 3-5, 2009 HDF/HDF-EOS Workshop XIII 6 www.hdfgroup.org
  7. 7. HDF5 Handler Update Aura Users IDV HDF5 HANDLER Update Hyrax OMI Swath Grid libnc-dap DAP Visualization Tools November 3-5, 2009 OPeNDAP Clients OPeNDAP Servers HDF/HDF-EOS Workshop XIII HDF5 Files 7 www.hdfgroup.org
  8. 8. HDF5-OPeNDAP Handler Update • Support Aura HDF-EOS5 OMI Swath • Access some files that follow HDF5 Dimension Scale Specification • Change in Short Name Option RadiativeCloudFraction A3RadiativeClou November 3-5, 2009 HDF/HDF-EOS Workshop XIII 8 www.hdfgroup.org
  9. 9. Summary: Updated HDF5 Handler • Official Release: October 6, 2009 • http://opendap.org/hdf5_handler November 3-5, 2009 HDF/HDF-EOS Workshop XIII 9 www.hdfgroup.org
  10. 10. What’s New? HDF4-OPeNDAP Handler Work November 3-5, 2009 HDF/HDF-EOS Workshop XIII 10 www.hdfgroup.org
  11. 11. HDF4-OPeNDAP Handler Work • Problems with the old HDF4 Handler • Our Work • Limitations • Future Work November 3-5, 2009 HDF/HDF-EOS Workshop XIII 11 www.hdfgroup.org
  12. 12. Old HDF4 Handler Problem • OPeNDAP Visualization Tools CAN NOT display most NASA HDF-EOS2 and HDF4 data November 3-5, 2009 HDF/HDF-EOS Workshop XIII 12 www.hdfgroup.org
  13. 13. Old HDF4 Handler Problem Aqua Users IDV Hyrax Old HDF4 HANDLER AIRS Grid libnc-dap DAP Visualization Tools November 3-5, 2009 OPeNDAP Clients OPeNDAP Servers HDF/HDF-EOS Workshop XIII HDF4 Files 13 www.hdfgroup.org
  14. 14. Our Work • Let Visualization Clients Access HDFEOS2 and HDF4 files November 3-5, 2009 HDF/HDF-EOS Workshop XIII 14 www.hdfgroup.org
  15. 15. What’s Missing? • My house costs 1. • Data – Computer Science • int my_house = 1; // Use boolean to save? • My house in Hawaii costs $1M. • Information – Earth Science • Geo-Location, Geo-Location, Geo-Location • Unit, Scale factor November 3-5, 2009 HDF/HDF-EOS Workshop XIII 15 www.hdfgroup.org
  16. 16. What’s Missing? • Clients expect certain data structures o DAP Grids and Shared dimensions • Clients expect some key attributes o CF conventions November 3-5, 2009 HDF/HDF-EOS Workshop XIII 16 www.hdfgroup.org
  17. 17. Solution: Metadata Parser • Goal: Meet the Client’s Requirements • Parse the structMetadata to o generate the DAP Grids and Shared Dims o generate the CF-compliant attributes November 3-5, 2009 HDF/HDF-EOS Workshop XIII 17 www.hdfgroup.org
  18. 18. New HDF4 Handler (parser) Aqua Users IDV libnc-dap New HDF4 HANDLER (parser) Hyrax Old HDF4 HANDLER AIRS Grid DAP Visualization Tools November 3-5, 2009 OPeNDAP Clients OPeNDAP Servers HDF/HDF-EOS Workshop XIII HDF4 Files 18 www.hdfgroup.org
  19. 19. Demo: Parser Solution Problem: Only a few HDF-EOS2 files work! AIRS Grid AIRS Swath MODIS Swath MODIS Grid AIRS.2008.10.27.L3.RetStd001.v5.2.2.0.G08303124144.hdf ftp://airspar1u.ecs.nasa.gov/data/s4pa//Aqua_AIRS_Level2/AIRX2RET.005//2008/330/AIRS.2008.11.25.143.L2.RetStd.v5.2.2.0.G0 http://daac.gsfc.nasa.gov/daac-bin/whom/mk_page_cgi.pl?PATH=datapool/AIRS/Level3_V005/01_AIRX3STD/2008/10 MOD08_D3.A2008284.005.2008289105354.hdf MOD06_L2.A2001222.0025.004.2003127111841.hdf 8331121849.hdf http://modis-atmos.gsfc.nasa.gov/MOD08_D3/acquiring.html XIII November 3-5, 2009 HDF/HDF-EOS Workshop 19 www.hdfgroup.org http://modis-atmos.gsfc.nasa.gov/MOD06_L2/acquiring.html
  20. 20. New Challenges for HDF4 Handler • Hundreds of HDF-EOS2 and HDF4 products o Grid Projections o Swath Dimension Map November 3-5, 2009 HDF/HDF-EOS Workshop XIII 20 www.hdfgroup.org
  21. 21. New Challenges HDF-EOS2 Grid projections • • • • Geographic Sinusoidal Polar Stereographic etc. HDF-EOS2 Swath Dimension Map November 3-5, 2009 HDF/HDF-EOS Workshop XIII 21 www.hdfgroup.org
  22. 22. Solution: Use HDF-EOS2 Library • Goal: Support more HDF-EOS2 products • Use HDF-EOS2 Library to o handle different Grid projections o handle Swath dimension map November 3-5, 2009 HDF/HDF-EOS Workshop XIII 22 www.hdfgroup.org
  23. 23. New HDF4 Handler (Parser) New HDF4 Handler (EOS2 library) New HDF4 HANDLER (parser) (EOS2 library) Aura Users IDV MODIS Swath w/ Dim. Map MODIS Swath Hyrax libnc-dap DAP Visualization Tools November 3-5, 2009 OPeNDAP Clients OPeNDAP Servers HDF/HDF-EOS Workshop XIII HDF4 Files 23 www.hdfgroup.org
  24. 24. Demo: HDF-EOS2 Library Solution Swath with Dimension Map November 3-5, 2009 HDF/HDF-EOS Workshop XIII 24 www.hdfgroup.org
  25. 25. Two Solutions Metadata Parser HDF-EOS2 Library • Pro: Simple installation • Pro: More products • Con: Limited products • Con: HE2 library installation November 3-5, 2009 HDF/HDF-EOS Workshop XIII 25 www.hdfgroup.org
  26. 26. Two Solutions - Implementation • Adapted the implementation of previous software packages Metadata Parser HDF-EOS2 Library • hdf5_handler • h4toh5 tool November 3-5, 2009 HDF/HDF-EOS Workshop XIII 26 www.hdfgroup.org
  27. 27. Solution Comparison November 3-5, 2009 HDF/HDF-EOS Workshop XIII 27 www.hdfgroup.org
  28. 28. Live Demo • Tutorial Example o http://hdfeos.org/software/idv.php • Demo Servers o http://hdfgroup.org/projects/opendap o http://hdfgroup.org/projects/opendap4 November 3-5, 2009 HDF/HDF-EOS Workshop XIII 28 www.hdfgroup.org
  29. 29. Limitations • 7 Reasons for Visualization Failures • Sample Files for Visualization Failures o http://hdfgroup.org/projects/opendap4/limits.html November 3-5, 2009 HDF/HDF-EOS Workshop XIII 29 www.hdfgroup.org
  30. 30. The Tip of the Iceberg The current HDF4 handler CAN NOT visualize most of NASA HDF-EOS2 / HDF4 files November 3-5, 2009 HDF/HDF-EOS Workshop XIII 30 www.hdfgroup.org
  31. 31. More Challenges for HDF4 Handler • Tackle limitations and other challenges. • Handle NASA HDF4 (e.g.,CERES) files. • In 2010 November 3-5, 2009 HDF/HDF-EOS Workshop XIII 31 www.hdfgroup.org
  32. 32. Summary: Enhanced HDF4 Handler • Official Release on Oct. 6, 2009. • http://opendap.org/hdf4_handler • Support some HDF-EOS2 Grid / Swath files • Use HDF-EOS2 library • Future work: support more NASA HDF-EOS2 / HDF4 files November 3-5, 2009 HDF/HDF-EOS Workshop XIII 32 www.hdfgroup.org
  33. 33. The HDF Group Thank You! November 3-5, 2009 HDF/HDF-EOS Workshop XIII 33 www.hdfgroup.org
  34. 34. Acknowledgements This work was supported by cooperative agreement number NNX06AC83A, NNX08AO77A and NNX06AG75A from the National Aeronautics and Space Administration (NASA). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author[s] and do not necessarily reflect the views of the National Aeronautics and Space Administration. November 3-5, 2009 HDF/HDF-EOS Workshop XIII 34 www.hdfgroup.org
  35. 35. The HDF Group Questions/comments? November 3-5, 2009 HDF/HDF-EOS Workshop XIII 35 www.hdfgroup.org

×