Access HDF5 Datasets via OPeNDAP's Data Access Protocol (DAP)

137
-1

Published on

The Data Access Protocol(DAP) is a protocol for requesting and transporting data across the web. By using OPeNDAP servers data can be made accessible to remote locations regardless of local storage format. A prototyped HDF5-OPeNDAP server was implemented several years ago. With the HDF5 being selected as the primary data format of NASA Aura satellite, a robust, fully supported HDF5-OPeNDAP server is necessary.

Recently NASA Headquarters Science Mission Directorate Advancing Collaborative Connections For Earth-Sun System Science (ACCESS) program has awarded a joint proposal to implement a fully supported HDF5-OPeNDAP server. This poster will discuss the design and implementation plan for this project.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
137
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • I will introduce a new joint project between the HDF Group and OPeNDAP. The goal of this project is to provide a server and utility so that people can have access to HDF5 Datasets via OPeNDAP.
  • First a brief introductions of OPeNDAP.
  • Access HDF5 Datasets via OPeNDAP's Data Access Protocol (DAP)

    1. 1. Access HDF5 Datasets via OPeNDAP’s Data Access Protocol (DAP) MuQun Yang The HDF Group 02/18/14 HDF and HDF-EOS Workshop X, Landover, MD 1
    2. 2. OPeNDAP • • • • • • A software framework that allows simple access to remote data Client-server model Servers make the local data accessible remotely Clients make applications easily access to remote served data Widely used by Earth Science Community URL: http://www.opendap.org/ 02/18/14 HDF and HDF-EOS Workshop X, Landover, MD 2
    3. 3. The Data Request  Data are requested with a URL. http://www.cdc.noaa.gov/cgi-bin/nph-nc/datasets/Reynolds_sst ?sst[10:10][0:90][0:180] Protocol Machine name OPeNDAP server Directory File name Constraint  The user can impose a constraint on the data that s/he would like to acquire from a data set by appending a constraint expression to the end of the URL. 02/18/14 HDF and HDF-EOS Workshop X, Landover, MD 3
    4. 4. OPeNDAP Data Types OPeNDAP provides three basic data objects:  data descriptor structure (.dds) - syntactic metadata - rigid  data attribute structure (.das) - semantic metadata - flexible  data (.dods) - the actual data in a binary structure 02/18/14 HDF and HDF-EOS Workshop X, Landover, MD 4
    5. 5. OPeNDAP Servers CODAR netCDF HDF4 Data Data CODAR netCDF Data Matlab DSP Tables SQL FITS CDF Flat Binary Data Data Data Data Data Data Data JGOFS Matlab HDF4 DSP FITS JDBC CEDAR General Data Data ESML FreeFrom CDF CEDAR No fully-supported HDF5 server 02/18/14 HDF and HDF-EOS Workshop X, Landover, MD 5
    6. 6. OPeNDAP Clients netCDF C Ferret GrADS netCDF Java IDV VisAD ncBrowse Matlab Client IDL Client Matlab IDL Access Excel No HDF5 client 02/18/14 HDF and HDF-EOS Workshop X, Landover, MD 6
    7. 7. Why HDF5-OPeNDAP server and client? Users! Users!! Users!!! 02/18/14 HDF and HDF-EOS Workshop X, Landover, MD 7
    8. 8. A User Case Task: •10 elements of 17 MB HDF5 data •200 elements of 10 MB NetCDF data •150 elements of 5 MB HDF4 data 02/18/14 HDF and HDF-EOS Workshop X, Landover, MD 8
    9. 9. General approach HDF5 data FTP HDF4 data FTP NetCDF data FTP 02/18/14 HDF and HDF-EOS Workshop X, Landover, MD 9
    10. 10. Approach with OPeNDAP HDF5 data FTP HDF4 data server client AP D NetCDF data DA P server 02/18/14 HDF and HDF-EOS Workshop X, Landover, MD 10
    11. 11. Conclusion • HDF5-DAP server is required • HDF5-DAP client is strongly recommended 02/18/14 HDF and HDF-EOS Workshop X, Landover, MD 11
    12. 12. Previous work - A Prototype of DAP2-HDF5 server • • • • Incomplete mapping from HDF5 to OPeNDAP No comprehensive test suite Only tested a few examples on two platforms Incomplete documentation 02/18/14 HDF and HDF-EOS Workshop X, Landover, MD 12
    13. 13. An example from SSMI From HDF4 server at JPL 02/18/14 From HDF5 server at NCSA HDF and HDF-EOS Workshop X, Landover, MD 13
    14. 14. Current HDF5-OPeNDAP Support HDF5 DAP No Client 02/18/14 Prototyped Server HDF and HDF-EOS Workshop X, Landover, MD 14
    15. 15. HDF5-OPeNDAP Project • Two-year project awarded by NASA ROSES NRA • Funding much less than expected • Original proposed work kept • with significant contributions from ESDIS • Starting date: Nov. 15th, 2006 02/18/14 HDF and HDF-EOS Workshop X, Landover, MD 15
    16. 16. Proposed Work – software development • Implementation • • • HDF5-DAP2 server HDF5-DAP4 server Deployment • 02/18/14 Release servers to OPeNDAP community HDF and HDF-EOS Workshop X, Landover, MD 16
    17. 17. HDF5-DAP2 server • • Update and complete the prototype, including make files, instructions, documentation, etc Add a few selected features to handle NASA Aura data and NPOESS data - • • • Map HDF5 group to DAP(?) Map Object/regional reference to DAP(?) Implement a comprehensive test suite Test on platforms supported by OPeNDAP Demonstrate with sample NASA data 02/18/14 HDF and HDF-EOS Workshop X, Landover, MD 17
    18. 18. Fully-Supported HDF5 Server HDF5 server HDF5 DAP HDF5 server Matlab/Ferret Client HDF5 DAP 02/18/14 HDF and HDF-EOS Workshop X, Landover, MD 18
    19. 19. Proposed Work – software development • Implementation • • DAP4 to HDF5 conversion utility Deployment • • Release utility to OPeNDAP community Design study • 02/18/14 Feasibility of DAP-aware HDF5 library HDF and HDF-EOS Workshop X, Landover, MD 19
    20. 20. Proposed Work - DAP to HDF5 utility • A command line tool that converts DAP to HDF5 file • • individual DAP variable subsetting Conversion tool HDF5 server HDF5 DAP 02/18/14 HDF and HDF-EOS Workshop X, Landover, MD 20
    21. 21. Proposed work - documentations • • • • • DAP4 specification HDF5-DAP4 mapping document DAP4 to HDF5 specification RFC for DAP-aware HDF5 library Submit specifications to ESDSWG 02/18/14 HDF and HDF-EOS Workshop X, Landover, MD 21
    22. 22. Acknowledgement This work was supported basing upon the Cooperative Agreement with the National Aeronautics and Space Administration (NASA) under NASA grant NNG05GC60A.  Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NASA.  02/18/14 HDF and HDF-EOS Workshop X, Landover, MD 22

    ×