Your SlideShare is downloading. ×
0
HDF5 OPeNDAP Project
Update and Demo
MuQun Yang and Hyo-Kyung Lee
(The HDF Group)
James Gallagher
(OPeNDAP, Inc.)
10/17/20...
OPeNDAP Review
• A software framework that allows
simple access to remote data
• Data Access Protocol
• Client-server mode...
OPeNDAP Concept
Typical Remote Access

View Data
(User)

FTP/HTTP

Remote Data
(HDF4)

Visualization
Tools
tools
(e.g. gra...
OPeNDAP HDF5 Handler

View Data
(User)

Remote HDF5

HDF5 Handler

Server
(apache)
DAP Protocol (via http)
10/17/2008

HDF...
Challenges
Mapping HDF5 to DAP
• Compound Datatype

• Groups
• Object/Regional References

10/17/2008

HDF and HDF-EOS Wor...
OPeNDAP HDF5 Handler with HDF-EOS5

View Data
(User)

Remote
HDF-EOS5
HDF5

Visualization
tools
(grads)

HDF5 Handler

Cli...
Challenges: HDF- EOS5
• Grid with No Geolocation data
 Clients expect Grid with Geolocation data
• Some Attributes stored...
Tweaks for HDF-EOS5
Added two default HDF5 handler
configuration options:
• --enable-eos-grid
• --enable-eos-meta

10/17/2...
Challenges: OPeNDAP Clients
• Need special attributes on dataset.
• Need shared geolocation variables.

10/17/2008

HDF an...
Tweaks for OPeNDAP Clients
Added two optional handler
configuration options:
• --enable-short-name
• --enable-CF

10/17/20...
OPeNDAP HDF5 Handler with HDF-EOS5

View Data
(User)

Remote
HDF-EOS5
HDF5

Visualization
tools
(grads)

HDF5 Handler
w/ C...
Day After Server Tweaks
• Finally, Happy Clients!

10/17/2008

HDF and HDF-EOS Workshop XII

12
Problems of Tweaking HDF5 Handler
Remember that we added two optional
HDF5 handler configuration options:

1. --enable-sho...
HDF5-Friendly OPeNDAP Client Library
View???
Groups

HDF5 Groups

View Data
(User)

Remote
HDF5

Visualization
tools
(grad...
Traditional Groups
Example:OPeNDAP clientin HDF5
library: It’s an attribute that I
don’t understand. I’ll ignore it.

HDF5...
Example: Reference in HDF5

• Important for NPOESS
• Object / Regional Reference
• Map to DAP URL

10/17/2008

HDF and HDF...
Example: dap2h5
View Group/Ref.

Group/Ref.

View Data
(User)

Remote
HDF5

• HDF5-F. Client
A test application for the cl...
Our Client Library(prototype)
• One more reason:
Help Clients to view Swath
Visualizing HDF-EOS5 Swath Problem
Visualizing HDF-EOS5 Grids
View Swath
???

Swath

View Data
(User)

Remote
HDF-EOS5
HDF...
Demo: MLS swath via GrADS
• GrADS coupled with our client
library

• The client library provides grid
mapping from swath d...
Summary of Client Library (Prototype)

• Finished coding
• Tested with the GrADS visualization
client
• Documented
• Worki...
Caution
• Our Client Library is ONLY A PROTOTYPE!
 It does NOT support all DAP data types
 It does NOT support all AURA ...
Summary
• HDF5 Access via OPeNDAP
 Is Easy
 Is used by GES DISC to serve Aura
files
• HDF5-Friendly OPeNDAP Client Libra...
Future Work
• HDF5 to DAP2 Mapping Document
• Release HDF5-friendly OPeNDAP
Client Library (Prototype)
• URL: http://hdfgr...
Credits

Mike Folk (The HDF Group)
Robert McGrath (NCSA)
Peter Leonard, Daniel Kahn, Marghi Hopkins (ADNET)
Christopher Ly...
Acknowledgement
This work was supported basing upon the
Cooperative Agreement with the National
Aeronautics and Space Admi...
Upcoming SlideShare
Loading in...5
×

HDF5 OPeNDAP project update and demo

101

Published on

In this talk, we will give an update on the HDF5 OPeNDAP project. We will update the new features inside OPeNDAP HDF5 data handler. We will also introduce a new HDF5-Friendly OPeNDAP client library and demonstrate how it can help users to view and analyze remote HDF-EOS5 data served by OPeNDAP HDF5 handler. A demo will be presented with a customized OPeNDAP visualization client (GrADS) that uses the library.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
101
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Here’s a summary of our effort regarding client library.
  • Transcript of "HDF5 OPeNDAP project update and demo"

    1. 1. HDF5 OPeNDAP Project Update and Demo MuQun Yang and Hyo-Kyung Lee (The HDF Group) James Gallagher (OPeNDAP, Inc.) 10/17/2008 HDF and HDF-EOS Workshop XII 1
    2. 2. OPeNDAP Review • A software framework that allows simple access to remote data • Data Access Protocol • Client-server model • Widely used by Earth Science Community 10/17/2008 HDF and HDF-EOS Workshop XII 2
    3. 3. OPeNDAP Concept Typical Remote Access View Data (User) FTP/HTTP Remote Data (HDF4) Visualization Tools tools (e.g. grads) (gradsdap) Handler (hdf4_handler) Client Library (libnc-dap) Server (apache) DAP Protocol (via http) 10/17/2008 HDF and HDF-EOS Workshop XII 3
    4. 4. OPeNDAP HDF5 Handler View Data (User) Remote HDF5 HDF5 Handler Server (apache) DAP Protocol (via http) 10/17/2008 HDF and HDF-EOS Workshop XII 4
    5. 5. Challenges Mapping HDF5 to DAP • Compound Datatype • Groups • Object/Regional References 10/17/2008 HDF and HDF-EOS Workshop XII 5
    6. 6. OPeNDAP HDF5 Handler with HDF-EOS5 View Data (User) Remote HDF-EOS5 HDF5 Visualization tools (grads) HDF5 Handler Client Library (libnc-dap) Server (apache) DAP Protocol (via http) 10/17/2008 HDF and HDF-EOS Workshop XII 6
    7. 7. Challenges: HDF- EOS5 • Grid with No Geolocation data  Clients expect Grid with Geolocation data • Some Attributes stored as Extremely Long String(s) (e.g. StructMetada.0)  Clients expect structured attributes 10/17/2008 HDF and HDF-EOS Workshop XII 7
    8. 8. Tweaks for HDF-EOS5 Added two default HDF5 handler configuration options: • --enable-eos-grid • --enable-eos-meta 10/17/2008 HDF and HDF-EOS Workshop XII 8
    9. 9. Challenges: OPeNDAP Clients • Need special attributes on dataset. • Need shared geolocation variables. 10/17/2008 HDF and HDF-EOS Workshop XII 9
    10. 10. Tweaks for OPeNDAP Clients Added two optional handler configuration options: • --enable-short-name • --enable-CF 10/17/2008 HDF and HDF-EOS Workshop XII 10
    11. 11. OPeNDAP HDF5 Handler with HDF-EOS5 View Data (User) Remote HDF-EOS5 HDF5 Visualization tools (grads) HDF5 Handler w/ CF options Client Library (libnc-dap) Server (apache) DAP Protocol (via http) 10/17/2008 HDF and HDF-EOS Workshop XII 11
    12. 12. Day After Server Tweaks • Finally, Happy Clients! 10/17/2008 HDF and HDF-EOS Workshop XII 12
    13. 13. Problems of Tweaking HDF5 Handler Remember that we added two optional HDF5 handler configuration options: 1. --enable-short-name 2. --enable-CF Causes ambiguity among variable names (e.g. /GroupA/ozone vs. /GroupB/ozone) Drops some key attributes (e.g. StructMetdata, HDF_ROOT_GROUP) 10/17/2008 HDF and HDF-EOS Workshop XII 13
    14. 14. HDF5-Friendly OPeNDAP Client Library View??? Groups HDF5 Groups View Data (User) Remote HDF5 Visualization tools (grads) HDF5 Handler w/ CF options HDF5-Friendly Client Library (liboc-dap) (libnc-dap) Server (apache) DAP Protocol (via http) 10/17/2008 HDF and HDF-EOS Workshop XII 14
    15. 15. Traditional Groups Example:OPeNDAP clientin HDF5 library: It’s an attribute that I don’t understand. I’ll ignore it. HDF5-Friendly OPeNDAP client library: I was waiting for this key attribute to re-construct HDF5 10/17/2008 HDF and HDF-EOS Workshop XII 15
    16. 16. Example: Reference in HDF5 • Important for NPOESS • Object / Regional Reference • Map to DAP URL 10/17/2008 HDF and HDF-EOS Workshop XII 16
    17. 17. Example: dap2h5 View Group/Ref. Group/Ref. View Data (User) Remote HDF5 • HDF5-F. Client A test application for the client library Library construct HDF5 from DAP output HDF5 Handler • It canApp. (dap2h5) HDF5-Friendly Client Library (liboc-dap) Server (apache) DAP Protocol (via http) 10/17/2008 HDF and HDF-EOS Workshop XII 17
    18. 18. Our Client Library(prototype) • One more reason: Help Clients to view Swath
    19. 19. Visualizing HDF-EOS5 Swath Problem Visualizing HDF-EOS5 Grids View Swath ??? Swath View Data (User) Remote HDF-EOS5 HDF5 Visualization Tools tools (gradsoc) (grads) HDF5 Handler w/ CF option HDF5-Friendly Client Library (liboc-dap) (libnc-dap) Server (apache) DAP Protocol (via http) 10/17/2008 HDF and HDF-EOS Workshop XII 19
    20. 20. Demo: MLS swath via GrADS • GrADS coupled with our client library • The client library provides grid mapping from swath data • GrADS displays swath through HDF-EOS5 specific client library API calls
    21. 21. Summary of Client Library (Prototype) • Finished coding • Tested with the GrADS visualization client • Documented • Working on a demo DAP to HDF5 tool by using the client library prototype • Will test with NCL if time allows 10/17/2008 HDF and HDF-EOS Workshop XII 21
    22. 22. Caution • Our Client Library is ONLY A PROTOTYPE!  It does NOT support all DAP data types  It does NOT support all AURA files  It does NOT support all Visualization clients 10/17/2008 HDF and HDF-EOS Workshop XII 22
    23. 23. Summary • HDF5 Access via OPeNDAP  Is Easy  Is used by GES DISC to serve Aura files • HDF5-Friendly OPeNDAP Client Library  Serves HDF5 better(EOS swath)  Benefits visualization clients 10/17/2008 HDF and HDF-EOS Workshop XII 23
    24. 24. Future Work • HDF5 to DAP2 Mapping Document • Release HDF5-friendly OPeNDAP Client Library (Prototype) • URL: http://hdfgroup.org/projects/opendap 10/17/2008 HDF and HDF-EOS Workshop XII 24
    25. 25. Credits Mike Folk (The HDF Group) Robert McGrath (NCSA) Peter Leonard, Daniel Kahn, Marghi Hopkins (ADNET) Christopher Lynnes, James Johnson, Denis Nadeau, Fan Fang (NASA) Jennifer Adams (GrADS) Dave Brown (UCAR) 10/17/2008 HDF and HDF-EOS Workshop XII 25
    26. 26. Acknowledgement This work was supported basing upon the Cooperative Agreement with the National Aeronautics and Space Administration (NASA) under NASA grant NNX06AC83A, NNX08A077A and NNX06AG75A. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NASA. 10/17/2008 HDF and HDF-EOS Workshop XII 26
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×