Your SlideShare is downloading. ×
0
The HDF Group
www.hdfgroup.orgJuly 8, 2014 HDF Workshop @ 2014 ESIP Summer Meeting
HDF Product Designer
Aleksandar Jelenak...
www.hdfgroup.orgHDF Workshop @ 2014 ESIP Summer Meeting
HDF Data Producer’s Conundrum
2
HDF Features Interoperability
Miss...
www.hdfgroup.orgHDF Workshop @ 2014 ESIP Summer Meeting
HDF Product Designer Background
• Originated from the HDF5 Earth S...
www.hdfgroup.orgHDF Workshop @ 2014 ESIP Summer Meeting
What is HDF Product Designer?
4
• Application for organizing HDF5 ...
www.hdfgroup.orgHDF Workshop @ 2014 ESIP Summer Meeting
Design Process
5
Create and organize content:
• Files, groups, dat...
www.hdfgroup.orgHDF Workshop @ 2014 ESIP Summer Meeting
After a File is Designed…
6
• Export description of the file’s con...
www.hdfgroup.orgHDF Workshop @ 2014 ESIP Summer Meeting
After a File is Designed…
7
• A template HDF5 file can be generate...
www.hdfgroup.orgHDF Workshop @ 2014 ESIP Summer Meeting
Typical HDF Product Designer Workflow
8
Design a
product file
Expo...
www.hdfgroup.orgHDF Workshop @ 2014 ESIP Summer Meeting
Expected Benefits
• Easy way to design HDF file content
• Interope...
www.hdfgroup.orgHDF Workshop @ 2014 ESIP Summer Meeting
Future Work
10
• Local (desktop) or central (web site) app?
• Redu...
www.hdfgroup.orgHDF Workshop @ 2014 ESIP Summer Meeting
Future Work
• Support for compound datatypes
• Vdata in HDF4 maps ...
www.hdfgroup.orgHDF Workshop @ 2014 ESIP Summer Meeting
Local vs. Central Application?
• Local:
• No need for Internet con...
www.hdfgroup.orgHDF Workshop @ 2014 ESIP Summer Meeting
Looking for Partners
• New missions
• Projects upgrading from HDF4...
www.hdfgroup.orgHDF Workshop @ 2014 ESIP Summer Meeting
Acknowledgement
15
This work was supported by Subcontract number
1...
Upcoming SlideShare
Loading in...5
×

HDF Product Designer

505

Published on

HDF and HDF-EOS Workshop XVII (2014)
Aleksandar Jelenak

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
505
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "HDF Product Designer"

  1. 1. The HDF Group www.hdfgroup.orgJuly 8, 2014 HDF Workshop @ 2014 ESIP Summer Meeting HDF Product Designer Aleksandar Jelenak, H. Joe Lee, Ted Habermann The HDF Group 1
  2. 2. www.hdfgroup.orgHDF Workshop @ 2014 ESIP Summer Meeting HDF Data Producer’s Conundrum 2 HDF Features Interoperability Mission Requirements HDF Product Designer • Datatypes • Groups • Attributes • Dimension scales • Compression • Chunking • Scale/offset • Etc. • Conventions • Metadata • Software • netCDF • Science objectives • Data processing, discovery & distribution • Data documentation • User engagement, preparedness, feedback
  3. 3. www.hdfgroup.orgHDF Workshop @ 2014 ESIP Summer Meeting HDF Product Designer Background • Originated from the HDF5 Earth Science (H5ES) Builder/Creator toolset developed by Jeffrey Lee for the ICESat-2 mission • Built using the MySQL, Apache, PHP software stack • Runs as a desktop application • Under review by The HDF Group to generalize the concept 3
  4. 4. www.hdfgroup.orgHDF Workshop @ 2014 ESIP Summer Meeting What is HDF Product Designer? 4 • Application for organizing HDF5 file content to data and metadata • Built-in support for conventions • Create and re-use parts of file content • Collaborative design process • Generate HDF5 template file, or code that produces the designed file for several programming languages • HDF5 template has structure and metadata
  5. 5. www.hdfgroup.orgHDF Workshop @ 2014 ESIP Summer Meeting Design Process 5 Create and organize content: • Files, groups, datasets, attributes • Many parameters of datasets and attributes can be set • Re-use any part of content elsewhere in the same or another file • All the work is saved in a database
  6. 6. www.hdfgroup.orgHDF Workshop @ 2014 ESIP Summer Meeting After a File is Designed… 6 • Export description of the file’s content into a tab-delimited text file • This description file can be opened with Excel • Content description can be edited • Description file can be imported back into the Product Designer
  7. 7. www.hdfgroup.orgHDF Workshop @ 2014 ESIP Summer Meeting After a File is Designed… 7 • A template HDF5 file can be generated • Code can be produced for reading and writing designed file for: • FORTRAN • IDL • The generated write code can be edited to fill in real data
  8. 8. www.hdfgroup.orgHDF Workshop @ 2014 ESIP Summer Meeting Typical HDF Product Designer Workflow 8 Design a product file Export description file Stakeholders review/edit content Import description file Generate template file and/or code Testing
  9. 9. www.hdfgroup.orgHDF Workshop @ 2014 ESIP Summer Meeting Expected Benefits • Easy way to design HDF file content • Interoperability issues addressed during the design phase • End-to-end data flow testing can start earlier • Reduced risk for costly redesign of processing pipeline based on stakeholder feedback later • Team collaboration • Promotion of design best practices across mission teams 9
  10. 10. www.hdfgroup.orgHDF Workshop @ 2014 ESIP Summer Meeting Future Work 10 • Local (desktop) or central (web site) app? • Reduce app’s required software footprint • MySQL already replaced with SQLite • Support Python, MATLAB for template file code • Web service for generating HDF5 templates and code • Generate portable/free production code using Python and h5py • Re-use content info from old products: • Import HDF4 Content Map.
  11. 11. www.hdfgroup.orgHDF Workshop @ 2014 ESIP Summer Meeting Future Work • Support for compound datatypes • Vdata in HDF4 maps to attribute or dataset? • Convention compliance checkers • CF, HDF-EOS, etc. • Interoperability checkers • netCDF-4, visualization tools, etc. 11
  12. 12. www.hdfgroup.orgHDF Workshop @ 2014 ESIP Summer Meeting Local vs. Central Application? • Local: • No need for Internet connection • Richer/more responsive user interface • No user accounts management • Requires a developer’s computer • Central: • Only web browser required • Better collaboration workflow • Only one place to update the app • Dual mode – like email software? 12
  13. 13. www.hdfgroup.orgHDF Workshop @ 2014 ESIP Summer Meeting Looking for Partners • New missions • Projects upgrading from HDF4 to HDF5 • Projects reprocessing their data to increase interoperability • Use cases Contact: eoshelp@hdfgroup.org 13
  14. 14. www.hdfgroup.orgHDF Workshop @ 2014 ESIP Summer Meeting Acknowledgement 15 This work was supported by Subcontract number 114820 under Raytheon Contract number NNG10HP02C, funded by the National Aeronautics and Space Administration (NASA). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of Raytheon or the National Aeronautics and Space Administration.
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×