HDF and Augmentation
•Download as PPTX, PDF•
0 likes•687 views
The document discusses augmenting HDF files to make them more accessible. It describes adding metadata and restructuring HDF-EOS5 files to allow them to be read as netCDF-4 files. It also talks about modifying JPSS HDF5 files and HDF4 files to include additional metadata and variables needed by tools like IDV, and mapping HDF4 files with XML to expose their structure and contents.
Report
Share
Report
Share
Recommended
HDF Studio
HDF Studio is a set of server and client software that enables users to visualize and share large structured datasets like HDF5 files. It focuses on persisting views and graphs, handling large data collections, and extending through plugins. The architecture includes a cloud repository for data accessed through a REST API and modified HDF5 libraries. Challenges include authentication, performance comparable to local files, scalability to many users and large data, and high reliability of the service. The presentation welcomed feedback as the software is just starting development.
Make CERES products user-friendly
The HDF Group gave a presentation at the ESIP Summer Meeting about working with the CERES team to upgrade their HDF4 climate data products and make them more user-friendly. The HDF Group provided suggestions to the CERES team to structure the data according to Climate and Forecast Metadata conventions and update their HDF4 to HDF5 conversion tool. Working closely with science teams is a good model, as it allows The HDF Group to leverage their expertise in HDF features and product usability. A demo was also given of the new CERES data products.
HDF Workshop SlideShare Experiment
The HDF Group held a workshop at the 2014 ESIP Summer Meeting to discuss HDF, a set of file formats and software libraries used to manage scientific data. Over 340 presentations from the entire history of HDF Workshops are now available on SlideShare and have been viewed over 27,000 times since December. The workshop addressed the challenges data producers face in managing and sharing scientific data sets using HDF formats and tools.
Adding new servicees for HDF in THREDDS Data Server (TDS)
The document discusses adding new web services to the THREDDS Data Server (TDS) for analyzing HDF files. It describes how a user could access and analyze a large HDF file X using a command line tool Y on their mobile device by running Y on a server Z that supports it and sending the output over the web. It provides steps for adding new services to TDS, including creating a controller class and catalog. New HDF services demonstrated include HDF file content mapping, HDF to JSON conversion, and netCDF checker.
HDF and ENVI Services Engine
The ENVI Services Engine (ESE) allows developers to build applications that leverage ENVI image analysis algorithms. ESE applications are deployed on a server so analytics can be performed without client-side processing. Results are returned to clients like webpages, IDL, or mobile devices. The HDF Group customized ESE for NASA HDF-EOS data by developing visualization applications from existing IDL examples and deploying them on ESE servers so results can be displayed on various clients. A demo showed running IDL programs as web services through ESE. User input was requested on desired web services for NASA HDF data.
HDF Project Update
The document provides an update from The HDF Group on their activities related to Earth science data. The HDF Group maintains HDF software and provides services to users. They work on projects for NASA, NOAA, and other government agencies to help manage large Earth science data using HDF formats. Recent activities include support for EOSDIS, JPSS, and other missions through tool development, web services, and data standards work.
HDF OPeNDAP update
The presentation summarized updates to HDF support for OPeNDAP data access. It discussed improvements to HDF4 and HDF5 handlers that enhance performance and support additional Earth science datasets. It demonstrated using OPeNDAP to access HDF files from satellites like TRMM and GPM, as well as a tool for converting HDF files to netCDF format. Challenges discussed included keeping up with evolving data standards and ensuring adequate performance for data aggregation services.
HDF-Java Overview
The presentation provided an overview of HDF-Java and HDFView. It discussed the history and architecture of HDF-Java, including the Java Object Layer API and format-specific Java Native Interface implementations. HDFView was described as providing image, table, and metadata views and editing capabilities for HDF-EOS2/5 and netCDF files through a plug-in architecture. While currently functional, HDFView has an outdated interface and scope issues. The future direction is to fully realize HDF data types and functionality in the layers, decouple the layers, redesign the memory model, and prepare for HDF5 1.10 features.
Recommended
HDF Studio
HDF Studio is a set of server and client software that enables users to visualize and share large structured datasets like HDF5 files. It focuses on persisting views and graphs, handling large data collections, and extending through plugins. The architecture includes a cloud repository for data accessed through a REST API and modified HDF5 libraries. Challenges include authentication, performance comparable to local files, scalability to many users and large data, and high reliability of the service. The presentation welcomed feedback as the software is just starting development.
Make CERES products user-friendly
The HDF Group gave a presentation at the ESIP Summer Meeting about working with the CERES team to upgrade their HDF4 climate data products and make them more user-friendly. The HDF Group provided suggestions to the CERES team to structure the data according to Climate and Forecast Metadata conventions and update their HDF4 to HDF5 conversion tool. Working closely with science teams is a good model, as it allows The HDF Group to leverage their expertise in HDF features and product usability. A demo was also given of the new CERES data products.
HDF Workshop SlideShare Experiment
The HDF Group held a workshop at the 2014 ESIP Summer Meeting to discuss HDF, a set of file formats and software libraries used to manage scientific data. Over 340 presentations from the entire history of HDF Workshops are now available on SlideShare and have been viewed over 27,000 times since December. The workshop addressed the challenges data producers face in managing and sharing scientific data sets using HDF formats and tools.
Adding new servicees for HDF in THREDDS Data Server (TDS)
The document discusses adding new web services to the THREDDS Data Server (TDS) for analyzing HDF files. It describes how a user could access and analyze a large HDF file X using a command line tool Y on their mobile device by running Y on a server Z that supports it and sending the output over the web. It provides steps for adding new services to TDS, including creating a controller class and catalog. New HDF services demonstrated include HDF file content mapping, HDF to JSON conversion, and netCDF checker.
HDF and ENVI Services Engine
The ENVI Services Engine (ESE) allows developers to build applications that leverage ENVI image analysis algorithms. ESE applications are deployed on a server so analytics can be performed without client-side processing. Results are returned to clients like webpages, IDL, or mobile devices. The HDF Group customized ESE for NASA HDF-EOS data by developing visualization applications from existing IDL examples and deploying them on ESE servers so results can be displayed on various clients. A demo showed running IDL programs as web services through ESE. User input was requested on desired web services for NASA HDF data.
HDF Project Update
The document provides an update from The HDF Group on their activities related to Earth science data. The HDF Group maintains HDF software and provides services to users. They work on projects for NASA, NOAA, and other government agencies to help manage large Earth science data using HDF formats. Recent activities include support for EOSDIS, JPSS, and other missions through tool development, web services, and data standards work.
HDF OPeNDAP update
The presentation summarized updates to HDF support for OPeNDAP data access. It discussed improvements to HDF4 and HDF5 handlers that enhance performance and support additional Earth science datasets. It demonstrated using OPeNDAP to access HDF files from satellites like TRMM and GPM, as well as a tool for converting HDF files to netCDF format. Challenges discussed included keeping up with evolving data standards and ensuring adequate performance for data aggregation services.
HDF-Java Overview
The presentation provided an overview of HDF-Java and HDFView. It discussed the history and architecture of HDF-Java, including the Java Object Layer API and format-specific Java Native Interface implementations. HDFView was described as providing image, table, and metadata views and editing capabilities for HDF-EOS2/5 and netCDF files through a plug-in architecture. While currently functional, HDFView has an outdated interface and scope issues. The future direction is to fully realize HDF data types and functionality in the layers, decouple the layers, redesign the memory model, and prepare for HDF5 1.10 features.
HDF Product Designer
The HDF Product Designer is an application for organizing HDF5 file content and metadata to support data standards conventions. It allows users to collaboratively design HDF5 file structure, then generate template files and code to write files in different programming languages. The tool originated from work on the ICESat-2 mission. Future work may include a web-based version, additional language support, and checks for standard compliance and interoperability. The goal is to help projects design HDF5 files earlier to reduce risks from late redesigns.
HDF Product Designer
The document describes the HDF Product Designer software tool. It was created to facilitate the design of interoperable scientific data products in HDF5 format. The tool allows intuitive editing of HDF5 objects and supports conventions like CF and ACDD. It also provides validation services to test file compliance. The goal is to help scientists design data products that follow standards and are easy for others to use.
HDF Project Update
The HDF Group provides software for managing large, complex data and services to support users of this technology. It derives most of its revenue from projects related to earth science, including supporting HDF-EOS, JPSS, and other earth science projects. It maintains various tools for working with HDF files and conducts maintenance, support, and development activities to support new versions and capabilities of HDF libraries and software.
HDF Cloud Services
HDF Cloud Services aims to bring HDF5 to the cloud by defining a REST API for HDF5 and implementing related services. The HDF REST API allows HDF5 data to be accessed via HTTP requests and responses. H5serv is an open source reference implementation of the HDF REST API. The HDF Scalable Data Service (HSDS) is being developed to support large HDF5 repositories in a scalable, cost effective manner using object storage like AWS S3.
Putting some Spark into HDF5
This presentation discusses putting HDF5 files into Apache Spark for analysis. It describes the differences between traditional file systems and Hadoop's HDFS, and how Spark provides a more accessible way to exploit data parallelism without using MapReduce. The presentation outlines experiments loading HDF5 climate data files into Spark to calculate statistics. It suggests variations like providing a file list instead of traversing directories. The conclusion is that Spark can effectively analyze HDF5 files under the right circumstances but current methods are imperfect, and future work with The HDF Group could build better integrations.
Geospatial Data Abstraction Library (GDAL) Enhancement for ESDIS (GEE)
Geospatial Data Abstraction Library (GDAL) Enhancement for ESDIS (GEE)The HDF-EOS Tools and Information Center
The document discusses enhancing the Geospatial Data Abstraction Library (GDAL) to improve accessibility and interoperability of NASA data products with GIS tools. It developed plugins for three NASA data products to demonstrate reading multidimensional datasets into GIS applications like ArcGIS. Next steps include providing outreach, enhancing the framework to be more flexible and production-ready, and developing guides to help other data centers build GDAL plugins to address their issues with geospatial data. The overall goal is to improve analysis and visualization of NASA scientific data in GIS tools and web applications.Open-source Scientific Computing and Data Analytics using HDF
Open-source Scientific Computing and Data Analytics using HDFThe HDF-EOS Tools and Information Center
Kitware uses HDF as a widely adopted data format for scientific computing and visualization across several domains. HDF supports climate modeling, geospatial data, medical imaging, and more. Kitware is looking to improve HDF support for streaming big data, cloud computing, and web applications to enable more advanced analytics and sharing of scientific data. Future work may include pure JavaScript implementations of HDF tools and optimizing performance for cloud storage.Indexing HDF5: A Survey
The document discusses indexing methods for HDF5 files to enable efficient searching and access of data based on data values. It describes several existing implementations of HDF5 indexing, including PyTables, FastQuery/FastBit, Alacrity, and prototypes from The HDF Group. It also outlines current and future work to develop indexing and querying capabilities within HDF5 to allow complex multi-dimensional searches across metadata and datasets.
Hierarchical Data Formats (HDF) Update
The document summarizes updates on Hierarchical Data Formats (HDF) software releases and tools. It discusses the latest releases of HDF5 1.8.19 and 1.10.1, compatibility issues when moving to newer versions, updates on tools like HDF-Java and HDFView 3.0, supported compilers and systems, and a new compression library for interoperability. It invites readers to provide feedback on their needs.
HDF Update 2016
The HDF Group provides updates on new features in HDF including faster compression, single writer/multiple reader file access, virtual datasets, and dynamically loaded filters. They also discuss tools like HDFView, nagg for data aggregation, and a new HDF5 ODBC driver. The work is supported by NASA.
Incorporating ISO Metadata Using HDF Product Designer
This document discusses incorporating ISO metadata standards into HDF files using the HDF Product Designer tool. It describes how the HDF Product Designer allows users to import pre-built ISO metadata components from a separate project into their HDF file designs. This allows essential contextual data or metadata to be stored in HDF5 files according to ISO 19115 standards.
NEON HDF5
This document discusses NEON's use of HDF5 file format for its ecological data. The goals are to implement a fast and efficient file format, develop a standardized data delivery structure, and provide metadata. It describes the HDF5 file structure, metadata inclusion, and an example workflow for processing eddy covariance data into HDF5 files. Future work includes integrating R code for HDF5 file generation and embedding ecological metadata.
HDF Product Designer: Using Templates to Achieve Interoperability
HDF Product Designer: Using Templates to Achieve InteroperabilityThe HDF-EOS Tools and Information Center
This document discusses how HDF Product Designer (HPD) uses templates to achieve interoperability. HPD is an application for consistently developing interoperable data content in HDF5 files. It has a client-server architecture and desktop app. Templates allow users to copy design examples that incorporate best practices and are curated by the HPD development team. Available templates include NCEI collections and CF templates, with more to be added based on community review and suggestions. Templates allow users to initialize new designs by mixing and matching content from different template examples.Data Are from Mars, Tools Are from Venus
This document discusses challenges with data and tools coming from different "planets" (Mars and Venus), frustrating users. It identifies gaps in file formats, libraries, tools, and services for working with Earth science data. The author provides solutions through the HDFEOS.org project, including file format conversion and examples, and suggestions for data producers, tool developers, and end users to help address interoperability issues. Emerging topics like big data, machine learning, and voice assistants are also discussed.
HDF Group Support for NPP/NPOESS/JPSS
The HDF Group presented on their support for the National Polar-orbiting Partnership/Joint Polar Satellite System (NPP/JPSS) program. Their goals included providing HDF5 support for distributing data from the Visible Infrared Imaging Radiometer Suite (VIIRS) and other sensors. They outlined priorities for testing software on critical platforms, developing tools to access and manage NPP/JPSS products, and providing rapid support. The presentation described software released or under development like h5edit, h5augjpss, and nagg, an aggregation tool.
HDF5 Performance Enhancements with the Elimination of Unlimited Dimension
HDF5 Performance Enhancements with the Elimination of Unlimited DimensionThe HDF-EOS Tools and Information Center
This document discusses changes made to the MOPITT instrument's HDF data products over multiple versions. MOPITT measures carbon monoxide on the Terra spacecraft. Version 7 products, which use HDF-EOS5 format without unlimited dimensions, were found to be read faster than Version 6 products despite being larger in size. Experiments comparing read times of Version 7 data with and without unlimited dimensions showed a small 2% difference initially, but up to 40% faster reads for Version 7 without unlimited dimensions at first, with the difference decreasing over multiple reads. Eliminating unlimited dimensions thus improved access performance for some HDF applications like MOPITT data.GES DISC Eexperiences with HDF Formats for MEaSUREs Projects
GES DISC Eexperiences with HDF Formats for MEaSUREs ProjectsThe HDF-EOS Tools and Information Center
The document discusses GES DISC's experience supporting 7 MEaSUREs projects, which create long-term Earth science data records. It recommends file formats like HDF-EOS5 and netCDF4 to promote consistency and interoperability. While HDF5 was initially recommended, ad-hoc implementations have caused issues. The document provides information on GES DISC's recommendations for file naming, formats and metadata, as well as problems encountered and resources for learning more.Knime customer intelligence on social media odsc london
The document discusses how KNIME was used to analyze social media data from Slashdot forums to understand user sentiment and influence. Text analytics identified the most positive and negative users, while network analysis identified influential users. Integrating both tools found that most influencers were neutral, but one positive and talkative user would be worth engaging further. The document provides resources for learning more about KNIME's capabilities.
HDF5 High Level and Lite Libraries
Overview Of the HDF5 Lite and High Level interfaces.
Source: http://hdfeos.org/workshops/ws07/presentations/McGrath3/McGrath_HDF5_High_Level_and_Lite_Libraries_Intro.ppt
Moving form HDF4 to HDF5/netCDF-4
This presentation discusses moving data and applications from HDF4 to HDF5/netCDF-4. It covers the differences between HDF4 and HDF5 data models and capabilities, tools for converting HDF4 data to HDF5, advantages of HDF5 like unlimited dimensions and compression, and ways to ensure compatibility with netCDF-4 like avoiding HDF5-specific features. The work was supported by a NASA contract.
ArcGIS and Multi-D: Tools & Roadmap
The document discusses Esri's tools and roadmap for working with multi-dimensional (MD) scientific data in ArcGIS. It outlines Esri's efforts to directly read HDF, GRIB, and netCDF files as raster layers or feature/table views in ArcGIS. MD mosaic datasets allow users to manage variables and dimensions across multiple files and perform on-the-fly computations and visualization of MD data. New functions have been added to improve MD data analysis and visualization, including a vector field renderer to depict raster data as vectors. Esri is also working to better support OPeNDAP data sources.
Python and HDF5
This document discusses Python and HDF5. It mentions that Python has over 45,000 packages including SciPy and h5py, a general-purpose HDF5 library. HDF5 is a fast database that uses datasets like arrays on disk, groups like filesystem folders, and attributes for key-value metadata. The document ends asking if there are any questions and providing a website for more information.
More Related Content
What's hot
HDF Product Designer
The HDF Product Designer is an application for organizing HDF5 file content and metadata to support data standards conventions. It allows users to collaboratively design HDF5 file structure, then generate template files and code to write files in different programming languages. The tool originated from work on the ICESat-2 mission. Future work may include a web-based version, additional language support, and checks for standard compliance and interoperability. The goal is to help projects design HDF5 files earlier to reduce risks from late redesigns.
HDF Product Designer
The document describes the HDF Product Designer software tool. It was created to facilitate the design of interoperable scientific data products in HDF5 format. The tool allows intuitive editing of HDF5 objects and supports conventions like CF and ACDD. It also provides validation services to test file compliance. The goal is to help scientists design data products that follow standards and are easy for others to use.
HDF Project Update
The HDF Group provides software for managing large, complex data and services to support users of this technology. It derives most of its revenue from projects related to earth science, including supporting HDF-EOS, JPSS, and other earth science projects. It maintains various tools for working with HDF files and conducts maintenance, support, and development activities to support new versions and capabilities of HDF libraries and software.
HDF Cloud Services
HDF Cloud Services aims to bring HDF5 to the cloud by defining a REST API for HDF5 and implementing related services. The HDF REST API allows HDF5 data to be accessed via HTTP requests and responses. H5serv is an open source reference implementation of the HDF REST API. The HDF Scalable Data Service (HSDS) is being developed to support large HDF5 repositories in a scalable, cost effective manner using object storage like AWS S3.
Putting some Spark into HDF5
This presentation discusses putting HDF5 files into Apache Spark for analysis. It describes the differences between traditional file systems and Hadoop's HDFS, and how Spark provides a more accessible way to exploit data parallelism without using MapReduce. The presentation outlines experiments loading HDF5 climate data files into Spark to calculate statistics. It suggests variations like providing a file list instead of traversing directories. The conclusion is that Spark can effectively analyze HDF5 files under the right circumstances but current methods are imperfect, and future work with The HDF Group could build better integrations.
Geospatial Data Abstraction Library (GDAL) Enhancement for ESDIS (GEE)
Geospatial Data Abstraction Library (GDAL) Enhancement for ESDIS (GEE)The HDF-EOS Tools and Information Center
The document discusses enhancing the Geospatial Data Abstraction Library (GDAL) to improve accessibility and interoperability of NASA data products with GIS tools. It developed plugins for three NASA data products to demonstrate reading multidimensional datasets into GIS applications like ArcGIS. Next steps include providing outreach, enhancing the framework to be more flexible and production-ready, and developing guides to help other data centers build GDAL plugins to address their issues with geospatial data. The overall goal is to improve analysis and visualization of NASA scientific data in GIS tools and web applications.Open-source Scientific Computing and Data Analytics using HDF
Open-source Scientific Computing and Data Analytics using HDFThe HDF-EOS Tools and Information Center
Kitware uses HDF as a widely adopted data format for scientific computing and visualization across several domains. HDF supports climate modeling, geospatial data, medical imaging, and more. Kitware is looking to improve HDF support for streaming big data, cloud computing, and web applications to enable more advanced analytics and sharing of scientific data. Future work may include pure JavaScript implementations of HDF tools and optimizing performance for cloud storage.Indexing HDF5: A Survey
The document discusses indexing methods for HDF5 files to enable efficient searching and access of data based on data values. It describes several existing implementations of HDF5 indexing, including PyTables, FastQuery/FastBit, Alacrity, and prototypes from The HDF Group. It also outlines current and future work to develop indexing and querying capabilities within HDF5 to allow complex multi-dimensional searches across metadata and datasets.
Hierarchical Data Formats (HDF) Update
The document summarizes updates on Hierarchical Data Formats (HDF) software releases and tools. It discusses the latest releases of HDF5 1.8.19 and 1.10.1, compatibility issues when moving to newer versions, updates on tools like HDF-Java and HDFView 3.0, supported compilers and systems, and a new compression library for interoperability. It invites readers to provide feedback on their needs.
HDF Update 2016
The HDF Group provides updates on new features in HDF including faster compression, single writer/multiple reader file access, virtual datasets, and dynamically loaded filters. They also discuss tools like HDFView, nagg for data aggregation, and a new HDF5 ODBC driver. The work is supported by NASA.
Incorporating ISO Metadata Using HDF Product Designer
This document discusses incorporating ISO metadata standards into HDF files using the HDF Product Designer tool. It describes how the HDF Product Designer allows users to import pre-built ISO metadata components from a separate project into their HDF file designs. This allows essential contextual data or metadata to be stored in HDF5 files according to ISO 19115 standards.
NEON HDF5
This document discusses NEON's use of HDF5 file format for its ecological data. The goals are to implement a fast and efficient file format, develop a standardized data delivery structure, and provide metadata. It describes the HDF5 file structure, metadata inclusion, and an example workflow for processing eddy covariance data into HDF5 files. Future work includes integrating R code for HDF5 file generation and embedding ecological metadata.
HDF Product Designer: Using Templates to Achieve Interoperability
HDF Product Designer: Using Templates to Achieve InteroperabilityThe HDF-EOS Tools and Information Center
This document discusses how HDF Product Designer (HPD) uses templates to achieve interoperability. HPD is an application for consistently developing interoperable data content in HDF5 files. It has a client-server architecture and desktop app. Templates allow users to copy design examples that incorporate best practices and are curated by the HPD development team. Available templates include NCEI collections and CF templates, with more to be added based on community review and suggestions. Templates allow users to initialize new designs by mixing and matching content from different template examples.Data Are from Mars, Tools Are from Venus
This document discusses challenges with data and tools coming from different "planets" (Mars and Venus), frustrating users. It identifies gaps in file formats, libraries, tools, and services for working with Earth science data. The author provides solutions through the HDFEOS.org project, including file format conversion and examples, and suggestions for data producers, tool developers, and end users to help address interoperability issues. Emerging topics like big data, machine learning, and voice assistants are also discussed.
HDF Group Support for NPP/NPOESS/JPSS
The HDF Group presented on their support for the National Polar-orbiting Partnership/Joint Polar Satellite System (NPP/JPSS) program. Their goals included providing HDF5 support for distributing data from the Visible Infrared Imaging Radiometer Suite (VIIRS) and other sensors. They outlined priorities for testing software on critical platforms, developing tools to access and manage NPP/JPSS products, and providing rapid support. The presentation described software released or under development like h5edit, h5augjpss, and nagg, an aggregation tool.
HDF5 Performance Enhancements with the Elimination of Unlimited Dimension
HDF5 Performance Enhancements with the Elimination of Unlimited DimensionThe HDF-EOS Tools and Information Center
This document discusses changes made to the MOPITT instrument's HDF data products over multiple versions. MOPITT measures carbon monoxide on the Terra spacecraft. Version 7 products, which use HDF-EOS5 format without unlimited dimensions, were found to be read faster than Version 6 products despite being larger in size. Experiments comparing read times of Version 7 data with and without unlimited dimensions showed a small 2% difference initially, but up to 40% faster reads for Version 7 without unlimited dimensions at first, with the difference decreasing over multiple reads. Eliminating unlimited dimensions thus improved access performance for some HDF applications like MOPITT data.GES DISC Eexperiences with HDF Formats for MEaSUREs Projects
GES DISC Eexperiences with HDF Formats for MEaSUREs ProjectsThe HDF-EOS Tools and Information Center
The document discusses GES DISC's experience supporting 7 MEaSUREs projects, which create long-term Earth science data records. It recommends file formats like HDF-EOS5 and netCDF4 to promote consistency and interoperability. While HDF5 was initially recommended, ad-hoc implementations have caused issues. The document provides information on GES DISC's recommendations for file naming, formats and metadata, as well as problems encountered and resources for learning more.Knime customer intelligence on social media odsc london
The document discusses how KNIME was used to analyze social media data from Slashdot forums to understand user sentiment and influence. Text analytics identified the most positive and negative users, while network analysis identified influential users. Integrating both tools found that most influencers were neutral, but one positive and talkative user would be worth engaging further. The document provides resources for learning more about KNIME's capabilities.
HDF5 High Level and Lite Libraries
Overview Of the HDF5 Lite and High Level interfaces.
Source: http://hdfeos.org/workshops/ws07/presentations/McGrath3/McGrath_HDF5_High_Level_and_Lite_Libraries_Intro.ppt
Moving form HDF4 to HDF5/netCDF-4
This presentation discusses moving data and applications from HDF4 to HDF5/netCDF-4. It covers the differences between HDF4 and HDF5 data models and capabilities, tools for converting HDF4 data to HDF5, advantages of HDF5 like unlimited dimensions and compression, and ways to ensure compatibility with netCDF-4 like avoiding HDF5-specific features. The work was supported by a NASA contract.
What's hot (20)
Geospatial Data Abstraction Library (GDAL) Enhancement for ESDIS (GEE)
Geospatial Data Abstraction Library (GDAL) Enhancement for ESDIS (GEE)
Open-source Scientific Computing and Data Analytics using HDF
Open-source Scientific Computing and Data Analytics using HDF
Incorporating ISO Metadata Using HDF Product Designer
Incorporating ISO Metadata Using HDF Product Designer
HDF Product Designer: Using Templates to Achieve Interoperability
HDF Product Designer: Using Templates to Achieve Interoperability
HDF5 Performance Enhancements with the Elimination of Unlimited Dimension
HDF5 Performance Enhancements with the Elimination of Unlimited Dimension
GES DISC Eexperiences with HDF Formats for MEaSUREs Projects
GES DISC Eexperiences with HDF Formats for MEaSUREs Projects
Knime customer intelligence on social media odsc london
Knime customer intelligence on social media odsc london
Viewers also liked
ArcGIS and Multi-D: Tools & Roadmap
The document discusses Esri's tools and roadmap for working with multi-dimensional (MD) scientific data in ArcGIS. It outlines Esri's efforts to directly read HDF, GRIB, and netCDF files as raster layers or feature/table views in ArcGIS. MD mosaic datasets allow users to manage variables and dimensions across multiple files and perform on-the-fly computations and visualization of MD data. New functions have been added to improve MD data analysis and visualization, including a vector field renderer to depict raster data as vectors. Esri is also working to better support OPeNDAP data sources.
Python and HDF5
This document discusses Python and HDF5. It mentions that Python has over 45,000 packages including SciPy and h5py, a general-purpose HDF5 library. HDF5 is a fast database that uses datasets like arrays on disk, groups like filesystem folders, and attributes for key-value metadata. The document ends asking if there are any questions and providing a website for more information.
The National Oceanographic Data Center’s NetCDF Templates
The document describes how the National Oceanographic Data Center (NODC) has incorporated Climate and Forecast (CF) and Attribute Conventions for Dataset Discovery (ACDD) conventions into their netCDF templates. The templates include attributes for variable and global metadata, follow CF feature types, and provide examples of how the conventions are applied to different types of oceanographic datasets to improve discoverability and reuse of data. Key benefits of using the templates include long-term data preservation, scientific quality control, and enabling multiple reuse of data beyond its original purpose.
The HDF Group: Community models and outreach
The HDF Group aims to build community around its data platform through engagement goals like improving metadata, developing sharing tools, and hosting workshops. It seeks to bring together disciplines by addressing needs, maintaining websites and forums, publishing a blog, and coordinating networks among NASA missions, data centers, research facilities, and models to facilitate cross-disciplinary data sharing, use, and understanding.
His Expert's Voice
This document discusses expert systems and conventions for data modeling. It provides an overview of expert systems and how rule-based systems work using examples like traffic lights and Conway's Game of Life. It describes how an expert system could be used for the HDF Product Designer to enforce data modeling conventions. The status, challenges, and future work are outlined for integrating an expert system into the HDF Product Designer to provide convention knowledge and guide the data modeling process.
MATLAB and Scientific Data: New Features and Capabilities
This document discusses new features and capabilities in MATLAB for working with scientific data formats and performing technical computing. It highlights enhancements to reading HDF5 and NetCDF files with both high-level and low-level interfaces. It also covers new capabilities for handling dates/times, big data, and accessing web services through RESTful APIs.
Improved Query Performance With Variant Indexes - review presentation
This document summarizes different types of database indexes that can improve query performance, including value-list indexes, bitmap indexes, projection indexes, and bit-sliced indexes. It compares how each index type can be used to efficiently evaluate single-column sum aggregates and range predicates. The document also discusses how join indexes and bitmap join indexes can help support ad-hoc OLAP-style queries on dimensional data when the query criteria are not known in advance. Segmentation and clustering techniques are also presented to further improve grouping efficiency in data warehousing environments.
HDF Server
The document describes HDF Server (h5serv), which exposes HDF5 files via a RESTful API. H5serv allows full read/write access and supports HDF5 features like compression and hyperslab selection. It uses the Tornado framework to implement a stateless, cacheable API accessed through HTTP requests in JSON. This provides a web interface for HDF5 data while maintaining HDF5 functionality. Future plans include client libraries, authentication/authorization, and improving performance for large repositories.
NASA HDF/HDF-EOS Data for Dummies (and Developers)
The document discusses NASA HDF/HDF-EOS data formats and how to work with them for Earth science data analysis. It begins with an overview of the complexity of NASA HDF data for new users. It then covers common HDF data formats like HDF4, HDF5, HDF-EOS2 and HDF-EOS5, explaining the differences and which APIs to use for each. It provides code examples for reading HDF-EOS data in C and Fortran. It also introduces tools that can help dump and extract HDF data in ASCII format for broader tool compatibility. Overall, the document aims to help new users understand and start working with NASA's HDF data products.
HDF Compass
HDF Compass is a new viewer for HDF and related formats that provides a highly focused tool for casual browsing and plotting of files in these formats. It has a fully native user interface experience, is cross-platform with no dependencies, and is easily extensible through a plugin layer. HDF Compass itself is a "frozen" Python application that can also support other formats beyond HDF through the plugin system.
Pilot Project for HDF5 Metadata Structures for SWOT
This document discusses a pilot project to incorporate ISO 19115-2 metadata attributes at the granule level for the NASA SWOT mission. The metadata will be stored in HDF5 groups and generated in two ways - via XML serialization or XML style sheet conversions. The project aims to capture essential metadata attributes from the SWOT information architecture and ISO metadata model, and generate an HDF5 structure specification and example metadata snippets.
How to Meet the CF Conventions with NcML for NASA HDF/HDF-EOS
How to Meet the CF Conventions with NcML for NASA HDF/HDF-EOSThe HDF-EOS Tools and Information Center
The document discusses how to modify a NASA HDF/HDF-EOS file using NetCDF Markup Language (NcML) to meet the Climate and Forecast (CF) conventions. It provides examples of fixes needed for the file, including adding coordinate variables, modifying attribute values, and adding scale and offset attributes. While the modified file meets the CF conventions, different tools may interpret it differently depending on whether coordinate variables are inside or outside the data group.ICESat-2 Metadata and Status
ICESat-2 is a NASA satellite mission scheduled to launch in December 2017 that will use photon counting laser altimetry to measure ice sheet and sea ice elevations. It will carry an advanced laser altimeter that splits each laser pulse into 6 beams in a cross-track pattern to provide dense sampling. This will allow for improved elevation estimates over rough terrain. The document discusses ICESat-2's science objectives, measurement concept, data products, processing workflow, and approach to managing metadata across different levels of data products.
America Runs on Excel and HDF5 - Glued together by Python
This document discusses the development of PyHexad, a Python-based add-in for Excel that allows users to access and analyze HDF5 data directly in Excel. PyHexad 0.1 allows users to display HDF5 file contents, read arrays and tables into Excel, and read HDF5 images. The developer is seeking feedback on usability, the Python dependency, and interest in helping advance the project to version 1.0. A prototype will be released in early August for testing and further input.
Viewers also liked (14)
The National Oceanographic Data Center’s NetCDF Templates
The National Oceanographic Data Center’s NetCDF Templates
MATLAB and Scientific Data: New Features and Capabilities
MATLAB and Scientific Data: New Features and Capabilities
Improved Query Performance With Variant Indexes - review presentation
Improved Query Performance With Variant Indexes - review presentation
NASA HDF/HDF-EOS Data for Dummies (and Developers)
NASA HDF/HDF-EOS Data for Dummies (and Developers)
Pilot Project for HDF5 Metadata Structures for SWOT
Pilot Project for HDF5 Metadata Structures for SWOT
How to Meet the CF Conventions with NcML for NASA HDF/HDF-EOS
How to Meet the CF Conventions with NcML for NASA HDF/HDF-EOS
America Runs on Excel and HDF5 - Glued together by Python
America Runs on Excel and HDF5 - Glued together by Python
Similar to HDF and Augmentation
HDF Update
This document discusses the HDF (Hierarchical Data Format) software. It provides an overview of HDF4 and HDF5, including past and future work. HDF is a format and software for scientific data storage and exchange that is free and supported by various organizations. The document outlines activities in 2000 to support users, develop tools like HDF5View, and refine the XML description of HDF5. It concludes by focusing on areas for 2001 like supporting new satellite missions and expanding HDF5 applications and performance.
HDF Update
An update on HDF, including a status report on the HDF Group, an overview of recent changes to the HDF4 and HDF5 libraries and tools, plans for future releases, HDF Group projects and collaborations, and future plans.
HDF4 Mapping Project Update
The document discusses the HDF4 Mapping Project which aims to ensure long-term access to Earth Observing System (EOS) data stored in HDF4 files. It provides an overview of the project scope, including developing a proof of concept prototype and production quality mapping tools. It also describes verification studies conducted with NASA data centers to identify requirements for verifying correctness of HDF4 file content maps produced by the mapping tools. The project aims to generate content maps for HDF4 files containing valuable EOS data before the HDF4 library and tools are no longer maintained.
HDF Update
This document summarizes HDF software activities in 2002, including support and funding sources for HDF, recent and upcoming releases of HDF4 and HDF5 libraries and tools, and other HDF-related projects. The HDF5 library saw improvements to performance, compilers supported, and tools like HDFView and converters. The next major HDF5 release in 2003 will focus on new features, performance enhancements, and special platform support. High level APIs and the parallel HDF5 programming model were also under development.
HDF Status and Development
The document summarizes updates from The HDF Group. It discusses that The HDF Group was established in 1988 and owns HDF4 and HDF5 formats and libraries. It provides services like helpdesk, support, consulting and training to users. The HDF Group aims to ensure long-term accessibility of HDF data through development and support of HDF technologies. Recent improvements include new HDF5 and HDF4 releases, tools updates, HDF-Java and SWMR file access work. Future work involves parallel I/O, indexing methods and EOS, OPeNDAP and NPP/NPOESS support.
HDF Project Status and Plans
This document summarizes activities related to the HDF project. It discusses the status of The HDF Group as a nonprofit organization dedicated to supporting HDF. It reviews ESDIS activities including maintenance of HDF and HDF-EOS code and user support. It provides statistics on downloads, helpdesk requests, and forum usage. It also outlines maintenance and testing of HDF4, HDF5 and related software releases. Platform support issues are also addressed. The document covers recent improvements and previews future work for the HDF software suite.
HDF Update
This document summarizes Mike Folk's presentation at the Science Data Processing Workshop from February 26-28, 2002. The presentation provided updates on HDF4 and HDF5, including recent releases and future plans. HDF4 and HDF5 are open source data formats and software libraries for scientific data that support efficient storage of arrays, images, and tables. The presentation outlined ongoing work to improve performance, add new features, and facilitate the transition from HDF4 to HDF5.
Introduction to HDF5
This tutorial is designed for new HDF5 users. We will cover HDF5 abstractions such as datasets, groups, attributes, and datatypes. Simple C examples will cover the programming model and basic features of the API, and will give new users the knowledge they need to navigate through the rich collection of HDF5 interfaces. Participants will be guided through an interactive demonstration of the fundamentals of HDF5.
This tutorial is for new HDF5 users.
HDF Updae
Update on HDF, including recent changes to the software, upcoming releases, collaborations, future plans. Will include an overview of the upcoming HDF5 1.8 release, and updates on the netCDF4/HDF5 merge, HDF5 support for indexing, BioHDF, the HDF5-Storage Resource Broker project, and the HDF spin-off THG.
Transitioning from HDF4 to HDF5
The document discusses the transition from HDF4 to HDF5. There are three main challenges: 1) supporting both HDF4 and HDF5, 2) enabling interoperability between HDF4 and HDF5 files and libraries, and 3) converting data from HDF4 to HDF5. NCSA has developed tools like h4toh5 and libh4toh5 to help with the conversion process. Experiments show that default conversion of HDF4 files to HDF5 is feasible and conversion of HDF-EOS files can be achieved through custom tools built with libh4toh5.
Improving long-term preservation of EOS data by independently mapping HDF4 da...
Improving long-term preservation of EOS data by independently mapping HDF4 da...The HDF-EOS Tools and Information Center
The document discusses a project to improve long-term preservation of Earth Observing System (EOS) data by creating independent maps of HDF4 data objects. The project aims to map HDF4 files to allow simple readers to access data without relying on HDF software. It involves categorizing NASA HDF4 data, prototyping an XML mapping file format, and building tools to create maps and read data based on maps. The project will investigate integrating the mapping schema with standards, address HDF-EOS2 requirements, redesign the schema, implement production mapping and reading tools, and deploy the tools at NASA data centers.HDF Update for DAAC Managers (2017-02-27)
This document provides information about HDF (Hierarchical Data Format) tools and resources for working with Earth observation data. It summarizes HDF's focus on helping users at different stages of working with data, from initial product design to long-term archiving. It also describes specific HDF tools for viewing, comparing, converting between formats and adding metadata to scientific data files.
HDF Tools Updates and Discussions
Peter Cao from The HDF Group presented updates on HDF tools and Java products at the 15th HDF and HDF-EOS Workshop from April 17-19, 2012. Recent improvements were made to h5dump, h5repack, and h5diff. Future developments discussed included h5edit for editing HDF5 files, H5Ocompare for comparing HDF5 objects, and h5augjpss for modifying JPSS files. Feedback from an HDFView user survey showed top user priorities were improving the GUI, searching by name and attributes, and viewing large datasets.
HDF Update
Mike Folk from the National Center for Supercomputing Applications gave an update on HDF software in 2003. HDF is supported by several government agencies for applications in earth science, simulations, and data-intensive computing. Version 4.2 Release 1 was planned for October 2003 with bug fixes and new features. HDF5 1.6.0 was released in July 2003 with new filters, properties, and performance improvements. Work was also being done on high-level APIs, parallel HDF5, tools, and collaborations with other projects.
Hdf5 current future
An introduction to HDF5, including descriptions of current and future work, presented at the HPC Oil & Gas Workshop on March 5, 2015.
Ensuring Long Term Access to Remotely Sensed HDF4 Data with Layout Maps
Ensuring Long Term Access to Remotely Sensed HDF4 Data with Layout MapsThe HDF-EOS Tools and Information Center
A preponderance of data from NASA's Earth Observing System (EOS) is archived in the HDF Version 4 (HDF4) format. The long-term preservation of these data is critical for climate and other scientific studies going many decades into the future. HDF4 is very effective for working with the large and complex collection of EOS data products. Unfortunately, because of the complex internal byte layout of HDF4 files, future readability of HDF4 data depends on preserving a complex software library that can interpret that layout. Having a way to access HDF4 data independent of a library could improve its viability as an archive format, and consequently give confidence that HDF4 data will be readily accessible forever, even if the HDF4 library is gone.
To address the need to simplify long-term access to EOS data stored in HDF4, a collaborative project between The HDF Group and NASA Earth Science Data Centers is implementing an approach to accessing data in HDF4 files based on the use of independent maps that describe the data in HDF4 files and tools that can use these maps to recover data from those files. With this approach, relatively simple programs will be able to extract the data from an HDF4 file, bypassing the need for the HDF4 library.
A demonstration project has shown that this approach is feasible. This involved an assessment of NASA�s HDF4 data holdings, and development of a prototype XML-based layout mapping language and tools to read layout maps and read HDF4 files using layout maps. Future plans call for a second phase of the project, in which the mapping tools and XML schema are made production quality, the mapping schema are integrated with existing XML metadata files in several data centers, and outreach activities are carried out to encourage and facilitate acceptance of the technology. Support for NPP/NPOESS by The HDF Group
Accessibility and usability of NPP/NPOESS data in HDF5 can be enhanced by providing tools that simplify and standardize how data is accessed and presented. In this project, The HDF Group is creating such tools in the form of software to read and write certain key data types and data aggregates used in NPP/NPOESS data products, and extending HDFView to extract, present and export these data effectively. In particular, the work will focus on NPP/NPOESS use of HDF5 region references and quality flags. The HDF Group will also provide high quality user support for the project.
Images of HDF5
The document summarizes Gerd Heber's presentation from The HDF Group's 15th HDF and HDF-EOS Workshop held from April 17-19, 2012. The presentation discusses five topics related to HDF5: a model of HDF5 file information, a new XML representation of HDF5, HDF5 as a service, improving the HDF5 user experience, and the relationship between HDF5 and databases. Each topic is distilled into short summaries to provide the key information discussed in the multi-page document.
HDF Town Hall
The document summarizes changes at The HDF Group, including new staff members and their roles. It also outlines recent and upcoming HDF software releases, new operating system and compiler support, tools for HDF and netCDF interoperability, ongoing research projects involving parallel I/O and analysis, and potential projects of interest involving scientific domains like plasma physics, particle accelerators, and digital twin technology.
Support for NPP/NPOESS/JPSS by The HDF Group
The HDF Group provides support for the National Polar-orbiting Partnership/National Polar-orbiting Operational Environmental Satellite System (NPP/NPOESS) through developing tools and libraries to help users access and work with NPP/NPOESS data. Key areas of focus include making data access intuitive, allowing data viewing and conversion, and providing helpdesk support. The HDF Group develops high-level APIs, tests HDF5 on NASA systems, enhances command line tools like h5dump to support NPP/NPOESS data, and is working on a new tool called h5edit to edit HDF5 files. Future work includes finishing h5edit, maintaining software releases, testing compatibility with netCDF, and providing user
Similar to HDF and Augmentation (20)
Improving long-term preservation of EOS data by independently mapping HDF4 da...
Improving long-term preservation of EOS data by independently mapping HDF4 da...
Ensuring Long Term Access to Remotely Sensed HDF4 Data with Layout Maps
Ensuring Long Term Access to Remotely Sensed HDF4 Data with Layout Maps
More from The HDF-EOS Tools and Information Center
Cloud-Optimized HDF5 Files
This document discusses how to optimize HDF5 files for efficient access in cloud object stores. Key optimizations include using large dataset chunk sizes of 1-4 MiB, consolidating internal file metadata, and minimizing variable-length datatypes. The document recommends creating files with paged aggregation and storing file content information in the user block to enable fast discovery of file contents when stored in object stores.
Accessing HDF5 data in the cloud with HSDS
This document provides an overview of HSDS (Highly Scalable Data Service), which is a REST-based service that allows accessing HDF5 data stored in the cloud. It discusses how HSDS maps HDF5 objects like datasets and groups to individual cloud storage objects to optimize performance. The document also describes how HSDS was used to improve access performance for NASA ICESat-2 HDF5 data on AWS S3 by hyper-chunking datasets into larger chunks spanning multiple original HDF5 chunks. Benchmark results showed that accessing the data through HSDS provided over 2x faster performance than other methods like ROS3 or S3FS that directly access the cloud storage.
The State of HDF
This document summarizes the current status and focus of the HDF Group. It discusses that the HDF Group is located in Champaign, IL and is a non-profit organization focused on developing and maintaining HDF software and data formats. It provides an overview of recent HDF5, HDF4 and HDFView releases and notes areas of focus for software quality improvements, increased transparency, strengthening the community, and modernizing HDF products. It invites support and participation in upcoming user group meetings.
Highly Scalable Data Service (HSDS) Performance Features
This document provides an overview of HSDS (HDF Server and Data Service), which allows HDF5 files to be stored and accessed from the cloud. Key points include:
- HSDS maps HDF5 objects like datasets and groups to individual cloud storage objects for scalability and parallelism.
- Features include streaming support, fancy indexing for complex queries, and caching for improved performance.
- HSDS can be deployed on Docker, Kubernetes, or AWS Lambda depending on needs.
- Case studies show HSDS is used by organizations like NREL and NSF to make petabytes of scientific data publicly accessible in the cloud.
Creating Cloud-Optimized HDF5 Files
This document discusses creating cloud-optimized HDF5 files by rearranging internal structures for more efficient data access in cloud object stores. It describes cloud-native and cloud-optimized storage formats, with the latter involving storing the entire HDF5 file as a single object. The benefits of cloud-optimized HDF5 include fast scanning and using the HDF5 library. Key aspects covered include using optimal chunk sizes, compression, and minimizing variable-length datatypes.
HDF5 OPeNDAP Handler Updates, and Performance Discussion
This document discusses updates and performance improvements to the HDF5 OPeNDAP data handler. It provides a history of the handler since 2001 and describes recent updates including supporting DAP4, new data types, and NetCDF data models. A performance study showed that passing compressed HDF5 data through the handler without decompressing/recompressing led to speedups of around 17-30x by leveraging HDF5 direct I/O APIs. This allows outputting HDF5 files as NetCDF files much faster through the handler.
Hyrax: Serving Data from S3
This document provides instructions for using the Hyrax software to serve scientific data files stored on Amazon S3 using the OPeNDAP data access protocol. It describes how to generate ancillary metadata files called DMR++ files using the get_dmrpp tool that provide information about the data file structure and locations. The document explains how to run get_dmrpp inside a Docker container to process data files on S3 and generate customized DMR++ files that the Hyrax server can use to serve the files to clients.
Accessing Cloud Data and Services Using EDL, Pydap, MATLAB
This document provides an overview and examples of accessing cloud data and services using the Earthdata Login (EDL), Pydap, and MATLAB. It discusses some common problems users encounter, such as being unable to access HDF5 data on AWS S3 using MATLAB or read data from OPeNDAP servers using Pydap. Solutions presented include using EDL to get temporary AWS tokens for S3 access in MATLAB and providing code examples on the HDFEOS website to help users access S3 data and OPeNDAP services. The document also notes some limitations, such as tokens being valid for only 1 hour, and workarounds like requesting new tokens or using the MATLAB HDF5 API instead of the netCDF API.
HDF - Current status and Future Directions
The HDF5 Roadmap and New Features document outlines upcoming changes and improvements to the HDF5 library. Key points include:
- HDF5 1.13.x releases will include new features like selection I/O, the Onion VFD for versioned files, improved VFD SWMR for single-writer multiple-reader access, and subfiling for parallel I/O.
- The Virtual Object Layer allows customizing HDF5 object storage and introduces terminal and pass-through connectors.
- The Onion VFD stores versions of HDF5 files in a separate onion file for versioned access.
- VFD SWMR improves on legacy SWMR by implementing single-writer multiple-reader capabilities
HDFEOS.org User Analsys, Updates, and Future
This document discusses user analysis of the HDFEOS.org website and plans for future improvements. It finds that the majority of the site's 100 daily users are "quiet", not posting on forums or other interactive elements. The main user types are locators, who search for examples or data; mergers, who combine or mosaic datasets; and converters, who change file formats. The document outlines recent updates focused on these user types, like adding Python examples for subsetting and calculating latitude and longitude. It proposes future work on artificial intelligence/machine learning uses of HDF files and examples for processing HDF data in the cloud.
HDF - Current status and Future Directions
This document summarizes a presentation about the current status and future directions of the Hierarchical Data Format (HDF) software. It provides updates on recent HDF5 releases, development efforts including new compression methods and ways to access HDF5 data, and outreach resources. It concludes by inviting the audience to share wishes for future HDF development.
H5Coro: The Cloud-Optimized Read-Only Library
The document describes H5Coro, a new C++ library for reading HDF5 files from cloud storage. H5Coro was created to optimize HDF5 reading for cloud environments by minimizing I/O operations through caching and efficient HTTP requests. Performance tests showed H5Coro was 77-132x faster than the previous HDF5 library at reading HDF5 data from Amazon S3 for NASA's SlideRule project. H5Coro supports common HDF5 elements but does not support writing or some complex HDF5 data types and messages to focus on optimized read-only performance for time series data stored sequentially in memory.
MATLAB Modernization on HDF5 1.10
This document summarizes MathWorks' work to modernize MATLAB's support for HDF5. Key points include:
1) MATLAB now supports HDF5 1.10.7 features like single-writer/multiple-reader access and virtual datasets through new and updated low-level functions.
2) Performance benchmarks show some improvements but also regressions compared to the previous HDF5 version, and work continues to optimize code and support future versions.
3) There are compatibility considerations for Linux filter plugins, but interim solutions are provided until MathWorks can ship a single HDF5 version.
HDF for the Cloud - Serverless HDF
HSDS provides HDF as a service through a REST API that can scale across nodes. New releases will enable serverless operation using AWS Lambda or direct client access without a server. This allows HDF data to be accessed remotely without managing servers. HSDS stores each HDF object separately, making it compatible with cloud object storage. Performance on AWS Lambda is slower than a dedicated server but has no management overhead. Direct client access has better performance but limits collaboration between clients.
HDF5 <-> Zarr
HDF5 and Zarr are data formats that can be used to store and access scientific data. This presentation discusses approaches to translating between the two formats. It describes how HDF5 files were translated to the Zarr format by creating a separate Zarr store to hold HDF5 file chunks, and storing chunk location metadata. It also discusses an implementation that translates Zarr data to the HDF5 format by using a special chunking layout and storing chunk information in an HDF5 compound dataset. Limitations of the translations include lack of support for some HDF5 dataset properties in Zarr, and lack of support for some Zarr compression methods in the HDF5 implementation.
HDF for the Cloud - New HDF Server Features
The document discusses HDF for the cloud, including new features of the HDF Server and what's next. Key points:
- HDF Server uses a "sharded schema" that maps HDF5 objects to individual storage objects, allowing parallel access and updates without transferring entire files.
- Implementations include HSDS software that uses the sharded schema with an API and SDKs for different languages like h5pyd for Python.
- New features of HSDS 0.6 include support for POSIX, Azure, AWS Lambda, and role-based access control.
- Future work includes direct access to storage without a server intermediary for some use cases.
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3The HDF-EOS Tools and Information Center
This document compares different methods for accessing HDF and netCDF files stored on Amazon S3, including Apache Drill, THREDDS Data Server (TDS), and HDF5 Virtual File Driver (VFD). A benchmark test of accessing a 24GB HDF5/netCDF-4 file on S3 from Amazon EC2 found that TDS performed the best, responding within 2 minutes, while Apache Drill failed after 7 minutes. The document concludes that TDS 5.0 is the clear winner based on performance and support for role-based access control and HDF4 files, but the best solution depends on use case and software.STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...
STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...The HDF-EOS Tools and Information Center
This document discusses STARE-PODS, a proposal to NASA/ACCESS-19 to develop a scalable data store for earth science data using the SpatioTemporal Adaptive Resolution Encoding (STARE) indexing scheme. STARE allows diverse earth science data to be unified and indexed, enabling the data to be partitioned and stored in a Parallel Optimized Data Store (PODS) for efficient analysis. The HDF Virtual Object Layer and Virtual Data Set technologies can then provide interfaces to access the data in STARE-PODS in a familiar way. The goal is for STARE-PODS to organize diverse data for alignment and parallel/distributed storage and processing to enable integrative analysis at scale.HDF5 and Ecosystem: What Is New?
This document provides an overview and update on HDF5 and its ecosystem. Key points include:
- HDF5 1.12.0 was recently released with new features like the Virtual Object Layer and external references.
- The HDF5 library now supports accessing data in the cloud using connectors like S3 VFD and REST VOL without needing to modify applications.
- Projects like HDFql and H5CPP provide additional interfaces for querying and working with HDF5 files from languages like SQL, C++, and Python.
- The HDF5 community is moving development to GitHub and improving documentation resources on the HDF wiki site.
HDF5 Roadmap 2019-2020
This document summarizes new features in HDF5 1.12.0, including support for storing references to objects and attributes across files, new storage backends using a virtual object layer (VOL), and virtual file drivers (VFDs) for Amazon S3 and HDFS. It outlines the HDF5 roadmap for 2019-2022, which includes continued support for HDF5 1.8 and 1.10, and new features in future 1.12.x releases like querying, indexing, and provenance tracking.
More from The HDF-EOS Tools and Information Center (20)
Highly Scalable Data Service (HSDS) Performance Features
Highly Scalable Data Service (HSDS) Performance Features
HDF5 OPeNDAP Handler Updates, and Performance Discussion
HDF5 OPeNDAP Handler Updates, and Performance Discussion
Accessing Cloud Data and Services Using EDL, Pydap, MATLAB
Accessing Cloud Data and Services Using EDL, Pydap, MATLAB
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3
STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...
STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...
Recently uploaded
Demystifying Knowledge Management through Storytelling
The Department of Veteran Affairs (VA) invited Taylor Paschal, Knowledge & Information Management Consultant at Enterprise Knowledge, to speak at a Knowledge Management Lunch and Learn hosted on June 12, 2024. All Office of Administration staff were invited to attend and received professional development credit for participating in the voluntary event.
The objectives of the Lunch and Learn presentation were to:
- Review what KM ‘is’ and ‘isn’t’
- Understand the value of KM and the benefits of engaging
- Define and reflect on your “what’s in it for me?”
- Share actionable ways you can participate in Knowledge - - Capture & Transfer
AppSec PNW: Android and iOS Application Security with MobSF
Mobile Security Framework - MobSF is a free and open source automated mobile application security testing environment designed to help security engineers, researchers, developers, and penetration testers to identify security vulnerabilities, malicious behaviours and privacy concerns in mobile applications using static and dynamic analysis. It supports all the popular mobile application binaries and source code formats built for Android and iOS devices. In addition to automated security assessment, it also offers an interactive testing environment to build and execute scenario based test/fuzz cases against the application.
This talk covers:
Using MobSF for static analysis of mobile applications.
Interactive dynamic security assessment of Android and iOS applications.
Solving Mobile app CTF challenges.
Reverse engineering and runtime analysis of Mobile malware.
How to shift left and integrate MobSF/mobsfscan SAST and DAST in your build pipeline.
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Gursev Pirge, PhD
Senior Data Scientist - JohnSnowLabs
MySQL InnoDB Storage Engine: Deep Dive - Mydbops
This presentation, titled "MySQL - InnoDB" and delivered by Mayank Prasad at the Mydbops Open Source Database Meetup 16 on June 8th, 2024, covers dynamic configuration of REDO logs and instant ADD/DROP columns in InnoDB.
This presentation dives deep into the world of InnoDB, exploring two ground-breaking features introduced in MySQL 8.0:
• Dynamic Configuration of REDO Logs: Enhance your database's performance and flexibility with on-the-fly adjustments to REDO log capacity. Unleash the power of the snake metaphor to visualize how InnoDB manages REDO log files.
• Instant ADD/DROP Columns: Say goodbye to costly table rebuilds! This presentation unveils how InnoDB now enables seamless addition and removal of columns without compromising data integrity or incurring downtime.
Key Learnings:
• Grasp the concept of REDO logs and their significance in InnoDB's transaction management.
• Discover the advantages of dynamic REDO log configuration and how to leverage it for optimal performance.
• Understand the inner workings of instant ADD/DROP columns and their impact on database operations.
• Gain valuable insights into the row versioning mechanism that empowers instant column modifications.
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
Tomaz Bratanic
Graph ML and GenAI Expert - Neo4j
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
The typical problem in product engineering is not bad strategy, so much as “no strategy”. This leads to confusion, lack of motivation, and incoherent action. The next time you look for a strategy and find an empty space, instead of waiting for it to be filled, I will show you how to fill it in yourself. If you’re wrong, it forces a correction. If you’re right, it helps create focus. I’ll share how I’ve approached this in the past, both what works and lessons for what didn’t work so well.
Mutation Testing for Task-Oriented Chatbots
Conversational agents, or chatbots, are increasingly used to access all sorts of services using natural language. While open-domain chatbots - like ChatGPT - can converse on any topic, task-oriented chatbots - the focus of this paper - are designed for specific tasks, like booking a flight, obtaining customer support, or setting an appointment. Like any other software, task-oriented chatbots need to be properly tested, usually by defining and executing test scenarios (i.e., sequences of user-chatbot interactions). However, there is currently a lack of methods to quantify the completeness and strength of such test scenarios, which can lead to low-quality tests, and hence to buggy chatbots.
To fill this gap, we propose adapting mutation testing (MuT) for task-oriented chatbots. To this end, we introduce a set of mutation operators that emulate faults in chatbot designs, an architecture that enables MuT on chatbots built using heterogeneous technologies, and a practical realisation as an Eclipse plugin. Moreover, we evaluate the applicability, effectiveness and efficiency of our approach on open-source chatbots, with promising results.
ScyllaDB Tablets: Rethinking Replication
ScyllaDB is making a major architecture shift. We’re moving from vNode replication to tablets – fragments of tables that are distributed independently, enabling dynamic data distribution and extreme elasticity. In this keynote, ScyllaDB co-founder and CTO Avi Kivity explains the reason for this shift, provides a look at the implementation and roadmap, and shares how this shift benefits ScyllaDB users.
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
What began over 115 years ago as a supplier of precision gauges to the automotive industry has evolved into being an industry leader in the manufacture of product branding, automotive cockpit trim and decorative appliance trim. Value-added services include in-house Design, Engineering, Program Management, Test Lab and Tool Shops.
From Natural Language to Structured Solr Queries using LLMs
This talk draws on experimentation to enable AI applications with Solr. One important use case is to use AI for better accessibility and discoverability of the data: while User eXperience techniques, lexical search improvements, and data harmonization can take organizations to a good level of accessibility, a structural (or “cognitive” gap) remains between the data user needs and the data producer constraints.
That is where AI – and most importantly, Natural Language Processing and Large Language Model techniques – could make a difference. This natural language, conversational engine could facilitate access and usage of the data leveraging the semantics of any data source.
The objective of the presentation is to propose a technical approach and a way forward to achieve this goal.
The key concept is to enable users to express their search queries in natural language, which the LLM then enriches, interprets, and translates into structured queries based on the Solr index’s metadata.
This approach leverages the LLM’s ability to understand the nuances of natural language and the structure of documents within Apache Solr.
The LLM acts as an intermediary agent, offering a transparent experience to users automatically and potentially uncovering relevant documents that conventional search methods might overlook. The presentation will include the results of this experimental work, lessons learned, best practices, and the scope of future work that should improve the approach and make it production-ready.
QR Secure: A Hybrid Approach Using Machine Learning and Security Validation F...
QR Secure: A Hybrid Approach Using Machine Learning and Security Validation Functions to Prevent Interaction with Malicious QR Codes.
Aim of the Study: The goal of this research was to develop a robust hybrid approach for identifying malicious and insecure URLs derived from QR codes, ensuring safe interactions.
This is achieved through:
Machine Learning Model: Predicts the likelihood of a URL being malicious.
Security Validation Functions: Ensures the derived URL has a valid certificate and proper URL format.
This innovative blend of technology aims to enhance cybersecurity measures and protect users from potential threats hidden within QR codes 🖥 🔒
This study was my first introduction to using ML which has shown me the immense potential of ML in creating more secure digital environments!
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
Direct losses from downtime in 1 minute = $5-$10 thousand dollars. Reputation is priceless.
As part of the talk, we will consider the architectural strategies necessary for the development of highly loaded fintech solutions. We will focus on using queues and streaming to efficiently work and manage large amounts of data in real-time and to minimize latency.
We will focus special attention on the architectural patterns used in the design of the fintech system, microservices and event-driven architecture, which ensure scalability, fault tolerance, and consistency of the entire system.
Containers & AI - Beauty and the Beast!?!
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
Keywords: AI, Containeres, Kubernetes, Cloud Native
Event Link: https://meine.doag.org/events/cloudland/2024/agenda/#agendaId.4211
Dandelion Hashtable: beyond billion requests per second on a commodity server
This slide deck presents DLHT, a concurrent in-memory hashtable. Despite efforts to optimize hashtables, that go as far as sacrificing core functionality, state-of-the-art designs still incur multiple memory accesses per request and block request processing in three cases. First, most hashtables block while waiting for data to be retrieved from memory. Second, open-addressing designs, which represent the current state-of-the-art, either cannot free index slots on deletes or must block all requests to do so. Third, index resizes block every request until all objects are copied to the new index. Defying folklore wisdom, DLHT forgoes open-addressing and adopts a fully-featured and memory-aware closed-addressing design based on bounded cache-line-chaining. This design offers lock-free index operations and deletes that free slots instantly, (2) completes most requests with a single memory access, (3) utilizes software prefetching to hide memory latencies, and (4) employs a novel non-blocking and parallel resizing. In a commodity server and a memory-resident workload, DLHT surpasses 1.6B requests per second and provides 3.5x (12x) the throughput of the state-of-the-art closed-addressing (open-addressing) resizable hashtable on Gets (Deletes).
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
How we managed to grow and scale a RAG application from zero to thousands of users in 7 months. Lessons from technical challenges around managing high load for LLMs, RAGs and Vector databases.
Leveraging the Graph for Clinical Trials and Standards
Katja Glaß
OpenStudyBuilder Community Manager - Katja Glaß Consulting
Marius Conjeaud
Principal Consultant - Neo4j
A Deep Dive into ScyllaDB's Architecture
This talk will cover ScyllaDB Architecture from the cluster-level view and zoom in on data distribution and internal node architecture. In the process, we will learn the secret sauce used to get ScyllaDB's high availability and superior performance. We will also touch on the upcoming changes to ScyllaDB architecture, moving to strongly consistent metadata and tablets.
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Dmitrii Kamaev, PhD
Senior Product Owner - QIAGEN
QA or the Highway - Component Testing: Bridging the gap between frontend appl...
These are the slides for the presentation, "Component Testing: Bridging the gap between frontend applications" that was presented at QA or the Highway 2024 in Columbus, OH by Zachary Hamm.
Recently uploaded (20)
Demystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through Storytelling
AppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSF
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
From Natural Language to Structured Solr Queries using LLMs
From Natural Language to Structured Solr Queries using LLMs
QR Secure: A Hybrid Approach Using Machine Learning and Security Validation F...
QR Secure: A Hybrid Approach Using Machine Learning and Security Validation F...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
Leveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and Standards
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
QA or the Highway - Component Testing: Bridging the gap between frontend appl...
QA or the Highway - Component Testing: Bridging the gap between frontend appl...
HDF and Augmentation
- 1. 1July 8 – 11, 2014 HDF and Augmentation Mike Folk The HDF Group ESIP 2014
- 2. www.hdfgroup.orgESIP Summer MeetingJuly 8 – 11, 2014 2 * *shoes that remove ice and snow
- 3. www.hdfgroup.orgESIP Summer MeetingJuly 8 – 11, 2014 3 * *shoes that remove ice and snow Augment: Add to, amplify, extend, enhance. thesaurus.com
- 4. www.hdfgroup.orgESIP Summer Meeting Augmentation An HDF design philosophy July 8 – 11, 2014 4
- 5. www.hdfgroup.orgESIP Summer Meeting Why we augment July 8 – 11, 2014 5 • Serve intended purpose better. • Whole new purpose. • Available to new tools and applications • More “standard,” hence more widely accessible. • Preservation
- 6. www.hdfgroup.orgESIP Summer Meeting HDF augmentation • Preserve the original use • Add needed extra information • Rename things • Move things around • Reduce complexity • Hide things July 8 – 11, 2014 6
- 7. July 8 – 11, 2014 7 AUGMENTING HDF-EOS5 FILES FOR NETCDF-4 ACCESS
- 8. www.hdfgroup.orgESIP Summer Meeting HDF5 HDF-EOS5 HDF-EOS5 file netCDF4 to access HDF-EOS5 files • HDF-EOS5 uses HDF5 as its storage layer • Adds EOS structural and content metadata July 8 – 11, 2014 8
- 9. www.hdfgroup.orgESIP Summer Meeting HDF5 HDF-EOS5 HDF-EOS5 file HDF5 NetCDF4 NetCDF-4 file netCDF4 to access HDF-EOS5 files • netCDF-4 uses HDF5 as its storage layer • Adds netCDF-4 structural and content metadata July 8 – 11, 2014 9
- 10. www.hdfgroup.orgESIP Summer Meeting HDF5 HDF-EOS5 HDF-EOS5 file HDF5 NetCDF4 NetCDF-4 file Augmented HDF-EOS5 file netCDF4 to access HDF-EOS5 files • netCDF-4 metadata added to HDF-EOS5 file • EOS5 use not affected. July 8 – 11, 2014 10
- 11. July 8 – 11, 2014 11 H5AUGJPSS
- 12. www.hdfgroup.orgESIP Summer Meeting IDV • IDV • Based on netCDF classic model (no groups) • IDV required attributes and variables • Scale factors, offsets • Geolocation metadata • “Radiance” variable July 8 – 11, 2014 12
- 13. www.hdfgroup.orgESIP Summer Meeting H5augjpss will … • Make a JPSS HDF5 file look like netCDF-4 • Hide HDF5 elements not supported by netCDF • Add coordinate variables • Remove groups from view to conform to netCDF classic • Add information needed by tools such as IDV • Geolocation metadata • “Radiance” variable July 8 – 11, 2014 13
- 14. HDF4 FILE CONTENT MAPS
- 15. www.hdfgroup.orgESIP Summer Meeting HDF4 File Contents – User View July 8 – 11, 2014 15
- 16. www.hdfgroup.orgESIP Summer Meeting HDF4 File Contents – User View Objects & Relationships July 8 – 11, 2014 16
- 17. www.hdfgroup.orgESIP Summer Meeting HDF4 File Contents – User View User Metadata July 8 – 11, 2014 17
- 18. www.hdfgroup.orgESIP Summer Meeting HDF4 File Contents – User View Object Data July 8 – 11, 2014 18
- 19. www.hdfgroup.orgESIP Summer Meeting HDF4 files are complex July 8 – 11, 2014 19
- 20. www.hdfgroup.orgESIP Summer Meeting Project Motivation HDF4 file HDF4 Library HDFView
- 21. www.hdfgroup.orgESIP Summer Meeting Project Motivation HDF4 file HDF4 Library HDFView X
- 22. www.hdfgroup.orgESIP Summer Meeting HDF4 mapping workflow HDF4 File HDF4 Mapping File (XML document) hmap linked with HDF4 library July 8 – 11, 2014 22
- 23. www.hdfgroup.orgESIP Summer Meeting HDF4 mapping workflow HDF4 File HDF4 Mapping File (XML document) hmap linked with HDF4 library Reader program Object Data Groups, Data Objects, Structural and Application Metadata; Locations of Object Data July 8 – 11, 2014 23
- 24. July 8 – 11, 2014 24