HDF5 In Support of Database Applications
•Download as PPT, PDF•
0 likes•988 views
This document discusses three approaches to integrating HDF5 data with databases: I) Using PyTables to treat HDF5 like a database, II) Using HDF5 and a relational database side-by-side, and III) Fully integrating HDF5 and a relational database where the HDF5 data model is exposed to SQL and other database technologies.
Report
Share
Report
Share
Recommended
HDF Update
Update on HDF, including recent changes to the software, upcoming releases, collaborations, future plans. Will include an overview of the upcoming HDF5 1.8 release, and updates on the netCDF4/HDF5 merge, HDF5 support for indexing, BioHDF, the HDF5-Storage Resource Broker project, the NPOESS BAA, HDF5-OPeNDAP project, HDF-EOS library and website supports and the HDF spin-off THG.
MATLAB Modernization on HDF5 1.10
This document summarizes MathWorks' work to modernize MATLAB's support for HDF5. Key points include:
1) MATLAB now supports HDF5 1.10.7 features like single-writer/multiple-reader access and virtual datasets through new and updated low-level functions.
2) Performance benchmarks show some improvements but also regressions compared to the previous HDF5 version, and work continues to optimize code and support future versions.
3) There are compatibility considerations for Linux filter plugins, but interim solutions are provided until MathWorks can ship a single HDF5 version.
HDFEOS.org User Analsys, Updates, and Future
This document discusses user analysis of the HDFEOS.org website and plans for future improvements. It finds that the majority of the site's 100 daily users are "quiet", not posting on forums or other interactive elements. The main user types are locators, who search for examples or data; mergers, who combine or mosaic datasets; and converters, who change file formats. The document outlines recent updates focused on these user types, like adding Python examples for subsetting and calculating latitude and longitude. It proposes future work on artificial intelligence/machine learning uses of HDF files and examples for processing HDF data in the cloud.
HDF5 and Ecosystem: What Is New?
This document provides an overview and update on HDF5 and its ecosystem. Key points include:
- HDF5 1.12.0 was recently released with new features like the Virtual Object Layer and external references.
- The HDF5 library now supports accessing data in the cloud using connectors like S3 VFD and REST VOL without needing to modify applications.
- Projects like HDFql and H5CPP provide additional interfaces for querying and working with HDF5 files from languages like SQL, C++, and Python.
- The HDF5 community is moving development to GitHub and improving documentation resources on the HDF wiki site.
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3The HDF-EOS Tools and Information Center
This document compares different methods for accessing HDF and netCDF files stored on Amazon S3, including Apache Drill, THREDDS Data Server (TDS), and HDF5 Virtual File Driver (VFD). A benchmark test of accessing a 24GB HDF5/netCDF-4 file on S3 from Amazon EC2 found that TDS performed the best, responding within 2 minutes, while Apache Drill failed after 7 minutes. The document concludes that TDS 5.0 is the clear winner based on performance and support for role-based access control and HDF4 files, but the best solution depends on use case and software.HDF for the Cloud - Serverless HDF
HSDS provides HDF as a service through a REST API that can scale across nodes. New releases will enable serverless operation using AWS Lambda or direct client access without a server. This allows HDF data to be accessed remotely without managing servers. HSDS stores each HDF object separately, making it compatible with cloud object storage. Performance on AWS Lambda is slower than a dedicated server but has no management overhead. Direct client access has better performance but limits collaboration between clients.
Product Designer Hub - Taking HPD to the Web
The document discusses the Product Designer Hub, which aims to take the HDF data format to the web by developing a system with the following key components:
1. A data store to manage user accounts, projects, file structures and metadata.
2. A RESTful server to enable exporting HDF files and data to various formats like JSON, CSV, and importing from formats like XML, MATLAB, and Python.
3. A web service that can ingest metadata from various XML dialects, transform it to HDF groups and attributes, and generate HDF files, JSON, and code templates from the metadata.
4. A client-server architecture where client software like web applications can access HDF files and data through the REST
H5Coro: The Cloud-Optimized Read-Only Library
The document describes H5Coro, a new C++ library for reading HDF5 files from cloud storage. H5Coro was created to optimize HDF5 reading for cloud environments by minimizing I/O operations through caching and efficient HTTP requests. Performance tests showed H5Coro was 77-132x faster than the previous HDF5 library at reading HDF5 data from Amazon S3 for NASA's SlideRule project. H5Coro supports common HDF5 elements but does not support writing or some complex HDF5 data types and messages to focus on optimized read-only performance for time series data stored sequentially in memory.
Recommended
HDF Update
Update on HDF, including recent changes to the software, upcoming releases, collaborations, future plans. Will include an overview of the upcoming HDF5 1.8 release, and updates on the netCDF4/HDF5 merge, HDF5 support for indexing, BioHDF, the HDF5-Storage Resource Broker project, the NPOESS BAA, HDF5-OPeNDAP project, HDF-EOS library and website supports and the HDF spin-off THG.
MATLAB Modernization on HDF5 1.10
This document summarizes MathWorks' work to modernize MATLAB's support for HDF5. Key points include:
1) MATLAB now supports HDF5 1.10.7 features like single-writer/multiple-reader access and virtual datasets through new and updated low-level functions.
2) Performance benchmarks show some improvements but also regressions compared to the previous HDF5 version, and work continues to optimize code and support future versions.
3) There are compatibility considerations for Linux filter plugins, but interim solutions are provided until MathWorks can ship a single HDF5 version.
HDFEOS.org User Analsys, Updates, and Future
This document discusses user analysis of the HDFEOS.org website and plans for future improvements. It finds that the majority of the site's 100 daily users are "quiet", not posting on forums or other interactive elements. The main user types are locators, who search for examples or data; mergers, who combine or mosaic datasets; and converters, who change file formats. The document outlines recent updates focused on these user types, like adding Python examples for subsetting and calculating latitude and longitude. It proposes future work on artificial intelligence/machine learning uses of HDF files and examples for processing HDF data in the cloud.
HDF5 and Ecosystem: What Is New?
This document provides an overview and update on HDF5 and its ecosystem. Key points include:
- HDF5 1.12.0 was recently released with new features like the Virtual Object Layer and external references.
- The HDF5 library now supports accessing data in the cloud using connectors like S3 VFD and REST VOL without needing to modify applications.
- Projects like HDFql and H5CPP provide additional interfaces for querying and working with HDF5 files from languages like SQL, C++, and Python.
- The HDF5 community is moving development to GitHub and improving documentation resources on the HDF wiki site.
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3The HDF-EOS Tools and Information Center
This document compares different methods for accessing HDF and netCDF files stored on Amazon S3, including Apache Drill, THREDDS Data Server (TDS), and HDF5 Virtual File Driver (VFD). A benchmark test of accessing a 24GB HDF5/netCDF-4 file on S3 from Amazon EC2 found that TDS performed the best, responding within 2 minutes, while Apache Drill failed after 7 minutes. The document concludes that TDS 5.0 is the clear winner based on performance and support for role-based access control and HDF4 files, but the best solution depends on use case and software.HDF for the Cloud - Serverless HDF
HSDS provides HDF as a service through a REST API that can scale across nodes. New releases will enable serverless operation using AWS Lambda or direct client access without a server. This allows HDF data to be accessed remotely without managing servers. HSDS stores each HDF object separately, making it compatible with cloud object storage. Performance on AWS Lambda is slower than a dedicated server but has no management overhead. Direct client access has better performance but limits collaboration between clients.
Product Designer Hub - Taking HPD to the Web
The document discusses the Product Designer Hub, which aims to take the HDF data format to the web by developing a system with the following key components:
1. A data store to manage user accounts, projects, file structures and metadata.
2. A RESTful server to enable exporting HDF files and data to various formats like JSON, CSV, and importing from formats like XML, MATLAB, and Python.
3. A web service that can ingest metadata from various XML dialects, transform it to HDF groups and attributes, and generate HDF files, JSON, and code templates from the metadata.
4. A client-server architecture where client software like web applications can access HDF files and data through the REST
H5Coro: The Cloud-Optimized Read-Only Library
The document describes H5Coro, a new C++ library for reading HDF5 files from cloud storage. H5Coro was created to optimize HDF5 reading for cloud environments by minimizing I/O operations through caching and efficient HTTP requests. Performance tests showed H5Coro was 77-132x faster than the previous HDF5 library at reading HDF5 data from Amazon S3 for NASA's SlideRule project. H5Coro supports common HDF5 elements but does not support writing or some complex HDF5 data types and messages to focus on optimized read-only performance for time series data stored sequentially in memory.
HDF - Current status and Future Directions
This document summarizes a presentation about the current status and future directions of the Hierarchical Data Format (HDF) software. It provides updates on recent HDF5 releases, development efforts including new compression methods and ways to access HDF5 data, and outreach resources. It concludes by inviting the audience to share wishes for future HDF development.
HDF-EOS 2/5 to netCDF Converter
The tool takes HDF-EOS 5 data as input, and generates COARDS-compatible output - if the input file has enough metadata to be COARDS-compliant, the output file will be COARDS-compliant. The tool is written in portable C, and ought to run on any platform where the HDF-EOS and netCDF libraries are available.
This year, we have made two major enhancements to the converter:
It now automatically detects whether its input is HDF-EOS2 or HDF-EOS5 format, and handles either one. The previous tool worked with HDF-EOS5 only.
Its netCDF output attempts to conform to the new CF conventions (a superset of the COARDS conventions). This is primarily an improvement in its translation of Swath datasets, which CF handles much better than COARDS.
HDF Update
Mike Folk from the National Center for Supercomputing Applications gave an update on HDF software in 2003. HDF is supported by several government agencies for applications in earth science, simulations, and data-intensive computing. Version 4.2 Release 1 was planned for October 2003 with bug fixes and new features. HDF5 1.6.0 was released in July 2003 with new filters, properties, and performance improvements. Work was also being done on high-level APIs, parallel HDF5, tools, and collaborations with other projects.
HDF Update
This document summarizes HDF software activities in 2002, including support and funding sources for HDF, recent and upcoming releases of HDF4 and HDF5 libraries and tools, and other HDF-related projects. The HDF5 library saw improvements to performance, compilers supported, and tools like HDFView and converters. The next major HDF5 release in 2003 will focus on new features, performance enhancements, and special platform support. High level APIs and the parallel HDF5 programming model were also under development.
Parallel Computing with HDF Server
The document discusses Parallel Computing with HDF Server. The key points are:
1. HDF Server (HSDS) allows efficient access to HDF5 data stored in AWS S3. It runs as containers on Kubernetes and supports parallel access across containers.
2. HSDS uses S3 as the data store for HDF5 files. Individual HDF5 objects like datasets and chunks are stored as separate S3 objects. This allows parallel read/write and only modifying what changes.
3. HDF Kita Lab is a hosted Jupyter environment on AWS that provides access to HSDS for reading and writing HDF5 data on S3. It allows scaling the server and provides tools for HDF5 on S3.
Hierarchical Data Formats (HDF) Update
The document summarizes updates on Hierarchical Data Formats (HDF) software releases and tools. It discusses the latest releases of HDF5 1.8.19 and 1.10.1, compatibility issues when moving to newer versions, updates on tools like HDF-Java and HDFView 3.0, supported compilers and systems, and a new compression library for interoperability. It invites readers to provide feedback on their needs.
HDF Update for DAAC Managers (2017-02-27)
This document provides information about HDF (Hierarchical Data Format) tools and resources for working with Earth observation data. It summarizes HDF's focus on helping users at different stages of working with data, from initial product design to long-term archiving. It also describes specific HDF tools for viewing, comparing, converting between formats and adding metadata to scientific data files.
Migrating from HDF5 1.6 to 1.8
The document discusses migrating from HDF5 1.6 to HDF5 1.8. It provides an overview of new features in HDF5 1.8, including a revised file format, improvements to group storage, new link types like external links, and enhanced error handling. The document recommends helping with the transition to HDF5 1.8 by discussing beneficial new features and awareness of compatibility issues when moving from 1.6 to 1.8.
Efficiently serving HDF5 via OPeNDAP
This presentation discusses serving HDF5 data via OPeNDAP for efficient access and visualization. The HDF5 handler with CF and NcML modules translates HDF5 file layout and metadata to comply with CF conventions and provides the data via OPeNDAP to tools like Hyrax. Caching and optimizations improve performance. Future work includes supporting additional data types and file formats in the HDF5 handler.
Introduction to HDF5 Data and Programming Models
This document provides an overview of HDF5 (Hierarchical Data Format version 5) and introduces its core concepts. HDF5 is an open source file format and software library designed for storing and managing large amounts of numerical data. It supports a data model with objects such as datasets, groups, attributes, and datatypes. HDF5 files can be accessed through its software library and APIs from languages like C, Fortran, C++, Python and more. The document covers HDF5's data model, file format, programming interfaces, tools and example code.
Caching and Buffering in HDF5
In this talk we will discuss caching and buffering strategies in HDF5. The information presented will help developers write more efficient applications and avoid performance bottlenecks.
HDF Product Designer: Using Templates to Achieve Interoperability
HDF Product Designer: Using Templates to Achieve InteroperabilityThe HDF-EOS Tools and Information Center
This document discusses how HDF Product Designer (HPD) uses templates to achieve interoperability. HPD is an application for consistently developing interoperable data content in HDF5 files. It has a client-server architecture and desktop app. Templates allow users to copy design examples that incorporate best practices and are curated by the HPD development team. Available templates include NCEI collections and CF templates, with more to be added based on community review and suggestions. Templates allow users to initialize new designs by mixing and matching content from different template examples.HDF-EOS to GeoTIFF Conversion Tool & HDF-EOS Plug-in for HDFView
HDF-EOS to GeoTIFF Conversion Tool & HDF-EOS Plug-in for HDFViewThe HDF-EOS Tools and Information Center
This document describes tools for working with HDF-EOS data formats: the HDF-EOS to GeoTIFF Conversion Tool (HEG) and an HDF-EOS plugin for the HDFView browser. HEG allows conversion between HDF-EOS and GeoTIFF formats and supports subsetting, reprojection, stitching and other functions. The HDFView plugin allows browsing and conversion between HDF-EOS2 and HDF-EOS5 formats within HDFView. Both tools are available for download and support a variety of Earth science datasets.Moving form HDF4 to HDF5/netCDF-4
This presentation discusses moving data and applications from HDF4 to HDF5/netCDF-4. It covers the differences between HDF4 and HDF5 data models and capabilities, tools for converting HDF4 data to HDF5, advantages of HDF5 like unlimited dimensions and compression, and ways to ensure compatibility with netCDF-4 like avoiding HDF5-specific features. The work was supported by a NASA contract.
Integrating HDF5 with SRB
Fast partial access to objects from very large files in the SDSC Storage Resource Broker (SRB5) can be extremely challenging, even when those objects are small. The HDF-SRB project integrates the SRB and NCSA Hierarchical Data Format (HDF5), to create an access mechanism within the SRB that is can be orders of magnitude more efficient than current methods for accessing object-based file formats.
The project provides interactive and efficient access to datasets or subsets of datasets in large files without bringing entire files into local machines. A new set of data structures and APIs have been implemented to the SRB support such object-level data access. A working prototype of the HDF5-SRB data system has been developed and tested. The SRB support is implemented in HDFView as a client application.
Parallel HDF5 Developments
The document discusses recent and upcoming improvements to parallel HDF5 for improved I/O performance on HPC systems. Recent improvements include reducing file truncations, distributing metadata writes across processes, and improved selection matching. Upcoming work includes a high-level HPC API, funding for Exascale-focused enhancements, and future improvements like asynchronous I/O and auto-tuning to parallel file systems. Performance tips are also provided like passing MPI hints and using collective I/O.
HDF and netCDF Data Support in ArcGIS
The document discusses HDF and netCDF data support in ArcGIS. It provides an overview of how HDF and netCDF data can be directly ingested and used as raster datasets, mosaic datasets, feature layers, and tables in ArcGIS. This allows for scientific data to be displayed, analyzed, and shared using common GIS tools and services. It also describes existing Python tools for working with netCDF data and outlines future areas of development, including improved support for HDF5, THREDDS/OPeNDAP access, and evolving data standards.
HDF Product Designer
The document describes the HDF Product Designer software tool. It was created to facilitate the design of interoperable scientific data products in HDF5 format. The tool allows intuitive editing of HDF5 objects and supports conventions like CF and ACDD. It also provides validation services to test file compliance. The goal is to help scientists design data products that follow standards and are easy for others to use.
HDF5 <-> Zarr
HDF5 and Zarr are data formats that can be used to store and access scientific data. This presentation discusses approaches to translating between the two formats. It describes how HDF5 files were translated to the Zarr format by creating a separate Zarr store to hold HDF5 file chunks, and storing chunk location metadata. It also discusses an implementation that translates Zarr data to the HDF5 format by using a special chunking layout and storing chunk information in an HDF5 compound dataset. Limitations of the translations include lack of support for some HDF5 dataset properties in Zarr, and lack of support for some Zarr compression methods in the HDF5 implementation.
MODIS Land and HDF-EOS
The document discusses MODIS Land products and their distribution format. MODIS Land products include radiation budget, ecosystem, and land cover variables. Products are distributed in HDF-EOS format with fine resolution grids in Integerized Sinusoidal or Lambert Azimuthal Equal-Area projections and coarse grids in a geographic Climate Modeling Grid. HDF-EOS allows for collaboration and standard geolocation representation but toolkit support is still limited.
Cloud-Optimized HDF5 Files
This document discusses how to optimize HDF5 files for efficient access in cloud object stores. Key optimizations include using large dataset chunk sizes of 1-4 MiB, consolidating internal file metadata, and minimizing variable-length datatypes. The document recommends creating files with paged aggregation and storing file content information in the user block to enable fast discovery of file contents when stored in object stores.
Accessing HDF5 data in the cloud with HSDS
This document provides an overview of HSDS (Highly Scalable Data Service), which is a REST-based service that allows accessing HDF5 data stored in the cloud. It discusses how HSDS maps HDF5 objects like datasets and groups to individual cloud storage objects to optimize performance. The document also describes how HSDS was used to improve access performance for NASA ICESat-2 HDF5 data on AWS S3 by hyper-chunking datasets into larger chunks spanning multiple original HDF5 chunks. Benchmark results showed that accessing the data through HSDS provided over 2x faster performance than other methods like ROS3 or S3FS that directly access the cloud storage.
More Related Content
What's hot
HDF - Current status and Future Directions
This document summarizes a presentation about the current status and future directions of the Hierarchical Data Format (HDF) software. It provides updates on recent HDF5 releases, development efforts including new compression methods and ways to access HDF5 data, and outreach resources. It concludes by inviting the audience to share wishes for future HDF development.
HDF-EOS 2/5 to netCDF Converter
The tool takes HDF-EOS 5 data as input, and generates COARDS-compatible output - if the input file has enough metadata to be COARDS-compliant, the output file will be COARDS-compliant. The tool is written in portable C, and ought to run on any platform where the HDF-EOS and netCDF libraries are available.
This year, we have made two major enhancements to the converter:
It now automatically detects whether its input is HDF-EOS2 or HDF-EOS5 format, and handles either one. The previous tool worked with HDF-EOS5 only.
Its netCDF output attempts to conform to the new CF conventions (a superset of the COARDS conventions). This is primarily an improvement in its translation of Swath datasets, which CF handles much better than COARDS.
HDF Update
Mike Folk from the National Center for Supercomputing Applications gave an update on HDF software in 2003. HDF is supported by several government agencies for applications in earth science, simulations, and data-intensive computing. Version 4.2 Release 1 was planned for October 2003 with bug fixes and new features. HDF5 1.6.0 was released in July 2003 with new filters, properties, and performance improvements. Work was also being done on high-level APIs, parallel HDF5, tools, and collaborations with other projects.
HDF Update
This document summarizes HDF software activities in 2002, including support and funding sources for HDF, recent and upcoming releases of HDF4 and HDF5 libraries and tools, and other HDF-related projects. The HDF5 library saw improvements to performance, compilers supported, and tools like HDFView and converters. The next major HDF5 release in 2003 will focus on new features, performance enhancements, and special platform support. High level APIs and the parallel HDF5 programming model were also under development.
Parallel Computing with HDF Server
The document discusses Parallel Computing with HDF Server. The key points are:
1. HDF Server (HSDS) allows efficient access to HDF5 data stored in AWS S3. It runs as containers on Kubernetes and supports parallel access across containers.
2. HSDS uses S3 as the data store for HDF5 files. Individual HDF5 objects like datasets and chunks are stored as separate S3 objects. This allows parallel read/write and only modifying what changes.
3. HDF Kita Lab is a hosted Jupyter environment on AWS that provides access to HSDS for reading and writing HDF5 data on S3. It allows scaling the server and provides tools for HDF5 on S3.
Hierarchical Data Formats (HDF) Update
The document summarizes updates on Hierarchical Data Formats (HDF) software releases and tools. It discusses the latest releases of HDF5 1.8.19 and 1.10.1, compatibility issues when moving to newer versions, updates on tools like HDF-Java and HDFView 3.0, supported compilers and systems, and a new compression library for interoperability. It invites readers to provide feedback on their needs.
HDF Update for DAAC Managers (2017-02-27)
This document provides information about HDF (Hierarchical Data Format) tools and resources for working with Earth observation data. It summarizes HDF's focus on helping users at different stages of working with data, from initial product design to long-term archiving. It also describes specific HDF tools for viewing, comparing, converting between formats and adding metadata to scientific data files.
Migrating from HDF5 1.6 to 1.8
The document discusses migrating from HDF5 1.6 to HDF5 1.8. It provides an overview of new features in HDF5 1.8, including a revised file format, improvements to group storage, new link types like external links, and enhanced error handling. The document recommends helping with the transition to HDF5 1.8 by discussing beneficial new features and awareness of compatibility issues when moving from 1.6 to 1.8.
Efficiently serving HDF5 via OPeNDAP
This presentation discusses serving HDF5 data via OPeNDAP for efficient access and visualization. The HDF5 handler with CF and NcML modules translates HDF5 file layout and metadata to comply with CF conventions and provides the data via OPeNDAP to tools like Hyrax. Caching and optimizations improve performance. Future work includes supporting additional data types and file formats in the HDF5 handler.
Introduction to HDF5 Data and Programming Models
This document provides an overview of HDF5 (Hierarchical Data Format version 5) and introduces its core concepts. HDF5 is an open source file format and software library designed for storing and managing large amounts of numerical data. It supports a data model with objects such as datasets, groups, attributes, and datatypes. HDF5 files can be accessed through its software library and APIs from languages like C, Fortran, C++, Python and more. The document covers HDF5's data model, file format, programming interfaces, tools and example code.
Caching and Buffering in HDF5
In this talk we will discuss caching and buffering strategies in HDF5. The information presented will help developers write more efficient applications and avoid performance bottlenecks.
HDF Product Designer: Using Templates to Achieve Interoperability
HDF Product Designer: Using Templates to Achieve InteroperabilityThe HDF-EOS Tools and Information Center
This document discusses how HDF Product Designer (HPD) uses templates to achieve interoperability. HPD is an application for consistently developing interoperable data content in HDF5 files. It has a client-server architecture and desktop app. Templates allow users to copy design examples that incorporate best practices and are curated by the HPD development team. Available templates include NCEI collections and CF templates, with more to be added based on community review and suggestions. Templates allow users to initialize new designs by mixing and matching content from different template examples.HDF-EOS to GeoTIFF Conversion Tool & HDF-EOS Plug-in for HDFView
HDF-EOS to GeoTIFF Conversion Tool & HDF-EOS Plug-in for HDFViewThe HDF-EOS Tools and Information Center
This document describes tools for working with HDF-EOS data formats: the HDF-EOS to GeoTIFF Conversion Tool (HEG) and an HDF-EOS plugin for the HDFView browser. HEG allows conversion between HDF-EOS and GeoTIFF formats and supports subsetting, reprojection, stitching and other functions. The HDFView plugin allows browsing and conversion between HDF-EOS2 and HDF-EOS5 formats within HDFView. Both tools are available for download and support a variety of Earth science datasets.Moving form HDF4 to HDF5/netCDF-4
This presentation discusses moving data and applications from HDF4 to HDF5/netCDF-4. It covers the differences between HDF4 and HDF5 data models and capabilities, tools for converting HDF4 data to HDF5, advantages of HDF5 like unlimited dimensions and compression, and ways to ensure compatibility with netCDF-4 like avoiding HDF5-specific features. The work was supported by a NASA contract.
Integrating HDF5 with SRB
Fast partial access to objects from very large files in the SDSC Storage Resource Broker (SRB5) can be extremely challenging, even when those objects are small. The HDF-SRB project integrates the SRB and NCSA Hierarchical Data Format (HDF5), to create an access mechanism within the SRB that is can be orders of magnitude more efficient than current methods for accessing object-based file formats.
The project provides interactive and efficient access to datasets or subsets of datasets in large files without bringing entire files into local machines. A new set of data structures and APIs have been implemented to the SRB support such object-level data access. A working prototype of the HDF5-SRB data system has been developed and tested. The SRB support is implemented in HDFView as a client application.
Parallel HDF5 Developments
The document discusses recent and upcoming improvements to parallel HDF5 for improved I/O performance on HPC systems. Recent improvements include reducing file truncations, distributing metadata writes across processes, and improved selection matching. Upcoming work includes a high-level HPC API, funding for Exascale-focused enhancements, and future improvements like asynchronous I/O and auto-tuning to parallel file systems. Performance tips are also provided like passing MPI hints and using collective I/O.
HDF and netCDF Data Support in ArcGIS
The document discusses HDF and netCDF data support in ArcGIS. It provides an overview of how HDF and netCDF data can be directly ingested and used as raster datasets, mosaic datasets, feature layers, and tables in ArcGIS. This allows for scientific data to be displayed, analyzed, and shared using common GIS tools and services. It also describes existing Python tools for working with netCDF data and outlines future areas of development, including improved support for HDF5, THREDDS/OPeNDAP access, and evolving data standards.
HDF Product Designer
The document describes the HDF Product Designer software tool. It was created to facilitate the design of interoperable scientific data products in HDF5 format. The tool allows intuitive editing of HDF5 objects and supports conventions like CF and ACDD. It also provides validation services to test file compliance. The goal is to help scientists design data products that follow standards and are easy for others to use.
HDF5 <-> Zarr
HDF5 and Zarr are data formats that can be used to store and access scientific data. This presentation discusses approaches to translating between the two formats. It describes how HDF5 files were translated to the Zarr format by creating a separate Zarr store to hold HDF5 file chunks, and storing chunk location metadata. It also discusses an implementation that translates Zarr data to the HDF5 format by using a special chunking layout and storing chunk information in an HDF5 compound dataset. Limitations of the translations include lack of support for some HDF5 dataset properties in Zarr, and lack of support for some Zarr compression methods in the HDF5 implementation.
MODIS Land and HDF-EOS
The document discusses MODIS Land products and their distribution format. MODIS Land products include radiation budget, ecosystem, and land cover variables. Products are distributed in HDF-EOS format with fine resolution grids in Integerized Sinusoidal or Lambert Azimuthal Equal-Area projections and coarse grids in a geographic Climate Modeling Grid. HDF-EOS allows for collaboration and standard geolocation representation but toolkit support is still limited.
What's hot (20)
HDF Product Designer: Using Templates to Achieve Interoperability
HDF Product Designer: Using Templates to Achieve Interoperability
HDF-EOS to GeoTIFF Conversion Tool & HDF-EOS Plug-in for HDFView
HDF-EOS to GeoTIFF Conversion Tool & HDF-EOS Plug-in for HDFView
More from The HDF-EOS Tools and Information Center
Cloud-Optimized HDF5 Files
This document discusses how to optimize HDF5 files for efficient access in cloud object stores. Key optimizations include using large dataset chunk sizes of 1-4 MiB, consolidating internal file metadata, and minimizing variable-length datatypes. The document recommends creating files with paged aggregation and storing file content information in the user block to enable fast discovery of file contents when stored in object stores.
Accessing HDF5 data in the cloud with HSDS
This document provides an overview of HSDS (Highly Scalable Data Service), which is a REST-based service that allows accessing HDF5 data stored in the cloud. It discusses how HSDS maps HDF5 objects like datasets and groups to individual cloud storage objects to optimize performance. The document also describes how HSDS was used to improve access performance for NASA ICESat-2 HDF5 data on AWS S3 by hyper-chunking datasets into larger chunks spanning multiple original HDF5 chunks. Benchmark results showed that accessing the data through HSDS provided over 2x faster performance than other methods like ROS3 or S3FS that directly access the cloud storage.
The State of HDF
This document summarizes the current status and focus of the HDF Group. It discusses that the HDF Group is located in Champaign, IL and is a non-profit organization focused on developing and maintaining HDF software and data formats. It provides an overview of recent HDF5, HDF4 and HDFView releases and notes areas of focus for software quality improvements, increased transparency, strengthening the community, and modernizing HDF products. It invites support and participation in upcoming user group meetings.
Highly Scalable Data Service (HSDS) Performance Features
This document provides an overview of HSDS (HDF Server and Data Service), which allows HDF5 files to be stored and accessed from the cloud. Key points include:
- HSDS maps HDF5 objects like datasets and groups to individual cloud storage objects for scalability and parallelism.
- Features include streaming support, fancy indexing for complex queries, and caching for improved performance.
- HSDS can be deployed on Docker, Kubernetes, or AWS Lambda depending on needs.
- Case studies show HSDS is used by organizations like NREL and NSF to make petabytes of scientific data publicly accessible in the cloud.
Creating Cloud-Optimized HDF5 Files
This document discusses creating cloud-optimized HDF5 files by rearranging internal structures for more efficient data access in cloud object stores. It describes cloud-native and cloud-optimized storage formats, with the latter involving storing the entire HDF5 file as a single object. The benefits of cloud-optimized HDF5 include fast scanning and using the HDF5 library. Key aspects covered include using optimal chunk sizes, compression, and minimizing variable-length datatypes.
HDF5 OPeNDAP Handler Updates, and Performance Discussion
This document discusses updates and performance improvements to the HDF5 OPeNDAP data handler. It provides a history of the handler since 2001 and describes recent updates including supporting DAP4, new data types, and NetCDF data models. A performance study showed that passing compressed HDF5 data through the handler without decompressing/recompressing led to speedups of around 17-30x by leveraging HDF5 direct I/O APIs. This allows outputting HDF5 files as NetCDF files much faster through the handler.
Hyrax: Serving Data from S3
This document provides instructions for using the Hyrax software to serve scientific data files stored on Amazon S3 using the OPeNDAP data access protocol. It describes how to generate ancillary metadata files called DMR++ files using the get_dmrpp tool that provide information about the data file structure and locations. The document explains how to run get_dmrpp inside a Docker container to process data files on S3 and generate customized DMR++ files that the Hyrax server can use to serve the files to clients.
Accessing Cloud Data and Services Using EDL, Pydap, MATLAB
This document provides an overview and examples of accessing cloud data and services using the Earthdata Login (EDL), Pydap, and MATLAB. It discusses some common problems users encounter, such as being unable to access HDF5 data on AWS S3 using MATLAB or read data from OPeNDAP servers using Pydap. Solutions presented include using EDL to get temporary AWS tokens for S3 access in MATLAB and providing code examples on the HDFEOS website to help users access S3 data and OPeNDAP services. The document also notes some limitations, such as tokens being valid for only 1 hour, and workarounds like requesting new tokens or using the MATLAB HDF5 API instead of the netCDF API.
HDF - Current status and Future Directions
The HDF5 Roadmap and New Features document outlines upcoming changes and improvements to the HDF5 library. Key points include:
- HDF5 1.13.x releases will include new features like selection I/O, the Onion VFD for versioned files, improved VFD SWMR for single-writer multiple-reader access, and subfiling for parallel I/O.
- The Virtual Object Layer allows customizing HDF5 object storage and introduces terminal and pass-through connectors.
- The Onion VFD stores versions of HDF5 files in a separate onion file for versioned access.
- VFD SWMR improves on legacy SWMR by implementing single-writer multiple-reader capabilities
HDF for the Cloud - New HDF Server Features
The document discusses HDF for the cloud, including new features of the HDF Server and what's next. Key points:
- HDF Server uses a "sharded schema" that maps HDF5 objects to individual storage objects, allowing parallel access and updates without transferring entire files.
- Implementations include HSDS software that uses the sharded schema with an API and SDKs for different languages like h5pyd for Python.
- New features of HSDS 0.6 include support for POSIX, Azure, AWS Lambda, and role-based access control.
- Future work includes direct access to storage without a server intermediary for some use cases.
STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...
STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...The HDF-EOS Tools and Information Center
This document discusses STARE-PODS, a proposal to NASA/ACCESS-19 to develop a scalable data store for earth science data using the SpatioTemporal Adaptive Resolution Encoding (STARE) indexing scheme. STARE allows diverse earth science data to be unified and indexed, enabling the data to be partitioned and stored in a Parallel Optimized Data Store (PODS) for efficient analysis. The HDF Virtual Object Layer and Virtual Data Set technologies can then provide interfaces to access the data in STARE-PODS in a familiar way. The goal is for STARE-PODS to organize diverse data for alignment and parallel/distributed storage and processing to enable integrative analysis at scale.HDF5 Roadmap 2019-2020
This document summarizes new features in HDF5 1.12.0, including support for storing references to objects and attributes across files, new storage backends using a virtual object layer (VOL), and virtual file drivers (VFDs) for Amazon S3 and HDFS. It outlines the HDF5 roadmap for 2019-2022, which includes continued support for HDF5 1.8 and 1.10, and new features in future 1.12.x releases like querying, indexing, and provenance tracking.
Leveraging the Cloud for HDF Software Testing
The document discusses leveraging cloud resources like Amazon Web Services to improve software testing for the HDF group. Currently HDF software is tested on various in-house systems, but moving more testing to the cloud could provide better coverage of operating systems and distributions at a lower cost. AWS spot instances are being used to run HDF5 build and regression tests across different Linux distributions in around 30 minutes for approximately $0.02 per hour.
Google Colaboratory for HDF-EOS
Google Colaboratory allows users to write and execute Python code in the cloud using Jupyter notebooks. It provides a free GPU and TPU for accelerating code. The document discusses how HDF-EOS is a standard format for satellite data and provides many examples for converting and processing HDF-EOS data. It then demonstrates how to install necessary packages and run an example zoo code in Colab to plot HDF-EOS data in the cloud without installing anything locally.
HDF-EOS Data Product Developer's Guide
This document summarizes a presentation on best practices for creating Earth observation data products that follow the Hierarchical Data Format for Earth Observing System (HDF-EOS) standard. It provides guidance on including geo-location variables, using named dimensions, following Climate and Forecast Metadata conventions, testing products with various tools, and using the HDF Product Designer tool to help design and validate compliant products. The work aims to improve the usability of data products and reduce questions received through help desks.
HDF Status Update
This document provides an update on HDF (Hierarchical Data Format) releases and features. It discusses the current HDF5 1.10 series which allows controlling HDF5 file versioning and enables compression for parallel writes. The upcoming HDF5 1.12 release will include non-POSIX I/O using a Virtual Object Layer, UTF-8 encoding, and other file format changes. The document shows examples of compressing Sentinel and SeaSat data files in HDF5 using different compression methods.
NASA Terra Data Fusion
The Terra Data Fusion Project aims to fuse data from NASA's Terra satellite's five instruments - ASTER, CERES, MISR, MODIS, and MOPITT - into a single product. This presents challenges due to the huge data volumes, different locations of input data, instrument granularities, data storage methods, and file formats. The project overcame these by using NCSA supercomputing facilities, converting files to HDF5, organizing data by instrument and granule, and adding metadata. The final fused data product contains over 1 million input files totaling 2.3 petabytes, reduced to 1 petabyte using HDF5 compression.
HDF Cloud: HDF5 at Scale
HDF Cloud provides scalable HDF5 data access in the cloud. It uses a RESTful interface to store HDF5 files and metadata on object storage like AWS S3. This allows datasets to be accessed elastically from anywhere, avoiding hardware costs while gaining redundancy, scalability and other cloud benefits. The architecture maps HDF5 objects to individual storage objects, caching frequently used data in memory for high performance. Client libraries provide transparent access whether files are local or remote.
HDF for the Cloud
This document discusses strategies for storing and accessing HDF5 data files in cloud object storage like Amazon S3. It describes an HDF5 Virtual File Driver (VFD) developed by The HDF Group that allows reading HDF5 files directly from S3 without downloading. For better performance, the document recommends optimizing HDF5 files stored in S3 by using chunking, compression, and aggregating smaller files. It also introduces the HDF Cloud Schema which maps HDF5 objects to individual object storage objects for parallel access.
S3 VFD
The document discusses an S3 VFD that allows HDF5 files to be served from object storage. It uses existing HDF5 libraries and new VFD drivers to access HDF5 files stored in S3. The S3 VFD uses range gets to read desired data and optimization is performed to avoid small metadata accesses. A new API and data structure are introduced to set credentials for the S3 VFD when opening HDF5 files.
More from The HDF-EOS Tools and Information Center (20)
Highly Scalable Data Service (HSDS) Performance Features
Highly Scalable Data Service (HSDS) Performance Features
HDF5 OPeNDAP Handler Updates, and Performance Discussion
HDF5 OPeNDAP Handler Updates, and Performance Discussion
Accessing Cloud Data and Services Using EDL, Pydap, MATLAB
Accessing Cloud Data and Services Using EDL, Pydap, MATLAB
STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...
STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...
Recently uploaded
Demystifying Knowledge Management through Storytelling
The Department of Veteran Affairs (VA) invited Taylor Paschal, Knowledge & Information Management Consultant at Enterprise Knowledge, to speak at a Knowledge Management Lunch and Learn hosted on June 12, 2024. All Office of Administration staff were invited to attend and received professional development credit for participating in the voluntary event.
The objectives of the Lunch and Learn presentation were to:
- Review what KM ‘is’ and ‘isn’t’
- Understand the value of KM and the benefits of engaging
- Define and reflect on your “what’s in it for me?”
- Share actionable ways you can participate in Knowledge - - Capture & Transfer
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
Tomaz Bratanic
Graph ML and GenAI Expert - Neo4j
QR Secure: A Hybrid Approach Using Machine Learning and Security Validation F...
QR Secure: A Hybrid Approach Using Machine Learning and Security Validation Functions to Prevent Interaction with Malicious QR Codes.
Aim of the Study: The goal of this research was to develop a robust hybrid approach for identifying malicious and insecure URLs derived from QR codes, ensuring safe interactions.
This is achieved through:
Machine Learning Model: Predicts the likelihood of a URL being malicious.
Security Validation Functions: Ensures the derived URL has a valid certificate and proper URL format.
This innovative blend of technology aims to enhance cybersecurity measures and protect users from potential threats hidden within QR codes 🖥 🔒
This study was my first introduction to using ML which has shown me the immense potential of ML in creating more secure digital environments!
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
HERE IS YOUR WEBINAR CONTENT! 'Mastering Customer Journey Management with Dr. Graham Hill'. We hope you find the webinar recording both insightful and enjoyable.
In this webinar, we explored essential aspects of Customer Journey Management and personalization. Here’s a summary of the key insights and topics discussed:
Key Takeaways:
Understanding the Customer Journey: Dr. Hill emphasized the importance of mapping and understanding the complete customer journey to identify touchpoints and opportunities for improvement.
Personalization Strategies: We discussed how to leverage data and insights to create personalized experiences that resonate with customers.
Technology Integration: Insights were shared on how inQuba’s advanced technology can streamline customer interactions and drive operational efficiency.
JavaLand 2024: Application Development Green Masterplan
My presentation slides I used at JavaLand 2024
"Choosing proper type of scaling", Olena Syrota
Imagine an IoT processing system that is already quite mature and production-ready and for which client coverage is growing and scaling and performance aspects are life and death questions. The system has Redis, MongoDB, and stream processing based on ksqldb. In this talk, firstly, we will analyze scaling approaches and then select the proper ones for our system.
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
The typical problem in product engineering is not bad strategy, so much as “no strategy”. This leads to confusion, lack of motivation, and incoherent action. The next time you look for a strategy and find an empty space, instead of waiting for it to be filled, I will show you how to fill it in yourself. If you’re wrong, it forces a correction. If you’re right, it helps create focus. I’ll share how I’ve approached this in the past, both what works and lessons for what didn’t work so well.
Northern Engraving | Nameplate Manufacturing Process - 2024
Manufacturing custom quality metal nameplates and badges involves several standard operations. Processes include sheet prep, lithography, screening, coating, punch press and inspection. All decoration is completed in the flat sheet with adhesive and tooling operations following. The possibilities for creating unique durable nameplates are endless. How will you create your brand identity? We can help!
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
Direct losses from downtime in 1 minute = $5-$10 thousand dollars. Reputation is priceless.
As part of the talk, we will consider the architectural strategies necessary for the development of highly loaded fintech solutions. We will focus on using queues and streaming to efficiently work and manage large amounts of data in real-time and to minimize latency.
We will focus special attention on the architectural patterns used in the design of the fintech system, microservices and event-driven architecture, which ensure scalability, fault tolerance, and consistency of the entire system.
A Deep Dive into ScyllaDB's Architecture
This talk will cover ScyllaDB Architecture from the cluster-level view and zoom in on data distribution and internal node architecture. In the process, we will learn the secret sauce used to get ScyllaDB's high availability and superior performance. We will also touch on the upcoming changes to ScyllaDB architecture, moving to strongly consistent metadata and tablets.
"NATO Hackathon Winner: AI-Powered Drug Search", Taras Kloba
This is a session that details how PostgreSQL's features and Azure AI Services can be effectively used to significantly enhance the search functionality in any application.
In this session, we'll share insights on how we used PostgreSQL to facilitate precise searches across multiple fields in our mobile application. The techniques include using LIKE and ILIKE operators and integrating a trigram-based search to handle potential misspellings, thereby increasing the search accuracy.
We'll also discuss how the azure_ai extension on PostgreSQL databases in Azure and Azure AI Services were utilized to create vectors from user input, a feature beneficial when users wish to find specific items based on text prompts. While our application's case study involves a drug search, the techniques and principles shared in this session can be adapted to improve search functionality in a wide range of applications. Join us to learn how PostgreSQL and Azure AI can be harnessed to enhance your application's search capability.
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Join us to learn how UiPath Apps can directly and easily interact with prebuilt connectors via Integration Service--including Salesforce, ServiceNow, Open GenAI, and more.
The best part is you can achieve this without building a custom workflow! Say goodbye to the hassle of using separate automations to call APIs. By seamlessly integrating within App Studio, you can now easily streamline your workflow, while gaining direct access to our Connector Catalog of popular applications.
We’ll discuss and demo the benefits of UiPath Apps and connectors including:
Creating a compelling user experience for any software, without the limitations of APIs.
Accelerating the app creation process, saving time and effort
Enjoying high-performance CRUD (create, read, update, delete) operations, for
seamless data management.
Speakers:
Russell Alfeche, Technology Leader, RPA at qBotic and UiPath MVP
Charlie Greenberg, host
Discover the Unseen: Tailored Recommendation of Unwatched Content
The session shares how JioCinema approaches ""watch discounting."" This capability ensures that if a user watched a certain amount of a show/movie, the platform no longer recommends that particular content to the user. Flawless operation of this feature promotes the discover of new content, improving the overall user experience.
JioCinema is an Indian over-the-top media streaming service owned by Viacom18.
Dandelion Hashtable: beyond billion requests per second on a commodity server
This slide deck presents DLHT, a concurrent in-memory hashtable. Despite efforts to optimize hashtables, that go as far as sacrificing core functionality, state-of-the-art designs still incur multiple memory accesses per request and block request processing in three cases. First, most hashtables block while waiting for data to be retrieved from memory. Second, open-addressing designs, which represent the current state-of-the-art, either cannot free index slots on deletes or must block all requests to do so. Third, index resizes block every request until all objects are copied to the new index. Defying folklore wisdom, DLHT forgoes open-addressing and adopts a fully-featured and memory-aware closed-addressing design based on bounded cache-line-chaining. This design offers lock-free index operations and deletes that free slots instantly, (2) completes most requests with a single memory access, (3) utilizes software prefetching to hide memory latencies, and (4) employs a novel non-blocking and parallel resizing. In a commodity server and a memory-resident workload, DLHT surpasses 1.6B requests per second and provides 3.5x (12x) the throughput of the state-of-the-art closed-addressing (open-addressing) resizable hashtable on Gets (Deletes).
QA or the Highway - Component Testing: Bridging the gap between frontend appl...
These are the slides for the presentation, "Component Testing: Bridging the gap between frontend applications" that was presented at QA or the Highway 2024 in Columbus, OH by Zachary Hamm.
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Gursev Pirge, PhD
Senior Data Scientist - JohnSnowLabs
Principle of conventional tomography-Bibash Shahi ppt..pptx
before the computed tomography, it had been widely used.
The Microsoft 365 Migration Tutorial For Beginner.pptx
This presentation will help you understand the power of Microsoft 365. However, we have mentioned every productivity app included in Office 365. Additionally, we have suggested the migration situation related to Office 365 and how we can help you.
You can also read: https://www.systoolsgroup.com/updates/office-365-tenant-to-tenant-migration-step-by-step-complete-guide/
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
How we managed to grow and scale a RAG application from zero to thousands of users in 7 months. Lessons from technical challenges around managing high load for LLMs, RAGs and Vector databases.
Recently uploaded (20)
Demystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through Storytelling
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
QR Secure: A Hybrid Approach Using Machine Learning and Security Validation F...
QR Secure: A Hybrid Approach Using Machine Learning and Security Validation F...
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving | Nameplate Manufacturing Process - 2024
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"NATO Hackathon Winner: AI-Powered Drug Search", Taras Kloba
"NATO Hackathon Winner: AI-Powered Drug Search", Taras Kloba
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Discover the Unseen: Tailored Recommendation of Unwatched Content
Discover the Unseen: Tailored Recommendation of Unwatched Content
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
QA or the Highway - Component Testing: Bridging the gap between frontend appl...
QA or the Highway - Component Testing: Bridging the gap between frontend appl...
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Principle of conventional tomography-Bibash Shahi ppt..pptx
Principle of conventional tomography-Bibash Shahi ppt..pptx
The Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptx
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
HDF5 In Support of Database Applications
- 1. HDF5 and Databases Three Approaches
- 2. I. HDF5 as DB – PyTables
- 3. II. HDF5 and RDBMS Side-by-Side Engineering Units DB and HDF-5 File
- 4. III. HDF5 and RDBMS Integration Visualization Libraries (MATLAB,…) Web Services (XML, REST, RSS) OLAP and Data Mining Reporting .NET Languages with Language Integrated Query HDF5 EDM model Entity Framework (EDM, eSQL, O-R mapping) SQL Server HDF5 Index HDF5 type HDF5 TVFs HDF5 FS blob HDF5 files