Visualizing and Analyzing HDF-EOS5 and HDF5 data with NCL
•Download as PPT, PDF•
0 likes•1,621 views
NCAR Command Language (NCL) is an interpreted language designed for sceintific data analysis and visualization with high quality graphics, espeially for atmospherice scince. NCL has been support NetCDF 3/4, GRIB 1/2, HDF-SDS, HDF_EOS, shapefiles, binary, and ASCII files for years. Now HDF-EOS5 is the released version, and HDF5 in beta-test stage. Now NCL team are developing NCL to write HDF5 files, and to read HDF-EOS5 data with OPeNDAP. NCL team will share with people their experience to visualize and analyze HDF-EOS5 and HDF5 data.
Report
Share
Report
Share
Recommended
Implementation of HDF-EOS5 and HDF5 into NCL
This document discusses the implementation of HDF5 and HDF-EOS5 data formats in the NCAR Command Language (NCL). It describes how NCL handles HDF5 files differently than other file formats by reading data on demand and representing the hierarchical structure of HDF5 groups, datasets, attributes and links. It provides sample NCL scripts to read HDF5 data files, print out file information and variable summaries, and access HDF5 data variables.
Usage of NCL, IDL, and MATLAB to access NASA HDF4/HDF-EOS2/HDF-EOS5 data
Usage of NCL, IDL, and MATLAB to access NASA HDF4/HDF-EOS2/HDF-EOS5 dataThe HDF-EOS Tools and Information Center
The document discusses providing easy access to HDF data via NCL, IDL, and MATLAB. It presents examples and code snippets for reading HDF data from various NASA data centers like GES DISC, MODAPS, NSIDC, and LP-DAAC into the three software packages. Common issues when working with HDF files like HDF-EOS2 swaths with dimension maps and different ways metadata is stored are also addressed. The overall goal is to help lower the learning curve for users who want to analyze HDF data in their favorite analysis packages.Easy Access of NASA HDF data via OPeNDAP
This slide will demonstrate how to use OPeNDAP Java clients such as IDV and Panoply via HDF OPeNDAP data handlers to access various NASA HDF products such as AIRS, OMI, MLS, MODIS, TRMM, CERES, SeaWIFS etc. Various features of these tools that can help users easy access the HDF data will also be explored.
Easy Remote Access Via OPeNDAP
The document discusses updates to OPeNDAP handlers for HDF4 and HDF5 data formats. The HDF4 handler was improved to allow more NASA HDF and HDF-EOS data to be visualized by translating the data structure to follow climate and forecast metadata conventions. The HDF5 handler was updated to support additional HDF-EOS5 data from NASA missions. The handlers address issues that previously prevented data visualization and make the data more interoperable. Limitations include unsupported additional HDF4 objects and untested HDF4 products.
Data Interoperability
This document summarizes discussions from the HDF Group's HDF/HDF-EOS Workshop XIV about data interoperability. It covered topics like enabling one set of APIs to handle multiple data formats through projects like netCDF4 and CDM. It also discussed format conversions and translations between formats like HDF4, HDF5, netCDF and others. Finally, it addressed semantic and content interoperability challenges like representing latitude and longitude in different formats and how standards like CF conventions help with interpretation of metadata across tools and applications. Interoperability issues that can arise from simultaneous access of HDF5 files via HDF5 and netCDF-4 libraries were also presented.
NASA HDF/HDF-EOS Data for Dummies (and Developers)
The document discusses NASA HDF/HDF-EOS data formats and how to work with them for Earth science data analysis. It begins with an overview of the complexity of NASA HDF data for new users. It then covers common HDF data formats like HDF4, HDF5, HDF-EOS2 and HDF-EOS5, explaining the differences and which APIs to use for each. It provides code examples for reading HDF-EOS data in C and Fortran. It also introduces tools that can help dump and extract HDF data in ASCII format for broader tool compatibility. Overall, the document aims to help new users understand and start working with NASA's HDF data products.
NetCDF and HDF5
NetCDF and HDF5 are data formats and software libraries used for scientific data. NetCDF began in 1989 and allows for array-oriented data with dimensions, variables, and attributes. NetCDF-4 introduced new features while maintaining backward compatibility. It uses HDF5 for data storage and can read HDF4/HDF5 files. NetCDF provides APIs for C, Fortran, Java, and is widely used for earth science and climate data. It supports conventions, parallel I/O, and reading many data formats.
Working with HDF and netCDF Data in ArcGIS: Tools and Case Studies
Working with HDF and netCDF Data in ArcGIS: Tools and Case StudiesThe HDF-EOS Tools and Information Center
This slide will provide an overview of current functionality, techniques, and tips for visualization and query of HDF and netCDF data in ArcGIS, as well as future plans. Hierarchical Data Format (HDF) and netCDF (network Common Data Form) are two widely used data formats for storing and manipulating scientific data. The NetCDF format also supports temporal data by using multidimensional arrays. The basic structure of data in this format and how to work with it will be covered in the context of standardized data structures and conventions. This slide will demonstrate the tools and techniques for ingesting HDF and netCDF data efficiently in ArcGIS, as well as some common workflows to employ the visualization capabilities of ArcGIS for effective animation and analysis of your data. Recommended
Implementation of HDF-EOS5 and HDF5 into NCL
This document discusses the implementation of HDF5 and HDF-EOS5 data formats in the NCAR Command Language (NCL). It describes how NCL handles HDF5 files differently than other file formats by reading data on demand and representing the hierarchical structure of HDF5 groups, datasets, attributes and links. It provides sample NCL scripts to read HDF5 data files, print out file information and variable summaries, and access HDF5 data variables.
Usage of NCL, IDL, and MATLAB to access NASA HDF4/HDF-EOS2/HDF-EOS5 data
Usage of NCL, IDL, and MATLAB to access NASA HDF4/HDF-EOS2/HDF-EOS5 dataThe HDF-EOS Tools and Information Center
The document discusses providing easy access to HDF data via NCL, IDL, and MATLAB. It presents examples and code snippets for reading HDF data from various NASA data centers like GES DISC, MODAPS, NSIDC, and LP-DAAC into the three software packages. Common issues when working with HDF files like HDF-EOS2 swaths with dimension maps and different ways metadata is stored are also addressed. The overall goal is to help lower the learning curve for users who want to analyze HDF data in their favorite analysis packages.Easy Access of NASA HDF data via OPeNDAP
This slide will demonstrate how to use OPeNDAP Java clients such as IDV and Panoply via HDF OPeNDAP data handlers to access various NASA HDF products such as AIRS, OMI, MLS, MODIS, TRMM, CERES, SeaWIFS etc. Various features of these tools that can help users easy access the HDF data will also be explored.
Easy Remote Access Via OPeNDAP
The document discusses updates to OPeNDAP handlers for HDF4 and HDF5 data formats. The HDF4 handler was improved to allow more NASA HDF and HDF-EOS data to be visualized by translating the data structure to follow climate and forecast metadata conventions. The HDF5 handler was updated to support additional HDF-EOS5 data from NASA missions. The handlers address issues that previously prevented data visualization and make the data more interoperable. Limitations include unsupported additional HDF4 objects and untested HDF4 products.
Data Interoperability
This document summarizes discussions from the HDF Group's HDF/HDF-EOS Workshop XIV about data interoperability. It covered topics like enabling one set of APIs to handle multiple data formats through projects like netCDF4 and CDM. It also discussed format conversions and translations between formats like HDF4, HDF5, netCDF and others. Finally, it addressed semantic and content interoperability challenges like representing latitude and longitude in different formats and how standards like CF conventions help with interpretation of metadata across tools and applications. Interoperability issues that can arise from simultaneous access of HDF5 files via HDF5 and netCDF-4 libraries were also presented.
NASA HDF/HDF-EOS Data for Dummies (and Developers)
The document discusses NASA HDF/HDF-EOS data formats and how to work with them for Earth science data analysis. It begins with an overview of the complexity of NASA HDF data for new users. It then covers common HDF data formats like HDF4, HDF5, HDF-EOS2 and HDF-EOS5, explaining the differences and which APIs to use for each. It provides code examples for reading HDF-EOS data in C and Fortran. It also introduces tools that can help dump and extract HDF data in ASCII format for broader tool compatibility. Overall, the document aims to help new users understand and start working with NASA's HDF data products.
NetCDF and HDF5
NetCDF and HDF5 are data formats and software libraries used for scientific data. NetCDF began in 1989 and allows for array-oriented data with dimensions, variables, and attributes. NetCDF-4 introduced new features while maintaining backward compatibility. It uses HDF5 for data storage and can read HDF4/HDF5 files. NetCDF provides APIs for C, Fortran, Java, and is widely used for earth science and climate data. It supports conventions, parallel I/O, and reading many data formats.
Working with HDF and netCDF Data in ArcGIS: Tools and Case Studies
Working with HDF and netCDF Data in ArcGIS: Tools and Case StudiesThe HDF-EOS Tools and Information Center
This slide will provide an overview of current functionality, techniques, and tips for visualization and query of HDF and netCDF data in ArcGIS, as well as future plans. Hierarchical Data Format (HDF) and netCDF (network Common Data Form) are two widely used data formats for storing and manipulating scientific data. The NetCDF format also supports temporal data by using multidimensional arrays. The basic structure of data in this format and how to work with it will be covered in the context of standardized data structures and conventions. This slide will demonstrate the tools and techniques for ingesting HDF and netCDF data efficiently in ArcGIS, as well as some common workflows to employ the visualization capabilities of ArcGIS for effective animation and analysis of your data. HDF Group Support for NPP/NPOESS/JPSS
The HDF Group presented on their support for the National Polar-orbiting Partnership/Joint Polar Satellite System (NPP/JPSS) program. Their goals included providing HDF5 support for distributing data from the Visible Infrared Imaging Radiometer Suite (VIIRS) and other sensors. They outlined priorities for testing software on critical platforms, developing tools to access and manage NPP/JPSS products, and providing rapid support. The presentation described software released or under development like h5edit, h5augjpss, and nagg, an aggregation tool.
Parallel HDF5 Developments
The document discusses recent and upcoming improvements to parallel HDF5 for improved I/O performance on HPC systems. Recent improvements include reducing file truncations, distributing metadata writes across processes, and improved selection matching. Upcoming work includes a high-level HPC API, funding for Exascale-focused enhancements, and future improvements like asynchronous I/O and auto-tuning to parallel file systems. Performance tips are also provided like passing MPI hints and using collective I/O.
Data Are from Mars, Tools Are from Venus
This document discusses challenges with data and tools coming from different "planets" (Mars and Venus), frustrating users. It identifies gaps in file formats, libraries, tools, and services for working with Earth science data. The author provides solutions through the HDFEOS.org project, including file format conversion and examples, and suggestions for data producers, tool developers, and end users to help address interoperability issues. Emerging topics like big data, machine learning, and voice assistants are also discussed.
Implementing HDF5 in MATLAB
The MathWorks introduced MATLAB support for HDF5 in 2002 via three high-level functions: HDF5INFO, HDF5READ, and HDF5WRITE. These functions worked well for their purpose-providing simple interfaces to a complicated file format-but MATLAB users requested finer control over their HDF5 files and the HDF5 library. MATLAB 7.3 (R2006b) adds this precise level of support for version 1.6.5 of the HDF5 library via a close mapping of the HDF5 C API to MATLAB function calls.
This presentation will briefly introduce the earlier, high-level HDF5 interface (and its limitations) before showing in detail the low-level HDF5 functions. It will show how to interact with the HDF5 library and files using the thirteen classes of functions in MATLAB, which encapsulate groupings of functionality found in the HDF5 C API. But because MATLAB is itself a higher-level language than C, we will also present MATLAB's extensions and modifications of the HDF5 C API that make it more MATLAB-like, work with defined values, and perform ID and memory management.
Wrapping a library like HDF5 requires a great deal of effort and design, and we will briefly present a general-purpose mechanism for creating close mappings between library interfaces and an application like MATLAB. One of our goals in this presentation is to facilitate communication with The HDF Group about how The MathWorks builds our HDF5 interfaces in order to ease adoption of future versions of the HDF5 library in large, general-purpose applications.
HDFEOS.org User Analsys, Updates, and Future
This document discusses user analysis of the HDFEOS.org website and plans for future improvements. It finds that the majority of the site's 100 daily users are "quiet", not posting on forums or other interactive elements. The main user types are locators, who search for examples or data; mergers, who combine or mosaic datasets; and converters, who change file formats. The document outlines recent updates focused on these user types, like adding Python examples for subsetting and calculating latitude and longitude. It proposes future work on artificial intelligence/machine learning uses of HDF files and examples for processing HDF data in the cloud.
MODIS Reprojection Tool
This document summarizes a presentation about the MODIS Reprojection Tool software. The tool was developed to reproject raw MODIS data from its native Integerized Sinusoidal map projection to standard projections like Geographic and UTM. It supports ingesting HDF-EOS and raw binary file formats, reprojecting to 9 standard projections using 3 resampling methods, and producing output in HDF-EOS, GeoTIFF or raw binary formats. The tool utilizes a graphical user interface and includes components like a resampler, support for the GCTP library and various file input/output libraries. It has been tested on Windows, Linux, Sun and SGI platforms.
HDF & HDF-EOS Data & Support at NSIDC
The document discusses the National Snow and Ice Data Center's (NSIDC) use of HDF and HDF-EOS file formats to manage and distribute scientific cryosphere data. NSIDC collects data from satellites like MODIS, AMSR-E and aircraft missions, processes the data, and makes it available in HDF(5) formats. The HDF formats allow for efficient storage of multi-dimensional scientific datasets along with metadata. NSIDC develops tools to allow users to access, analyze and visualize data stored in HDF files.
Using IDL with Suomi NPP VIIRS Data
This document discusses using IDL to work with Suomi NPP VIIRS data. It presents an overview of VIIRS data and examples of constructing false color composite images and cloud masks from VIIRS imagery files. The examples demonstrate reading VIIRS data from HDF5 files, performing image processing, and displaying results. In summary, HDF5 makes it easy to work with Suomi NPP VIIRS data in IDL.
GES DISC Eexperiences with HDF Formats for MEaSUREs Projects
GES DISC Eexperiences with HDF Formats for MEaSUREs ProjectsThe HDF-EOS Tools and Information Center
The document discusses GES DISC's experience supporting 7 MEaSUREs projects, which create long-term Earth science data records. It recommends file formats like HDF-EOS5 and netCDF4 to promote consistency and interoperability. While HDF5 was initially recommended, ad-hoc implementations have caused issues. The document provides information on GES DISC's recommendations for file naming, formats and metadata, as well as problems encountered and resources for learning more.The MATLAB Low-Level HDF5 Interface
HDF5 is a powerful and feature-rich creature, and getting the most out of it requires powerful tools. The MathWorks provides a "low-level" interface to the HDF5 library that closely corresponds to the C API and exposes much of its richness. This short tutorial will present ways to use the low-level MATLAB interface to build those tools and tackle such topics as subsetting, chunking, and compression.
HDF and netCDF Data Support in ArcGIS
The document discusses HDF and netCDF data support in ArcGIS. It provides an overview of how HDF and netCDF data can be directly ingested and used as raster datasets, mosaic datasets, feature layers, and tables in ArcGIS. This allows for scientific data to be displayed, analyzed, and shared using common GIS tools and services. It also describes existing Python tools for working with netCDF data and outlines future areas of development, including improved support for HDF5, THREDDS/OPeNDAP access, and evolving data standards.
HDF OPeNDAP Project Update and Demo
The HDF Group gave a presentation on updates to the HDF-OPeNDAP project. They summarized the redesign of the HDF5 handler to support more NASA HDF5 data products, including grids, swaths, and zonal averages. They also discussed updates to the HDF4 handler to improve support for MODIS products and add new map projections. Live demonstrations showed example data from instruments like OMI, MLS, and Aquarius being served through the updated HDF handlers.
HDF5 Performance Enhancements with the Elimination of Unlimited Dimension
HDF5 Performance Enhancements with the Elimination of Unlimited DimensionThe HDF-EOS Tools and Information Center
This document discusses changes made to the MOPITT instrument's HDF data products over multiple versions. MOPITT measures carbon monoxide on the Terra spacecraft. Version 7 products, which use HDF-EOS5 format without unlimited dimensions, were found to be read faster than Version 6 products despite being larger in size. Experiments comparing read times of Version 7 data with and without unlimited dimensions showed a small 2% difference initially, but up to 40% faster reads for Version 7 without unlimited dimensions at first, with the difference decreasing over multiple reads. Eliminating unlimited dimensions thus improved access performance for some HDF applications like MOPITT data.HDF5 <-> Zarr
HDF5 and Zarr are data formats that can be used to store and access scientific data. This presentation discusses approaches to translating between the two formats. It describes how HDF5 files were translated to the Zarr format by creating a separate Zarr store to hold HDF5 file chunks, and storing chunk location metadata. It also discusses an implementation that translates Zarr data to the HDF5 format by using a special chunking layout and storing chunk information in an HDF5 compound dataset. Limitations of the translations include lack of support for some HDF5 dataset properties in Zarr, and lack of support for some Zarr compression methods in the HDF5 implementation.
Status of HDF-EOS, Related Software and Tools
This document provides an overview of the status of HDF-EOS software and tools. It describes HDF-EOS5, a rewrite of HDF-EOS2 based on HDF5, which is used operationally by EOS instrument teams. The document also outlines software releases, major developments including bug fixes, and future plans, and provides contact information for support.
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3The HDF-EOS Tools and Information Center
This document compares different methods for accessing HDF and netCDF files stored on Amazon S3, including Apache Drill, THREDDS Data Server (TDS), and HDF5 Virtual File Driver (VFD). A benchmark test of accessing a 24GB HDF5/netCDF-4 file on S3 from Amazon EC2 found that TDS performed the best, responding within 2 minutes, while Apache Drill failed after 7 minutes. The document concludes that TDS 5.0 is the clear winner based on performance and support for role-based access control and HDF4 files, but the best solution depends on use case and software.Efficiently serving HDF5 via OPeNDAP
This presentation discusses serving HDF5 data via OPeNDAP for efficient access and visualization. The HDF5 handler with CF and NcML modules translates HDF5 file layout and metadata to comply with CF conventions and provides the data via OPeNDAP to tools like Hyrax. Caching and optimizations improve performance. Future work includes supporting additional data types and file formats in the HDF5 handler.
MATLAB and Scientific Data: New Features and Capabilities
This document discusses new features and capabilities in MATLAB for working with scientific data formats and performing technical computing. It highlights enhancements to reading HDF5 and NetCDF files with both high-level and low-level interfaces. It also covers new capabilities for handling dates/times, big data, and accessing web services through RESTful APIs.
Google Colaboratory for HDF-EOS
Google Colaboratory allows users to write and execute Python code in the cloud using Jupyter notebooks. It provides a free GPU and TPU for accelerating code. The document discusses how HDF-EOS is a standard format for satellite data and provides many examples for converting and processing HDF-EOS data. It then demonstrates how to install necessary packages and run an example zoo code in Colab to plot HDF-EOS data in the cloud without installing anything locally.
Advanced HDF5 Features
This document provides an overview and outline of topics related to advanced features in HDF5, including:
- HDF5 supports various datatypes like atomic, compound, array, and variable-length datatypes. It allows creation of complex user-defined datatypes.
- Partial I/O in HDF5 allows reading and writing subsets of datasets using hyperslab selections, which describe subsets through properties like start point, stride, count, and block size.
- Chunking and compression can be used to improve performance and reduce storage needs when working with subsets of large HDF5 datasets.
Geoscience Data Analysis and Visualization Tools from NCAR
The document provides an overview of geoscience data analysis and visualization tools from the National Center for Atmospheric Research (NCAR). It describes two interfaces - the NCL scripting language and the Python modules PyNGL and PyNIO - that provide capabilities for data input/output, analysis, and visualization. Key topics covered include an introduction to NCL and PyNGL/PyNIO, the data model and file format support, visualization examples, and the status and distribution of the software.
HDF5 Tools Update
We will introduce the basic features and recent updates on the HDF5 tools. The intention of the talk is to keep users updated on the HDF5 tool development such as new features and new tools. It also helps new users to get familiar with the HDF5 tools. Three new tools h5check, h5copy and h5stat will be introduced.
It is important that HDF5 files created and modified by the HDF5 library are fully compliant with the defined HDF5 File Format Specification (File Format) to ensure the data model integrity and long term compatibility between evolving versions of the HDF5 library. The h5check tool verifies that the content of an HDF5 file is encoded according to the File Format. The verification role also makes h5check act as a watchdog for the implementation correctness of the HDF5 library. This presentation explains the features of the tool and time permitting, shows a demonstration of the tool.
h5copy is a command line tool that allows to make a copy of an HDF5 object (group, dataset or named datatype) from one location to another location within a file or across files. h5copy uses the HDF5 API function H5Gcopy to copy objects, which provides users with several options on how data object is copied, and high efficiency in data copy.
h5stat is a new utility for viewing different statistics about an HDF5 file. Tool is still under development. The goal of the talk will be to introduce the tool and get HDF5 users feedback.
More Related Content
What's hot
HDF Group Support for NPP/NPOESS/JPSS
The HDF Group presented on their support for the National Polar-orbiting Partnership/Joint Polar Satellite System (NPP/JPSS) program. Their goals included providing HDF5 support for distributing data from the Visible Infrared Imaging Radiometer Suite (VIIRS) and other sensors. They outlined priorities for testing software on critical platforms, developing tools to access and manage NPP/JPSS products, and providing rapid support. The presentation described software released or under development like h5edit, h5augjpss, and nagg, an aggregation tool.
Parallel HDF5 Developments
The document discusses recent and upcoming improvements to parallel HDF5 for improved I/O performance on HPC systems. Recent improvements include reducing file truncations, distributing metadata writes across processes, and improved selection matching. Upcoming work includes a high-level HPC API, funding for Exascale-focused enhancements, and future improvements like asynchronous I/O and auto-tuning to parallel file systems. Performance tips are also provided like passing MPI hints and using collective I/O.
Data Are from Mars, Tools Are from Venus
This document discusses challenges with data and tools coming from different "planets" (Mars and Venus), frustrating users. It identifies gaps in file formats, libraries, tools, and services for working with Earth science data. The author provides solutions through the HDFEOS.org project, including file format conversion and examples, and suggestions for data producers, tool developers, and end users to help address interoperability issues. Emerging topics like big data, machine learning, and voice assistants are also discussed.
Implementing HDF5 in MATLAB
The MathWorks introduced MATLAB support for HDF5 in 2002 via three high-level functions: HDF5INFO, HDF5READ, and HDF5WRITE. These functions worked well for their purpose-providing simple interfaces to a complicated file format-but MATLAB users requested finer control over their HDF5 files and the HDF5 library. MATLAB 7.3 (R2006b) adds this precise level of support for version 1.6.5 of the HDF5 library via a close mapping of the HDF5 C API to MATLAB function calls.
This presentation will briefly introduce the earlier, high-level HDF5 interface (and its limitations) before showing in detail the low-level HDF5 functions. It will show how to interact with the HDF5 library and files using the thirteen classes of functions in MATLAB, which encapsulate groupings of functionality found in the HDF5 C API. But because MATLAB is itself a higher-level language than C, we will also present MATLAB's extensions and modifications of the HDF5 C API that make it more MATLAB-like, work with defined values, and perform ID and memory management.
Wrapping a library like HDF5 requires a great deal of effort and design, and we will briefly present a general-purpose mechanism for creating close mappings between library interfaces and an application like MATLAB. One of our goals in this presentation is to facilitate communication with The HDF Group about how The MathWorks builds our HDF5 interfaces in order to ease adoption of future versions of the HDF5 library in large, general-purpose applications.
HDFEOS.org User Analsys, Updates, and Future
This document discusses user analysis of the HDFEOS.org website and plans for future improvements. It finds that the majority of the site's 100 daily users are "quiet", not posting on forums or other interactive elements. The main user types are locators, who search for examples or data; mergers, who combine or mosaic datasets; and converters, who change file formats. The document outlines recent updates focused on these user types, like adding Python examples for subsetting and calculating latitude and longitude. It proposes future work on artificial intelligence/machine learning uses of HDF files and examples for processing HDF data in the cloud.
MODIS Reprojection Tool
This document summarizes a presentation about the MODIS Reprojection Tool software. The tool was developed to reproject raw MODIS data from its native Integerized Sinusoidal map projection to standard projections like Geographic and UTM. It supports ingesting HDF-EOS and raw binary file formats, reprojecting to 9 standard projections using 3 resampling methods, and producing output in HDF-EOS, GeoTIFF or raw binary formats. The tool utilizes a graphical user interface and includes components like a resampler, support for the GCTP library and various file input/output libraries. It has been tested on Windows, Linux, Sun and SGI platforms.
HDF & HDF-EOS Data & Support at NSIDC
The document discusses the National Snow and Ice Data Center's (NSIDC) use of HDF and HDF-EOS file formats to manage and distribute scientific cryosphere data. NSIDC collects data from satellites like MODIS, AMSR-E and aircraft missions, processes the data, and makes it available in HDF(5) formats. The HDF formats allow for efficient storage of multi-dimensional scientific datasets along with metadata. NSIDC develops tools to allow users to access, analyze and visualize data stored in HDF files.
Using IDL with Suomi NPP VIIRS Data
This document discusses using IDL to work with Suomi NPP VIIRS data. It presents an overview of VIIRS data and examples of constructing false color composite images and cloud masks from VIIRS imagery files. The examples demonstrate reading VIIRS data from HDF5 files, performing image processing, and displaying results. In summary, HDF5 makes it easy to work with Suomi NPP VIIRS data in IDL.
GES DISC Eexperiences with HDF Formats for MEaSUREs Projects
GES DISC Eexperiences with HDF Formats for MEaSUREs ProjectsThe HDF-EOS Tools and Information Center
The document discusses GES DISC's experience supporting 7 MEaSUREs projects, which create long-term Earth science data records. It recommends file formats like HDF-EOS5 and netCDF4 to promote consistency and interoperability. While HDF5 was initially recommended, ad-hoc implementations have caused issues. The document provides information on GES DISC's recommendations for file naming, formats and metadata, as well as problems encountered and resources for learning more.The MATLAB Low-Level HDF5 Interface
HDF5 is a powerful and feature-rich creature, and getting the most out of it requires powerful tools. The MathWorks provides a "low-level" interface to the HDF5 library that closely corresponds to the C API and exposes much of its richness. This short tutorial will present ways to use the low-level MATLAB interface to build those tools and tackle such topics as subsetting, chunking, and compression.
HDF and netCDF Data Support in ArcGIS
The document discusses HDF and netCDF data support in ArcGIS. It provides an overview of how HDF and netCDF data can be directly ingested and used as raster datasets, mosaic datasets, feature layers, and tables in ArcGIS. This allows for scientific data to be displayed, analyzed, and shared using common GIS tools and services. It also describes existing Python tools for working with netCDF data and outlines future areas of development, including improved support for HDF5, THREDDS/OPeNDAP access, and evolving data standards.
HDF OPeNDAP Project Update and Demo
The HDF Group gave a presentation on updates to the HDF-OPeNDAP project. They summarized the redesign of the HDF5 handler to support more NASA HDF5 data products, including grids, swaths, and zonal averages. They also discussed updates to the HDF4 handler to improve support for MODIS products and add new map projections. Live demonstrations showed example data from instruments like OMI, MLS, and Aquarius being served through the updated HDF handlers.
HDF5 Performance Enhancements with the Elimination of Unlimited Dimension
HDF5 Performance Enhancements with the Elimination of Unlimited DimensionThe HDF-EOS Tools and Information Center
This document discusses changes made to the MOPITT instrument's HDF data products over multiple versions. MOPITT measures carbon monoxide on the Terra spacecraft. Version 7 products, which use HDF-EOS5 format without unlimited dimensions, were found to be read faster than Version 6 products despite being larger in size. Experiments comparing read times of Version 7 data with and without unlimited dimensions showed a small 2% difference initially, but up to 40% faster reads for Version 7 without unlimited dimensions at first, with the difference decreasing over multiple reads. Eliminating unlimited dimensions thus improved access performance for some HDF applications like MOPITT data.HDF5 <-> Zarr
HDF5 and Zarr are data formats that can be used to store and access scientific data. This presentation discusses approaches to translating between the two formats. It describes how HDF5 files were translated to the Zarr format by creating a separate Zarr store to hold HDF5 file chunks, and storing chunk location metadata. It also discusses an implementation that translates Zarr data to the HDF5 format by using a special chunking layout and storing chunk information in an HDF5 compound dataset. Limitations of the translations include lack of support for some HDF5 dataset properties in Zarr, and lack of support for some Zarr compression methods in the HDF5 implementation.
Status of HDF-EOS, Related Software and Tools
This document provides an overview of the status of HDF-EOS software and tools. It describes HDF-EOS5, a rewrite of HDF-EOS2 based on HDF5, which is used operationally by EOS instrument teams. The document also outlines software releases, major developments including bug fixes, and future plans, and provides contact information for support.
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3The HDF-EOS Tools and Information Center
This document compares different methods for accessing HDF and netCDF files stored on Amazon S3, including Apache Drill, THREDDS Data Server (TDS), and HDF5 Virtual File Driver (VFD). A benchmark test of accessing a 24GB HDF5/netCDF-4 file on S3 from Amazon EC2 found that TDS performed the best, responding within 2 minutes, while Apache Drill failed after 7 minutes. The document concludes that TDS 5.0 is the clear winner based on performance and support for role-based access control and HDF4 files, but the best solution depends on use case and software.Efficiently serving HDF5 via OPeNDAP
This presentation discusses serving HDF5 data via OPeNDAP for efficient access and visualization. The HDF5 handler with CF and NcML modules translates HDF5 file layout and metadata to comply with CF conventions and provides the data via OPeNDAP to tools like Hyrax. Caching and optimizations improve performance. Future work includes supporting additional data types and file formats in the HDF5 handler.
MATLAB and Scientific Data: New Features and Capabilities
This document discusses new features and capabilities in MATLAB for working with scientific data formats and performing technical computing. It highlights enhancements to reading HDF5 and NetCDF files with both high-level and low-level interfaces. It also covers new capabilities for handling dates/times, big data, and accessing web services through RESTful APIs.
Google Colaboratory for HDF-EOS
Google Colaboratory allows users to write and execute Python code in the cloud using Jupyter notebooks. It provides a free GPU and TPU for accelerating code. The document discusses how HDF-EOS is a standard format for satellite data and provides many examples for converting and processing HDF-EOS data. It then demonstrates how to install necessary packages and run an example zoo code in Colab to plot HDF-EOS data in the cloud without installing anything locally.
Advanced HDF5 Features
This document provides an overview and outline of topics related to advanced features in HDF5, including:
- HDF5 supports various datatypes like atomic, compound, array, and variable-length datatypes. It allows creation of complex user-defined datatypes.
- Partial I/O in HDF5 allows reading and writing subsets of datasets using hyperslab selections, which describe subsets through properties like start point, stride, count, and block size.
- Chunking and compression can be used to improve performance and reduce storage needs when working with subsets of large HDF5 datasets.
What's hot (20)
GES DISC Eexperiences with HDF Formats for MEaSUREs Projects
GES DISC Eexperiences with HDF Formats for MEaSUREs Projects
HDF5 Performance Enhancements with the Elimination of Unlimited Dimension
HDF5 Performance Enhancements with the Elimination of Unlimited Dimension
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3
MATLAB and Scientific Data: New Features and Capabilities
MATLAB and Scientific Data: New Features and Capabilities
Similar to Visualizing and Analyzing HDF-EOS5 and HDF5 data with NCL
Geoscience Data Analysis and Visualization Tools from NCAR
The document provides an overview of geoscience data analysis and visualization tools from the National Center for Atmospheric Research (NCAR). It describes two interfaces - the NCL scripting language and the Python modules PyNGL and PyNIO - that provide capabilities for data input/output, analysis, and visualization. Key topics covered include an introduction to NCL and PyNGL/PyNIO, the data model and file format support, visualization examples, and the status and distribution of the software.
HDF5 Tools Update
We will introduce the basic features and recent updates on the HDF5 tools. The intention of the talk is to keep users updated on the HDF5 tool development such as new features and new tools. It also helps new users to get familiar with the HDF5 tools. Three new tools h5check, h5copy and h5stat will be introduced.
It is important that HDF5 files created and modified by the HDF5 library are fully compliant with the defined HDF5 File Format Specification (File Format) to ensure the data model integrity and long term compatibility between evolving versions of the HDF5 library. The h5check tool verifies that the content of an HDF5 file is encoded according to the File Format. The verification role also makes h5check act as a watchdog for the implementation correctness of the HDF5 library. This presentation explains the features of the tool and time permitting, shows a demonstration of the tool.
h5copy is a command line tool that allows to make a copy of an HDF5 object (group, dataset or named datatype) from one location to another location within a file or across files. h5copy uses the HDF5 API function H5Gcopy to copy objects, which provides users with several options on how data object is copied, and high efficiency in data copy.
h5stat is a new utility for viewing different statistics about an HDF5 file. Tool is still under development. The goal of the talk will be to introduce the tool and get HDF5 users feedback.
Introduction to NetCDF-4
The document provides an introduction to NetCDF4 and covers its key features and performance. It discusses NetCDF4's history as a joint project between Unidata and HDF Group to combine the strengths of netCDF and HDF5. NetCDF4 uses HDF5 as its storage layer and allows writing netCDF files with HDF5 features like compression, groups and parallel I/O. It provides an overview of NetCDF4's features and APIs, and shows performance benchmarks demonstrating the significant size reductions and minor performance impacts of using compression. The document concludes with suggestions for users regarding chunking for performance and using the classic model for backward compatibility.
Putting some Spark into HDF5
This presentation discusses putting HDF5 files into Apache Spark for analysis. It describes the differences between traditional file systems and Hadoop's HDFS, and how Spark provides a more accessible way to exploit data parallelism without using MapReduce. The presentation outlines experiments loading HDF5 climate data files into Spark to calculate statistics. It suggests variations like providing a file list instead of traversing directories. The conclusion is that Spark can effectively analyze HDF5 files under the right circumstances but current methods are imperfect, and future work with The HDF Group could build better integrations.
view_hdf
The document discusses view_hdf, a visualization and analysis tool developed to access data from HDF products generated by NASA's CERES Data Management System. view_hdf allows users to select and plot variables from CERES Science Data Sets without needing knowledge of HDF formats. It provides capabilities such as 2D and 3D graphics, geographic mapping, statistics computation, and saving/printing plots. Contact information is provided for accessing the CERES data center and documentation for view_hdf.
HDF Town Hall
The document summarizes changes at The HDF Group, including new staff members and their roles. It also outlines recent and upcoming HDF software releases, new operating system and compiler support, tools for HDF and netCDF interoperability, ongoing research projects involving parallel I/O and analysis, and potential projects of interest involving scientific domains like plasma physics, particle accelerators, and digital twin technology.
NASA HDF/HDF-EOS Data Access Challenges
This slide summarizes the common data access challenges that data users and producers may experience in NASA HDF/HDF-EOS data.
On demand file-caching_-_gustavo_brand
The document discusses using GlusterFS to build a proof of concept for a group-based distributed file system (GBFS). It describes experiments conducted with HDFS and Lustre to analyze the impact of metadata and data traffic over wide area networks. The findings show that using local nodes often outperforms accessing data over WAN links. The presentation outlines initial plans for GBFS to leverage GlusterFS for implicit striping, local placement, and caching across groups. It also discusses potential future scenarios involving dynamic node addition and multiple cooperative storage and computing groups.
Access HDF5 Datasets via OPeNDAP's Data Access Protocol (DAP)
Access HDF5 Datasets via OPeNDAP's Data Access Protocol (DAP)The HDF-EOS Tools and Information Center
The Data Access Protocol(DAP) is a protocol for requesting and transporting data across the web. By using OPeNDAP servers data can be made accessible to remote locations regardless of local storage format. A prototyped HDF5-OPeNDAP server was implemented several years ago. With the HDF5 being selected as the primary data format of NASA Aura satellite, a robust, fully supported HDF5-OPeNDAP server is necessary.
Recently NASA Headquarters Science Mission Directorate Advancing Collaborative Connections For Earth-Sun System Science (ACCESS) program has awarded a joint proposal to implement a fully supported HDF5-OPeNDAP server. This poster will discuss the design and implementation plan for this project. EOS MLS software
The document summarizes the EOS MLS software and data processing levels. The EOS MLS instrument measures atmospheric temperature, water, ozone and other molecules from 5-80km on the AURA spacecraft. The software processes the instrument data through four levels: Level 1 converts telemetry to radiances, Level 2 converts radiances to abundances, Level 3 produces daily and monthly mean maps and zonal means. The data flows between levels using HDF4 and HDF-EOS formats, with products archived and described by metadata to enable ordering of Level 1-3 products. Future plans include transitioning to HDF5 and HDF-EOS5.
HDF5 Tools Updates
This tutorial is designed for anyone who needs to work with data stored in HDF5 files. The tutorial will cover functionality and useful features of the HDF5 utilities h5dump, h5diff, h5repack, h5stat, h5copy, h5check and h5repart. It will also cover HDFView, the HDF5 Java browsing and editing tool.
ESDIS Status (2002)
The document provides an overview and status update of the Earth Science Data and Information System (ESDIS). ESDIS has successfully supported numerous Earth science satellite missions and currently manages over 2 petabytes of science data. In fiscal year 2002, ESDIS delivered over 16 million data products to more than 1.8 million users. ESDIS is working to enhance its capabilities through initiatives like Data Pools and the EOS ClearingHOuse (ECHO) metadata broker. HDF-EOS 5 development is nearly complete and the workshop will discuss next steps for HDF-EOS tools and community adoption of HDF-EOS 5.
Ensuring Long Term Access to Remotely Sensed HDF4 Data with Layout Maps
Ensuring Long Term Access to Remotely Sensed HDF4 Data with Layout MapsThe HDF-EOS Tools and Information Center
A preponderance of data from NASA's Earth Observing System (EOS) is archived in the HDF Version 4 (HDF4) format. The long-term preservation of these data is critical for climate and other scientific studies going many decades into the future. HDF4 is very effective for working with the large and complex collection of EOS data products. Unfortunately, because of the complex internal byte layout of HDF4 files, future readability of HDF4 data depends on preserving a complex software library that can interpret that layout. Having a way to access HDF4 data independent of a library could improve its viability as an archive format, and consequently give confidence that HDF4 data will be readily accessible forever, even if the HDF4 library is gone.
To address the need to simplify long-term access to EOS data stored in HDF4, a collaborative project between The HDF Group and NASA Earth Science Data Centers is implementing an approach to accessing data in HDF4 files based on the use of independent maps that describe the data in HDF4 files and tools that can use these maps to recover data from those files. With this approach, relatively simple programs will be able to extract the data from an HDF4 file, bypassing the need for the HDF4 library.
A demonstration project has shown that this approach is feasible. This involved an assessment of NASA�s HDF4 data holdings, and development of a prototype XML-based layout mapping language and tools to read layout maps and read HDF4 files using layout maps. Future plans call for a second phase of the project, in which the mapping tools and XML schema are made production quality, the mapping schema are integrated with existing XML metadata files in several data centers, and outreach activities are carried out to encourage and facilitate acceptance of the technology. The New HDF-EOS WebSite - How it can help you
The HDF Group is in the process of updating HDF-EOS web site. During the workshop, we would like to share with audiences some useful information in the new website that can help users to have easy access of NASA HDF and HDF-EOS data.
The presentation includes three parts:
EOS User Forum: will introduce the EOS user forum and how users can benefit from this forum.
Tools: will present information on how to use several widely-used tools to access NASA HDF and HDF-EOS data.
Examples: will present several examples on how to use C, Fortran and IDL to access NASA HDF and HDF-EOS data.
Building the Next Generation Earth System Grid Federation (ESGF2)
The document discusses plans for the next generation of the Earth System Grid Federation (ESGF2). ESGF2 will have improved capabilities to handle exponentially larger climate and Earth system model data volumes. It will leverage the Globus platform for scalable central services and use cases. The project has already used Globus to replicate 7.5 PB of climate data between laboratories, demonstrating ESGF2's focus on reliable, high-performance data distribution. Governance structures and user support services will also be enhanced to support scientific discovery.
Q4 2016 GeoTrellis Presentation
This is a slide deck that I have been using to present on GeoTrellis for various meetings and workshops. The information is speaks to GeoTrellis pre-1.0 release in Q4 of 2016.
Kkeithley ufonfs-gluster summit
The document discusses the status and history of NFS and UFO protocols. It notes that many bugs have been fixed in NFS and details UFO's origins as being based on Swift 1.4.8, its packaging in Fedora 18, and plans for future performance enhancements and integration with Keystone authentication. The document contains several sections written by Kaleb KEITHLEY covering NFS status, UFO history, and its future.
HDF5 Tools
This tutorial is designed for anyone who needs to work with data stored in HDF5 files. It will cover functionality and useful features of the HDF5 utilities, which include h5dump, h5diff, h5repack, h5stat, h5copy, h5check and h5repart. The tutorial will also introduce recently changes and new features of the utilities.
The HDFView is a visual tool for browsing and editing HDF4 and HDF5 files. Some basic features and new changes of HDFView will be presented. Details of recent development in HDF-Java products will be discussed in a separate presentation.
HDF5 for NPOESS Data Products
HDF-EOS Workshop VI (2002)
Source: http://hdfeos.org/workshops/ws06/presentations/Goldberg/HDF5_for_NPOESS.ppt
Pivotal Greenplum 次世代マルチクラウド・データ分析プラットフォーム
Pivotal Greenplum is a massively parallel processing (MPP) database for analytics. It provides high performance for data warehousing and big data analytics workloads. Key features include its ability to load and query data in parallel across multiple CPUs and disks, support for SQL and analytical functions and libraries like MADlib, and deployment on public clouds or on-premises. Pivotal Greenplum can be used for both structured and unstructured data and integrates with other Pivotal products like GemFire, Data Flow, and the Pivotal Data Suite for analytics workflows.
Similar to Visualizing and Analyzing HDF-EOS5 and HDF5 data with NCL (20)
Geoscience Data Analysis and Visualization Tools from NCAR
Geoscience Data Analysis and Visualization Tools from NCAR
Access HDF5 Datasets via OPeNDAP's Data Access Protocol (DAP)
Access HDF5 Datasets via OPeNDAP's Data Access Protocol (DAP)
Ensuring Long Term Access to Remotely Sensed HDF4 Data with Layout Maps
Ensuring Long Term Access to Remotely Sensed HDF4 Data with Layout Maps
Building the Next Generation Earth System Grid Federation (ESGF2)
Building the Next Generation Earth System Grid Federation (ESGF2)
More from The HDF-EOS Tools and Information Center
Cloud-Optimized HDF5 Files
This document discusses how to optimize HDF5 files for efficient access in cloud object stores. Key optimizations include using large dataset chunk sizes of 1-4 MiB, consolidating internal file metadata, and minimizing variable-length datatypes. The document recommends creating files with paged aggregation and storing file content information in the user block to enable fast discovery of file contents when stored in object stores.
Accessing HDF5 data in the cloud with HSDS
This document provides an overview of HSDS (Highly Scalable Data Service), which is a REST-based service that allows accessing HDF5 data stored in the cloud. It discusses how HSDS maps HDF5 objects like datasets and groups to individual cloud storage objects to optimize performance. The document also describes how HSDS was used to improve access performance for NASA ICESat-2 HDF5 data on AWS S3 by hyper-chunking datasets into larger chunks spanning multiple original HDF5 chunks. Benchmark results showed that accessing the data through HSDS provided over 2x faster performance than other methods like ROS3 or S3FS that directly access the cloud storage.
The State of HDF
This document summarizes the current status and focus of the HDF Group. It discusses that the HDF Group is located in Champaign, IL and is a non-profit organization focused on developing and maintaining HDF software and data formats. It provides an overview of recent HDF5, HDF4 and HDFView releases and notes areas of focus for software quality improvements, increased transparency, strengthening the community, and modernizing HDF products. It invites support and participation in upcoming user group meetings.
Highly Scalable Data Service (HSDS) Performance Features
This document provides an overview of HSDS (HDF Server and Data Service), which allows HDF5 files to be stored and accessed from the cloud. Key points include:
- HSDS maps HDF5 objects like datasets and groups to individual cloud storage objects for scalability and parallelism.
- Features include streaming support, fancy indexing for complex queries, and caching for improved performance.
- HSDS can be deployed on Docker, Kubernetes, or AWS Lambda depending on needs.
- Case studies show HSDS is used by organizations like NREL and NSF to make petabytes of scientific data publicly accessible in the cloud.
Creating Cloud-Optimized HDF5 Files
This document discusses creating cloud-optimized HDF5 files by rearranging internal structures for more efficient data access in cloud object stores. It describes cloud-native and cloud-optimized storage formats, with the latter involving storing the entire HDF5 file as a single object. The benefits of cloud-optimized HDF5 include fast scanning and using the HDF5 library. Key aspects covered include using optimal chunk sizes, compression, and minimizing variable-length datatypes.
HDF5 OPeNDAP Handler Updates, and Performance Discussion
This document discusses updates and performance improvements to the HDF5 OPeNDAP data handler. It provides a history of the handler since 2001 and describes recent updates including supporting DAP4, new data types, and NetCDF data models. A performance study showed that passing compressed HDF5 data through the handler without decompressing/recompressing led to speedups of around 17-30x by leveraging HDF5 direct I/O APIs. This allows outputting HDF5 files as NetCDF files much faster through the handler.
Hyrax: Serving Data from S3
This document provides instructions for using the Hyrax software to serve scientific data files stored on Amazon S3 using the OPeNDAP data access protocol. It describes how to generate ancillary metadata files called DMR++ files using the get_dmrpp tool that provide information about the data file structure and locations. The document explains how to run get_dmrpp inside a Docker container to process data files on S3 and generate customized DMR++ files that the Hyrax server can use to serve the files to clients.
Accessing Cloud Data and Services Using EDL, Pydap, MATLAB
This document provides an overview and examples of accessing cloud data and services using the Earthdata Login (EDL), Pydap, and MATLAB. It discusses some common problems users encounter, such as being unable to access HDF5 data on AWS S3 using MATLAB or read data from OPeNDAP servers using Pydap. Solutions presented include using EDL to get temporary AWS tokens for S3 access in MATLAB and providing code examples on the HDFEOS website to help users access S3 data and OPeNDAP services. The document also notes some limitations, such as tokens being valid for only 1 hour, and workarounds like requesting new tokens or using the MATLAB HDF5 API instead of the netCDF API.
HDF - Current status and Future Directions
The HDF5 Roadmap and New Features document outlines upcoming changes and improvements to the HDF5 library. Key points include:
- HDF5 1.13.x releases will include new features like selection I/O, the Onion VFD for versioned files, improved VFD SWMR for single-writer multiple-reader access, and subfiling for parallel I/O.
- The Virtual Object Layer allows customizing HDF5 object storage and introduces terminal and pass-through connectors.
- The Onion VFD stores versions of HDF5 files in a separate onion file for versioned access.
- VFD SWMR improves on legacy SWMR by implementing single-writer multiple-reader capabilities
HDF - Current status and Future Directions
This document summarizes a presentation about the current status and future directions of the Hierarchical Data Format (HDF) software. It provides updates on recent HDF5 releases, development efforts including new compression methods and ways to access HDF5 data, and outreach resources. It concludes by inviting the audience to share wishes for future HDF development.
H5Coro: The Cloud-Optimized Read-Only Library
The document describes H5Coro, a new C++ library for reading HDF5 files from cloud storage. H5Coro was created to optimize HDF5 reading for cloud environments by minimizing I/O operations through caching and efficient HTTP requests. Performance tests showed H5Coro was 77-132x faster than the previous HDF5 library at reading HDF5 data from Amazon S3 for NASA's SlideRule project. H5Coro supports common HDF5 elements but does not support writing or some complex HDF5 data types and messages to focus on optimized read-only performance for time series data stored sequentially in memory.
MATLAB Modernization on HDF5 1.10
This document summarizes MathWorks' work to modernize MATLAB's support for HDF5. Key points include:
1) MATLAB now supports HDF5 1.10.7 features like single-writer/multiple-reader access and virtual datasets through new and updated low-level functions.
2) Performance benchmarks show some improvements but also regressions compared to the previous HDF5 version, and work continues to optimize code and support future versions.
3) There are compatibility considerations for Linux filter plugins, but interim solutions are provided until MathWorks can ship a single HDF5 version.
HDF for the Cloud - Serverless HDF
HSDS provides HDF as a service through a REST API that can scale across nodes. New releases will enable serverless operation using AWS Lambda or direct client access without a server. This allows HDF data to be accessed remotely without managing servers. HSDS stores each HDF object separately, making it compatible with cloud object storage. Performance on AWS Lambda is slower than a dedicated server but has no management overhead. Direct client access has better performance but limits collaboration between clients.
HDF for the Cloud - New HDF Server Features
The document discusses HDF for the cloud, including new features of the HDF Server and what's next. Key points:
- HDF Server uses a "sharded schema" that maps HDF5 objects to individual storage objects, allowing parallel access and updates without transferring entire files.
- Implementations include HSDS software that uses the sharded schema with an API and SDKs for different languages like h5pyd for Python.
- New features of HSDS 0.6 include support for POSIX, Azure, AWS Lambda, and role-based access control.
- Future work includes direct access to storage without a server intermediary for some use cases.
STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...
STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...The HDF-EOS Tools and Information Center
This document discusses STARE-PODS, a proposal to NASA/ACCESS-19 to develop a scalable data store for earth science data using the SpatioTemporal Adaptive Resolution Encoding (STARE) indexing scheme. STARE allows diverse earth science data to be unified and indexed, enabling the data to be partitioned and stored in a Parallel Optimized Data Store (PODS) for efficient analysis. The HDF Virtual Object Layer and Virtual Data Set technologies can then provide interfaces to access the data in STARE-PODS in a familiar way. The goal is for STARE-PODS to organize diverse data for alignment and parallel/distributed storage and processing to enable integrative analysis at scale.HDF5 and Ecosystem: What Is New?
This document provides an overview and update on HDF5 and its ecosystem. Key points include:
- HDF5 1.12.0 was recently released with new features like the Virtual Object Layer and external references.
- The HDF5 library now supports accessing data in the cloud using connectors like S3 VFD and REST VOL without needing to modify applications.
- Projects like HDFql and H5CPP provide additional interfaces for querying and working with HDF5 files from languages like SQL, C++, and Python.
- The HDF5 community is moving development to GitHub and improving documentation resources on the HDF wiki site.
HDF5 Roadmap 2019-2020
This document summarizes new features in HDF5 1.12.0, including support for storing references to objects and attributes across files, new storage backends using a virtual object layer (VOL), and virtual file drivers (VFDs) for Amazon S3 and HDFS. It outlines the HDF5 roadmap for 2019-2022, which includes continued support for HDF5 1.8 and 1.10, and new features in future 1.12.x releases like querying, indexing, and provenance tracking.
Leveraging the Cloud for HDF Software Testing
The document discusses leveraging cloud resources like Amazon Web Services to improve software testing for the HDF group. Currently HDF software is tested on various in-house systems, but moving more testing to the cloud could provide better coverage of operating systems and distributions at a lower cost. AWS spot instances are being used to run HDF5 build and regression tests across different Linux distributions in around 30 minutes for approximately $0.02 per hour.
Parallel Computing with HDF Server
The document discusses Parallel Computing with HDF Server. The key points are:
1. HDF Server (HSDS) allows efficient access to HDF5 data stored in AWS S3. It runs as containers on Kubernetes and supports parallel access across containers.
2. HSDS uses S3 as the data store for HDF5 files. Individual HDF5 objects like datasets and chunks are stored as separate S3 objects. This allows parallel read/write and only modifying what changes.
3. HDF Kita Lab is a hosted Jupyter environment on AWS that provides access to HSDS for reading and writing HDF5 data on S3. It allows scaling the server and provides tools for HDF5 on S3.
HDF-EOS Data Product Developer's Guide
This document summarizes a presentation on best practices for creating Earth observation data products that follow the Hierarchical Data Format for Earth Observing System (HDF-EOS) standard. It provides guidance on including geo-location variables, using named dimensions, following Climate and Forecast Metadata conventions, testing products with various tools, and using the HDF Product Designer tool to help design and validate compliant products. The work aims to improve the usability of data products and reduce questions received through help desks.
More from The HDF-EOS Tools and Information Center (20)
Highly Scalable Data Service (HSDS) Performance Features
Highly Scalable Data Service (HSDS) Performance Features
HDF5 OPeNDAP Handler Updates, and Performance Discussion
HDF5 OPeNDAP Handler Updates, and Performance Discussion
Accessing Cloud Data and Services Using EDL, Pydap, MATLAB
Accessing Cloud Data and Services Using EDL, Pydap, MATLAB
STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...
STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...
Recently uploaded
20240605 QFM017 Machine Intelligence Reading List May 2024
Everything I found interesting about machines behaving intelligently during May 2024
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for life science domain, where you retriever information from biomedical knowledge graphs using LLMs to increase the accuracy and performance of generated answers
Building Production Ready Search Pipelines with Spark and Milvus
Spark is the widely used ETL tool for processing, indexing and ingesting data to serving stack for search. Milvus is the production-ready open-source vector database. In this talk we will show how to use Spark to process unstructured data to extract vector representations, and push the vectors to Milvus vector database for search serving.
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Discover the seamless integration of RPA (Robotic Process Automation), COMPOSER, and APM with AWS IDP enhanced with Slack notifications. Explore how these technologies converge to streamline workflows, optimize performance, and ensure secure access, all while leveraging the power of AWS IDP and real-time communication via Slack notifications.
UiPath Test Automation using UiPath Test Suite series, part 6
Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI.
UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities.
Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes.
What will you get from this session?
1. Insights into integrating generative AI.
2. Understanding how this integration enhances test automation within the UiPath platform
3. Practical demonstrations
4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath
Topics covered:
What is generative AI
Test Automation with generative AI and Open AI.
UiPath integration with generative AI
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
20240607 QFM018 Elixir Reading List May 2024
Everything I found interesting about the Elixir programming ecosystem in May 2024
TrustArc Webinar - 2024 Global Privacy Survey
How does your privacy program stack up against your peers? What challenges are privacy teams tackling and prioritizing in 2024?
In the fifth annual Global Privacy Benchmarks Survey, we asked over 1,800 global privacy professionals and business executives to share their perspectives on the current state of privacy inside and outside of their organizations. This year’s report focused on emerging areas of importance for privacy and compliance professionals, including considerations and implications of Artificial Intelligence (AI) technologies, building brand trust, and different approaches for achieving higher privacy competence scores.
See how organizational priorities and strategic approaches to data security and privacy are evolving around the globe.
This webinar will review:
- The top 10 privacy insights from the fifth annual Global Privacy Benchmarks Survey
- The top challenges for privacy leaders, practitioners, and organizations in 2024
- Key themes to consider in developing and maintaining your privacy program
Best 20 SEO Techniques To Improve Website Visibility In SERP
Boost your website's visibility with proven SEO techniques! Our latest blog dives into essential strategies to enhance your online presence, increase traffic, and rank higher on search engines. From keyword optimization to quality content creation, learn how to make your site stand out in the crowded digital landscape. Discover actionable tips and expert insights to elevate your SEO game.
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...Edge AI and Vision Alliance
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/building-and-scaling-ai-applications-with-the-nx-ai-manager-a-presentation-from-network-optix/
Robin van Emden, Senior Director of Data Science at Network Optix, presents the “Building and Scaling AI Applications with the Nx AI Manager,” tutorial at the May 2024 Embedded Vision Summit.
In this presentation, van Emden covers the basics of scaling edge AI solutions using the Nx tool kit. He emphasizes the process of developing AI models and deploying them globally. He also showcases the conversion of AI models and the creation of effective edge AI pipelines, with a focus on pre-processing, model conversion, selecting the appropriate inference engine for the target hardware and post-processing.
van Emden shows how Nx can simplify the developer’s life and facilitate a rapid transition from concept to production-ready applications.He provides valuable insights into developing scalable and efficient edge AI solutions, with a strong focus on practical implementation.Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift
Overview
Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices.
Key Topics Covered
1. Introduction to Anomaly Detection
- Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems.
2. Understanding Edge (IoT)
- Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source.
3. What is ArgoCD?
- Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices.
4. Deployment Using ArgoCD for Edge Devices
- Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD.
5. Introduction to Apache Kafka and S3
- Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions.
6. Viewing Kafka Messages in the Data Lake
- Learn how to view and analyze Kafka messages stored in a data lake for better insights.
7. What is Prometheus?
- Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices.
8. Monitoring Application Metrics with Prometheus
- Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system.
9. What is Camel K?
- Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes.
10. Configuring Camel K Integrations for Data Pipelines
- Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow.
11. What is a Jupyter Notebook?
- Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text.
12. Jupyter Notebooks with Code Examples
- Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.
CAKE: Sharing Slices of Confidential Data on Blockchain
Presented at the CAiSE 2024 Forum, Intelligent Information Systems, June 6th, Limassol, Cyprus.
Synopsis: Cooperative information systems typically involve various entities in a collaborative process within a distributed environment. Blockchain technology offers a mechanism for automating such processes, even when only partial trust exists among participants. The data stored on the blockchain is replicated across all nodes in the network, ensuring accessibility to all participants. While this aspect facilitates traceability, integrity, and persistence, it poses challenges for adopting public blockchains in enterprise settings due to confidentiality issues. In this paper, we present a software tool named Control Access via Key Encryption (CAKE), designed to ensure data confidentiality in scenarios involving public blockchains. After outlining its core components and functionalities, we showcase the application of CAKE in the context of a real-world cyber-security project within the logistics domain.
Paper: https://doi.org/10.1007/978-3-031-61000-4_16
Removing Uninteresting Bytes in Software Fuzzing
Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process.
In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds.
- These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.
Presentation of the OECD Artificial Intelligence Review of Germany
Consult the full report at https://www.oecd.org/digital/oecd-artificial-intelligence-review-of-germany-609808d6-en.htm
Things to Consider When Choosing a Website Developer for your Website | FODUU
Choosing the right website developer is crucial for your business. This article covers essential factors to consider, including experience, portfolio, technical skills, communication, pricing, reputation & reviews, cost and budget considerations and post-launch support. Make an informed decision to ensure your website meets your business goals.
Infrastructure Challenges in Scaling RAG with Custom AI models
Building Retrieval-Augmented Generation (RAG) systems with open-source and custom AI models is a complex task. This talk explores the challenges in productionizing RAG systems, including retrieval performance, response synthesis, and evaluation. We’ll discuss how to leverage open-source models like text embeddings, language models, and custom fine-tuned models to enhance RAG performance. Additionally, we’ll cover how BentoML can help orchestrate and scale these AI components efficiently, ensuring seamless deployment and management of RAG systems in the cloud.
National Security Agency - NSA mobile device best practices
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
Artificial Intelligence for XMLDevelopment
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
Essentials of Automations: The Art of Triggers and Actions in FME
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
Recently uploaded (20)
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
CAKE: Sharing Slices of Confidential Data on Blockchain
CAKE: Sharing Slices of Confidential Data on Blockchain
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
Things to Consider When Choosing a Website Developer for your Website | FODUU
Things to Consider When Choosing a Website Developer for your Website | FODUU
Infrastructure Challenges in Scaling RAG with Custom AI models
Infrastructure Challenges in Scaling RAG with Custom AI models
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
Visualizing and Analyzing HDF-EOS5 and HDF5 data with NCL
- 1. Visualizing and Analyzing of HDF-EOS and HDF data with NCL Wei Huang, Dave Brown, Mary Haley, Rick Brownrigg CISL/NCAR Dennis Shea, Adam Philips CGD/NCAR huangwei@ucar.edu
- 2. What is NCL ◦ NCL stands for NCAR Command Language http://www.ncl.ucar.edu for more information ◦ NCL is developed at CISL/NCAR, for scientific data analysis and visualization ◦ Free available as binaries, open source ◦ Supports NetCDF (3 and 4), GRIB (1 and 2), HDF, HDF-EOS 2 and 5 (as Spring of 2010) ◦ Newly added HDF5 and OpenDap (in 2010) ◦ High quality graphics ◦ Lots of example scripts 02/17/14 CISL/NCAR
- 3. HDF-EOS5 Released April, 2010 HDF-EOS5 included NCL ◦ 5.2.0 released in April, 2010 ◦ 5.2.1 released in July, 2010 HDF, and HDF-EOS2/5 Examples ◦ http://www.ncl.ucar.edu/Applications/HDF.shtml ◦ Use “ncl_filedump” to check variables in HDF, and HDF-EOS2/5 files ◦ Use “ncl_convert2nc” to convert HDF-EOS2/5 and HDF files to NetCDF files 02/17/14 CISL/NCAR
- 6. HDF-EOS5 Sample (continue) Reflectivity of the ground pixel 02/17/14 CISL/NCAR
- 7. HDF5 in Beta Release 02/17/14 CISL/NCAR
- 8. Sample Script to Read HDF5 f = addfile(”./data/MOD13A2A.h5", "r”) ◦ printVarSummary(f) grps = getfilegroups(f, "/", 0) ◦ print(grps) grps2 = getfilegroups(f, "/MODIS_Grid_16DAY_1km_VI", 0) ◦ print(grps2) v1 = f->1_km_16_days_red_reflectance ◦ printVarSummary(v1) vn = grps2(0)+"/1 km 16 days blue reflectance” v2 = f->$vn$ ◦ printVarSummary(v2) 02/17/14 CISL/NCAR
- 9. Group in HDF5 and Sample Script f = addfile(”./data/NISESSMI.h5", "r") ◦ print(f) gn = "/Northern Hemisphere” g1 = f=>$gn$ ◦ print(g1) g2n = "/Northern Hemisphere/Data Fields” g2 = g1=>$g2n$ ◦ print(g2) vn = "/Northern Hemisphere/Data Fields/Age" v1 = g1->$vn$ ◦ printVarSummary(v1) v2 = g2->$vn$ ◦ printVarSummary(v2) 02/17/14 CISL/NCAR
- 11. HDF5 Sample Plot (continue) 02/17/14 CISL/NCAR
- 12. HDF5 Sample Plot (continue) 02/17/14 CISL/NCAR
- 13. HDF5 Sample Plot (continue) 02/17/14 CISL/NCAR
- 14. Read Compound data from HDF5 fn = "K1VHR_23SEP2008_2330_L02_SST.h5” f = addfile(fn, "r”) ◦ print(f) lon1d = f->/SST/SST_Composite.Longitude lat1d = f->/SST/SST_Composite.Latitude sst1d = f->/SST/SST_Dataset.SST 02/17/14 CISL/NCAR
- 15. Print-out filename: K1VHR_23SEP2008_2330_L02_SST file global attributes: dimensions: DIM_000 = 36381 variables: group </PRODUCT_INFORMATION> group </PRODUCT_METADATA> group </PRODUCT_METADATA/PRODUCT_DETAILS> group </PRODUCT_METADATA/PROJECTION_PARAMETERS> group </SST> compound <SST_Composite> (Latitude, Longitude, SST) (DIM_000) compound <SST_Dataset> (Latitude, Longitude, SST) (DIM_000) group </SST/GP_PARAM_INFO> 02/17/14 CISL/NCAR
- 17. HDF-EOS5 OpenDap Using liboc-dap-0.0.2 In Alpha Version 02/17/14 CISL/NCAR
- 18. HDF-EOS5 OpenDap -- a sample script url = "http://acdisc.gsfc.nasa.gov/opendap/HDFEOS5/Aura_OMI_Level3/OMAEROe.003/2010/" filename = url + "OMI-Aura_L3-OMAEROe_2010m0712_v0032010m0714t203639.he5” f = addfile(filename,"r”) ◦ print(f) vis_aerosol = f->SingleScatteringAlbedoPassedThresholdMean ◦ printVarSummary(vis_aerosol) lat = f->Latitude ◦ printVarSummary(lat) lon = f->Longitude ◦ printVarSummary(lon) 02/17/14 CISL/NCAR
- 19. HDF-EOS5 OpenDap –sample plot 02/17/14 CISL/NCAR
- 20. Conclusion HDF-EOS5 is already released ◦ 5.2.0 in April, 2010 ◦ 5.2.1 in July, 2010 HDF5 will be released soon ◦ Being tested with lots of data ◦ In Beta Version HDF-EOS5 OpenDap is on the way ◦ With limited test ◦ In Alpha Version 02/17/14 CISL/NCAR
- 22. Write HDF5 with NCL Write HDF5 with Options: ◦ Compression At File Level At Variable Level ◦ Chunking At File Level At Variable Level 02/17/14 CISL/NCAR
- 23. Sample Script to Write HDF5 02/17/14 CISL/NCAR