An Overview of HDF-EOS (Part II)
•Download as PPT, PDF•
2 likes•867 views
An HDF-EOS file contains one or more HDF-EOS data objects along with metadata. It provides geolocation information coupled to data, standardized subsetting by geography or time, and ECS metadata for archiving and distribution. The difference between HDF and HDF-EOS is at the object level within a file, not the file type itself. A file can contain both HDF and HDF-EOS objects.
Report
Share
Report
Share
Recommended
NUIG LOSD tools
This document describes tools for publishing Linked Open Statistical Data (LOSD). It outlines the LOSD publishing pipeline, which includes stages for mapping government datasets to the RDF Data Cube schema, building RDF data cubes from the datasets, and exploring the data cubes. Tools are demonstrated for assisted schema mapping using OpenRefine, building RDF data cubes from CSV files, and exploring data cubes using a web-based explorer with pivot tables, maps, and other visualizations. The tools are designed to help publish open government statistical data according to the RDF Data Cube standard.
Summary of HDF-EOS5 Files, Data Model and File Format
The document provides an overview of the HDF-EOS5 file format including:
- HDF-EOS5 files contain coremetadata, archivemetadata, and StructMetadata global attributes that provide information on the file structure and contents.
- Files can contain Grid, Swath, Point, Zonal Average, and Profile data structures with no size limits.
- Swath data is organized by time or track with irregular spacing, storing geolocation, time, and data in arrays. Grid data is organized by regular geographic spacing specified by projection parameters with data and geolocation information separated. Point data specifies locations but with no organization of the data.
Reading HDF family of formats via NetCDF-Java / CDM
NetCDF-Java is an open source Java library for reading scientific data formats like NetCDF, HDF5, HDF4, and OPeNDAP. It has been used as a component in many software projects. The library provides an object-oriented API for reading data from these file formats and exposing it to Java programs. It works by providing format readers for specific file types that can read data into the Common Data Model used by the library. The library has been tested against many file examples but could benefit from more systematic testing. Proper use of dimensions, variables, units, and metadata is important for self-documenting scientific data files.
Advancing Scientific Data Support in ArcGIS
The document discusses how ArcGIS can be used to ingest, visualize, analyze, and share scientific data stored in formats like netCDF, HDF, and GRIB, including directly reading these files, creating multidimensional mosaics for aggregation, analyzing spatial and temporal patterns, publishing services and maps, and extending capabilities through Python tools and custom geoprocessing. ArcGIS supports the full scientific data workflow from ingesting data to sharing final results and apps on the web and with other platforms like WMS and Dapple Earth Explorer.
Working with HDF and netCDF Data in ArcGIS: Tools and Case Studies
Working with HDF and netCDF Data in ArcGIS: Tools and Case StudiesThe HDF-EOS Tools and Information Center
This slide will provide an overview of current functionality, techniques, and tips for visualization and query of HDF and netCDF data in ArcGIS, as well as future plans. Hierarchical Data Format (HDF) and netCDF (network Common Data Form) are two widely used data formats for storing and manipulating scientific data. The NetCDF format also supports temporal data by using multidimensional arrays. The basic structure of data in this format and how to work with it will be covered in the context of standardized data structures and conventions. This slide will demonstrate the tools and techniques for ingesting HDF and netCDF data efficiently in ArcGIS, as well as some common workflows to employ the visualization capabilities of ArcGIS for effective animation and analysis of your data. Data Are from Mars, Tools Are from Venus
This document discusses challenges with data and tools coming from different "planets" (Mars and Venus), frustrating users. It identifies gaps in file formats, libraries, tools, and services for working with Earth science data. The author provides solutions through the HDFEOS.org project, including file format conversion and examples, and suggestions for data producers, tool developers, and end users to help address interoperability issues. Emerging topics like big data, machine learning, and voice assistants are also discussed.
SPD and KEA: HDF5 based file formats for Earth Observation
This document discusses two HDF5-based file formats for storing Earth observation data:
1. The Sorted Pulse Data (SPD) format stores laser scanning data including pulses and point data with attributes. It was created in 2008 and updated to version 4 to improve flexibility.
2. The KEA image file format implements the GDAL raster data model in HDF5, allowing large raster datasets and attribute tables to be stored together with compression. It was created in 2012 to address limitations of other formats.
Both formats take advantage of HDF5 features like compression but also discuss some limitations and lessons learned for effectively designing scientific data formats.
HDF5 Performance Enhancements with the Elimination of Unlimited Dimension
HDF5 Performance Enhancements with the Elimination of Unlimited DimensionThe HDF-EOS Tools and Information Center
This document discusses changes made to the MOPITT instrument's HDF data products over multiple versions. MOPITT measures carbon monoxide on the Terra spacecraft. Version 7 products, which use HDF-EOS5 format without unlimited dimensions, were found to be read faster than Version 6 products despite being larger in size. Experiments comparing read times of Version 7 data with and without unlimited dimensions showed a small 2% difference initially, but up to 40% faster reads for Version 7 without unlimited dimensions at first, with the difference decreasing over multiple reads. Eliminating unlimited dimensions thus improved access performance for some HDF applications like MOPITT data.Recommended
NUIG LOSD tools
This document describes tools for publishing Linked Open Statistical Data (LOSD). It outlines the LOSD publishing pipeline, which includes stages for mapping government datasets to the RDF Data Cube schema, building RDF data cubes from the datasets, and exploring the data cubes. Tools are demonstrated for assisted schema mapping using OpenRefine, building RDF data cubes from CSV files, and exploring data cubes using a web-based explorer with pivot tables, maps, and other visualizations. The tools are designed to help publish open government statistical data according to the RDF Data Cube standard.
Summary of HDF-EOS5 Files, Data Model and File Format
The document provides an overview of the HDF-EOS5 file format including:
- HDF-EOS5 files contain coremetadata, archivemetadata, and StructMetadata global attributes that provide information on the file structure and contents.
- Files can contain Grid, Swath, Point, Zonal Average, and Profile data structures with no size limits.
- Swath data is organized by time or track with irregular spacing, storing geolocation, time, and data in arrays. Grid data is organized by regular geographic spacing specified by projection parameters with data and geolocation information separated. Point data specifies locations but with no organization of the data.
Reading HDF family of formats via NetCDF-Java / CDM
NetCDF-Java is an open source Java library for reading scientific data formats like NetCDF, HDF5, HDF4, and OPeNDAP. It has been used as a component in many software projects. The library provides an object-oriented API for reading data from these file formats and exposing it to Java programs. It works by providing format readers for specific file types that can read data into the Common Data Model used by the library. The library has been tested against many file examples but could benefit from more systematic testing. Proper use of dimensions, variables, units, and metadata is important for self-documenting scientific data files.
Advancing Scientific Data Support in ArcGIS
The document discusses how ArcGIS can be used to ingest, visualize, analyze, and share scientific data stored in formats like netCDF, HDF, and GRIB, including directly reading these files, creating multidimensional mosaics for aggregation, analyzing spatial and temporal patterns, publishing services and maps, and extending capabilities through Python tools and custom geoprocessing. ArcGIS supports the full scientific data workflow from ingesting data to sharing final results and apps on the web and with other platforms like WMS and Dapple Earth Explorer.
Working with HDF and netCDF Data in ArcGIS: Tools and Case Studies
Working with HDF and netCDF Data in ArcGIS: Tools and Case StudiesThe HDF-EOS Tools and Information Center
This slide will provide an overview of current functionality, techniques, and tips for visualization and query of HDF and netCDF data in ArcGIS, as well as future plans. Hierarchical Data Format (HDF) and netCDF (network Common Data Form) are two widely used data formats for storing and manipulating scientific data. The NetCDF format also supports temporal data by using multidimensional arrays. The basic structure of data in this format and how to work with it will be covered in the context of standardized data structures and conventions. This slide will demonstrate the tools and techniques for ingesting HDF and netCDF data efficiently in ArcGIS, as well as some common workflows to employ the visualization capabilities of ArcGIS for effective animation and analysis of your data. Data Are from Mars, Tools Are from Venus
This document discusses challenges with data and tools coming from different "planets" (Mars and Venus), frustrating users. It identifies gaps in file formats, libraries, tools, and services for working with Earth science data. The author provides solutions through the HDFEOS.org project, including file format conversion and examples, and suggestions for data producers, tool developers, and end users to help address interoperability issues. Emerging topics like big data, machine learning, and voice assistants are also discussed.
SPD and KEA: HDF5 based file formats for Earth Observation
This document discusses two HDF5-based file formats for storing Earth observation data:
1. The Sorted Pulse Data (SPD) format stores laser scanning data including pulses and point data with attributes. It was created in 2008 and updated to version 4 to improve flexibility.
2. The KEA image file format implements the GDAL raster data model in HDF5, allowing large raster datasets and attribute tables to be stored together with compression. It was created in 2012 to address limitations of other formats.
Both formats take advantage of HDF5 features like compression but also discuss some limitations and lessons learned for effectively designing scientific data formats.
HDF5 Performance Enhancements with the Elimination of Unlimited Dimension
HDF5 Performance Enhancements with the Elimination of Unlimited DimensionThe HDF-EOS Tools and Information Center
This document discusses changes made to the MOPITT instrument's HDF data products over multiple versions. MOPITT measures carbon monoxide on the Terra spacecraft. Version 7 products, which use HDF-EOS5 format without unlimited dimensions, were found to be read faster than Version 6 products despite being larger in size. Experiments comparing read times of Version 7 data with and without unlimited dimensions showed a small 2% difference initially, but up to 40% faster reads for Version 7 without unlimited dimensions at first, with the difference decreasing over multiple reads. Eliminating unlimited dimensions thus improved access performance for some HDF applications like MOPITT data.Geo Analytics Canada Overview - May 2020
This document summarizes a new cloud-based platform for analyzing large satellite Earth observation (EO) data. Key points:
- Traditional desktop tools cannot handle the growing size of EO data, requiring a shift to cloud-native analytics using big data tools.
- The demonstrated platform brings algorithms to distributed data using open-source tools like Kubernetes, Docker, and Jupyter notebooks running on Google Cloud.
- It provides scalable infrastructure for processing and analyzing petabytes of EO data through pipelines, supports open data standards, and ensures data sovereignty with Canadian storage.
Why is postgis awesome?
We explain you three problems the Postgres SQL extention Postgis solves (Geometric data storage, Spatial operators, Spatial index)
Bridging ICESat and ICESat-2 Standard Data Products
This document discusses efforts to standardize data products from three NASA laser altimeter missions - ICESat, ICESat-2, and MABEL. It describes designing similar data products for all three missions to promote interoperability. Products are being developed for ICESat data using lessons from MABEL. Code is also being generated to help create products from specifications to reduce development time. The goal is to make the multi-rate point data from all three missions easily accessible and usable.
Guided Tour of Pythonian Museum
This document provides a summary of a guided tour of the Pythonian Museum focused on analyzing NASA HDF data products using Python. The tour covered 4 main Python packages for working with HDF4, HDF5, netCDF4 and GDAL data formats. It provided examples of reading and visualizing different NASA Earth science data products stored in these formats, including tools for handling latitude/longitude variables, projections, scaling and subsetting large datasets. The goal was to demonstrate common patterns and tips for working with NASA HDF Earth observation data in Python.
STAC, ZARR, COG, K8S and Data Cubes: The brave new world of satellite EO anal...
STAC, ZARR, COG, K8S and Data Cubes: The brave new world of satellite EO anal...GEO Analytics Canada
The document discusses new technologies and approaches for analyzing satellite earth observation (EO) data in the cloud, including file formats like COG and ZARR that optimize data access, metadata standards like STAC for discovery, and platforms like Kubernetes and data cubes that enable scalable analytics. It argues that traditional approaches are now obsolete, and that Canada should embrace these new cloud native techniques to become a leader in using satellite data to improve society, as the country's space agency president advocates.Multidimensional Scientific Data in ArcGIS
This document discusses how ArcGIS supports scientific multidimensional data. It can directly ingest data in formats like netCDF, HDF, and GRIB, and represent the data as raster layers, feature layers, or tables. Users can visualize, analyze, and share the data through tools in ArcGIS Desktop and services. Python can also be used to extend analytical capabilities. ArcGIS is evolving to better support scientific data through capabilities like multidimensional raster and feature layers, on-the-fly processing, and disseminating content as web services.
GES DISC Eexperiences with HDF Formats for MEaSUREs Projects
GES DISC Eexperiences with HDF Formats for MEaSUREs ProjectsThe HDF-EOS Tools and Information Center
The document discusses GES DISC's experience supporting 7 MEaSUREs projects, which create long-term Earth science data records. It recommends file formats like HDF-EOS5 and netCDF4 to promote consistency and interoperability. While HDF5 was initially recommended, ad-hoc implementations have caused issues. The document provides information on GES DISC's recommendations for file naming, formats and metadata, as well as problems encountered and resources for learning more.HDFCloud Workshop: HDF5 in the Cloud
John Readey presented on HDF5 in the cloud using HDFCloud. HDF5 can provide a cost-effective cloud infrastructure by paying for what is used rather than what may be needed. HDFCloud uses an HDF5 server to enable accessing HDF5 data through a REST API, allowing users to access large datasets without downloading entire files. It maps HDF5 objects to cloud object storage for scalable performance and uses Docker containers for elastic scaling.
HDF and netCDF Data Support in ArcGIS
The document discusses HDF and netCDF data support in ArcGIS. It provides an overview of how HDF and netCDF data can be directly ingested and used as raster datasets, mosaic datasets, feature layers, and tables in ArcGIS. This allows for scientific data to be displayed, analyzed, and shared using common GIS tools and services. It also describes existing Python tools for working with netCDF data and outlines future areas of development, including improved support for HDF5, THREDDS/OPeNDAP access, and evolving data standards.
Ccsds based file delivery protocol (cfdp) v1p3
This document discusses implementing the CCSDS File Delivery Protocol (CFDP) for future Indian Space Research Organization (ISRO) small satellites. It involves studying CFDP features, specifying a design for CFDP implementation, and developing both hardware and software platforms. The goals are to port an existing real-time operating system to a field programmable gate array with file management and implement a basic version of the CFDP standard for data storage and transmission.
Usage of NCL, IDL, and MATLAB to access NASA HDF4/HDF-EOS2/HDF-EOS5 data
Usage of NCL, IDL, and MATLAB to access NASA HDF4/HDF-EOS2/HDF-EOS5 dataThe HDF-EOS Tools and Information Center
The document discusses providing easy access to HDF data via NCL, IDL, and MATLAB. It presents examples and code snippets for reading HDF data from various NASA data centers like GES DISC, MODAPS, NSIDC, and LP-DAAC into the three software packages. Common issues when working with HDF files like HDF-EOS2 swaths with dimension maps and different ways metadata is stored are also addressed. The overall goal is to help lower the learning curve for users who want to analyze HDF data in their favorite analysis packages.ICESat-2 Metadata and Status
ICESat-2 is a NASA satellite mission scheduled to launch in December 2017 that will use photon counting laser altimetry to measure ice sheet and sea ice elevations. It will carry an advanced laser altimeter that splits each laser pulse into 6 beams in a cross-track pattern to provide dense sampling. This will allow for improved elevation estimates over rough terrain. The document discusses ICESat-2's science objectives, measurement concept, data products, processing workflow, and approach to managing metadata across different levels of data products.
Improving long-term preservation of EOS data by independently mapping HDF4 da...
Improving long-term preservation of EOS data by independently mapping HDF4 da...The HDF-EOS Tools and Information Center
The document discusses a project to improve long-term preservation of Earth Observing System (EOS) data by creating independent maps of HDF4 data objects. The project aims to map HDF4 files to allow simple readers to access data without relying on HDF software. It involves categorizing NASA HDF4 data, prototyping an XML mapping file format, and building tools to create maps and read data based on maps. The project will investigate integrating the mapping schema with standards, address HDF-EOS2 requirements, redesign the schema, implement production mapping and reading tools, and deploy the tools at NASA data centers.Working with Scientific Data in MATLAB
For WMS demo code, please visit http://hdfeos.org/workshops/ws16/presentations/day1/MATLAB/esip_demo.m
HDF OPeNDAP Project Update and Demo
The HDF Group gave a presentation on updates to the HDF-OPeNDAP project. They summarized the redesign of the HDF5 handler to support more NASA HDF5 data products, including grids, swaths, and zonal averages. They also discussed updates to the HDF4 handler to improve support for MODIS products and add new map projections. Live demonstrations showed example data from instruments like OMI, MLS, and Aquarius being served through the updated HDF handlers.
Tag-Based Browsing of Digital Collections with Inverted Indexes and Browsing ...
Tag-Based Browsing of Digital Collections with Inverted Indexes and Browsing ...Technological Ecosystems for Enhancing Multiculturality
This document proposes a tag-based browsing system for digital collections that uses inverted indexes and a browsing cache to improve performance. Tags representing element-value pairs are used to filter resources. A browsing cache stores browsing states like filtered resources and selectable tags to speed up navigating when tag filters change. Preliminary experiments show the cache can substantially improve browsing speed over an uncached system using just inverted indexes, though with increased memory usage. Future work aims to integrate browsing automata and links between resources.Improved Methods for Accessing Scientific Data for the Masses
Improved Methods for Accessing Scientific Data for the MassesThe HDF-EOS Tools and Information Center
Harris Corporation provides geospatial software and analytics tools to access and analyze scientific data from remote sensing platforms. Their ENVI and IDL software support common data formats like HDF and NetCDF and provide capabilities for calibration, bowtie correction, reprojection, and visualization of data from sensors including GOES-16, VIIRS, and ocean and weather satellites. The tools allow scientists and analysts to efficiently process large volumes of earth observation data and extract valuable information to support applications in weather forecasting, agriculture, infrastructure monitoring, and more.Leverage data analytics to reduce human space mission risks Presentation
The document discusses Lockheed Martin's Orion Multi-Purpose Crew Vehicle program. It provides an overview of Orion, which is designed to transport humans in deep space exploration missions beyond low Earth orbit. It then discusses Lockheed Martin's efforts to develop analytics capabilities to process and analyze the large amounts of telemetry data from Orion test missions in order to identify anomalies early and improve the vehicle's development.
TGS GPS- Russian well database
TGS has built an extensive database of digital well logs and related data from Russia over many years. They have well packages including scanned log images where available, standardized LAS files, and processed LAS+ files ready for interpretation. Their database contains about 24,000 wells covering major Russian basins, including over 21,000 from West Siberia. Additionally, TGS has digital databases with information on around 55,000 and 50,000 Russian wells respectively, plus seismic and license data.
HDF-EOS to GeoTIFF Conversion Tool & HDF-EOS Plug-in for HDFView
HDF-EOS to GeoTIFF Conversion Tool & HDF-EOS Plug-in for HDFViewThe HDF-EOS Tools and Information Center
This document describes tools for working with HDF-EOS data formats: the HDF-EOS to GeoTIFF Conversion Tool (HEG) and an HDF-EOS plugin for the HDFView browser. HEG allows conversion between HDF-EOS and GeoTIFF formats and supports subsetting, reprojection, stitching and other functions. The HDFView plugin allows browsing and conversion between HDF-EOS2 and HDF-EOS5 formats within HDFView. Both tools are available for download and support a variety of Earth science datasets.What is HDF-EOS?
HDF-EOS defines standard profiles for using HDF to organize Earth observation data. It builds on HDF by adding geolocation information coupled to data for point, swath, and grid collection profiles. HDF-EOS is used to define over 200 products from instruments on Terra to be distributed through EOSDIS. It provides full geolocation support and metadata to allow search and access of Earth science data.
ESDIS Status (2002)
The document provides an overview and status update of the Earth Science Data and Information System (ESDIS). ESDIS has successfully supported numerous Earth science satellite missions and currently manages over 2 petabytes of science data. In fiscal year 2002, ESDIS delivered over 16 million data products to more than 1.8 million users. ESDIS is working to enhance its capabilities through initiatives like Data Pools and the EOS ClearingHOuse (ECHO) metadata broker. HDF-EOS 5 development is nearly complete and the workshop will discuss next steps for HDF-EOS tools and community adoption of HDF-EOS 5.
More Related Content
What's hot
Geo Analytics Canada Overview - May 2020
This document summarizes a new cloud-based platform for analyzing large satellite Earth observation (EO) data. Key points:
- Traditional desktop tools cannot handle the growing size of EO data, requiring a shift to cloud-native analytics using big data tools.
- The demonstrated platform brings algorithms to distributed data using open-source tools like Kubernetes, Docker, and Jupyter notebooks running on Google Cloud.
- It provides scalable infrastructure for processing and analyzing petabytes of EO data through pipelines, supports open data standards, and ensures data sovereignty with Canadian storage.
Why is postgis awesome?
We explain you three problems the Postgres SQL extention Postgis solves (Geometric data storage, Spatial operators, Spatial index)
Bridging ICESat and ICESat-2 Standard Data Products
This document discusses efforts to standardize data products from three NASA laser altimeter missions - ICESat, ICESat-2, and MABEL. It describes designing similar data products for all three missions to promote interoperability. Products are being developed for ICESat data using lessons from MABEL. Code is also being generated to help create products from specifications to reduce development time. The goal is to make the multi-rate point data from all three missions easily accessible and usable.
Guided Tour of Pythonian Museum
This document provides a summary of a guided tour of the Pythonian Museum focused on analyzing NASA HDF data products using Python. The tour covered 4 main Python packages for working with HDF4, HDF5, netCDF4 and GDAL data formats. It provided examples of reading and visualizing different NASA Earth science data products stored in these formats, including tools for handling latitude/longitude variables, projections, scaling and subsetting large datasets. The goal was to demonstrate common patterns and tips for working with NASA HDF Earth observation data in Python.
STAC, ZARR, COG, K8S and Data Cubes: The brave new world of satellite EO anal...
STAC, ZARR, COG, K8S and Data Cubes: The brave new world of satellite EO anal...GEO Analytics Canada
The document discusses new technologies and approaches for analyzing satellite earth observation (EO) data in the cloud, including file formats like COG and ZARR that optimize data access, metadata standards like STAC for discovery, and platforms like Kubernetes and data cubes that enable scalable analytics. It argues that traditional approaches are now obsolete, and that Canada should embrace these new cloud native techniques to become a leader in using satellite data to improve society, as the country's space agency president advocates.Multidimensional Scientific Data in ArcGIS
This document discusses how ArcGIS supports scientific multidimensional data. It can directly ingest data in formats like netCDF, HDF, and GRIB, and represent the data as raster layers, feature layers, or tables. Users can visualize, analyze, and share the data through tools in ArcGIS Desktop and services. Python can also be used to extend analytical capabilities. ArcGIS is evolving to better support scientific data through capabilities like multidimensional raster and feature layers, on-the-fly processing, and disseminating content as web services.
GES DISC Eexperiences with HDF Formats for MEaSUREs Projects
GES DISC Eexperiences with HDF Formats for MEaSUREs ProjectsThe HDF-EOS Tools and Information Center
The document discusses GES DISC's experience supporting 7 MEaSUREs projects, which create long-term Earth science data records. It recommends file formats like HDF-EOS5 and netCDF4 to promote consistency and interoperability. While HDF5 was initially recommended, ad-hoc implementations have caused issues. The document provides information on GES DISC's recommendations for file naming, formats and metadata, as well as problems encountered and resources for learning more.HDFCloud Workshop: HDF5 in the Cloud
John Readey presented on HDF5 in the cloud using HDFCloud. HDF5 can provide a cost-effective cloud infrastructure by paying for what is used rather than what may be needed. HDFCloud uses an HDF5 server to enable accessing HDF5 data through a REST API, allowing users to access large datasets without downloading entire files. It maps HDF5 objects to cloud object storage for scalable performance and uses Docker containers for elastic scaling.
HDF and netCDF Data Support in ArcGIS
The document discusses HDF and netCDF data support in ArcGIS. It provides an overview of how HDF and netCDF data can be directly ingested and used as raster datasets, mosaic datasets, feature layers, and tables in ArcGIS. This allows for scientific data to be displayed, analyzed, and shared using common GIS tools and services. It also describes existing Python tools for working with netCDF data and outlines future areas of development, including improved support for HDF5, THREDDS/OPeNDAP access, and evolving data standards.
Ccsds based file delivery protocol (cfdp) v1p3
This document discusses implementing the CCSDS File Delivery Protocol (CFDP) for future Indian Space Research Organization (ISRO) small satellites. It involves studying CFDP features, specifying a design for CFDP implementation, and developing both hardware and software platforms. The goals are to port an existing real-time operating system to a field programmable gate array with file management and implement a basic version of the CFDP standard for data storage and transmission.
Usage of NCL, IDL, and MATLAB to access NASA HDF4/HDF-EOS2/HDF-EOS5 data
Usage of NCL, IDL, and MATLAB to access NASA HDF4/HDF-EOS2/HDF-EOS5 dataThe HDF-EOS Tools and Information Center
The document discusses providing easy access to HDF data via NCL, IDL, and MATLAB. It presents examples and code snippets for reading HDF data from various NASA data centers like GES DISC, MODAPS, NSIDC, and LP-DAAC into the three software packages. Common issues when working with HDF files like HDF-EOS2 swaths with dimension maps and different ways metadata is stored are also addressed. The overall goal is to help lower the learning curve for users who want to analyze HDF data in their favorite analysis packages.ICESat-2 Metadata and Status
ICESat-2 is a NASA satellite mission scheduled to launch in December 2017 that will use photon counting laser altimetry to measure ice sheet and sea ice elevations. It will carry an advanced laser altimeter that splits each laser pulse into 6 beams in a cross-track pattern to provide dense sampling. This will allow for improved elevation estimates over rough terrain. The document discusses ICESat-2's science objectives, measurement concept, data products, processing workflow, and approach to managing metadata across different levels of data products.
Improving long-term preservation of EOS data by independently mapping HDF4 da...
Improving long-term preservation of EOS data by independently mapping HDF4 da...The HDF-EOS Tools and Information Center
The document discusses a project to improve long-term preservation of Earth Observing System (EOS) data by creating independent maps of HDF4 data objects. The project aims to map HDF4 files to allow simple readers to access data without relying on HDF software. It involves categorizing NASA HDF4 data, prototyping an XML mapping file format, and building tools to create maps and read data based on maps. The project will investigate integrating the mapping schema with standards, address HDF-EOS2 requirements, redesign the schema, implement production mapping and reading tools, and deploy the tools at NASA data centers.Working with Scientific Data in MATLAB
For WMS demo code, please visit http://hdfeos.org/workshops/ws16/presentations/day1/MATLAB/esip_demo.m
HDF OPeNDAP Project Update and Demo
The HDF Group gave a presentation on updates to the HDF-OPeNDAP project. They summarized the redesign of the HDF5 handler to support more NASA HDF5 data products, including grids, swaths, and zonal averages. They also discussed updates to the HDF4 handler to improve support for MODIS products and add new map projections. Live demonstrations showed example data from instruments like OMI, MLS, and Aquarius being served through the updated HDF handlers.
Tag-Based Browsing of Digital Collections with Inverted Indexes and Browsing ...
Tag-Based Browsing of Digital Collections with Inverted Indexes and Browsing ...Technological Ecosystems for Enhancing Multiculturality
This document proposes a tag-based browsing system for digital collections that uses inverted indexes and a browsing cache to improve performance. Tags representing element-value pairs are used to filter resources. A browsing cache stores browsing states like filtered resources and selectable tags to speed up navigating when tag filters change. Preliminary experiments show the cache can substantially improve browsing speed over an uncached system using just inverted indexes, though with increased memory usage. Future work aims to integrate browsing automata and links between resources.Improved Methods for Accessing Scientific Data for the Masses
Improved Methods for Accessing Scientific Data for the MassesThe HDF-EOS Tools and Information Center
Harris Corporation provides geospatial software and analytics tools to access and analyze scientific data from remote sensing platforms. Their ENVI and IDL software support common data formats like HDF and NetCDF and provide capabilities for calibration, bowtie correction, reprojection, and visualization of data from sensors including GOES-16, VIIRS, and ocean and weather satellites. The tools allow scientists and analysts to efficiently process large volumes of earth observation data and extract valuable information to support applications in weather forecasting, agriculture, infrastructure monitoring, and more.Leverage data analytics to reduce human space mission risks Presentation
The document discusses Lockheed Martin's Orion Multi-Purpose Crew Vehicle program. It provides an overview of Orion, which is designed to transport humans in deep space exploration missions beyond low Earth orbit. It then discusses Lockheed Martin's efforts to develop analytics capabilities to process and analyze the large amounts of telemetry data from Orion test missions in order to identify anomalies early and improve the vehicle's development.
TGS GPS- Russian well database
TGS has built an extensive database of digital well logs and related data from Russia over many years. They have well packages including scanned log images where available, standardized LAS files, and processed LAS+ files ready for interpretation. Their database contains about 24,000 wells covering major Russian basins, including over 21,000 from West Siberia. Additionally, TGS has digital databases with information on around 55,000 and 50,000 Russian wells respectively, plus seismic and license data.
HDF-EOS to GeoTIFF Conversion Tool & HDF-EOS Plug-in for HDFView
HDF-EOS to GeoTIFF Conversion Tool & HDF-EOS Plug-in for HDFViewThe HDF-EOS Tools and Information Center
This document describes tools for working with HDF-EOS data formats: the HDF-EOS to GeoTIFF Conversion Tool (HEG) and an HDF-EOS plugin for the HDFView browser. HEG allows conversion between HDF-EOS and GeoTIFF formats and supports subsetting, reprojection, stitching and other functions. The HDFView plugin allows browsing and conversion between HDF-EOS2 and HDF-EOS5 formats within HDFView. Both tools are available for download and support a variety of Earth science datasets.What's hot (20)
Bridging ICESat and ICESat-2 Standard Data Products
Bridging ICESat and ICESat-2 Standard Data Products
STAC, ZARR, COG, K8S and Data Cubes: The brave new world of satellite EO anal...
STAC, ZARR, COG, K8S and Data Cubes: The brave new world of satellite EO anal...
GES DISC Eexperiences with HDF Formats for MEaSUREs Projects
GES DISC Eexperiences with HDF Formats for MEaSUREs Projects
Usage of NCL, IDL, and MATLAB to access NASA HDF4/HDF-EOS2/HDF-EOS5 data
Usage of NCL, IDL, and MATLAB to access NASA HDF4/HDF-EOS2/HDF-EOS5 data
Improving long-term preservation of EOS data by independently mapping HDF4 da...
Improving long-term preservation of EOS data by independently mapping HDF4 da...
Tag-Based Browsing of Digital Collections with Inverted Indexes and Browsing ...
Tag-Based Browsing of Digital Collections with Inverted Indexes and Browsing ...
Improved Methods for Accessing Scientific Data for the Masses
Improved Methods for Accessing Scientific Data for the Masses
Leverage data analytics to reduce human space mission risks Presentation
Leverage data analytics to reduce human space mission risks Presentation
HDF-EOS to GeoTIFF Conversion Tool & HDF-EOS Plug-in for HDFView
HDF-EOS to GeoTIFF Conversion Tool & HDF-EOS Plug-in for HDFView
Similar to An Overview of HDF-EOS (Part II)
What is HDF-EOS?
HDF-EOS defines standard profiles for using HDF to organize Earth observation data. It builds on HDF by adding geolocation information coupled to data for point, swath, and grid collection profiles. HDF-EOS is used to define over 200 products from instruments on Terra to be distributed through EOSDIS. It provides full geolocation support and metadata to allow search and access of Earth science data.
ESDIS Status (2002)
The document provides an overview and status update of the Earth Science Data and Information System (ESDIS). ESDIS has successfully supported numerous Earth science satellite missions and currently manages over 2 petabytes of science data. In fiscal year 2002, ESDIS delivered over 16 million data products to more than 1.8 million users. ESDIS is working to enhance its capabilities through initiatives like Data Pools and the EOS ClearingHOuse (ECHO) metadata broker. HDF-EOS 5 development is nearly complete and the workshop will discuss next steps for HDF-EOS tools and community adoption of HDF-EOS 5.
Profile of HDF-EOS5 Files
The document describes the general structure of HDF-EOS5 files. HDF-EOS5 files can contain gridded data, swath data, point measurements, and other objects. Key aspects include required and optional metadata like coremetadata and archivemetadata, along with data structures like grids, swaths, and points. Grids contain projection information and dimensions to geolocate data fields. Swaths contain geolocation, data, and profile fields along with dimension maps. Points contain scattered records linked by a common field. Compression is supported at the field level within data structures.
LoCloud Geocoding Application, Runar Bergheim, Asplan Viak Internet
This presentation provides an introduction and walk-through of the LoCloud Geocoding application, used during the LoCloud training workshops. The application is a tool to add geographic coordinates to existing data (such as records describing items of content in digital libraries). The presentation includes a step-by-step walk through of the application
The Landsat 7 Processing System (LPS) Level Zero-R Science Products
The Landsat 7 Processing System (LPS) Level Zero-R Science ProductsThe HDF-EOS Tools and Information Center
The Landsat 7 Processing System (LPS) processes Landsat 7 data into science data products. It receives raw telemetry from the Landsat 7 satellite and constructs images from the data without applying corrections. The LPS divides the data into subintervals and produces multi-band files and metadata for each. It also generates browse images and assesses cloud cover for each subinterval. The LPS data products are separated into Format 1 and Format 2 files representing the two types of raw data received by the LPS. The Level Zero-R science products created by the LPS support further processing but are not yet calibrated or rectified.Geoservices Activities at EDINA
1) Postgres and PostGIS have been used at EDINA for over 8 years to power major geospatial services like Digimap.
2) It is used for data storage, mapping, spatial indexing, querying, and data downloads. Postgres allows EDINA to handle large amounts of geospatial data and large user bases.
3) EDINA finds Postgres reliable, performant, scalable, and standards-compliant with good support tools. It will continue being the core database for EDINA's geoservices.
GIS ANALYTICS-2011
The document discusses establishing geospatial analysis capabilities within an exploration environment. It outlines collecting all relevant exploration data, representing it properly in a GIS database, and generating continuous geospatial grids using deterministic and geostatistical methods. These grids would support spatial analysis and decision making by allowing visualization and queries of parameters like probability of success. Maintaining data quality and understanding relationships between different data types is important for effective analysis.
LoCloud Geolocation enrichment tools, Siri Slettvag, Asplan Viak Internet (Av...
Presentation about the Geolocation Enrichment tools developed by Avinet as part of the LoCloud project. The tool can be used to add geographic locations to existing datasets provided by cultural institutions. It can be used as part of existing workflows by curators, or in crowd-sourcing projects with users finding places and adding coordinates.
http://www.locloud.eu
HDF5, HDF-EOS and Geospatial Data Archives
The National Archives and Records Administration (NARA) has a duty to preserve federal records, including a large and diverse volume of electronic records. NARA evaluated HDF5 and HDF-EOS as potential formats for archiving geospatial data given their abilities to represent many data models, accommodate large data volumes, and enable high performance access and visualization applications. HDF5 and HDF-EOS are established open standards that efficiently store and provide access to huge data files using flexible, extendable, and portable libraries and APIs. The project explored using HDF5 and HDF-EOS data types to store non-remote sensed geospatial data like surfaces, volumes, rasters, vectors, and tables. It also examined converting data from other formats and
NWGISS: The Web GIS Software Suite for Interoperable Access and Manipulation ...
NWGISS: The Web GIS Software Suite for Interoperable Access and Manipulation ...The HDF-EOS Tools and Information Center
The document describes the NWGISS (NASA HDF-EOS Web GIS Software Suite), which aims to make HDF-EOS remote sensing data easily accessible to GIS users through open standards. It discusses key components of NWGISS including a map server, coverage server, catalog server, and client that implement Open Geospatial Consortium (OGC) specifications. These allow interoperable access and manipulation of HDF-EOS data in different formats. The suite provides on-demand data processing and reformatting to meet users' individual needs.Profile of HDF-EOS5 Files
HDF-EOS is a software library designed to support NASA Earth Observing System (EOS) science data. HDF is the Hierarchical Data Format developed by The HDF Group. Specific data structures in HDF-EOS5 which are containers for science data are: Grid, Point, Zonal Average and Swath. These data structures are constructed from standard HDF5 data objects, using EOS conventions, through the use of a software library. This presentation is intended to familiarize current HDF-EOS users with the structure of HDF-EOS5 files and the Grid, Swath, Point and Zonal Average structures used in these files.
HDF-EOS Development: Current Status and Tools
This document provides a summary of the current status and schedule of HDF-EOS development. It describes what HDF-EOS is, its data structures like Grid, Swath, Point, and Zonal Average, and applications that can be used to access and process HDF-EOS data. HDF-EOS 5, based on HDF5, is now used by EOS Aura producers and users, while HDF-EOS 2 will continue to be supported.
Metadata in EOSDIS
Metadata Models in EOSDIS (V0, ECS, ECHO, FGDC, ISO 19115)
Source: http://hdfeos.org/workshops/ws07/presentations/Ullman2/Ullman_meta.ppt
HDF-EOS Development Status and Maintenance Support
The document discusses the HDF-EOS data format and software toolkit. It summarizes recent additions to the toolkit, including new capabilities for handling swath data and accessing metadata. Near-future development plans include additional testing and preparing the software and test data for the EOS-AM1 launch. Potential enhancements discussed include supporting new data types and sensor geometries, incorporating HDF5, and adding more sophisticated subsetting capabilities. Maintenance and user support is provided by ECS through 2022.
Edward King SPEDDEXES 2014
The document discusses using OPeNDAP as a means to provide access to gridded remote sensing data from various sources. It describes setting up an OPeNDAP server to serve various data formats and implementing a system using a PostGIS database to add spatial and temporal metadata to help discover and access the data more easily. Key lessons learned include that while OPeNDAP performs well, significant infrastructure is needed to handle the variety of data formats and projections, and it is tempting to add too many features to the system.
iXblue - DELPH geophysical software
DELPH is a software suite for gophysical data acquisition, processing and interpretation. It features side-scan sonar, seismic and sub-bottom profiler as well as magnetometer analysis. Capable of working with very large multi-sensor datasets, it automates data processing and simplifies operations for geophysicists and hydrographers.
HDF-EOS Maintenance, Current Development and Tools
We will summarize current status of HDF-EOS and associated tools. Update on HDF-EOS, HDFView plug-in and The HDF-EOS to GeoTIFF (HEG) conversion tool, including recent changes to the software, ongoing maintenance, upcoming releases, future plans, and issues will be discussed.
We will also summarize the status of HDF-EOS RFC. The HDF-EOS plug-in for the THG-developed tool, HDFView, has been enhanced. The plug-in offers browse capability for both HDF4 and HDF5 - based HDF-EOS files. HDFView can also process vanilla HDF4 and HDF5 files. New features including support for Point and Zonal Average objects have been added. A port to Mac OS X version will be available in next release.
The HDF-EOS to GeoTIFF (HEG) conversion tool has been augmented to include new projections, and support for additional AMSR-E and AIRS products. Subsetting features have also been augmented. The tool is available in both stand-alone and EOS DAAC online versions.
Content Framework for Operational Environmental Remote Sensing Data Sets: NPO...
Content Framework for Operational Environmental Remote Sensing Data Sets: NPO...The HDF-EOS Tools and Information Center
This poster (or slides) describes the main concepts which relate data format (HDF5), metadata, and data organization, as they are being applied to NPOESS. It includes the motivation for selecting HDF5; the motivation and general implementation of FGDC base metadata and remote sensing extensions; and the current attempt to use best experiences with HDF-EOS swath and netCDF Climate & Forecast conventions to guide NPOESS product definition. The HDF-EOS5 Tutorial
The document describes HDF-EOS5, an extension of HDF used by NASA for Earth science data. HDF-EOS5 is based on HDF5 and contains standardized structures for gridded, swath, point, and zonal average data. It provides a library for reading, writing, and manipulating these data structures and their associated metadata. The library contains functions prefixed with "HE5_" for accessing, defining, input/output, inquiry, and subsetting HDF-EOS5 data.
HDF Explorer
HDF Explorer is a data visualization program that reads HDF and HDF5 file formats. It allows users to browse data hierarchically and visualize it numerically in grids, as pixel colors, or as vector data by combining two scalar datasets. The software has over 4000 users in 54 countries, including government agencies, national labs, companies, and universities. A demo of HDF Explorer opening sample data files is included.
Similar to An Overview of HDF-EOS (Part II) (20)
LoCloud Geocoding Application, Runar Bergheim, Asplan Viak Internet
LoCloud Geocoding Application, Runar Bergheim, Asplan Viak Internet
The Landsat 7 Processing System (LPS) Level Zero-R Science Products
The Landsat 7 Processing System (LPS) Level Zero-R Science Products
LoCloud Geolocation enrichment tools, Siri Slettvag, Asplan Viak Internet (Av...
LoCloud Geolocation enrichment tools, Siri Slettvag, Asplan Viak Internet (Av...
NWGISS: The Web GIS Software Suite for Interoperable Access and Manipulation ...
NWGISS: The Web GIS Software Suite for Interoperable Access and Manipulation ...
HDF-EOS Development Status and Maintenance Support
HDF-EOS Development Status and Maintenance Support
HDF-EOS Maintenance, Current Development and Tools
HDF-EOS Maintenance, Current Development and Tools
Content Framework for Operational Environmental Remote Sensing Data Sets: NPO...
Content Framework for Operational Environmental Remote Sensing Data Sets: NPO...
More from The HDF-EOS Tools and Information Center
Cloud-Optimized HDF5 Files
This document discusses how to optimize HDF5 files for efficient access in cloud object stores. Key optimizations include using large dataset chunk sizes of 1-4 MiB, consolidating internal file metadata, and minimizing variable-length datatypes. The document recommends creating files with paged aggregation and storing file content information in the user block to enable fast discovery of file contents when stored in object stores.
Accessing HDF5 data in the cloud with HSDS
This document provides an overview of HSDS (Highly Scalable Data Service), which is a REST-based service that allows accessing HDF5 data stored in the cloud. It discusses how HSDS maps HDF5 objects like datasets and groups to individual cloud storage objects to optimize performance. The document also describes how HSDS was used to improve access performance for NASA ICESat-2 HDF5 data on AWS S3 by hyper-chunking datasets into larger chunks spanning multiple original HDF5 chunks. Benchmark results showed that accessing the data through HSDS provided over 2x faster performance than other methods like ROS3 or S3FS that directly access the cloud storage.
The State of HDF
This document summarizes the current status and focus of the HDF Group. It discusses that the HDF Group is located in Champaign, IL and is a non-profit organization focused on developing and maintaining HDF software and data formats. It provides an overview of recent HDF5, HDF4 and HDFView releases and notes areas of focus for software quality improvements, increased transparency, strengthening the community, and modernizing HDF products. It invites support and participation in upcoming user group meetings.
Highly Scalable Data Service (HSDS) Performance Features
This document provides an overview of HSDS (HDF Server and Data Service), which allows HDF5 files to be stored and accessed from the cloud. Key points include:
- HSDS maps HDF5 objects like datasets and groups to individual cloud storage objects for scalability and parallelism.
- Features include streaming support, fancy indexing for complex queries, and caching for improved performance.
- HSDS can be deployed on Docker, Kubernetes, or AWS Lambda depending on needs.
- Case studies show HSDS is used by organizations like NREL and NSF to make petabytes of scientific data publicly accessible in the cloud.
Creating Cloud-Optimized HDF5 Files
This document discusses creating cloud-optimized HDF5 files by rearranging internal structures for more efficient data access in cloud object stores. It describes cloud-native and cloud-optimized storage formats, with the latter involving storing the entire HDF5 file as a single object. The benefits of cloud-optimized HDF5 include fast scanning and using the HDF5 library. Key aspects covered include using optimal chunk sizes, compression, and minimizing variable-length datatypes.
HDF5 OPeNDAP Handler Updates, and Performance Discussion
This document discusses updates and performance improvements to the HDF5 OPeNDAP data handler. It provides a history of the handler since 2001 and describes recent updates including supporting DAP4, new data types, and NetCDF data models. A performance study showed that passing compressed HDF5 data through the handler without decompressing/recompressing led to speedups of around 17-30x by leveraging HDF5 direct I/O APIs. This allows outputting HDF5 files as NetCDF files much faster through the handler.
Hyrax: Serving Data from S3
This document provides instructions for using the Hyrax software to serve scientific data files stored on Amazon S3 using the OPeNDAP data access protocol. It describes how to generate ancillary metadata files called DMR++ files using the get_dmrpp tool that provide information about the data file structure and locations. The document explains how to run get_dmrpp inside a Docker container to process data files on S3 and generate customized DMR++ files that the Hyrax server can use to serve the files to clients.
Accessing Cloud Data and Services Using EDL, Pydap, MATLAB
This document provides an overview and examples of accessing cloud data and services using the Earthdata Login (EDL), Pydap, and MATLAB. It discusses some common problems users encounter, such as being unable to access HDF5 data on AWS S3 using MATLAB or read data from OPeNDAP servers using Pydap. Solutions presented include using EDL to get temporary AWS tokens for S3 access in MATLAB and providing code examples on the HDFEOS website to help users access S3 data and OPeNDAP services. The document also notes some limitations, such as tokens being valid for only 1 hour, and workarounds like requesting new tokens or using the MATLAB HDF5 API instead of the netCDF API.
HDF - Current status and Future Directions
The HDF5 Roadmap and New Features document outlines upcoming changes and improvements to the HDF5 library. Key points include:
- HDF5 1.13.x releases will include new features like selection I/O, the Onion VFD for versioned files, improved VFD SWMR for single-writer multiple-reader access, and subfiling for parallel I/O.
- The Virtual Object Layer allows customizing HDF5 object storage and introduces terminal and pass-through connectors.
- The Onion VFD stores versions of HDF5 files in a separate onion file for versioned access.
- VFD SWMR improves on legacy SWMR by implementing single-writer multiple-reader capabilities
HDFEOS.org User Analsys, Updates, and Future
This document discusses user analysis of the HDFEOS.org website and plans for future improvements. It finds that the majority of the site's 100 daily users are "quiet", not posting on forums or other interactive elements. The main user types are locators, who search for examples or data; mergers, who combine or mosaic datasets; and converters, who change file formats. The document outlines recent updates focused on these user types, like adding Python examples for subsetting and calculating latitude and longitude. It proposes future work on artificial intelligence/machine learning uses of HDF files and examples for processing HDF data in the cloud.
HDF - Current status and Future Directions
This document summarizes a presentation about the current status and future directions of the Hierarchical Data Format (HDF) software. It provides updates on recent HDF5 releases, development efforts including new compression methods and ways to access HDF5 data, and outreach resources. It concludes by inviting the audience to share wishes for future HDF development.
H5Coro: The Cloud-Optimized Read-Only Library
The document describes H5Coro, a new C++ library for reading HDF5 files from cloud storage. H5Coro was created to optimize HDF5 reading for cloud environments by minimizing I/O operations through caching and efficient HTTP requests. Performance tests showed H5Coro was 77-132x faster than the previous HDF5 library at reading HDF5 data from Amazon S3 for NASA's SlideRule project. H5Coro supports common HDF5 elements but does not support writing or some complex HDF5 data types and messages to focus on optimized read-only performance for time series data stored sequentially in memory.
MATLAB Modernization on HDF5 1.10
This document summarizes MathWorks' work to modernize MATLAB's support for HDF5. Key points include:
1) MATLAB now supports HDF5 1.10.7 features like single-writer/multiple-reader access and virtual datasets through new and updated low-level functions.
2) Performance benchmarks show some improvements but also regressions compared to the previous HDF5 version, and work continues to optimize code and support future versions.
3) There are compatibility considerations for Linux filter plugins, but interim solutions are provided until MathWorks can ship a single HDF5 version.
HDF for the Cloud - Serverless HDF
HSDS provides HDF as a service through a REST API that can scale across nodes. New releases will enable serverless operation using AWS Lambda or direct client access without a server. This allows HDF data to be accessed remotely without managing servers. HSDS stores each HDF object separately, making it compatible with cloud object storage. Performance on AWS Lambda is slower than a dedicated server but has no management overhead. Direct client access has better performance but limits collaboration between clients.
HDF5 <-> Zarr
HDF5 and Zarr are data formats that can be used to store and access scientific data. This presentation discusses approaches to translating between the two formats. It describes how HDF5 files were translated to the Zarr format by creating a separate Zarr store to hold HDF5 file chunks, and storing chunk location metadata. It also discusses an implementation that translates Zarr data to the HDF5 format by using a special chunking layout and storing chunk information in an HDF5 compound dataset. Limitations of the translations include lack of support for some HDF5 dataset properties in Zarr, and lack of support for some Zarr compression methods in the HDF5 implementation.
HDF for the Cloud - New HDF Server Features
The document discusses HDF for the cloud, including new features of the HDF Server and what's next. Key points:
- HDF Server uses a "sharded schema" that maps HDF5 objects to individual storage objects, allowing parallel access and updates without transferring entire files.
- Implementations include HSDS software that uses the sharded schema with an API and SDKs for different languages like h5pyd for Python.
- New features of HSDS 0.6 include support for POSIX, Azure, AWS Lambda, and role-based access control.
- Future work includes direct access to storage without a server intermediary for some use cases.
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3The HDF-EOS Tools and Information Center
This document compares different methods for accessing HDF and netCDF files stored on Amazon S3, including Apache Drill, THREDDS Data Server (TDS), and HDF5 Virtual File Driver (VFD). A benchmark test of accessing a 24GB HDF5/netCDF-4 file on S3 from Amazon EC2 found that TDS performed the best, responding within 2 minutes, while Apache Drill failed after 7 minutes. The document concludes that TDS 5.0 is the clear winner based on performance and support for role-based access control and HDF4 files, but the best solution depends on use case and software.STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...
STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...The HDF-EOS Tools and Information Center
This document discusses STARE-PODS, a proposal to NASA/ACCESS-19 to develop a scalable data store for earth science data using the SpatioTemporal Adaptive Resolution Encoding (STARE) indexing scheme. STARE allows diverse earth science data to be unified and indexed, enabling the data to be partitioned and stored in a Parallel Optimized Data Store (PODS) for efficient analysis. The HDF Virtual Object Layer and Virtual Data Set technologies can then provide interfaces to access the data in STARE-PODS in a familiar way. The goal is for STARE-PODS to organize diverse data for alignment and parallel/distributed storage and processing to enable integrative analysis at scale.HDF5 and Ecosystem: What Is New?
This document provides an overview and update on HDF5 and its ecosystem. Key points include:
- HDF5 1.12.0 was recently released with new features like the Virtual Object Layer and external references.
- The HDF5 library now supports accessing data in the cloud using connectors like S3 VFD and REST VOL without needing to modify applications.
- Projects like HDFql and H5CPP provide additional interfaces for querying and working with HDF5 files from languages like SQL, C++, and Python.
- The HDF5 community is moving development to GitHub and improving documentation resources on the HDF wiki site.
HDF5 Roadmap 2019-2020
This document summarizes new features in HDF5 1.12.0, including support for storing references to objects and attributes across files, new storage backends using a virtual object layer (VOL), and virtual file drivers (VFDs) for Amazon S3 and HDFS. It outlines the HDF5 roadmap for 2019-2022, which includes continued support for HDF5 1.8 and 1.10, and new features in future 1.12.x releases like querying, indexing, and provenance tracking.
More from The HDF-EOS Tools and Information Center (20)
Highly Scalable Data Service (HSDS) Performance Features
Highly Scalable Data Service (HSDS) Performance Features
HDF5 OPeNDAP Handler Updates, and Performance Discussion
HDF5 OPeNDAP Handler Updates, and Performance Discussion
Accessing Cloud Data and Services Using EDL, Pydap, MATLAB
Accessing Cloud Data and Services Using EDL, Pydap, MATLAB
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3
STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...
STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...
Recently uploaded
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift
Overview
Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices.
Key Topics Covered
1. Introduction to Anomaly Detection
- Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems.
2. Understanding Edge (IoT)
- Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source.
3. What is ArgoCD?
- Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices.
4. Deployment Using ArgoCD for Edge Devices
- Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD.
5. Introduction to Apache Kafka and S3
- Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions.
6. Viewing Kafka Messages in the Data Lake
- Learn how to view and analyze Kafka messages stored in a data lake for better insights.
7. What is Prometheus?
- Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices.
8. Monitoring Application Metrics with Prometheus
- Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system.
9. What is Camel K?
- Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes.
10. Configuring Camel K Integrations for Data Pipelines
- Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow.
11. What is a Jupyter Notebook?
- Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text.
12. Jupyter Notebooks with Code Examples
- Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.
TrustArc Webinar - 2024 Global Privacy Survey
How does your privacy program stack up against your peers? What challenges are privacy teams tackling and prioritizing in 2024?
In the fifth annual Global Privacy Benchmarks Survey, we asked over 1,800 global privacy professionals and business executives to share their perspectives on the current state of privacy inside and outside of their organizations. This year’s report focused on emerging areas of importance for privacy and compliance professionals, including considerations and implications of Artificial Intelligence (AI) technologies, building brand trust, and different approaches for achieving higher privacy competence scores.
See how organizational priorities and strategic approaches to data security and privacy are evolving around the globe.
This webinar will review:
- The top 10 privacy insights from the fifth annual Global Privacy Benchmarks Survey
- The top challenges for privacy leaders, practitioners, and organizations in 2024
- Key themes to consider in developing and maintaining your privacy program
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Simplify your search for a reliable Python development partner! This list presents the top 10 trusted US providers offering comprehensive Python development services, ensuring your project's success from conception to completion.
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI, built on the robust SAP Business Technology Platform (SAP BTP), is the latest and most advanced version of our AI development, reaffirming our commitment to delivering top-tier AI solutions. Skybuffer AI harnesses all the innovative capabilities of the SAP BTP in the AI domain, from Conversational AI to cutting-edge Generative AI and Retrieval-Augmented Generation (RAG). It also helps SAP customers safeguard their investments into SAP Conversational AI and ensure a seamless, one-click transition to SAP Business AI.
With Skybuffer AI, various AI models can be integrated into a single communication channel such as Microsoft Teams. This integration empowers business users with insights drawn from SAP backend systems, enterprise documents, and the expansive knowledge of Generative AI. And the best part of it is that it is all managed through our intuitive no-code Action Server interface, requiring no extensive coding knowledge and making the advanced AI accessible to more users.
Taking AI to the Next Level in Manufacturing.pdf
Read Taking AI to the Next Level in Manufacturing to gain insights on AI adoption in the manufacturing industry, such as:
1. How quickly AI is being implemented in manufacturing.
2. Which barriers stand in the way of AI adoption.
3. How data quality and governance form the backbone of AI.
4. Organizational processes and structures that may inhibit effective AI adoption.
6. Ideas and approaches to help build your organization's AI strategy.
Azure API Management to expose backend services securely
How to use Azure API Management to expose backend service securely
Letter and Document Automation for Bonterra Impact Management (fka Social Sol...
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on automated letter generation for Bonterra Impact Management using Google Workspace or Microsoft 365.
Interested in deploying letter generation automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
Best 20 SEO Techniques To Improve Website Visibility In SERP
Boost your website's visibility with proven SEO techniques! Our latest blog dives into essential strategies to enhance your online presence, increase traffic, and rank higher on search engines. From keyword optimization to quality content creation, learn how to make your site stand out in the crowded digital landscape. Discover actionable tips and expert insights to elevate your SEO game.
System Design Case Study: Building a Scalable E-Commerce Platform - Hiike
This case study explores designing a scalable e-commerce platform, covering key requirements, system components, and best practices.
Generating privacy-protected synthetic data using Secludy and Milvus
During this demo, the founders of Secludy will demonstrate how their system utilizes Milvus to store and manipulate embeddings for generating privacy-protected synthetic data. Their approach not only maintains the confidentiality of the original data but also enhances the utility and scalability of LLMs under privacy constraints. Attendees, including machine learning engineers, data scientists, and data managers, will witness first-hand how Secludy's integration with Milvus empowers organizations to harness the power of LLMs securely and efficiently.
Artificial Intelligence for XMLDevelopment
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
Digital Marketing Trends in 2024 | Guide for Staying Ahead
https://www.wask.co/ebooks/digital-marketing-trends-in-2024
Feeling lost in the digital marketing whirlwind of 2024? Technology is changing, consumer habits are evolving, and staying ahead of the curve feels like a never-ending pursuit. This e-book is your compass. Dive into actionable insights to handle the complexities of modern marketing. From hyper-personalization to the power of user-generated content, learn how to build long-term relationships with your audience and unlock the secrets to success in the ever-shifting digital landscape.
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for life science domain, where you retriever information from biomedical knowledge graphs using LLMs to increase the accuracy and performance of generated answers
Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays, June 2024 with Maria Copot 20
June Patch Tuesday
Ivanti’s Patch Tuesday breakdown goes beyond patching your applications and brings you the intelligence and guidance needed to prioritize where to focus your attention first. Catch early analysis on our Ivanti blog, then join industry expert Chris Goettl for the Patch Tuesday Webinar Event. There we’ll do a deep dive into each of the bulletins and give guidance on the risks associated with the newly-identified vulnerabilities.
Choosing The Best AWS Service For Your Website + API.pptx
Have you ever been confused by the myriad of choices offered by AWS for hosting a website or an API?
Lambda, Elastic Beanstalk, Lightsail, Amplify, S3 (and more!) can each host websites + APIs. But which one should we choose?
Which one is cheapest? Which one is fastest? Which one will scale to meet our needs?
Join me in this session as we dive into each AWS hosting service to determine which one is best for your scenario and explain why!
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on integration of Salesforce with Bonterra Impact Management.
Interested in deploying an integration with Salesforce for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
Recently uploaded (20)
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Azure API Management to expose backend services securely
Azure API Management to expose backend services securely
Letter and Document Automation for Bonterra Impact Management (fka Social Sol...
Letter and Document Automation for Bonterra Impact Management (fka Social Sol...
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
System Design Case Study: Building a Scalable E-Commerce Platform - Hiike
System Design Case Study: Building a Scalable E-Commerce Platform - Hiike
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Digital Marketing Trends in 2024 | Guide for Staying Ahead
WeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation Techniques
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
An Overview of HDF-EOS (Part II)
- 1. An Overview of HDF-EOS (Part II) Doug Ilg Raytheon STX Doug.Ilg@gsfc.nasa.gov (301) 441-4089 1
- 2. Outline The Swath Interface What is an HDF-EOS File? 2
- 3. In str u me nt Pa th Swath Data Sets Instrument Profiles In str u m en t Pa th Instrument Scan Lines Along Track Along Track 3
- 4. A Swath Structure Dimension Name: Scan Size: 16 “Temp” “Time” Dimension Name: Track Size: 42 Dimension Name: Geotrack Size: 21 Map1 DataDimension: “Track” Geodimension: “Geotrack” Offset: 1 Increment: 2 4
- 5. Components of the Swath Interface Access Definition Basic I/O Inquiry Utility Subset 5
- 6. Tips on Writing a Swath Order of calls is significant: – define dimensions before using them to define fields of maps – setting a compression method affects all subsequently defined fields Unless a dimension map is defined, a 1to-1 mapping is assumed during subsetting. This can produce strange results. 6
- 7. Swath Subsetting Features By Time – SWdeftimeperiod/SWextractperiod By Geolocation – SWdefboxregion/SWextractregion By “Vertical” field – SWdefvrtregion/SWextractregion Tip: use Time or Geolocation first, then Vertical 7
- 8. Compression Methods for Swaths Run-Length Encoding Adaptive Huffman Gzip 8
- 9. Question: What’s an HDF-EOS File? 9
- 10. Some Answers ECS: An HDF file with ECS metadata Any file with an HDF-EOS object in it A file with only HDF-EOS objects in it 10
- 11. My Answer You’ve asked the wrong question. The difference between HDF and HDFEOS is at the object level, not the file level. 11
- 12. Key An HDF file HDF-EOS One or more HDF-EOS data objects with attendant structural metadata ECS Metadata Core and possibly Archive metadata HDF One or more HDF data objects 12
- 13. HDF •Portability •Elementary data access •Basic self-description HDF-EOS • Geolocation information tightly coupled to data • Fully standardized geographical subsetting • Moderate self-description ECS Metadata • ECS ingest, archiving, search, and distribution • Enhanced self-description 13
- 14. HDF An HDF file with a set of HDF data objects Services Provided •Portability •Elementary data access •Basic self-description Services Not Provided • Geolocation information tightly coupled to the data • Fully standardized geographical subsetting • ECS ingest, archiving, search, and distribution • Enhanced self-description 14
- 15. HDF-EOS An HDF file with a set of HDF-EOS data objects Services Provided • Portability • Elementary data access • Moderate self-description • Geolocation information tightly coupled to the data • Fully standardized geographical subsetting Services Not Provided • ECS ingest, archiving, search, and distribution • Enhanced self-description 15
- 16. Services Provided • Portability • Elementary data access HDF-EOS • Moderate selfdescription • Geolocation information HDF tightly coupled to some of the data • Partially standardized geographical subsetting An HDF file with a mixed Services Not Provided set of HDF-EOS and HDF • ECS ingest, archiving, data objects search, and distribution • Enhanced self-description • Fully standardized geographical subsetting 16
- 17. HDF ECS Metadata An HDF file with a set of HDF data objects with ECS metadata Services Provided • Portability • Elementary data access • ECS ingest, archiving, search, and distribution • Enhanced self-description Services Not Provided • Geolocation information tightly coupled to the data • Fully standardized geographical subsetting 17
- 18. HDF-EOS ECS Metadata An HDF file with a set of HDF-EOS data objects with ECS metadata Services Provided • Portability • Elementary data access • Geolocation information tightly coupled to the data • Fully standardized geographical subsetting • ECS ingest, archiving, search, and distribution • Enhanced self-description Services Not Provided • None 18
- 19. HDF-EOS HDF ECS Metadata An HDF file with a mixed set of HDF and HDF-EOS data objects with ECS metadata Services Provided • Portability • Elementary data access • Geolocation information tightly coupled to some of the data • Partially standardized geographical subsetting • ECS ingest, archiving, search, and distribution • Enhanced self-description Services Not Provided • Fully standardized geographical subsetting 19