ORNL DAAC MODIS Land Product Subsets
•Download as PPT, PDF•
1 like•838 views
MODIS (Moderate Resolution Imaging Spectroradiometer) sensor data are highly useful for field research. However, the volume of MODIS data and the complexity in data format makes MODIS data less usable for some communities. To expand the use of MODIS data beyond traditional remote sensing specialists, the Oak Ridge National Laboratory (ORNL) Distributed Active Archive Center (DAAC) prepares and distributes subsets of selected Land Products in a scale and format useful for undergraduate students and field researchers. MODIS subsets are provided for more than 1,000 sites across the globe. The subsets are offered in tabular ASCII format and in GIS compatible GeoTIFF format. Time series plots and grid visualizations to help characterize field sites are also provided. In addition to offering subsets for fixed sites, the ORNL DAAC also offers the capability to create user-defined subsets for any location worldwide. The MODIS Global subsetting tool provides subsets from a single pixel up to 201 x 201 km for user-defined time range. Statistics, time series plots and GIS compatible files for the customized subsets are also distributed through this tool. Users can also programmatically retrieve the subsets through a SOAP based Web Service.
Report
Share
Report
Share
Recommended
The Landsat 7 Processing System (LPS) Level Zero-R Science Products
The Landsat 7 Processing System (LPS) Level Zero-R Science ProductsThe HDF-EOS Tools and Information Center
The Landsat 7 Processing System (LPS) processes Landsat 7 data into science data products. It receives raw telemetry from the Landsat 7 satellite and constructs images from the data without applying corrections. The LPS divides the data into subintervals and produces multi-band files and metadata for each. It also generates browse images and assesses cloud cover for each subinterval. The LPS data products are separated into Format 1 and Format 2 files representing the two types of raw data received by the LPS. The Level Zero-R science products created by the LPS support further processing but are not yet calibrated or rectified.Indian Satellite Program(2001-2020)
Indian Satellite Program (2001-2020)
India has developed an extensive satellite program over the past two decades focused on earth observation, communication, space science, and navigation. Key satellites include resources satellites to monitor land and water resources, radar satellites like RISAT for all-weather imaging, Cartosat satellites for mapping applications, Oceansat for ocean and atmospheric monitoring, space science satellites like Chandrayaan and Astrosat, and navigation satellites. The program is led by ISRO and has supported applications in agriculture, disaster management, and development planning.
MODIS
MODIS is an instrument aboard the Terra and Aqua satellites that images the entire Earth every 1-2 days using 36 spectral bands. It provides data to study global land, ocean, and atmospheric processes. MODIS data products include measurements of sea surface temperature, snow cover, sea ice, vegetation indices, ocean color, and more. The data is available freely from NASA and USGS websites and can be used with GIS software for applications like monitoring wildfires, agriculture, water quality, and air pollution.
Introduction to HDFLook_MODIS
HDFLook is a software tool developed jointly by NASA GSFC and LOA USTL, France that allows users to view and analyze Earth science datasets. It provides capabilities to access, visualize, remap, reproject, subset, mosaic and convert files for MODIS, AIRS, and CERES products. HDFLook can be run in interactive, operational and batch modes and supports accessing geophysical and ancillary data for scientific analysis and operational uses. Over 3000 copies of HDFLook have been distributed worldwide to support the user community.
Applications of LIDAR Data
This document provides information on various LiDAR datasets available for Hawaii. It discusses bathymetric and terrain LiDAR datasets from sources such as the Army Corps of Engineers SHOALS program, NOAA, and the University of Hawaii. It also outlines some applications of LiDAR data such as vulnerability studies, feature extraction, and shoreline change analysis. Key contacts at the University of Hawaii Pacific GPS Facility are provided for additional information.
Polly use of remote sensing products for local water
This document summarizes the use of remote sensing techniques to monitor local water supply and use. It provides an overview of traditional monitoring methods and their limitations. Remote sensing allows for broader spatial and temporal coverage at a lower cost. Case studies are presented on using remote sensing to monitor urban irrigation, snowpack, and crop water use. Specific techniques discussed include using satellite imagery to map irrigated areas, generating snow water equivalent traces from SNODAS data, and estimating evapotranspiration using the METRIC model.
Hawaii LIDAR Datasets
This document provides an overview of LiDAR technology, including what it is, how it collects data, and common applications. It discusses the main types of LiDAR systems, how data is collected and stored, and applications like topographic mapping. It also covers data availability in Hawaii from sources like the USACE and USGS. The document aims to educate about LiDAR and provide resources for accessing LiDAR data.
Generalized EOS Data Converter: Making Data Products Accessible to GIS Tools
Generalized EOS Data Converter: Making Data Products Accessible to GIS ToolsThe HDF-EOS Tools and Information Center
The document describes a tool called the HDF-EOS to GeoTIFF conversion tool (HEG) that was developed to convert science data files in HDF-EOS format to GeoTIFF files. GeoTIFF files can be directly accessed by many GIS tools, unlike HDF-EOS files. HEG allows individual users to convert a wide variety of EOS data products, including MODIS, MISR, and ASTER data. It provides functionality like mosaicking, subsetting, reprojection, and metadata creation. HEG has both a graphical and command line interface and runs on Windows, Linux, Sun, and SGI platforms. It has been integrated into online NASA data archives to help make earth science data more accessible.Recommended
The Landsat 7 Processing System (LPS) Level Zero-R Science Products
The Landsat 7 Processing System (LPS) Level Zero-R Science ProductsThe HDF-EOS Tools and Information Center
The Landsat 7 Processing System (LPS) processes Landsat 7 data into science data products. It receives raw telemetry from the Landsat 7 satellite and constructs images from the data without applying corrections. The LPS divides the data into subintervals and produces multi-band files and metadata for each. It also generates browse images and assesses cloud cover for each subinterval. The LPS data products are separated into Format 1 and Format 2 files representing the two types of raw data received by the LPS. The Level Zero-R science products created by the LPS support further processing but are not yet calibrated or rectified.Indian Satellite Program(2001-2020)
Indian Satellite Program (2001-2020)
India has developed an extensive satellite program over the past two decades focused on earth observation, communication, space science, and navigation. Key satellites include resources satellites to monitor land and water resources, radar satellites like RISAT for all-weather imaging, Cartosat satellites for mapping applications, Oceansat for ocean and atmospheric monitoring, space science satellites like Chandrayaan and Astrosat, and navigation satellites. The program is led by ISRO and has supported applications in agriculture, disaster management, and development planning.
MODIS
MODIS is an instrument aboard the Terra and Aqua satellites that images the entire Earth every 1-2 days using 36 spectral bands. It provides data to study global land, ocean, and atmospheric processes. MODIS data products include measurements of sea surface temperature, snow cover, sea ice, vegetation indices, ocean color, and more. The data is available freely from NASA and USGS websites and can be used with GIS software for applications like monitoring wildfires, agriculture, water quality, and air pollution.
Introduction to HDFLook_MODIS
HDFLook is a software tool developed jointly by NASA GSFC and LOA USTL, France that allows users to view and analyze Earth science datasets. It provides capabilities to access, visualize, remap, reproject, subset, mosaic and convert files for MODIS, AIRS, and CERES products. HDFLook can be run in interactive, operational and batch modes and supports accessing geophysical and ancillary data for scientific analysis and operational uses. Over 3000 copies of HDFLook have been distributed worldwide to support the user community.
Applications of LIDAR Data
This document provides information on various LiDAR datasets available for Hawaii. It discusses bathymetric and terrain LiDAR datasets from sources such as the Army Corps of Engineers SHOALS program, NOAA, and the University of Hawaii. It also outlines some applications of LiDAR data such as vulnerability studies, feature extraction, and shoreline change analysis. Key contacts at the University of Hawaii Pacific GPS Facility are provided for additional information.
Polly use of remote sensing products for local water
This document summarizes the use of remote sensing techniques to monitor local water supply and use. It provides an overview of traditional monitoring methods and their limitations. Remote sensing allows for broader spatial and temporal coverage at a lower cost. Case studies are presented on using remote sensing to monitor urban irrigation, snowpack, and crop water use. Specific techniques discussed include using satellite imagery to map irrigated areas, generating snow water equivalent traces from SNODAS data, and estimating evapotranspiration using the METRIC model.
Hawaii LIDAR Datasets
This document provides an overview of LiDAR technology, including what it is, how it collects data, and common applications. It discusses the main types of LiDAR systems, how data is collected and stored, and applications like topographic mapping. It also covers data availability in Hawaii from sources like the USACE and USGS. The document aims to educate about LiDAR and provide resources for accessing LiDAR data.
Generalized EOS Data Converter: Making Data Products Accessible to GIS Tools
Generalized EOS Data Converter: Making Data Products Accessible to GIS ToolsThe HDF-EOS Tools and Information Center
The document describes a tool called the HDF-EOS to GeoTIFF conversion tool (HEG) that was developed to convert science data files in HDF-EOS format to GeoTIFF files. GeoTIFF files can be directly accessed by many GIS tools, unlike HDF-EOS files. HEG allows individual users to convert a wide variety of EOS data products, including MODIS, MISR, and ASTER data. It provides functionality like mosaicking, subsetting, reprojection, and metadata creation. HEG has both a graphical and command line interface and runs on Windows, Linux, Sun, and SGI platforms. It has been integrated into online NASA data archives to help make earth science data more accessible.EOSDIS Status
The document provides an overview of the status of EOSDIS (Earth Observing System Data and Information System) as of November 4, 2009. It discusses the distributed nature of EOSDIS with multiple DAACs (Distributed Active Archive Centers) and SCFs/SIPSs (Science Computing Facilities/Science Investigator-led Processing Systems). It also includes timelines showing the availability of data from past and current Earth science missions archived by EOSDIS. Metrics on data distribution and a breakdown of the top distributed data products are presented. Plans for continued evolution of the EOSDIS infrastructure through contracts and grant funding are outlined.
EOSDIS Status
This document provides an overview of the Earth Observing System Data and Information System (EOSDIS) including its components, metrics, trends in data access, and plans for the upcoming year. Key points include that EOSDIS is a distributed system consisting of data centers and processing systems that have archived over 4 petabytes of data and distributed over 254 million data products in fiscal year 2009. Popular data products distributed include MODIS and AIRS data. Plans for the upcoming year focus on improving data access and search capabilities.
FR1.L09.2 - ONBOARD RADAR PROCESSING CONCEPTS FOR THE DESDYNI MISSION
The document discusses onboard radar processing concepts for the DESDynI Earth observation mission. It describes how onboard processing can enable more frequent observations than currently feasible by reducing the volume of downlinked data through techniques like SAR image formation and compression. Onboard processing could generate targeted science products to enable rapid response to natural hazards. Examples of potential onboard products discussed include forest fire extent, forest fuel load, earthquake damage assessment, glacier melting, and vegetation classification.
MODIS (Moderate Resolution Imaging Spectrometer)
This slide provides a detail overview of MODIS sensors and it also mentions about the MODIS receiving station at Nepal.
TU2.T10.1.pptx
NOAA is transitioning SAR-derived sea surface wind products to operational status to provide high-resolution coastal wind data to users. The system ingests SAR data from various satellites, retrieves winds using geophysical models, and distributes products through CoastWatch. Validation shows accuracy of 1-2.5 m/s compared to buoy winds. Operational implementation began in 2009 and will be complete in 2012 to handle future SAR missions like Sentinel-1 and provide coastal wind information to users.
TU3.L09 - SPACEBORNE FULLY POLARIMETRIC TIME-SERIES DATASETS FOR LAND COVER A...
The document summarizes a project using spaceborne fully polarimetric time-series datasets from Radarsat-2 and ALOS for land cover analysis and monitoring of wetlands. The objectives were to delineate wetlands and map vegetation, identify agricultural practices, and determine water cycles and levels. Ground truth campaigns were conducted including soil moisture and roughness measurements. Initial qualitative analysis of the Radarsat-2 data showed polarimetric descriptors and classifications could distinguish flooded areas with 78-94% accuracy. Optical data from ALOS provided complementary information and 88-99% classification accuracy. Further inversion of parameters like soil moisture from time-series datasets and data fusion of radar and optical data was outlined.
ENVI/IDL for HDF
ENVI and IDL software support HDF and HDF-EOS. Capabilities and the HDF tools built on ENVI and IDL will be reviewed. The current development will be discussed and demonstrated.
The CFD General Notation System transition to HDF5
The document discusses the transition of the CFD General Notation System (CGNS) to using HDF5 as its main storage format instead of ADF. CGNS provides a standard for storing computational fluid dynamics (CFD) simulation data. It is switching to HDF5 to take advantage of HDF5's capabilities like parallel I/O and availability in many tools, though initial HDF5 implementations have larger file sizes and slower I/O performance than ADF. The CGNS steering committee is evaluating the HDF5 implementation and investigating performance problems to further improve the transition.
Profile of HDF-EOS5 Files
HDF-EOS is a software library designed to support NASA Earth Observing System (EOS) science data. HDF is the Hierarchical Data Format developed by The HDF Group. Specific data structures in HDF-EOS5 which are containers for science data are: Grid, Point, Zonal Average and Swath. These data structures are constructed from standard HDF5 data objects, using EOS conventions, through the use of a software library. This presentation is intended to familiarize current HDF-EOS users with the structure of HDF-EOS5 files and the Grid, Swath, Point and Zonal Average structures used in these files.
HDFView and HDF Java Products
This tutorial will introduce the three levels of the HDF-Java products: the HDF-Java wrapper (or Java Native Interfaces to the standard HDF libraries), the HDF-Java object package, and the HDFView. The Java wrapper provides standard Java APIs that allow applications to call the C HDF libraries from Java. The HDF-Java object package implements HDF data objects, e.g. Groups and Datasets, in an object-oriented form and makes it easy for applications to use the libraries. The HDFView is a visual tool for browsing and editing HDF4 and HDF5 files.
Workshop Discussion: HDF & HDF-EOS Future Direction
This is a slide from
HDF AND HDF-EOS WORKSHOP V
February 26 - 28, 2002
Source: http://hdfeos.org/workshops/ws05/presentations/Ullman/11c-Discussion_notes.ppt
Support for NPP/NPOESS by The HDF Group
Accessibility and usability of NPP/NPOESS data in HDF5 can be enhanced by providing tools that simplify and standardize how data is accessed and presented. In this project, The HDF Group is creating such tools in the form of software to read and write certain key data types and data aggregates used in NPP/NPOESS data products, and extending HDFView to extract, present and export these data effectively. In particular, the work will focus on NPP/NPOESS use of HDF5 region references and quality flags. The HDF Group will also provide high quality user support for the project.
Shifting the Burden from the User to the Data Provider
As the volume and complexity of data from myriad Earth Observing platforms, both remote sensing and in-situ increases so does the demand for access to both data and information products from these data. The audience no longer is restricted to an investigator team with specialist science credentials. Non-specialist users from scientists from other disciplines, science-literate public, to teachers, to the general public and decision makers want access. What prevents them from this access to resources? It is the very complexity and specialist developed data formats, data set organizations and specialist terminology. What can be done in response? We must shift the burden from the user to the data provider. To achieve this our developed data infrastructures are likely to need greater degrees of internal code and data structure complexity to achieve (relatively) simpler end-user complexity. Evidence from numerous technical and consumer markets supports this scenario. We will cover the elements of modern data environments, what the new use cases are and how we can respond to them.
Status of HDF-EOS, Related Software, and Tools
Current status of HDF-EOS and access tools will be summarized. Update on HDF-EOS, HDFView plug-in and The HDF-EOS to GeoTIFF (HEG) conversion tool, including recent changes to the software, ongoing maintenance, upcoming releases, future plans, and issues will be discussed.
What will be new in HDF5?
Are you curious what is coming next? Are you willing to try some prototyped software? If so, come to this talk to learn about new HDF5 features such as HDF5 FORTARN2003 APIs, metadata journaling etc. and how your application can benefit from using them.
HDF Update
An update on HDF, including a status report on the HDF Group, an overview of recent changes to the HDF4 and HDF5 libraries and tools, plans for future releases, HDF Group projects and collaborations, and future plans.
Proposal for adding Named Dimensions to HDF5 Arrays
Data produced by the Ozone PEATE from the Ozone Mapping and Profiler Suite (OMPS) instruments are to be stored in HDF5, not HDF-EOS, but will still need some features similar to those in HDF-EOS. In particular, a mechanism for handling dimension names will be needed. This poster proposes a method to handle dimension names for arrays in HDF5 in a manner commensurate with HDF-EOS5.
HDF5 OPeNDAP project update and demo
In this talk, we will give an update on the HDF5 OPeNDAP project. We will update the new features inside OPeNDAP HDF5 data handler. We will also introduce a new HDF5-Friendly OPeNDAP client library and demonstrate how it can help users to view and analyze remote HDF-EOS5 data served by OPeNDAP HDF5 handler. A demo will be presented with a customized OPeNDAP visualization client (GrADS) that uses the library.
EOSDIS Status
The document provides an overview and status update of the Earth Observing System Data and Information System (EOSDIS). It discusses that EOSDIS supports EOS missions by ingesting, processing, archiving, and distributing their data. It notes that the volume of archived data has grown to over 4.9 petabytes containing over 2700 datasets. It also outlines plans to transition to new systems and complete updates to the EOSDIS code by 2009.
Profile of NPOESS HDF5 Files
The document provides an overview of National Polar-orbiting Operational Satellite System (NPOESS) HDF5 files. Key points include:
1) NPOESS is a satellite system that collects environmental data related to weather, atmosphere, oceans, land, and near-space. Data products are distributed in HDF5 format.
2) NPOESS HDF5 files contain raw data records, sensor data records, intermediate products, application related products, and environmental data records.
3) Data is organized into granules and aggregations. Granules contain a segment of data and are referenced by aggregations, which group granules over a temporal range.
HDF and HDF-EOS Experiences and Applications
Slide 13 discusses interesting HDF-EOS issues.
Source: http://hdfeos.org/workshops/ws05/presentations/Johnson/10c-HDF_at_Goddard_DAAC.ppt
Migrating from HDF5 1.6 to 1.8
The document discusses migrating from HDF5 1.6 to HDF5 1.8. It provides an overview of new features in HDF5 1.8, including a revised file format, improvements to group storage, new link types like external links, and enhanced error handling. The document recommends helping with the transition to HDF5 1.8 by discussing beneficial new features and awareness of compatibility issues when moving from 1.6 to 1.8.
More Related Content
What's hot
EOSDIS Status
The document provides an overview of the status of EOSDIS (Earth Observing System Data and Information System) as of November 4, 2009. It discusses the distributed nature of EOSDIS with multiple DAACs (Distributed Active Archive Centers) and SCFs/SIPSs (Science Computing Facilities/Science Investigator-led Processing Systems). It also includes timelines showing the availability of data from past and current Earth science missions archived by EOSDIS. Metrics on data distribution and a breakdown of the top distributed data products are presented. Plans for continued evolution of the EOSDIS infrastructure through contracts and grant funding are outlined.
EOSDIS Status
This document provides an overview of the Earth Observing System Data and Information System (EOSDIS) including its components, metrics, trends in data access, and plans for the upcoming year. Key points include that EOSDIS is a distributed system consisting of data centers and processing systems that have archived over 4 petabytes of data and distributed over 254 million data products in fiscal year 2009. Popular data products distributed include MODIS and AIRS data. Plans for the upcoming year focus on improving data access and search capabilities.
FR1.L09.2 - ONBOARD RADAR PROCESSING CONCEPTS FOR THE DESDYNI MISSION
The document discusses onboard radar processing concepts for the DESDynI Earth observation mission. It describes how onboard processing can enable more frequent observations than currently feasible by reducing the volume of downlinked data through techniques like SAR image formation and compression. Onboard processing could generate targeted science products to enable rapid response to natural hazards. Examples of potential onboard products discussed include forest fire extent, forest fuel load, earthquake damage assessment, glacier melting, and vegetation classification.
MODIS (Moderate Resolution Imaging Spectrometer)
This slide provides a detail overview of MODIS sensors and it also mentions about the MODIS receiving station at Nepal.
TU2.T10.1.pptx
NOAA is transitioning SAR-derived sea surface wind products to operational status to provide high-resolution coastal wind data to users. The system ingests SAR data from various satellites, retrieves winds using geophysical models, and distributes products through CoastWatch. Validation shows accuracy of 1-2.5 m/s compared to buoy winds. Operational implementation began in 2009 and will be complete in 2012 to handle future SAR missions like Sentinel-1 and provide coastal wind information to users.
TU3.L09 - SPACEBORNE FULLY POLARIMETRIC TIME-SERIES DATASETS FOR LAND COVER A...
The document summarizes a project using spaceborne fully polarimetric time-series datasets from Radarsat-2 and ALOS for land cover analysis and monitoring of wetlands. The objectives were to delineate wetlands and map vegetation, identify agricultural practices, and determine water cycles and levels. Ground truth campaigns were conducted including soil moisture and roughness measurements. Initial qualitative analysis of the Radarsat-2 data showed polarimetric descriptors and classifications could distinguish flooded areas with 78-94% accuracy. Optical data from ALOS provided complementary information and 88-99% classification accuracy. Further inversion of parameters like soil moisture from time-series datasets and data fusion of radar and optical data was outlined.
What's hot (6)
FR1.L09.2 - ONBOARD RADAR PROCESSING CONCEPTS FOR THE DESDYNI MISSION
FR1.L09.2 - ONBOARD RADAR PROCESSING CONCEPTS FOR THE DESDYNI MISSION
TU3.L09 - SPACEBORNE FULLY POLARIMETRIC TIME-SERIES DATASETS FOR LAND COVER A...
TU3.L09 - SPACEBORNE FULLY POLARIMETRIC TIME-SERIES DATASETS FOR LAND COVER A...
Viewers also liked
ENVI/IDL for HDF
ENVI and IDL software support HDF and HDF-EOS. Capabilities and the HDF tools built on ENVI and IDL will be reviewed. The current development will be discussed and demonstrated.
The CFD General Notation System transition to HDF5
The document discusses the transition of the CFD General Notation System (CGNS) to using HDF5 as its main storage format instead of ADF. CGNS provides a standard for storing computational fluid dynamics (CFD) simulation data. It is switching to HDF5 to take advantage of HDF5's capabilities like parallel I/O and availability in many tools, though initial HDF5 implementations have larger file sizes and slower I/O performance than ADF. The CGNS steering committee is evaluating the HDF5 implementation and investigating performance problems to further improve the transition.
Profile of HDF-EOS5 Files
HDF-EOS is a software library designed to support NASA Earth Observing System (EOS) science data. HDF is the Hierarchical Data Format developed by The HDF Group. Specific data structures in HDF-EOS5 which are containers for science data are: Grid, Point, Zonal Average and Swath. These data structures are constructed from standard HDF5 data objects, using EOS conventions, through the use of a software library. This presentation is intended to familiarize current HDF-EOS users with the structure of HDF-EOS5 files and the Grid, Swath, Point and Zonal Average structures used in these files.
HDFView and HDF Java Products
This tutorial will introduce the three levels of the HDF-Java products: the HDF-Java wrapper (or Java Native Interfaces to the standard HDF libraries), the HDF-Java object package, and the HDFView. The Java wrapper provides standard Java APIs that allow applications to call the C HDF libraries from Java. The HDF-Java object package implements HDF data objects, e.g. Groups and Datasets, in an object-oriented form and makes it easy for applications to use the libraries. The HDFView is a visual tool for browsing and editing HDF4 and HDF5 files.
Workshop Discussion: HDF & HDF-EOS Future Direction
This is a slide from
HDF AND HDF-EOS WORKSHOP V
February 26 - 28, 2002
Source: http://hdfeos.org/workshops/ws05/presentations/Ullman/11c-Discussion_notes.ppt
Support for NPP/NPOESS by The HDF Group
Accessibility and usability of NPP/NPOESS data in HDF5 can be enhanced by providing tools that simplify and standardize how data is accessed and presented. In this project, The HDF Group is creating such tools in the form of software to read and write certain key data types and data aggregates used in NPP/NPOESS data products, and extending HDFView to extract, present and export these data effectively. In particular, the work will focus on NPP/NPOESS use of HDF5 region references and quality flags. The HDF Group will also provide high quality user support for the project.
Shifting the Burden from the User to the Data Provider
As the volume and complexity of data from myriad Earth Observing platforms, both remote sensing and in-situ increases so does the demand for access to both data and information products from these data. The audience no longer is restricted to an investigator team with specialist science credentials. Non-specialist users from scientists from other disciplines, science-literate public, to teachers, to the general public and decision makers want access. What prevents them from this access to resources? It is the very complexity and specialist developed data formats, data set organizations and specialist terminology. What can be done in response? We must shift the burden from the user to the data provider. To achieve this our developed data infrastructures are likely to need greater degrees of internal code and data structure complexity to achieve (relatively) simpler end-user complexity. Evidence from numerous technical and consumer markets supports this scenario. We will cover the elements of modern data environments, what the new use cases are and how we can respond to them.
Status of HDF-EOS, Related Software, and Tools
Current status of HDF-EOS and access tools will be summarized. Update on HDF-EOS, HDFView plug-in and The HDF-EOS to GeoTIFF (HEG) conversion tool, including recent changes to the software, ongoing maintenance, upcoming releases, future plans, and issues will be discussed.
What will be new in HDF5?
Are you curious what is coming next? Are you willing to try some prototyped software? If so, come to this talk to learn about new HDF5 features such as HDF5 FORTARN2003 APIs, metadata journaling etc. and how your application can benefit from using them.
HDF Update
An update on HDF, including a status report on the HDF Group, an overview of recent changes to the HDF4 and HDF5 libraries and tools, plans for future releases, HDF Group projects and collaborations, and future plans.
Proposal for adding Named Dimensions to HDF5 Arrays
Data produced by the Ozone PEATE from the Ozone Mapping and Profiler Suite (OMPS) instruments are to be stored in HDF5, not HDF-EOS, but will still need some features similar to those in HDF-EOS. In particular, a mechanism for handling dimension names will be needed. This poster proposes a method to handle dimension names for arrays in HDF5 in a manner commensurate with HDF-EOS5.
HDF5 OPeNDAP project update and demo
In this talk, we will give an update on the HDF5 OPeNDAP project. We will update the new features inside OPeNDAP HDF5 data handler. We will also introduce a new HDF5-Friendly OPeNDAP client library and demonstrate how it can help users to view and analyze remote HDF-EOS5 data served by OPeNDAP HDF5 handler. A demo will be presented with a customized OPeNDAP visualization client (GrADS) that uses the library.
EOSDIS Status
The document provides an overview and status update of the Earth Observing System Data and Information System (EOSDIS). It discusses that EOSDIS supports EOS missions by ingesting, processing, archiving, and distributing their data. It notes that the volume of archived data has grown to over 4.9 petabytes containing over 2700 datasets. It also outlines plans to transition to new systems and complete updates to the EOSDIS code by 2009.
Profile of NPOESS HDF5 Files
The document provides an overview of National Polar-orbiting Operational Satellite System (NPOESS) HDF5 files. Key points include:
1) NPOESS is a satellite system that collects environmental data related to weather, atmosphere, oceans, land, and near-space. Data products are distributed in HDF5 format.
2) NPOESS HDF5 files contain raw data records, sensor data records, intermediate products, application related products, and environmental data records.
3) Data is organized into granules and aggregations. Granules contain a segment of data and are referenced by aggregations, which group granules over a temporal range.
HDF and HDF-EOS Experiences and Applications
Slide 13 discusses interesting HDF-EOS issues.
Source: http://hdfeos.org/workshops/ws05/presentations/Johnson/10c-HDF_at_Goddard_DAAC.ppt
Migrating from HDF5 1.6 to 1.8
The document discusses migrating from HDF5 1.6 to HDF5 1.8. It provides an overview of new features in HDF5 1.8, including a revised file format, improvements to group storage, new link types like external links, and enhanced error handling. The document recommends helping with the transition to HDF5 1.8 by discussing beneficial new features and awareness of compatibility issues when moving from 1.6 to 1.8.
The MATLAB Low-Level HDF5 Interface
HDF5 is a powerful and feature-rich creature, and getting the most out of it requires powerful tools. The MathWorks provides a "low-level" interface to the HDF5 library that closely corresponds to the C API and exposes much of its richness. This short tutorial will present ways to use the low-level MATLAB interface to build those tools and tackle such topics as subsetting, chunking, and compression.
Reading HDF family of formats via NetCDF-Java / CDM
NetCDF-Java is an open source Java library for reading scientific data formats like NetCDF, HDF5, HDF4, and OPeNDAP. It has been used as a component in many software projects. The library provides an object-oriented API for reading data from these file formats and exposing it to Java programs. It works by providing format readers for specific file types that can read data into the Common Data Model used by the library. The library has been tested against many file examples but could benefit from more systematic testing. Proper use of dimensions, variables, units, and metadata is important for self-documenting scientific data files.
Using HDF5 Archive Information Package to preserve HDF-EOS2 data
Using HDF5 Archive Information Package to preserve HDF-EOS2 dataThe HDF-EOS Tools and Information Center
This one-year research project, funded by NOAA Climate Program Office (CPO) Scientific Data Stewardship (SDS), provides a solution to migrate data to a single standards-based archive format. Specifically, we investigate on how to store NASA ECS data and metadata into HDF5 Archival Information Packages (AIP). To achieve this, the HDF4 to HDF5 conversion tool has been enhanced so that converted ECS data can be read through the NetCDF4/CDM interface. In addition, metadata tools will be developed that convert ECS collection and granule level metadata to NOAA's collection level and NARA's METS standard. The enhanced HDF4 to HDF5 conversion tool has been released in May 2008 and it includes new functionality as the converted ECS data can be read through the NetCDF4 interface. We have tested 33 typical HDF-EOS2 swath, grid and point products at the National Snow and Ice Data Center (NSIDC). We also demonstrate the initial effort of the work to develop METS compliant metadata from granule metadata held in NASA's Earth Observing System (EOS) Data and Information System (EOSDIS) Core System (ECS). Advanced HDF5 Features
This tutorial is designed for the HDF5 users with some HDF5 experience.
It will cover advanced features of the HDF5 library for achieving better I/O performance and efficient storage. The following HDF5 features will be discussed: partial I/O, chunked storage layout, compression and other filters including new n-bit and scale+offset filters. Significant time will be devoted to the discussion of complex HDF5 datatypes such as strings, variable-length datatypes, array and compound datatypes.
Viewers also liked (20)
The CFD General Notation System transition to HDF5
The CFD General Notation System transition to HDF5
Workshop Discussion: HDF & HDF-EOS Future Direction
Workshop Discussion: HDF & HDF-EOS Future Direction
Shifting the Burden from the User to the Data Provider
Shifting the Burden from the User to the Data Provider
Proposal for adding Named Dimensions to HDF5 Arrays
Proposal for adding Named Dimensions to HDF5 Arrays
Reading HDF family of formats via NetCDF-Java / CDM
Reading HDF family of formats via NetCDF-Java / CDM
Using HDF5 Archive Information Package to preserve HDF-EOS2 data
Using HDF5 Archive Information Package to preserve HDF-EOS2 data
Similar to ORNL DAAC MODIS Land Product Subsets
Increasing Access to MODIS data through Web services and visualization tools
Remote sensing –Beyond images
Mexico 14-15 December 2013
The workshop was organized by CIMMYT Global Conservation Agriculture Program (GCAP) and funded by the Bill & Melinda Gates Foundation (BMGF), the Mexican Secretariat of Agriculture, Livestock, Rural Development, Fisheries and Food (SAGARPA), the International Maize and Wheat Improvement Center (CIMMYT), CGIAR Research Program on Maize, the Cereal System Initiative for South Asia (CSISA) and the Sustainable Modernization of the Traditional Agriculture (MasAgro)
WE1.L10 - TERRA, AQUA, AND AURA DIRECT BROADCAST – PROVIDING EARTH SCIENCE DA...
This document provides an overview of the Direct Broadcast (DB) capabilities of NASA's Terra, Aqua, and Aura Earth observing satellites. DB allows near real-time transmission of science data to ground stations around the world. Over 200 stations worldwide receive DB data, which is used for applications like weather forecasting, disaster monitoring, and fisheries. Examples of DB data users include government agencies, universities, and private companies using the data for purposes like wildfire detection, ozone monitoring, and natural resource management.
2017 ASPRS-RMR Big Data Track: Using NASA's AppEEARS to Slice and Dice Big Ea...
This document summarizes the AppEEARS tool, which was developed by NASA's LP DAAC to allow users to easily subset, reformat, and analyze large Earth observation datasets. AppEEARS provides interactive subsetting of spatial, temporal, and variable subsets of datasets. It outputs the subsets in common file formats like GeoTIFF and NetCDF while maintaining metadata and provenance. The document describes several use cases where researchers were able to efficiently extract relevant data for studies on vegetation productivity, population changes, snow zones, and wildfire impacts using AppEEARS. It highlights how the tool eliminates much of the data processing workload and enables more focus on analysis.
HDF & HDF-EOS Data & Support at NSIDC
The document discusses the National Snow and Ice Data Center's (NSIDC) use of HDF and HDF-EOS file formats to manage and distribute scientific cryosphere data. NSIDC collects data from satellites like MODIS, AMSR-E and aircraft missions, processes the data, and makes it available in HDF(5) formats. The HDF formats allow for efficient storage of multi-dimensional scientific datasets along with metadata. NSIDC develops tools to allow users to access, analyze and visualize data stored in HDF files.
MODIS Land and HDF-EOS
The document discusses MODIS Land products and their distribution format. MODIS Land products include radiation budget, ecosystem, and land cover variables. Products are distributed in HDF-EOS format with fine resolution grids in Integerized Sinusoidal or Lambert Azimuthal Equal-Area projections and coarse grids in a geographic Climate Modeling Grid. HDF-EOS allows for collaboration and standard geolocation representation but toolkit support is still limited.
HDF-EOS Development - Current Status and Schedule
The document discusses the current status and schedule of HDF-EOS development. HDF-EOS 2 is the current storage format for EOS standard products, using HDF4, and is supported by NASA. HDF-EOS 5 is a rewrite using HDF5 that is designed to resemble HDF-EOS 2. It will be used by the EOS Aura mission. The document outlines current archive holdings using HDF-EOS 2 and functionality included in HDF-EOS 5.
Scheduled Scientific Data Releases Using .backup Volumes
How the Mars Space Flight Facility uses (and abuses) the .backup snapshot feature of OpenAFS.
This presentation was given by Chris Kurtz and myself at the 2008 AFS and Kerberos Best Practices Workshop in Newark, NJ.
GES DISC Eexperiences with HDF Formats for MEaSUREs Projects
GES DISC Eexperiences with HDF Formats for MEaSUREs ProjectsThe HDF-EOS Tools and Information Center
The document discusses GES DISC's experience supporting 7 MEaSUREs projects, which create long-term Earth science data records. It recommends file formats like HDF-EOS5 and netCDF4 to promote consistency and interoperability. While HDF5 was initially recommended, ad-hoc implementations have caused issues. The document provides information on GES DISC's recommendations for file naming, formats and metadata, as well as problems encountered and resources for learning more.HDF-EOS Aura File Format Guidelines
The document presents guidelines for standardizing the HDF-EOS file format for the Aura satellite mission. It proposes organizing data fields and attributes in a consistent way, using standard dimension and field names, and reporting data on a common pressure grid. This will make files easier to understand, cross-platform compatible, and simplify software development. The guidelines cover topics such as data ordering, units, and attribute names. Standardizing these aspects aims to reduce mismatches between instruments. [/SUMMARY]
Fundamentals aquatic-web
Utilizando la información espectral y visión panorámica mediante la interpretación de imágenes satelitales se puede conocer la superficie, forma y distribución de la cobertura de agua y vomo la arcatan las alteraciones en las componentes vegetal y uso de la tierra de grandes áreas. En el trabajo específico con vegetación son empleadas para describir grandes tipos de comunidades (incluso determinar presencia de especies invasoras), estimar su estado hídrico, fenología, niveles de degradación y tasas de productividad. Basado en imágenes de satélites, en el año 2002 se presentó en Argentina el Primer Inventario de Bosques Nativos, que tuvo como objetivo obtener mapas temáticos de cobertura de uso de la tierra, del estado de los bosques (niveles de aprovechamiento y degradación), indicadores de factores ambientales responsables de la alteración de los recursos hidricos
NPOESS Program Overview
This document provides an overview of the NPOESS Program. NPOESS is a tri-agency program between NOAA, NASA, and the Department of Defense to develop the next generation of US polar-orbiting environmental satellites. The goal is to converge the DoD and NOAA satellite programs to achieve cost savings while incorporating new technologies. NPOESS will provide global environmental data for weather forecasting, climate monitoring, and other applications. The first NPOESS satellite is scheduled for launch in 2013 and the system is expected to operate through 2026. The NPP satellite launching in 2011 will help reduce risks for NPOESS.
Archive Information Packages for NASA HDF-EOS Data
This presentation discusses the development of Archival Information Packages (AIPs) for NASA HDF-EOS data. It outlines the components of an AIP according to the OAIS reference model. It then describes efforts to implement AIPs using standards like METS, PREMIS, ISO 19115 and HDF-5 to package HDF data files with associated metadata. The goals were to prototype AIPs at the data set and granule level and test the usability of digital library standards for geospatial data. Next steps involve further work applying these standards to create preservation-ready packages of NASA's HDF data holdings.
ESDIS Status (2002)
The document provides an overview and status update of the Earth Science Data and Information System (ESDIS). ESDIS has successfully supported numerous Earth science satellite missions and currently manages over 2 petabytes of science data. In fiscal year 2002, ESDIS delivered over 16 million data products to more than 1.8 million users. ESDIS is working to enhance its capabilities through initiatives like Data Pools and the EOS ClearingHOuse (ECHO) metadata broker. HDF-EOS 5 development is nearly complete and the workshop will discuss next steps for HDF-EOS tools and community adoption of HDF-EOS 5.
HDF-EOS Development Current Status
HDF-EOS 5 is a new format based on HDF5 that is designed to resemble the existing HDF-EOS 2 format. It supports the same data structures as HDF-EOS 2 but uses HDF5 functionality like compression and chunking. A conversion tool allows data to be changed from HDF-EOS 2 to HDF-EOS 5 format. While HDF-EOS 2 will still be supported, HDF-EOS 5 will be used by new missions like Aura and provides more options for data storage and filtering.
HDF5, HDF-EOS and Geospatial Data Archives
The National Archives and Records Administration (NARA) has a duty to preserve federal records, including a large and diverse volume of electronic records. NARA evaluated HDF5 and HDF-EOS as potential formats for archiving geospatial data given their abilities to represent many data models, accommodate large data volumes, and enable high performance access and visualization applications. HDF5 and HDF-EOS are established open standards that efficiently store and provide access to huge data files using flexible, extendable, and portable libraries and APIs. The project explored using HDF5 and HDF-EOS data types to store non-remote sensed geospatial data like surfaces, volumes, rasters, vectors, and tables. It also examined converting data from other formats and
AusCover
The AusCover Facility aims to produce nationally consistent satellite-based maps and remote sensing data validated for Australian conditions. It addresses key questions about environmental changes over time and the impacts of management programs. The facility will operate regional nodes in partnerships and engage users through workshops and an online portal. It will work closely with government agencies and provide remote sensing data, validated biophysical products, and ground instrumentation to establish national standards. Sample datasets will include time series maps of vegetation, land cover, and temperature as well as high resolution field measurements for validation.
Welcome to HDF Workshop V
This document summarizes the fifth annual HDF workshop sponsored by ESDIS and NCSA. It provides an overview of the status of ESDIS, HDF/HDF-EOS, and plans for the future. Over 750 terabytes of Terra and Landsat 7 data have been processed and made available. Some instruments like ASTER and CERES now have validated data while others like MODIS are still being reprocessed. Future plans include installing data pools at DAACs and procuring an EMD contract to support ongoing EOS operations. The community advisory process involves groups like UWGs, DAWG, and SWGD to provide feedback. HDF is a file format for scientific data while HDF-EOS is the
Evolving NASA’s Data and Information Systems for Earth Science
In this deck from the HPC User Forum, Rahul Ramachandran from NASA presents: Evolving NASA’s Data and Information Systems for Earth Science.
"NASA’s Earth Science Division (ESD) missions help us to understand our planet’s interconnected systems, from a global scale down to minute processes. Working in concert with a satellite network of international partners, ESD can measure precipitation around the world, and it can employ its own constellation of small satellites to look into the eye of a hurricane. ESD technology can track dust storms across continents and mosquito habitats across cities. ESD delivers the technology, expertise and global observations that help us to map the myriad connections between our planet’s vital processes and the effects of ongoing natural and human-caused changes."
Watch the video: https://wp.me/p3RLHQ-k8y
Learn more: https://science.nasa.gov/earth-science
and
http://hpcuserforum.com
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Cool Tools Esri ArcGIS
The document describes the Atmospheric Science Data Center (ASDC) and its pilot project to utilize Geographic Information Systems (GIS) to increase accessibility of earth science data. The ASDC archives over 2 petabytes of data from 42 science projects, distributing 590 terabytes in 2012 to over 142,000 customers. Its goals for the Esri pilot include overcoming challenges of using HDF and NetCDF files in GIS, strengthening partnerships, and facilitating easier discovery and greater use of ASDC data holdings through various portals. The pilot aims to allow ASDC to reach new user communities and enhance capabilities for current customers.
HDF-EOS Development: Current Status and Tools
This document provides a summary of the current status and schedule of HDF-EOS development. It describes what HDF-EOS is, its data structures like Grid, Swath, Point, and Zonal Average, and applications that can be used to access and process HDF-EOS data. HDF-EOS 5, based on HDF5, is now used by EOS Aura producers and users, while HDF-EOS 2 will continue to be supported.
Similar to ORNL DAAC MODIS Land Product Subsets (20)
Increasing Access to MODIS data through Web services and visualization tools
Increasing Access to MODIS data through Web services and visualization tools
WE1.L10 - TERRA, AQUA, AND AURA DIRECT BROADCAST – PROVIDING EARTH SCIENCE DA...
WE1.L10 - TERRA, AQUA, AND AURA DIRECT BROADCAST – PROVIDING EARTH SCIENCE DA...
2017 ASPRS-RMR Big Data Track: Using NASA's AppEEARS to Slice and Dice Big Ea...
2017 ASPRS-RMR Big Data Track: Using NASA's AppEEARS to Slice and Dice Big Ea...
Scheduled Scientific Data Releases Using .backup Volumes
Scheduled Scientific Data Releases Using .backup Volumes
GES DISC Eexperiences with HDF Formats for MEaSUREs Projects
GES DISC Eexperiences with HDF Formats for MEaSUREs Projects
Archive Information Packages for NASA HDF-EOS Data
Archive Information Packages for NASA HDF-EOS Data
Evolving NASA’s Data and Information Systems for Earth Science
Evolving NASA’s Data and Information Systems for Earth Science
More from The HDF-EOS Tools and Information Center
Cloud-Optimized HDF5 Files
This document discusses how to optimize HDF5 files for efficient access in cloud object stores. Key optimizations include using large dataset chunk sizes of 1-4 MiB, consolidating internal file metadata, and minimizing variable-length datatypes. The document recommends creating files with paged aggregation and storing file content information in the user block to enable fast discovery of file contents when stored in object stores.
Accessing HDF5 data in the cloud with HSDS
This document provides an overview of HSDS (Highly Scalable Data Service), which is a REST-based service that allows accessing HDF5 data stored in the cloud. It discusses how HSDS maps HDF5 objects like datasets and groups to individual cloud storage objects to optimize performance. The document also describes how HSDS was used to improve access performance for NASA ICESat-2 HDF5 data on AWS S3 by hyper-chunking datasets into larger chunks spanning multiple original HDF5 chunks. Benchmark results showed that accessing the data through HSDS provided over 2x faster performance than other methods like ROS3 or S3FS that directly access the cloud storage.
The State of HDF
This document summarizes the current status and focus of the HDF Group. It discusses that the HDF Group is located in Champaign, IL and is a non-profit organization focused on developing and maintaining HDF software and data formats. It provides an overview of recent HDF5, HDF4 and HDFView releases and notes areas of focus for software quality improvements, increased transparency, strengthening the community, and modernizing HDF products. It invites support and participation in upcoming user group meetings.
Highly Scalable Data Service (HSDS) Performance Features
This document provides an overview of HSDS (HDF Server and Data Service), which allows HDF5 files to be stored and accessed from the cloud. Key points include:
- HSDS maps HDF5 objects like datasets and groups to individual cloud storage objects for scalability and parallelism.
- Features include streaming support, fancy indexing for complex queries, and caching for improved performance.
- HSDS can be deployed on Docker, Kubernetes, or AWS Lambda depending on needs.
- Case studies show HSDS is used by organizations like NREL and NSF to make petabytes of scientific data publicly accessible in the cloud.
Creating Cloud-Optimized HDF5 Files
This document discusses creating cloud-optimized HDF5 files by rearranging internal structures for more efficient data access in cloud object stores. It describes cloud-native and cloud-optimized storage formats, with the latter involving storing the entire HDF5 file as a single object. The benefits of cloud-optimized HDF5 include fast scanning and using the HDF5 library. Key aspects covered include using optimal chunk sizes, compression, and minimizing variable-length datatypes.
HDF5 OPeNDAP Handler Updates, and Performance Discussion
This document discusses updates and performance improvements to the HDF5 OPeNDAP data handler. It provides a history of the handler since 2001 and describes recent updates including supporting DAP4, new data types, and NetCDF data models. A performance study showed that passing compressed HDF5 data through the handler without decompressing/recompressing led to speedups of around 17-30x by leveraging HDF5 direct I/O APIs. This allows outputting HDF5 files as NetCDF files much faster through the handler.
Hyrax: Serving Data from S3
This document provides instructions for using the Hyrax software to serve scientific data files stored on Amazon S3 using the OPeNDAP data access protocol. It describes how to generate ancillary metadata files called DMR++ files using the get_dmrpp tool that provide information about the data file structure and locations. The document explains how to run get_dmrpp inside a Docker container to process data files on S3 and generate customized DMR++ files that the Hyrax server can use to serve the files to clients.
Accessing Cloud Data and Services Using EDL, Pydap, MATLAB
This document provides an overview and examples of accessing cloud data and services using the Earthdata Login (EDL), Pydap, and MATLAB. It discusses some common problems users encounter, such as being unable to access HDF5 data on AWS S3 using MATLAB or read data from OPeNDAP servers using Pydap. Solutions presented include using EDL to get temporary AWS tokens for S3 access in MATLAB and providing code examples on the HDFEOS website to help users access S3 data and OPeNDAP services. The document also notes some limitations, such as tokens being valid for only 1 hour, and workarounds like requesting new tokens or using the MATLAB HDF5 API instead of the netCDF API.
HDF - Current status and Future Directions
The HDF5 Roadmap and New Features document outlines upcoming changes and improvements to the HDF5 library. Key points include:
- HDF5 1.13.x releases will include new features like selection I/O, the Onion VFD for versioned files, improved VFD SWMR for single-writer multiple-reader access, and subfiling for parallel I/O.
- The Virtual Object Layer allows customizing HDF5 object storage and introduces terminal and pass-through connectors.
- The Onion VFD stores versions of HDF5 files in a separate onion file for versioned access.
- VFD SWMR improves on legacy SWMR by implementing single-writer multiple-reader capabilities
HDFEOS.org User Analsys, Updates, and Future
This document discusses user analysis of the HDFEOS.org website and plans for future improvements. It finds that the majority of the site's 100 daily users are "quiet", not posting on forums or other interactive elements. The main user types are locators, who search for examples or data; mergers, who combine or mosaic datasets; and converters, who change file formats. The document outlines recent updates focused on these user types, like adding Python examples for subsetting and calculating latitude and longitude. It proposes future work on artificial intelligence/machine learning uses of HDF files and examples for processing HDF data in the cloud.
HDF - Current status and Future Directions
This document summarizes a presentation about the current status and future directions of the Hierarchical Data Format (HDF) software. It provides updates on recent HDF5 releases, development efforts including new compression methods and ways to access HDF5 data, and outreach resources. It concludes by inviting the audience to share wishes for future HDF development.
H5Coro: The Cloud-Optimized Read-Only Library
The document describes H5Coro, a new C++ library for reading HDF5 files from cloud storage. H5Coro was created to optimize HDF5 reading for cloud environments by minimizing I/O operations through caching and efficient HTTP requests. Performance tests showed H5Coro was 77-132x faster than the previous HDF5 library at reading HDF5 data from Amazon S3 for NASA's SlideRule project. H5Coro supports common HDF5 elements but does not support writing or some complex HDF5 data types and messages to focus on optimized read-only performance for time series data stored sequentially in memory.
MATLAB Modernization on HDF5 1.10
This document summarizes MathWorks' work to modernize MATLAB's support for HDF5. Key points include:
1) MATLAB now supports HDF5 1.10.7 features like single-writer/multiple-reader access and virtual datasets through new and updated low-level functions.
2) Performance benchmarks show some improvements but also regressions compared to the previous HDF5 version, and work continues to optimize code and support future versions.
3) There are compatibility considerations for Linux filter plugins, but interim solutions are provided until MathWorks can ship a single HDF5 version.
HDF for the Cloud - Serverless HDF
HSDS provides HDF as a service through a REST API that can scale across nodes. New releases will enable serverless operation using AWS Lambda or direct client access without a server. This allows HDF data to be accessed remotely without managing servers. HSDS stores each HDF object separately, making it compatible with cloud object storage. Performance on AWS Lambda is slower than a dedicated server but has no management overhead. Direct client access has better performance but limits collaboration between clients.
HDF5 <-> Zarr
HDF5 and Zarr are data formats that can be used to store and access scientific data. This presentation discusses approaches to translating between the two formats. It describes how HDF5 files were translated to the Zarr format by creating a separate Zarr store to hold HDF5 file chunks, and storing chunk location metadata. It also discusses an implementation that translates Zarr data to the HDF5 format by using a special chunking layout and storing chunk information in an HDF5 compound dataset. Limitations of the translations include lack of support for some HDF5 dataset properties in Zarr, and lack of support for some Zarr compression methods in the HDF5 implementation.
HDF for the Cloud - New HDF Server Features
The document discusses HDF for the cloud, including new features of the HDF Server and what's next. Key points:
- HDF Server uses a "sharded schema" that maps HDF5 objects to individual storage objects, allowing parallel access and updates without transferring entire files.
- Implementations include HSDS software that uses the sharded schema with an API and SDKs for different languages like h5pyd for Python.
- New features of HSDS 0.6 include support for POSIX, Azure, AWS Lambda, and role-based access control.
- Future work includes direct access to storage without a server intermediary for some use cases.
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3The HDF-EOS Tools and Information Center
This document compares different methods for accessing HDF and netCDF files stored on Amazon S3, including Apache Drill, THREDDS Data Server (TDS), and HDF5 Virtual File Driver (VFD). A benchmark test of accessing a 24GB HDF5/netCDF-4 file on S3 from Amazon EC2 found that TDS performed the best, responding within 2 minutes, while Apache Drill failed after 7 minutes. The document concludes that TDS 5.0 is the clear winner based on performance and support for role-based access control and HDF4 files, but the best solution depends on use case and software.STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...
STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...The HDF-EOS Tools and Information Center
This document discusses STARE-PODS, a proposal to NASA/ACCESS-19 to develop a scalable data store for earth science data using the SpatioTemporal Adaptive Resolution Encoding (STARE) indexing scheme. STARE allows diverse earth science data to be unified and indexed, enabling the data to be partitioned and stored in a Parallel Optimized Data Store (PODS) for efficient analysis. The HDF Virtual Object Layer and Virtual Data Set technologies can then provide interfaces to access the data in STARE-PODS in a familiar way. The goal is for STARE-PODS to organize diverse data for alignment and parallel/distributed storage and processing to enable integrative analysis at scale.HDF5 and Ecosystem: What Is New?
This document provides an overview and update on HDF5 and its ecosystem. Key points include:
- HDF5 1.12.0 was recently released with new features like the Virtual Object Layer and external references.
- The HDF5 library now supports accessing data in the cloud using connectors like S3 VFD and REST VOL without needing to modify applications.
- Projects like HDFql and H5CPP provide additional interfaces for querying and working with HDF5 files from languages like SQL, C++, and Python.
- The HDF5 community is moving development to GitHub and improving documentation resources on the HDF wiki site.
HDF5 Roadmap 2019-2020
This document summarizes new features in HDF5 1.12.0, including support for storing references to objects and attributes across files, new storage backends using a virtual object layer (VOL), and virtual file drivers (VFDs) for Amazon S3 and HDFS. It outlines the HDF5 roadmap for 2019-2022, which includes continued support for HDF5 1.8 and 1.10, and new features in future 1.12.x releases like querying, indexing, and provenance tracking.
More from The HDF-EOS Tools and Information Center (20)
Highly Scalable Data Service (HSDS) Performance Features
Highly Scalable Data Service (HSDS) Performance Features
HDF5 OPeNDAP Handler Updates, and Performance Discussion
HDF5 OPeNDAP Handler Updates, and Performance Discussion
Accessing Cloud Data and Services Using EDL, Pydap, MATLAB
Accessing Cloud Data and Services Using EDL, Pydap, MATLAB
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3
STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...
STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...
Recently uploaded
National Security Agency - NSA mobile device best practices
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
Taking AI to the Next Level in Manufacturing.pdf
Read Taking AI to the Next Level in Manufacturing to gain insights on AI adoption in the manufacturing industry, such as:
1. How quickly AI is being implemented in manufacturing.
2. Which barriers stand in the way of AI adoption.
3. How data quality and governance form the backbone of AI.
4. Organizational processes and structures that may inhibit effective AI adoption.
6. Ideas and approaches to help build your organization's AI strategy.
Webinar: Designing a schema for a Data Warehouse
Are you new to data warehouses (DWH)? Do you need to check whether your data warehouse follows the best practices for a good design? In both cases, this webinar is for you.
A data warehouse is a central relational database that contains all measurements about a business or an organisation. This data comes from a variety of heterogeneous data sources, which includes databases of any type that back the applications used by the company, data files exported by some applications, or APIs provided by internal or external services.
But designing a data warehouse correctly is a hard task, which requires gathering information about the business processes that need to be analysed in the first place. These processes must be translated into so-called star schemas, which means, denormalised databases where each table represents a dimension or facts.
We will discuss these topics:
- How to gather information about a business;
- Understanding dictionaries and how to identify business entities;
- Dimensions and facts;
- Setting a table granularity;
- Types of facts;
- Types of dimensions;
- Snowflakes and how to avoid them;
- Expanding existing dimensions and facts.
OpenID AuthZEN Interop Read Out - Authorization
During Identiverse 2024 and EIC 2024, members of the OpenID AuthZEN WG got together and demoed their authorization endpoints conforming to the AuthZEN API
How to Get CNIC Information System with Paksim Ga.pptx
Pakdata Cf is a groundbreaking system designed to streamline and facilitate access to CNIC information. This innovative platform leverages advanced technology to provide users with efficient and secure access to their CNIC details.
UiPath Test Automation using UiPath Test Suite series, part 6
Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI.
UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities.
Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes.
What will you get from this session?
1. Insights into integrating generative AI.
2. Understanding how this integration enhances test automation within the UiPath platform
3. Practical demonstrations
4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath
Topics covered:
What is generative AI
Test Automation with generative AI and Open AI.
UiPath integration with generative AI
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
A Mix Chart displays historical data of numbers in a graphical or tabular form. The Kalyan Rajdhani Mix Chart specifically shows the results of a sequence of numbers over different periods.
Artificial Intelligence for XMLDevelopment
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
Programming Foundation Models with DSPy - Meetup Slides
Prompting language models is hard, while programming language models is easy. In this talk, I will discuss the state-of-the-art framework DSPy for programming foundation models with its powerful optimizers and runtime constraint system.
Digital Marketing Trends in 2024 | Guide for Staying Ahead
https://www.wask.co/ebooks/digital-marketing-trends-in-2024
Feeling lost in the digital marketing whirlwind of 2024? Technology is changing, consumer habits are evolving, and staying ahead of the curve feels like a never-ending pursuit. This e-book is your compass. Dive into actionable insights to handle the complexities of modern marketing. From hyper-personalization to the power of user-generated content, learn how to build long-term relationships with your audience and unlock the secrets to success in the ever-shifting digital landscape.
Introduction of Cybersecurity with OSS at Code Europe 2024
I develop the Ruby programming language, RubyGems, and Bundler, which are package managers for Ruby. Today, I will introduce how to enhance the security of your application using open-source software (OSS) examples from Ruby and RubyGems.
The first topic is CVE (Common Vulnerabilities and Exposures). I have published CVEs many times. But what exactly is a CVE? I'll provide a basic understanding of CVEs and explain how to detect and handle vulnerabilities in OSS.
Next, let's discuss package managers. Package managers play a critical role in the OSS ecosystem. I'll explain how to manage library dependencies in your application.
I'll share insights into how the Ruby and RubyGems core team works to keep our ecosystem safe. By the end of this talk, you'll have a better understanding of how to safeguard your code.
Skybuffer SAM4U tool for SAP license adoption
Manage and optimize your license adoption and consumption with SAM4U, an SAP free customer software asset management tool.
SAM4U, an SAP complimentary software asset management tool for customers, delivers a detailed and well-structured overview of license inventory and usage with a user-friendly interface. We offer a hosted, cost-effective, and performance-optimized SAM4U setup in the Skybuffer Cloud environment. You retain ownership of the system and data, while we manage the ABAP 7.58 infrastructure, ensuring fixed Total Cost of Ownership (TCO) and exceptional services through the SAP Fiori interface.
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
ここ3000字までしか入らないけどタイトルの方がたくさん文字入ると思います。
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Discover the seamless integration of RPA (Robotic Process Automation), COMPOSER, and APM with AWS IDP enhanced with Slack notifications. Explore how these technologies converge to streamline workflows, optimize performance, and ensure secure access, all while leveraging the power of AWS IDP and real-time communication via Slack notifications.
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-und-domino-lizenzkostenreduzierung-in-der-welt-von-dlau/
DLAU und die Lizenzen nach dem CCB- und CCX-Modell sind für viele in der HCL-Community seit letztem Jahr ein heißes Thema. Als Notes- oder Domino-Kunde haben Sie vielleicht mit unerwartet hohen Benutzerzahlen und Lizenzgebühren zu kämpfen. Sie fragen sich vielleicht, wie diese neue Art der Lizenzierung funktioniert und welchen Nutzen sie Ihnen bringt. Vor allem wollen Sie sicherlich Ihr Budget einhalten und Kosten sparen, wo immer möglich. Das verstehen wir und wir möchten Ihnen dabei helfen!
Wir erklären Ihnen, wie Sie häufige Konfigurationsprobleme lösen können, die dazu führen können, dass mehr Benutzer gezählt werden als nötig, und wie Sie überflüssige oder ungenutzte Konten identifizieren und entfernen können, um Geld zu sparen. Es gibt auch einige Ansätze, die zu unnötigen Ausgaben führen können, z. B. wenn ein Personendokument anstelle eines Mail-Ins für geteilte Mailboxen verwendet wird. Wir zeigen Ihnen solche Fälle und deren Lösungen. Und natürlich erklären wir Ihnen das neue Lizenzmodell.
Nehmen Sie an diesem Webinar teil, bei dem HCL-Ambassador Marc Thomas und Gastredner Franz Walder Ihnen diese neue Welt näherbringen. Es vermittelt Ihnen die Tools und das Know-how, um den Überblick zu bewahren. Sie werden in der Lage sein, Ihre Kosten durch eine optimierte Domino-Konfiguration zu reduzieren und auch in Zukunft gering zu halten.
Diese Themen werden behandelt
- Reduzierung der Lizenzkosten durch Auffinden und Beheben von Fehlkonfigurationen und überflüssigen Konten
- Wie funktionieren CCB- und CCX-Lizenzen wirklich?
- Verstehen des DLAU-Tools und wie man es am besten nutzt
- Tipps für häufige Problembereiche, wie z. B. Team-Postfächer, Funktions-/Testbenutzer usw.
- Praxisbeispiele und Best Practices zum sofortigen Umsetzen
Generating privacy-protected synthetic data using Secludy and Milvus
During this demo, the founders of Secludy will demonstrate how their system utilizes Milvus to store and manipulate embeddings for generating privacy-protected synthetic data. Their approach not only maintains the confidentiality of the original data but also enhances the utility and scalability of LLMs under privacy constraints. Attendees, including machine learning engineers, data scientists, and data managers, will witness first-hand how Secludy's integration with Milvus empowers organizations to harness the power of LLMs securely and efficiently.
Recently uploaded (20)
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
ORNL DAAC MODIS Land Product Subsets
- 1. MODIS Land Product Subsets Suresh K. Santhana Vannan, Robert B. Cook, Bruce E. Wilson, Lisa M. Olsen HDF and HDF-EOS Workshop XII October 15 – October 17, 2008 ORNL DAAC Oak Ridge National Laboratory Distributed Active Archive Center Oak Ridge, Tennessee, USA
- 2. What is MODIS MODerate Resolution Imaging Spectroradiometer Hyper-Spectral sensor onboard two satellites (Terra and Aqua) Terra MODIS and Aqua MODIS are viewing the entire Earth's surface every 1 to 2 days, acquiring data in 36 spectral bands, or groups of wavelengths Data stored in HDF-EOS gridded file format Source : http://aqua.nasa.gov
- 3. MODIS is great! but …….. Users (Field researchers) have some inconveniences :
- 4. Volume
- 6. What ORNL-DAAC offers? User community requested that we prepare MODIS Land Products in an easy-to-use format and size To validate remote sensing products To characterize field sites
- 7. Tools Used • hdp dumpsds from The HDF group • Customized for our need • MODIS Reprojection tool from LPDAAC • GDAL (Geospatial Data Abstraction Library) • Minnesota Map server • Custom Perl code
- 8. MODIS Subsets for Selected sites MODIS Products offered at ORNL-DAAC Data Formats Offered ASCII – 7x7 km subsets GeoTIFF – 25x25 km subsets Surface Reflectance Surface Temperature Land Cover Vegetation Phenology NDVI / EVI LAI / fPAR Net Photosynthesis Annual NPP Albedo (calc) Reflectance – BRDF Adjusted
- 10. Subsetting and Visualization Tool for Any location on Globe Demo: http://daac.ornl.gov/modisglobal
- 12. Case Study: Deforestation in Amazon
- 13. 1975 2001
- 15. Dec 31 2010
- 16. Jan 1 2011
- 17. MODIS Subsetting Web Service f(x) Client ################ # ORNL DAAC MODIS Web service - Perl client ################# use SOAP::Lite @params= SOAP::Lite -> service(‘WSDL_URL') -> xmlschema('http://www.w3.org/2001/XMLSchema') -> getsubset(lat,lon,product,start_date,end_date,size); SOAP SOAP Server MOD11A2.A2003001.h26v05.005.2007264203857.Clear_sky_ MOD11A2.A2003001.h26v05.005.2007264203857.Clear_sky_ days,MOD11A2,A2003001,Lat37.7Lon110Samp1Line1,200726 days,MOD11A2,A2003001,Lat37.7Lon110Samp1Line1,200726 4203857,Clear_sky_days,NA,0.000,2 4203857,Clear_sky_days,NA,0.000,2 MOD11A2.A2003001.h26v05.005.2007264203857.Clear_sky_ MOD11A2.A2003001.h26v05.005.2007264203857.Clear_sky_ nights,MOD11A2,A2003001,Lat37.7Lon110Samp1Line1,20072 nights,MOD11A2,A2003001,Lat37.7Lon110Samp1Line1,20072 64203857,Clear_sky_nights,NA,0.000,254 64203857,Clear_sky_nights,NA,0.000,254 ……. …….
- 18. Scope Tool for scientific research (Environmental Impact assessment, carbon budgeting, etc.) Classroom education and projects for students Time series, trend analysis GIS overlay analysis Statistical analysis (Across many years and across different land cover types) Validation of MODIS products and other remote sensing products