HDF-EOS Development - Current Status and Schedule
•
0 likes•334 views
The document discusses the current status and schedule of HDF-EOS development. HDF-EOS 2 is the current storage format for EOS standard products, using HDF4, and is supported by NASA. HDF-EOS 5 is a rewrite using HDF5 that is designed to resemble HDF-EOS 2. It will be used by the EOS Aura mission. The document outlines current archive holdings using HDF-EOS 2 and functionality included in HDF-EOS 5.
Report
Share
Report
Share
Download to read offline
Recommended
HDF-EOS Development Current Status
HDF-EOS 5 is a new format based on HDF5 that is designed to resemble the existing HDF-EOS 2 format. It supports the same data structures as HDF-EOS 2 but uses HDF5 functionality like compression and chunking. A conversion tool allows data to be changed from HDF-EOS 2 to HDF-EOS 5 format. While HDF-EOS 2 will still be supported, HDF-EOS 5 will be used by new missions like Aura and provides more options for data storage and filtering.
HDF-EOS to GeoTIFF Conversion Tool & HDF-EOS Plug-in for HDFView
HDF-EOS to GeoTIFF Conversion Tool & HDF-EOS Plug-in for HDFViewThe HDF-EOS Tools and Information Center
HDF-EOS to GeoTIFF (HEG) conversion tool is a versatile tool that besides converting HDF-EOS to GeoTIFF it subsets, resamples, convert swath to grid and/or grids to different projections, subsamples, or does mosaicing. HDF-EOS plug-in is a library that extends HDFView, a visual tool for browsing and editing HDF4 and HDF5 files, for EOS applications. These tools will be discussed in detail and a demo will be presented.HDF-EOS Development: Current Status and Tools
This document provides a summary of the current status and schedule of HDF-EOS development. It describes what HDF-EOS is, its data structures like Grid, Swath, Point, and Zonal Average, and applications that can be used to access and process HDF-EOS data. HDF-EOS 5, based on HDF5, is now used by EOS Aura producers and users, while HDF-EOS 2 will continue to be supported.
What is HDF-EOS?
HDF-EOS defines standard profiles for using HDF to organize Earth observation data. It builds on HDF by adding geolocation information coupled to data for point, swath, and grid collection profiles. HDF-EOS is used to define over 200 products from instruments on Terra to be distributed through EOSDIS. It provides full geolocation support and metadata to allow search and access of Earth science data.
HDF Tools Tutorial
This tutorial is designed for anyone who needs to work with data stored in HDF and HDF5 files.
The first part of the tutorial will focus on the HDF5 utilities to display the contents of HDF5 files, to extract and to import data from and to HDF5 files, to compare two HDF5 files, and more. Participants will be guided through the hand-on examples and will learn about different tools options. New changes and advanced features will be covered in a separate session (Updates on HDF tools) on Wednesday.
The second part of tutorial includes a hands-on session to learn the HDF (4 & 5) Java browsing tool, HDFView. The tool and special plug-ins will be used to work with the existing HDF, HDF-EOS, and netCDF-4 files, and to create a new HDF5 file. The tutorial will cover basic features of HDFView.
HDF-EOS Status and Developments
This document summarizes the status and development of HDF-EOS software. It notes that maintenance support is funded through July 2008. It describes software releases from May 2005 to the planned December 2005 release. Major developments for HDF-EOS5 since October 2004 are also outlined, including performance improvements and added functionality. Future plans include additional platform support and performance optimizations. The HDF-EOS to GeoTIFF Converter (HEG) is also described, which allows conversion of HDF-EOS data to common GIS formats.
Advanced HDF5 Features
This document provides an overview and outline of topics related to advanced features in HDF5, including:
- HDF5 supports various datatypes like atomic, compound, array, and variable-length datatypes. It allows creation of complex user-defined datatypes.
- Partial I/O in HDF5 allows reading and writing subsets of datasets using hyperslab selections, which describe subsets through properties like start point, stride, count, and block size.
- Chunking and compression can be used to improve performance and reduce storage needs when working with subsets of large HDF5 datasets.
HDF-EOS Overview and Status
This document provides an overview of HDF-EOS, which is an extension to HDF that defines standard data structures for remote sensing and in-situ data with tightly coupled geolocation information. It describes the core components of HDF-EOS files, including Grid, Swath, and Point structures, and provides examples. It also outlines the development of an HDF5-based version to overcome limitations of the HDF4-based library and allow for larger files.
Recommended
HDF-EOS Development Current Status
HDF-EOS 5 is a new format based on HDF5 that is designed to resemble the existing HDF-EOS 2 format. It supports the same data structures as HDF-EOS 2 but uses HDF5 functionality like compression and chunking. A conversion tool allows data to be changed from HDF-EOS 2 to HDF-EOS 5 format. While HDF-EOS 2 will still be supported, HDF-EOS 5 will be used by new missions like Aura and provides more options for data storage and filtering.
HDF-EOS to GeoTIFF Conversion Tool & HDF-EOS Plug-in for HDFView
HDF-EOS to GeoTIFF Conversion Tool & HDF-EOS Plug-in for HDFViewThe HDF-EOS Tools and Information Center
HDF-EOS to GeoTIFF (HEG) conversion tool is a versatile tool that besides converting HDF-EOS to GeoTIFF it subsets, resamples, convert swath to grid and/or grids to different projections, subsamples, or does mosaicing. HDF-EOS plug-in is a library that extends HDFView, a visual tool for browsing and editing HDF4 and HDF5 files, for EOS applications. These tools will be discussed in detail and a demo will be presented.HDF-EOS Development: Current Status and Tools
This document provides a summary of the current status and schedule of HDF-EOS development. It describes what HDF-EOS is, its data structures like Grid, Swath, Point, and Zonal Average, and applications that can be used to access and process HDF-EOS data. HDF-EOS 5, based on HDF5, is now used by EOS Aura producers and users, while HDF-EOS 2 will continue to be supported.
What is HDF-EOS?
HDF-EOS defines standard profiles for using HDF to organize Earth observation data. It builds on HDF by adding geolocation information coupled to data for point, swath, and grid collection profiles. HDF-EOS is used to define over 200 products from instruments on Terra to be distributed through EOSDIS. It provides full geolocation support and metadata to allow search and access of Earth science data.
HDF Tools Tutorial
This tutorial is designed for anyone who needs to work with data stored in HDF and HDF5 files.
The first part of the tutorial will focus on the HDF5 utilities to display the contents of HDF5 files, to extract and to import data from and to HDF5 files, to compare two HDF5 files, and more. Participants will be guided through the hand-on examples and will learn about different tools options. New changes and advanced features will be covered in a separate session (Updates on HDF tools) on Wednesday.
The second part of tutorial includes a hands-on session to learn the HDF (4 & 5) Java browsing tool, HDFView. The tool and special plug-ins will be used to work with the existing HDF, HDF-EOS, and netCDF-4 files, and to create a new HDF5 file. The tutorial will cover basic features of HDFView.
HDF-EOS Status and Developments
This document summarizes the status and development of HDF-EOS software. It notes that maintenance support is funded through July 2008. It describes software releases from May 2005 to the planned December 2005 release. Major developments for HDF-EOS5 since October 2004 are also outlined, including performance improvements and added functionality. Future plans include additional platform support and performance optimizations. The HDF-EOS to GeoTIFF Converter (HEG) is also described, which allows conversion of HDF-EOS data to common GIS formats.
Advanced HDF5 Features
This document provides an overview and outline of topics related to advanced features in HDF5, including:
- HDF5 supports various datatypes like atomic, compound, array, and variable-length datatypes. It allows creation of complex user-defined datatypes.
- Partial I/O in HDF5 allows reading and writing subsets of datasets using hyperslab selections, which describe subsets through properties like start point, stride, count, and block size.
- Chunking and compression can be used to improve performance and reduce storage needs when working with subsets of large HDF5 datasets.
HDF-EOS Overview and Status
This document provides an overview of HDF-EOS, which is an extension to HDF that defines standard data structures for remote sensing and in-situ data with tightly coupled geolocation information. It describes the core components of HDF-EOS files, including Grid, Swath, and Point structures, and provides examples. It also outlines the development of an HDF5-based version to overcome limitations of the HDF4-based library and allow for larger files.
Status of HDF-EOS, Related Software and Tools
This document provides an overview of the status of HDF-EOS software and tools. It describes HDF-EOS5, a rewrite of HDF-EOS2 based on HDF5, which is used operationally by EOS instrument teams. The document also outlines software releases, major developments including bug fixes, and future plans, and provides contact information for support.
Parallel HDF5 Developments
The document discusses recent and upcoming improvements to parallel HDF5 for improved I/O performance on HPC systems. Recent improvements include reducing file truncations, distributing metadata writes across processes, and improved selection matching. Upcoming work includes a high-level HPC API, funding for Exascale-focused enhancements, and future improvements like asynchronous I/O and auto-tuning to parallel file systems. Performance tips are also provided like passing MPI hints and using collective I/O.
Easy Access of NASA HDF data via OPeNDAP
This slide will demonstrate how to use OPeNDAP Java clients such as IDV and Panoply via HDF OPeNDAP data handlers to access various NASA HDF products such as AIRS, OMI, MLS, MODIS, TRMM, CERES, SeaWIFS etc. Various features of these tools that can help users easy access the HDF data will also be explored.
Bridging ICESat and ICESat-2 Standard Data Products
This document discusses efforts to standardize data products from three NASA laser altimeter missions - ICESat, ICESat-2, and MABEL. It describes designing similar data products for all three missions to promote interoperability. Products are being developed for ICESat data using lessons from MABEL. Code is also being generated to help create products from specifications to reduce development time. The goal is to make the multi-rate point data from all three missions easily accessible and usable.
Easy Remote Access Via OPeNDAP
The document discusses updates to OPeNDAP handlers for HDF4 and HDF5 data formats. The HDF4 handler was improved to allow more NASA HDF and HDF-EOS data to be visualized by translating the data structure to follow climate and forecast metadata conventions. The HDF5 handler was updated to support additional HDF-EOS5 data from NASA missions. The handlers address issues that previously prevented data visualization and make the data more interoperable. Limitations include unsupported additional HDF4 objects and untested HDF4 products.
Subsetting at UAH
The presentation will provide an overview of subsetting software development activity at UAH. Updates have been made to all packages, reflecting the latest versions of HDF5 and HE5. The library of tools (HSE) for subsetting HDF-EOS data is up-to-date for SGI, Sun, and Linux platforms. Subsetting software is operational at NSIDC DAAC and GDAAC, in testing at LPDAAC. Ongoing work and plans will also be described, including row/column subsetting and index subsampling.
HDF OPeNDAP Project Update and Demo
The HDF Group gave a presentation on updates to the HDF-OPeNDAP project. They summarized the redesign of the HDF5 handler to support more NASA HDF5 data products, including grids, swaths, and zonal averages. They also discussed updates to the HDF4 handler to improve support for MODIS products and add new map projections. Live demonstrations showed example data from instruments like OMI, MLS, and Aquarius being served through the updated HDF handlers.
hadoop architecture -Big data hadoop
This document provides an overview of the Hadoop architecture, including its two main components: the Hadoop Distributed File System (HDFS) and the MapReduce engine. It describes HDFS' use of blocks and replication for reliability. It explains how files are written to HDFS and accessed via command line tools. It also outlines the different node types in Hadoop including the NameNode, DataNode, JobTracker and TaskTracker. Finally, it discusses topology awareness and how bandwidth decreases between processes on different nodes, racks and data centers.
NetCDF and HDF5
NetCDF and HDF5 are data formats and software libraries used for scientific data. NetCDF began in 1989 and allows for array-oriented data with dimensions, variables, and attributes. NetCDF-4 introduced new features while maintaining backward compatibility. It uses HDF5 for data storage and can read HDF4/HDF5 files. NetCDF provides APIs for C, Fortran, Java, and is widely used for earth science and climate data. It supports conventions, parallel I/O, and reading many data formats.
ESDIS Status (2002)
The document provides an overview and status update of the Earth Science Data and Information System (ESDIS). ESDIS has successfully supported numerous Earth science satellite missions and currently manages over 2 petabytes of science data. In fiscal year 2002, ESDIS delivered over 16 million data products to more than 1.8 million users. ESDIS is working to enhance its capabilities through initiatives like Data Pools and the EOS ClearingHOuse (ECHO) metadata broker. HDF-EOS 5 development is nearly complete and the workshop will discuss next steps for HDF-EOS tools and community adoption of HDF-EOS 5.
Hadoop 3.0 features
Hadoop 3.0.0 includes several new features including enhanced high availability for HDFS NameNode, Java 8 as the minimum runtime version, HDFS erasure coding for improved read and write performance, and improvements to YARN Timeline Service for better scalability and usability. The Hadoop shell scripts were also rewritten to unify behavior and address bugs.
EAS Data Flow lessons learnt
1. The document describes the data flow and file handling in a mock implementation of the Infrastructure Abstraction Layer (IAL) for the Euclid Archive System (EAS). It notes two key lessons learned: (1) file URLs need to be abstracted to reduce errors, and (2) reducing unnecessary file copies between systems.
2. It proposes a new Euclid File Access Service (EuFAS) that would provide a standardized way to look up, replicate, and manage file access across the EAS components through an abstract interface. This would replace direct file URLs and simplify data flow.
3. The service would also reduce unnecessary file copies by allowing processing to occur directly from cached copies on execution
Status of HDF-EOS, Related Software and Tools
Status of HDF-EOS and access tools will be summarized. Updates on HDF-EOS, TOOLKIT, HDFView plug-in and The HDF-EOS to GeoTIFF (HEG) conversion tool, including recent changes to the software, ongoing maintenance, upcoming releases, future plans, and issues will be discussed.
Introduction to HDF5
This tutorial is designed for new HDF5 users. We will cover basic HDF5 Data Model objects and their properties, give an overview of the HDF5 Libraries and APIs, and discuss the HDF5 programming model. Simple C and Fortran examples will be used to illustrate HDF5 concepts.
MODIS Land and HDF-EOS
The document discusses MODIS Land products and their distribution format. MODIS Land products include radiation budget, ecosystem, and land cover variables. Products are distributed in HDF-EOS format with fine resolution grids in Integerized Sinusoidal or Lambert Azimuthal Equal-Area projections and coarse grids in a geographic Climate Modeling Grid. HDF-EOS allows for collaboration and standard geolocation representation but toolkit support is still limited.
HDF Update for DAAC Managers (2017-02-27)
This document provides information about HDF (Hierarchical Data Format) tools and resources for working with Earth observation data. It summarizes HDF's focus on helping users at different stages of working with data, from initial product design to long-term archiving. It also describes specific HDF tools for viewing, comparing, converting between formats and adding metadata to scientific data files.
70a monitoring & troubleshooting
This document discusses monitoring and troubleshooting a Hadoop cluster running MapR. It outlines tools for cluster monitoring including the MapR Control System and MapR Metrics. Common troubleshooting scenarios like slow nodes, out of memory errors, and time skew issues are described. The document also provides guidance on working with MapR support and things to avoid when troubleshooting.
Analyzing Real-World Data with Apache Drill
The document describes a demo of analyzing real-world data using Apache Drill. The demo involves running Drill, configuring storage plugins for HDFS and MongoDB, and exploring sample data from Yelp including reviews, users, and business data stored in JSON files and MongoDB collections. Queries are run against this data using SQL to analyze basics as well as complex data structures.
Reference Architecture: Architecting Ceph Storage Solutions
This document discusses reference architectures for Ceph storage solutions. It provides guidance on key design considerations for Ceph clusters, including workload profiling, storage access methods, capacity planning, fault tolerance, and data protection schemes. Example hardware configurations are also presented for different performance and cost optimization targets.
Data Hacking with RHadoop
Rhadoop is an effective platform for doing exploratory data analysis over big data sets. The convenience of an interactive command-line interpreter and the overwhelming number of statistical and machine learning routines implemented in R libraries make a highly effective environment to perform elementary data science.
We'll discuss the basics of RHadoop: what it is, how to install it, and the API fundamentals. Next we'll discuss common use cases that you might want to use RHadoop for. Last, we'll run through an interactive example.
EOS MLS software
The document summarizes the EOS MLS software and data processing levels. The EOS MLS instrument measures atmospheric temperature, water, ozone and other molecules from 5-80km on the AURA spacecraft. The software processes the instrument data through four levels: Level 1 converts telemetry to radiances, Level 2 converts radiances to abundances, Level 3 produces daily and monthly mean maps and zonal means. The data flows between levels using HDF4 and HDF-EOS formats, with products archived and described by metadata to enable ordering of Level 1-3 products. Future plans include transitioning to HDF5 and HDF-EOS5.
Overview of Parallel HDF5 and Performance Tuning in HDF5 Library
Overview of Parallel HDF5 and Performance Tuning in HDF5 LibraryThe HDF-EOS Tools and Information Center
This document provides an overview of parallel HDF5 and performance tuning in the HDF5 library. It discusses the design and implementation of parallel HDF5 (PHDF5), including requirements, layers, programming restrictions, examples of APIs, and support for creating/accessing files and datasets in parallel. It also provides examples of writing and reading datasets in parallel using hyperslabs, including by rows, columns, pattern, and chunks. The goal of PHDF5 is to enable efficient parallel I/O while maintaining compatibility with serial HDF5 files.More Related Content
What's hot
Status of HDF-EOS, Related Software and Tools
This document provides an overview of the status of HDF-EOS software and tools. It describes HDF-EOS5, a rewrite of HDF-EOS2 based on HDF5, which is used operationally by EOS instrument teams. The document also outlines software releases, major developments including bug fixes, and future plans, and provides contact information for support.
Parallel HDF5 Developments
The document discusses recent and upcoming improvements to parallel HDF5 for improved I/O performance on HPC systems. Recent improvements include reducing file truncations, distributing metadata writes across processes, and improved selection matching. Upcoming work includes a high-level HPC API, funding for Exascale-focused enhancements, and future improvements like asynchronous I/O and auto-tuning to parallel file systems. Performance tips are also provided like passing MPI hints and using collective I/O.
Easy Access of NASA HDF data via OPeNDAP
This slide will demonstrate how to use OPeNDAP Java clients such as IDV and Panoply via HDF OPeNDAP data handlers to access various NASA HDF products such as AIRS, OMI, MLS, MODIS, TRMM, CERES, SeaWIFS etc. Various features of these tools that can help users easy access the HDF data will also be explored.
Bridging ICESat and ICESat-2 Standard Data Products
This document discusses efforts to standardize data products from three NASA laser altimeter missions - ICESat, ICESat-2, and MABEL. It describes designing similar data products for all three missions to promote interoperability. Products are being developed for ICESat data using lessons from MABEL. Code is also being generated to help create products from specifications to reduce development time. The goal is to make the multi-rate point data from all three missions easily accessible and usable.
Easy Remote Access Via OPeNDAP
The document discusses updates to OPeNDAP handlers for HDF4 and HDF5 data formats. The HDF4 handler was improved to allow more NASA HDF and HDF-EOS data to be visualized by translating the data structure to follow climate and forecast metadata conventions. The HDF5 handler was updated to support additional HDF-EOS5 data from NASA missions. The handlers address issues that previously prevented data visualization and make the data more interoperable. Limitations include unsupported additional HDF4 objects and untested HDF4 products.
Subsetting at UAH
The presentation will provide an overview of subsetting software development activity at UAH. Updates have been made to all packages, reflecting the latest versions of HDF5 and HE5. The library of tools (HSE) for subsetting HDF-EOS data is up-to-date for SGI, Sun, and Linux platforms. Subsetting software is operational at NSIDC DAAC and GDAAC, in testing at LPDAAC. Ongoing work and plans will also be described, including row/column subsetting and index subsampling.
HDF OPeNDAP Project Update and Demo
The HDF Group gave a presentation on updates to the HDF-OPeNDAP project. They summarized the redesign of the HDF5 handler to support more NASA HDF5 data products, including grids, swaths, and zonal averages. They also discussed updates to the HDF4 handler to improve support for MODIS products and add new map projections. Live demonstrations showed example data from instruments like OMI, MLS, and Aquarius being served through the updated HDF handlers.
hadoop architecture -Big data hadoop
This document provides an overview of the Hadoop architecture, including its two main components: the Hadoop Distributed File System (HDFS) and the MapReduce engine. It describes HDFS' use of blocks and replication for reliability. It explains how files are written to HDFS and accessed via command line tools. It also outlines the different node types in Hadoop including the NameNode, DataNode, JobTracker and TaskTracker. Finally, it discusses topology awareness and how bandwidth decreases between processes on different nodes, racks and data centers.
NetCDF and HDF5
NetCDF and HDF5 are data formats and software libraries used for scientific data. NetCDF began in 1989 and allows for array-oriented data with dimensions, variables, and attributes. NetCDF-4 introduced new features while maintaining backward compatibility. It uses HDF5 for data storage and can read HDF4/HDF5 files. NetCDF provides APIs for C, Fortran, Java, and is widely used for earth science and climate data. It supports conventions, parallel I/O, and reading many data formats.
ESDIS Status (2002)
The document provides an overview and status update of the Earth Science Data and Information System (ESDIS). ESDIS has successfully supported numerous Earth science satellite missions and currently manages over 2 petabytes of science data. In fiscal year 2002, ESDIS delivered over 16 million data products to more than 1.8 million users. ESDIS is working to enhance its capabilities through initiatives like Data Pools and the EOS ClearingHOuse (ECHO) metadata broker. HDF-EOS 5 development is nearly complete and the workshop will discuss next steps for HDF-EOS tools and community adoption of HDF-EOS 5.
Hadoop 3.0 features
Hadoop 3.0.0 includes several new features including enhanced high availability for HDFS NameNode, Java 8 as the minimum runtime version, HDFS erasure coding for improved read and write performance, and improvements to YARN Timeline Service for better scalability and usability. The Hadoop shell scripts were also rewritten to unify behavior and address bugs.
EAS Data Flow lessons learnt
1. The document describes the data flow and file handling in a mock implementation of the Infrastructure Abstraction Layer (IAL) for the Euclid Archive System (EAS). It notes two key lessons learned: (1) file URLs need to be abstracted to reduce errors, and (2) reducing unnecessary file copies between systems.
2. It proposes a new Euclid File Access Service (EuFAS) that would provide a standardized way to look up, replicate, and manage file access across the EAS components through an abstract interface. This would replace direct file URLs and simplify data flow.
3. The service would also reduce unnecessary file copies by allowing processing to occur directly from cached copies on execution
Status of HDF-EOS, Related Software and Tools
Status of HDF-EOS and access tools will be summarized. Updates on HDF-EOS, TOOLKIT, HDFView plug-in and The HDF-EOS to GeoTIFF (HEG) conversion tool, including recent changes to the software, ongoing maintenance, upcoming releases, future plans, and issues will be discussed.
Introduction to HDF5
This tutorial is designed for new HDF5 users. We will cover basic HDF5 Data Model objects and their properties, give an overview of the HDF5 Libraries and APIs, and discuss the HDF5 programming model. Simple C and Fortran examples will be used to illustrate HDF5 concepts.
MODIS Land and HDF-EOS
The document discusses MODIS Land products and their distribution format. MODIS Land products include radiation budget, ecosystem, and land cover variables. Products are distributed in HDF-EOS format with fine resolution grids in Integerized Sinusoidal or Lambert Azimuthal Equal-Area projections and coarse grids in a geographic Climate Modeling Grid. HDF-EOS allows for collaboration and standard geolocation representation but toolkit support is still limited.
HDF Update for DAAC Managers (2017-02-27)
This document provides information about HDF (Hierarchical Data Format) tools and resources for working with Earth observation data. It summarizes HDF's focus on helping users at different stages of working with data, from initial product design to long-term archiving. It also describes specific HDF tools for viewing, comparing, converting between formats and adding metadata to scientific data files.
70a monitoring & troubleshooting
This document discusses monitoring and troubleshooting a Hadoop cluster running MapR. It outlines tools for cluster monitoring including the MapR Control System and MapR Metrics. Common troubleshooting scenarios like slow nodes, out of memory errors, and time skew issues are described. The document also provides guidance on working with MapR support and things to avoid when troubleshooting.
Analyzing Real-World Data with Apache Drill
The document describes a demo of analyzing real-world data using Apache Drill. The demo involves running Drill, configuring storage plugins for HDFS and MongoDB, and exploring sample data from Yelp including reviews, users, and business data stored in JSON files and MongoDB collections. Queries are run against this data using SQL to analyze basics as well as complex data structures.
Reference Architecture: Architecting Ceph Storage Solutions
This document discusses reference architectures for Ceph storage solutions. It provides guidance on key design considerations for Ceph clusters, including workload profiling, storage access methods, capacity planning, fault tolerance, and data protection schemes. Example hardware configurations are also presented for different performance and cost optimization targets.
Data Hacking with RHadoop
Rhadoop is an effective platform for doing exploratory data analysis over big data sets. The convenience of an interactive command-line interpreter and the overwhelming number of statistical and machine learning routines implemented in R libraries make a highly effective environment to perform elementary data science.
We'll discuss the basics of RHadoop: what it is, how to install it, and the API fundamentals. Next we'll discuss common use cases that you might want to use RHadoop for. Last, we'll run through an interactive example.
What's hot (20)
Bridging ICESat and ICESat-2 Standard Data Products
Bridging ICESat and ICESat-2 Standard Data Products
Reference Architecture: Architecting Ceph Storage Solutions
Reference Architecture: Architecting Ceph Storage Solutions
Viewers also liked
EOS MLS software
The document summarizes the EOS MLS software and data processing levels. The EOS MLS instrument measures atmospheric temperature, water, ozone and other molecules from 5-80km on the AURA spacecraft. The software processes the instrument data through four levels: Level 1 converts telemetry to radiances, Level 2 converts radiances to abundances, Level 3 produces daily and monthly mean maps and zonal means. The data flows between levels using HDF4 and HDF-EOS formats, with products archived and described by metadata to enable ordering of Level 1-3 products. Future plans include transitioning to HDF5 and HDF-EOS5.
Overview of Parallel HDF5 and Performance Tuning in HDF5 Library
Overview of Parallel HDF5 and Performance Tuning in HDF5 LibraryThe HDF-EOS Tools and Information Center
This document provides an overview of parallel HDF5 and performance tuning in the HDF5 library. It discusses the design and implementation of parallel HDF5 (PHDF5), including requirements, layers, programming restrictions, examples of APIs, and support for creating/accessing files and datasets in parallel. It also provides examples of writing and reading datasets in parallel using hyperslabs, including by rows, columns, pattern, and chunks. The goal of PHDF5 is to enable efficient parallel I/O while maintaining compatibility with serial HDF5 files.misr_view 4.0: A Tool for Visualizing MISR data
misr_view 4.0 is a visualization and analysis tool for MISR data that has several new features including a data transform utility, map projection tool, vector overlay tool, and color bar utility. It allows users to view, analyze, manipulate and reproject MISR data.
Facilitating Access to EOS Data at the NSIDC DAAC
The document discusses the NSIDC DAAC's efforts to facilitate access to Earth Observing System (EOS) data in HDF-EOS format. It summarizes tools developed by NSIDC to help users access, visualize, and process HDF-EOS data, including the PHDIS tool for viewing metadata and subsetting data, a swath-to-grid tool for gridding MODIS data, and a bit viewer for MODIS quality flags. It also discusses future plans to enhance these tools and develop additional capabilities like output in HDF-EOS and GIS formats.
Generalized Conversion of HDF-EOS Products to GIS-Compatible Formats
Generalized Conversion of HDF-EOS Products to GIS-Compatible FormatsThe HDF-EOS Tools and Information Center
This document discusses the development of a tool to convert Earth Observing System (EOS) data products stored in HDF-EOS format to more widely used Geographic Information System (GIS) compatible formats like GeoTIFF. The tool aims to make EOS data more accessible to GIS users by handling various issues with HDF-EOS like different geolocation standards, allowing mosaic-ing of products across files, and integrating with common analysis programs. It will convert HDF-EOS swath and grid products to GeoTIFF, allow for reprojection, subsetting, stitching, and preservation of metadata. The tool is being developed with graphical and command line interfaces using Java and C to work across platforms and without proprietary software dependencies.HDF-EOS Aura File Format Guidelines
This document presents guidelines for formatting HDF-EOS files for NASA's Aura satellite mission, which is scheduled to launch in June 2003. It summarizes the Aura instruments, discusses changes from previous guidelines, and outlines the standardized structure for HDF-EOS files, including dimensions, geolocation fields, data fields, and attributes. The guidelines are intended to promote cross-platform use and understanding of Aura data files.
Subsetting
HDF AND HDF-EOS WORKSHOP V (2002)
HDF Work at UAH: ESML - Earth Science Markup Language, HEW-UAH & HDF-EOS Web-based Subsetter
Viewers also liked (7)
Overview of Parallel HDF5 and Performance Tuning in HDF5 Library
Overview of Parallel HDF5 and Performance Tuning in HDF5 Library
Generalized Conversion of HDF-EOS Products to GIS-Compatible Formats
Generalized Conversion of HDF-EOS Products to GIS-Compatible Formats
Similar to HDF-EOS Development - Current Status and Schedule
HDF-EOS APIs, tools, etc.
HDF-EOS is a standard developed by NASA for storing Earth science data in the Hierarchical Data Format (HDF). It defines structures for gridded data, swaths, points, and other arrays. The HDF-EOS APIs provide functions for reading, writing, and manipulating these data structures and attributes in both HDF4 and HDF5 formats. Tools like HDFView and the HDF-EOS to GeoTIFF Converter (HEG) allow users to browse, subset, and convert HDF-EOS data to other formats.
NASA HDF and HDF-EOS Status - Use in EOSDIS
The document summarizes HDF and HDF-EOS, data formats used by NASA's Earth Observing System Data and Information System (EOSDIS). EOSDIS acquires, archives, manages, and distributes Earth observation data from NASA satellites and other sources. It uses HDF and the EOS-specific profile HDF-EOS as its standard data formats. HDF-EOS facilitates data storage, access, analysis and discovery. The document provides background on HDF-EOS, its history and goals, experience using it, and the goals of the HDF/HDF-EOS workshop being summarized.
HDF-EOS Maintenance, Current Development and Tools
We will summarize current status of HDF-EOS and associated tools. Update on HDF-EOS, HDFView plug-in and The HDF-EOS to GeoTIFF (HEG) conversion tool, including recent changes to the software, ongoing maintenance, upcoming releases, future plans, and issues will be discussed.
We will also summarize the status of HDF-EOS RFC. The HDF-EOS plug-in for the THG-developed tool, HDFView, has been enhanced. The plug-in offers browse capability for both HDF4 and HDF5 - based HDF-EOS files. HDFView can also process vanilla HDF4 and HDF5 files. New features including support for Point and Zonal Average objects have been added. A port to Mac OS X version will be available in next release.
The HDF-EOS to GeoTIFF (HEG) conversion tool has been augmented to include new projections, and support for additional AMSR-E and AIRS products. Subsetting features have also been augmented. The tool is available in both stand-alone and EOS DAAC online versions.
HDF-EOS Tools
The document describes HDF-EOS tools developed to help users work with Earth Observing System (EOS) data stored in HDF-EOS format. It outlines libraries and tools created, including the HDF-EOS to GeoTIFF Converter (HEG) tool, which can convert HDF-EOS files to GeoTIFF and perform subsetting, stitching, and reprojection. It also describes the HDF-EOS plugin for HDFView, which allows browsing of HDF-EOS data and objects. The tools were created to make HDF-EOS data more accessible to users and reduce the need to learn HDF/HDF-EOS libraries.
Status of HDF-EOS, Related Software, and Tools
Current status of HDF-EOS and access tools will be summarized. Update on HDF-EOS, HDFView plug-in and The HDF-EOS to GeoTIFF (HEG) conversion tool, including recent changes to the software, ongoing maintenance, upcoming releases, future plans, and issues will be discussed.
Hdf eos status-workshp_xi_nov_2007
The document discusses the current status and future plans for HDF-EOS, a software toolkit for working with Earth science data files. It notes that maintenance support is funded through May 2009, with one full-time and one half-time developer. Major software components like the SDP Toolkit, HDF-EOS2, HDF-EOS5, and the HDF-EOS to GeoTIFF Converter are maintained and new releases planned. Future development focuses on auto-configuration, 64-bit support, and improved tools for viewing and converting HDF-EOS files.
HDF-EOS Workshop II Introduction
This document introduces an HDF-EOS workshop for data producers, users, and tool developers. HDF-EOS is the baseline standard data format for EOS data, based on HDF but adding features for EOS. The workshop aims to provide in-depth information about HDF and HDF-EOS through hands-on tutorials for tools, utilities, and programming. It is sponsored by NASA and the ECS team, and provides individual consultation. HDF-EOS supports a large community that will need various analysis tools, and resources like documentation, sample data, and experts are available to support the HDF-EOS format.
HDF-EOS to GeoTIFF Conversion Tool and HDF-EOS Plug-in for HDFView
HDF-EOS to GeoTIFF Conversion Tool and HDF-EOS Plug-in for HDFViewThe HDF-EOS Tools and Information Center
HDF-EOS to GeoTIFF (HEG) conversion tool is a versatile tool that besides converting HDF-EOS to GeoTIFF it subsets, resamples, convert swath to grid or grids to different projections, subsamples, or does mosaicing. HDF-EOS plug-in is a library that extends HDFView, a visual tool for browsing and editing HDF4 and HDF5 files, for EOS applications. These tools will be discussed in detail and a demo will be presented. The Landsat 7 Processing System (LPS) Level Zero-R Science Products
The Landsat 7 Processing System (LPS) Level Zero-R Science ProductsThe HDF-EOS Tools and Information Center
The Landsat 7 Processing System (LPS) processes Landsat 7 data into science data products. It receives raw telemetry from the Landsat 7 satellite and constructs images from the data without applying corrections. The LPS divides the data into subintervals and produces multi-band files and metadata for each. It also generates browse images and assesses cloud cover for each subinterval. The LPS data products are separated into Format 1 and Format 2 files representing the two types of raw data received by the LPS. The Level Zero-R science products created by the LPS support further processing but are not yet calibrated or rectified.HDF-EOS to GeoTIFF Conversion Tool & HDF-EOS Plug-in for HDFView
HDF-EOS to GeoTIFF Conversion Tool & HDF-EOS Plug-in for HDFViewThe HDF-EOS Tools and Information Center
This document describes tools for working with HDF-EOS data formats: the HDF-EOS to GeoTIFF Conversion Tool (HEG) and an HDF-EOS plugin for the HDFView browser. HEG allows conversion between HDF-EOS and GeoTIFF formats and supports subsetting, reprojection, stitching and other functions. The HDFView plugin allows browsing and conversion between HDF-EOS2 and HDF-EOS5 formats within HDFView. Both tools are available for download and support a variety of Earth science datasets.HDF-EOS 2/5 to netCDF Converter
The tool takes HDF-EOS 5 data as input, and generates COARDS-compatible output - if the input file has enough metadata to be COARDS-compliant, the output file will be COARDS-compliant. The tool is written in portable C, and ought to run on any platform where the HDF-EOS and netCDF libraries are available.
This year, we have made two major enhancements to the converter:
It now automatically detects whether its input is HDF-EOS2 or HDF-EOS5 format, and handles either one. The previous tool worked with HDF-EOS5 only.
Its netCDF output attempts to conform to the new CF conventions (a superset of the COARDS conventions). This is primarily an improvement in its translation of Swath datasets, which CF handles much better than COARDS.
Hadoop security
This document provides an overview of securing Hadoop applications and clusters. It discusses authentication using Kerberos, authorization using POSIX permissions and HDFS ACLs, encrypting HDFS data at rest, and configuring secure communication between Hadoop services and clients. The principles of least privilege and separating duties are important to apply for a secure Hadoop deployment. Application code may need changes to use Kerberos authentication when accessing Hadoop services.
HDF-EOS 3.0 Functional and Structural Design
This document discusses the design of HDF-EOS 3.0, which is based on HDF5 and improves on HDF-EOS 2.x. It overviews the functionality, including I/O, attributes, compression, chunking, and interfaces for swaths, grids, profiles, and points. The file structure is shown, with the root containing HDFEOS, HDFEOS INFORMATION, SWATHS, GRIDS, and POINTS groups. Swath, grid, profile and point structures each allow global, group and local attributes on fields. Some issues around data types, chunking and compression are also noted.
Aura HDF-EOS File Format Guidelines: Overview and Status
The instrument teams on Aura developed a data format standard for their data files using HDF-EOS5 as the storage mechanism. Now that data files are being shared between instrument teams, the success of this endeavor can be studied. Details of this standard, the process used for creating it, what we've learned and suggestions for future missions will be presented.
HDF-EOS Aura File Format Guidelines
The document presents guidelines for standardizing the HDF-EOS file format for the Aura satellite mission. It proposes organizing data fields and attributes in a consistent way, using standard dimension and field names, and reporting data on a common pressure grid. This will make files easier to understand, cross-platform compatible, and simplify software development. The guidelines cover topics such as data ordering, units, and attribute names. Standardizing these aspects aims to reduce mismatches between instruments. [/SUMMARY]
The HDF-EOS Aura Data Guidelines - "What's New"
The document discusses the Aura Guidelines, a common data format developed and adopted by four instruments on the Aura satellite. The Guidelines were developed using HDF5/HDF-EOS5 to store each instrument's data in a single format. New features include guidelines for zonal mean and gridded products, following similar naming conventions and metadata definitions as level 2 swath products. The Aura Guidelines have been rewritten and submitted as a technical note to NASA's Standards Process Group.
Introduction to HDF5
This tutorial is designed for new HDF5 users. We will go over a brief history of HDF and HDF5 software, and will cover basic HDF5 Data Model objects and their properties; we will give an overview of the HDF5 Libraries and APIs, and discuss the HDF5 programming model. Simple C and Fortran examples, and Java tool HDFView will be used to illustrate HDF5 concepts.
Status of HDF-EOS, Related Software, and Tools
HDF-EOS is a software library designed to support NASA Earth Observing System (EOS) science data, built on top of HDF (Hierarchical Data Format) developed by The HDF Group. Current status of HDF-EOS and access tools will be summarized and updates on TOOLKIT, HDFView plug-in and the HDF-EOS to GeoTIFF (HEG) conversion tool, including recent changes to the software, ongoing maintenance, upcoming releases, and issues will be discussed.
DAOS Middleware overview
DAOS (Distributed Application Object Storage) is a high-performance storage architecture and software stack that delivers scalable object storage capabilities. It uses Intel Optane memory and NVMe SSDs to provide high IOPS, bandwidth, and low latency storage. DAOS supports various data models and interfaces like POSIX, HDF5, Spark, and Python. It allows applications to access storage with library calls instead of system calls for high performance.
The HDF-EOS5 Tutorial
The document describes HDF-EOS5, an extension of HDF used by NASA for Earth science data. HDF-EOS5 is based on HDF5 and contains standardized structures for gridded, swath, point, and zonal average data. It provides a library for reading, writing, and manipulating these data structures and their associated metadata. The library contains functions prefixed with "HE5_" for accessing, defining, input/output, inquiry, and subsetting HDF-EOS5 data.
Similar to HDF-EOS Development - Current Status and Schedule (20)
HDF-EOS Maintenance, Current Development and Tools
HDF-EOS Maintenance, Current Development and Tools
HDF-EOS to GeoTIFF Conversion Tool and HDF-EOS Plug-in for HDFView
HDF-EOS to GeoTIFF Conversion Tool and HDF-EOS Plug-in for HDFView
The Landsat 7 Processing System (LPS) Level Zero-R Science Products
The Landsat 7 Processing System (LPS) Level Zero-R Science Products
HDF-EOS to GeoTIFF Conversion Tool & HDF-EOS Plug-in for HDFView
HDF-EOS to GeoTIFF Conversion Tool & HDF-EOS Plug-in for HDFView
Aura HDF-EOS File Format Guidelines: Overview and Status
Aura HDF-EOS File Format Guidelines: Overview and Status
More from The HDF-EOS Tools and Information Center
Cloud-Optimized HDF5 Files
This document discusses how to optimize HDF5 files for efficient access in cloud object stores. Key optimizations include using large dataset chunk sizes of 1-4 MiB, consolidating internal file metadata, and minimizing variable-length datatypes. The document recommends creating files with paged aggregation and storing file content information in the user block to enable fast discovery of file contents when stored in object stores.
Accessing HDF5 data in the cloud with HSDS
This document provides an overview of HSDS (Highly Scalable Data Service), which is a REST-based service that allows accessing HDF5 data stored in the cloud. It discusses how HSDS maps HDF5 objects like datasets and groups to individual cloud storage objects to optimize performance. The document also describes how HSDS was used to improve access performance for NASA ICESat-2 HDF5 data on AWS S3 by hyper-chunking datasets into larger chunks spanning multiple original HDF5 chunks. Benchmark results showed that accessing the data through HSDS provided over 2x faster performance than other methods like ROS3 or S3FS that directly access the cloud storage.
The State of HDF
This document summarizes the current status and focus of the HDF Group. It discusses that the HDF Group is located in Champaign, IL and is a non-profit organization focused on developing and maintaining HDF software and data formats. It provides an overview of recent HDF5, HDF4 and HDFView releases and notes areas of focus for software quality improvements, increased transparency, strengthening the community, and modernizing HDF products. It invites support and participation in upcoming user group meetings.
Highly Scalable Data Service (HSDS) Performance Features
This document provides an overview of HSDS (HDF Server and Data Service), which allows HDF5 files to be stored and accessed from the cloud. Key points include:
- HSDS maps HDF5 objects like datasets and groups to individual cloud storage objects for scalability and parallelism.
- Features include streaming support, fancy indexing for complex queries, and caching for improved performance.
- HSDS can be deployed on Docker, Kubernetes, or AWS Lambda depending on needs.
- Case studies show HSDS is used by organizations like NREL and NSF to make petabytes of scientific data publicly accessible in the cloud.
Creating Cloud-Optimized HDF5 Files
This document discusses creating cloud-optimized HDF5 files by rearranging internal structures for more efficient data access in cloud object stores. It describes cloud-native and cloud-optimized storage formats, with the latter involving storing the entire HDF5 file as a single object. The benefits of cloud-optimized HDF5 include fast scanning and using the HDF5 library. Key aspects covered include using optimal chunk sizes, compression, and minimizing variable-length datatypes.
HDF5 OPeNDAP Handler Updates, and Performance Discussion
This document discusses updates and performance improvements to the HDF5 OPeNDAP data handler. It provides a history of the handler since 2001 and describes recent updates including supporting DAP4, new data types, and NetCDF data models. A performance study showed that passing compressed HDF5 data through the handler without decompressing/recompressing led to speedups of around 17-30x by leveraging HDF5 direct I/O APIs. This allows outputting HDF5 files as NetCDF files much faster through the handler.
Hyrax: Serving Data from S3
This document provides instructions for using the Hyrax software to serve scientific data files stored on Amazon S3 using the OPeNDAP data access protocol. It describes how to generate ancillary metadata files called DMR++ files using the get_dmrpp tool that provide information about the data file structure and locations. The document explains how to run get_dmrpp inside a Docker container to process data files on S3 and generate customized DMR++ files that the Hyrax server can use to serve the files to clients.
Accessing Cloud Data and Services Using EDL, Pydap, MATLAB
This document provides an overview and examples of accessing cloud data and services using the Earthdata Login (EDL), Pydap, and MATLAB. It discusses some common problems users encounter, such as being unable to access HDF5 data on AWS S3 using MATLAB or read data from OPeNDAP servers using Pydap. Solutions presented include using EDL to get temporary AWS tokens for S3 access in MATLAB and providing code examples on the HDFEOS website to help users access S3 data and OPeNDAP services. The document also notes some limitations, such as tokens being valid for only 1 hour, and workarounds like requesting new tokens or using the MATLAB HDF5 API instead of the netCDF API.
HDF - Current status and Future Directions
The HDF5 Roadmap and New Features document outlines upcoming changes and improvements to the HDF5 library. Key points include:
- HDF5 1.13.x releases will include new features like selection I/O, the Onion VFD for versioned files, improved VFD SWMR for single-writer multiple-reader access, and subfiling for parallel I/O.
- The Virtual Object Layer allows customizing HDF5 object storage and introduces terminal and pass-through connectors.
- The Onion VFD stores versions of HDF5 files in a separate onion file for versioned access.
- VFD SWMR improves on legacy SWMR by implementing single-writer multiple-reader capabilities
HDFEOS.org User Analsys, Updates, and Future
This document discusses user analysis of the HDFEOS.org website and plans for future improvements. It finds that the majority of the site's 100 daily users are "quiet", not posting on forums or other interactive elements. The main user types are locators, who search for examples or data; mergers, who combine or mosaic datasets; and converters, who change file formats. The document outlines recent updates focused on these user types, like adding Python examples for subsetting and calculating latitude and longitude. It proposes future work on artificial intelligence/machine learning uses of HDF files and examples for processing HDF data in the cloud.
HDF - Current status and Future Directions
This document summarizes a presentation about the current status and future directions of the Hierarchical Data Format (HDF) software. It provides updates on recent HDF5 releases, development efforts including new compression methods and ways to access HDF5 data, and outreach resources. It concludes by inviting the audience to share wishes for future HDF development.
H5Coro: The Cloud-Optimized Read-Only Library
The document describes H5Coro, a new C++ library for reading HDF5 files from cloud storage. H5Coro was created to optimize HDF5 reading for cloud environments by minimizing I/O operations through caching and efficient HTTP requests. Performance tests showed H5Coro was 77-132x faster than the previous HDF5 library at reading HDF5 data from Amazon S3 for NASA's SlideRule project. H5Coro supports common HDF5 elements but does not support writing or some complex HDF5 data types and messages to focus on optimized read-only performance for time series data stored sequentially in memory.
MATLAB Modernization on HDF5 1.10
This document summarizes MathWorks' work to modernize MATLAB's support for HDF5. Key points include:
1) MATLAB now supports HDF5 1.10.7 features like single-writer/multiple-reader access and virtual datasets through new and updated low-level functions.
2) Performance benchmarks show some improvements but also regressions compared to the previous HDF5 version, and work continues to optimize code and support future versions.
3) There are compatibility considerations for Linux filter plugins, but interim solutions are provided until MathWorks can ship a single HDF5 version.
HDF for the Cloud - Serverless HDF
HSDS provides HDF as a service through a REST API that can scale across nodes. New releases will enable serverless operation using AWS Lambda or direct client access without a server. This allows HDF data to be accessed remotely without managing servers. HSDS stores each HDF object separately, making it compatible with cloud object storage. Performance on AWS Lambda is slower than a dedicated server but has no management overhead. Direct client access has better performance but limits collaboration between clients.
HDF5 <-> Zarr
HDF5 and Zarr are data formats that can be used to store and access scientific data. This presentation discusses approaches to translating between the two formats. It describes how HDF5 files were translated to the Zarr format by creating a separate Zarr store to hold HDF5 file chunks, and storing chunk location metadata. It also discusses an implementation that translates Zarr data to the HDF5 format by using a special chunking layout and storing chunk information in an HDF5 compound dataset. Limitations of the translations include lack of support for some HDF5 dataset properties in Zarr, and lack of support for some Zarr compression methods in the HDF5 implementation.
HDF for the Cloud - New HDF Server Features
The document discusses HDF for the cloud, including new features of the HDF Server and what's next. Key points:
- HDF Server uses a "sharded schema" that maps HDF5 objects to individual storage objects, allowing parallel access and updates without transferring entire files.
- Implementations include HSDS software that uses the sharded schema with an API and SDKs for different languages like h5pyd for Python.
- New features of HSDS 0.6 include support for POSIX, Azure, AWS Lambda, and role-based access control.
- Future work includes direct access to storage without a server intermediary for some use cases.
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3The HDF-EOS Tools and Information Center
This document compares different methods for accessing HDF and netCDF files stored on Amazon S3, including Apache Drill, THREDDS Data Server (TDS), and HDF5 Virtual File Driver (VFD). A benchmark test of accessing a 24GB HDF5/netCDF-4 file on S3 from Amazon EC2 found that TDS performed the best, responding within 2 minutes, while Apache Drill failed after 7 minutes. The document concludes that TDS 5.0 is the clear winner based on performance and support for role-based access control and HDF4 files, but the best solution depends on use case and software.STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...
STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...The HDF-EOS Tools and Information Center
This document discusses STARE-PODS, a proposal to NASA/ACCESS-19 to develop a scalable data store for earth science data using the SpatioTemporal Adaptive Resolution Encoding (STARE) indexing scheme. STARE allows diverse earth science data to be unified and indexed, enabling the data to be partitioned and stored in a Parallel Optimized Data Store (PODS) for efficient analysis. The HDF Virtual Object Layer and Virtual Data Set technologies can then provide interfaces to access the data in STARE-PODS in a familiar way. The goal is for STARE-PODS to organize diverse data for alignment and parallel/distributed storage and processing to enable integrative analysis at scale.HDF5 and Ecosystem: What Is New?
This document provides an overview and update on HDF5 and its ecosystem. Key points include:
- HDF5 1.12.0 was recently released with new features like the Virtual Object Layer and external references.
- The HDF5 library now supports accessing data in the cloud using connectors like S3 VFD and REST VOL without needing to modify applications.
- Projects like HDFql and H5CPP provide additional interfaces for querying and working with HDF5 files from languages like SQL, C++, and Python.
- The HDF5 community is moving development to GitHub and improving documentation resources on the HDF wiki site.
HDF5 Roadmap 2019-2020
This document summarizes new features in HDF5 1.12.0, including support for storing references to objects and attributes across files, new storage backends using a virtual object layer (VOL), and virtual file drivers (VFDs) for Amazon S3 and HDFS. It outlines the HDF5 roadmap for 2019-2022, which includes continued support for HDF5 1.8 and 1.10, and new features in future 1.12.x releases like querying, indexing, and provenance tracking.
More from The HDF-EOS Tools and Information Center (20)
Highly Scalable Data Service (HSDS) Performance Features
Highly Scalable Data Service (HSDS) Performance Features
HDF5 OPeNDAP Handler Updates, and Performance Discussion
HDF5 OPeNDAP Handler Updates, and Performance Discussion
Accessing Cloud Data and Services Using EDL, Pydap, MATLAB
Accessing Cloud Data and Services Using EDL, Pydap, MATLAB
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3
STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...
STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...
Recently uploaded
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI, built on the robust SAP Business Technology Platform (SAP BTP), is the latest and most advanced version of our AI development, reaffirming our commitment to delivering top-tier AI solutions. Skybuffer AI harnesses all the innovative capabilities of the SAP BTP in the AI domain, from Conversational AI to cutting-edge Generative AI and Retrieval-Augmented Generation (RAG). It also helps SAP customers safeguard their investments into SAP Conversational AI and ensure a seamless, one-click transition to SAP Business AI.
With Skybuffer AI, various AI models can be integrated into a single communication channel such as Microsoft Teams. This integration empowers business users with insights drawn from SAP backend systems, enterprise documents, and the expansive knowledge of Generative AI. And the best part of it is that it is all managed through our intuitive no-code Action Server interface, requiring no extensive coding knowledge and making the advanced AI accessible to more users.
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers:
* What is Vector Search?
* Importance and benefits of vector search
* Practical use cases across various industries
* Step-by-step implementation guide
* Live demos with code snippets
* Enhancing LLM capabilities with vector search
* Best practices and optimization strategies
Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications.
#MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology
Choosing The Best AWS Service For Your Website + API.pptx
Have you ever been confused by the myriad of choices offered by AWS for hosting a website or an API?
Lambda, Elastic Beanstalk, Lightsail, Amplify, S3 (and more!) can each host websites + APIs. But which one should we choose?
Which one is cheapest? Which one is fastest? Which one will scale to meet our needs?
Join me in this session as we dive into each AWS hosting service to determine which one is best for your scenario and explain why!
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
Predictive maintenance is a proactive approach that anticipates equipment failures before they happen. At the forefront of this innovative strategy is Artificial Intelligence (AI), which brings unprecedented precision and efficiency. AI in predictive maintenance is transforming industries by reducing downtime, minimizing costs, and enhancing productivity.
Digital Marketing Trends in 2024 | Guide for Staying Ahead
https://www.wask.co/ebooks/digital-marketing-trends-in-2024
Feeling lost in the digital marketing whirlwind of 2024? Technology is changing, consumer habits are evolving, and staying ahead of the curve feels like a never-ending pursuit. This e-book is your compass. Dive into actionable insights to handle the complexities of modern marketing. From hyper-personalization to the power of user-generated content, learn how to build long-term relationships with your audience and unlock the secrets to success in the ever-shifting digital landscape.
Programming Foundation Models with DSPy - Meetup Slides
Prompting language models is hard, while programming language models is easy. In this talk, I will discuss the state-of-the-art framework DSPy for programming foundation models with its powerful optimizers and runtime constraint system.
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for life science domain, where you retriever information from biomedical knowledge graphs using LLMs to increase the accuracy and performance of generated answers
5th LF Energy Power Grid Model Meet-up Slides
5th Power Grid Model Meet-up
It is with great pleasure that we extend to you an invitation to the 5th Power Grid Model Meet-up, scheduled for 6th June 2024. This event will adopt a hybrid format, allowing participants to join us either through an online Mircosoft Teams session or in person at TU/e located at Den Dolech 2, Eindhoven, Netherlands. The meet-up will be hosted by Eindhoven University of Technology (TU/e), a research university specializing in engineering science & technology.
Power Grid Model
The global energy transition is placing new and unprecedented demands on Distribution System Operators (DSOs). Alongside upgrades to grid capacity, processes such as digitization, capacity optimization, and congestion management are becoming vital for delivering reliable services.
Power Grid Model is an open source project from Linux Foundation Energy and provides a calculation engine that is increasingly essential for DSOs. It offers a standards-based foundation enabling real-time power systems analysis, simulations of electrical power grids, and sophisticated what-if analysis. In addition, it enables in-depth studies and analysis of the electrical power grid’s behavior and performance. This comprehensive model incorporates essential factors such as power generation capacity, electrical losses, voltage levels, power flows, and system stability.
Power Grid Model is currently being applied in a wide variety of use cases, including grid planning, expansion, reliability, and congestion studies. It can also help in analyzing the impact of renewable energy integration, assessing the effects of disturbances or faults, and developing strategies for grid control and optimization.
What to expect
For the upcoming meetup we are organizing, we have an exciting lineup of activities planned:
-Insightful presentations covering two practical applications of the Power Grid Model.
-An update on the latest advancements in Power Grid -Model technology during the first and second quarters of 2024.
-An interactive brainstorming session to discuss and propose new feature requests.
-An opportunity to connect with fellow Power Grid Model enthusiasts and users.
TrustArc Webinar - 2024 Global Privacy Survey
How does your privacy program stack up against your peers? What challenges are privacy teams tackling and prioritizing in 2024?
In the fifth annual Global Privacy Benchmarks Survey, we asked over 1,800 global privacy professionals and business executives to share their perspectives on the current state of privacy inside and outside of their organizations. This year’s report focused on emerging areas of importance for privacy and compliance professionals, including considerations and implications of Artificial Intelligence (AI) technologies, building brand trust, and different approaches for achieving higher privacy competence scores.
See how organizational priorities and strategic approaches to data security and privacy are evolving around the globe.
This webinar will review:
- The top 10 privacy insights from the fifth annual Global Privacy Benchmarks Survey
- The top challenges for privacy leaders, practitioners, and organizations in 2024
- Key themes to consider in developing and maintaining your privacy program
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
A Mix Chart displays historical data of numbers in a graphical or tabular form. The Kalyan Rajdhani Mix Chart specifically shows the results of a sequence of numbers over different periods.
Presentation of the OECD Artificial Intelligence Review of Germany
Consult the full report at https://www.oecd.org/digital/oecd-artificial-intelligence-review-of-germany-609808d6-en.htm
Best 20 SEO Techniques To Improve Website Visibility In SERP
Boost your website's visibility with proven SEO techniques! Our latest blog dives into essential strategies to enhance your online presence, increase traffic, and rank higher on search engines. From keyword optimization to quality content creation, learn how to make your site stand out in the crowded digital landscape. Discover actionable tips and expert insights to elevate your SEO game.
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Discover the seamless integration of RPA (Robotic Process Automation), COMPOSER, and APM with AWS IDP enhanced with Slack notifications. Explore how these technologies converge to streamline workflows, optimize performance, and ensure secure access, all while leveraging the power of AWS IDP and real-time communication via Slack notifications.
AWS Cloud Cost Optimization Presentation.pptx
This presentation provides valuable insights into effective cost-saving techniques on AWS. Learn how to optimize your AWS resources by rightsizing, increasing elasticity, picking the right storage class, and choosing the best pricing model. Additionally, discover essential governance mechanisms to ensure continuous cost efficiency. Whether you are new to AWS or an experienced user, this presentation provides clear and practical tips to help you reduce your cloud costs and get the most out of your budget.
Skybuffer SAM4U tool for SAP license adoption
Manage and optimize your license adoption and consumption with SAM4U, an SAP free customer software asset management tool.
SAM4U, an SAP complimentary software asset management tool for customers, delivers a detailed and well-structured overview of license inventory and usage with a user-friendly interface. We offer a hosted, cost-effective, and performance-optimized SAM4U setup in the Skybuffer Cloud environment. You retain ownership of the system and data, while we manage the ABAP 7.58 infrastructure, ensuring fixed Total Cost of Ownership (TCO) and exceptional services through the SAP Fiori interface.
Main news related to the CCS TSI 2023 (2023/1695)
An English 🇬🇧 translation of a presentation to the speech I gave about the main changes brought by CCS TSI 2023 at the biggest Czech conference on Communications and signalling systems on Railways, which was held in Clarion Hotel Olomouc from 7th to 9th November 2023 (konferenceszt.cz). Attended by around 500 participants and 200 on-line followers.
The original Czech 🇨🇿 version of the presentation can be found here: https://www.slideshare.net/slideshow/hlavni-novinky-souvisejici-s-ccs-tsi-2023-2023-1695/269688092 .
The videorecording (in Czech) from the presentation is available here: https://youtu.be/WzjJWm4IyPk?si=SImb06tuXGb30BEH .
Recently uploaded (20)
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
HDF-EOS Development - Current Status and Schedule
- 1. HDF-EOS Development Current Status and Schedule Larry Klein, Alex Muslimov & David Wynne Emergent Information Technologies, Inc. February 27, 2002 larry @eos.hitc.com, amuslimo@eos.hitc.com 1
- 2. HDF-EOS 2 • HDF4 based, storage format for EOS standard products. • Used operationally by MODIS, MISR, ASTER, Landsat, other EOS instruments • Will be used by EOS Aqua instruments. • Support for Grid/Point/Swath structures • HDF4, HDF-EOS 2 will be supported by NCSA/NASA for the fore-seeable future. 2
- 3. Current Archive Holdings • GSFC DAAC: MODIS L1, L2 Atmos./Ocean – 416 TB 5,182,000 granules (EOS Terra) • EDC DAAC: Landsat/ASTER/MODIS Land – 323 TB 3,551,000 granules • Langely DAAC: MISR – 141 TB 1,266,000 granules • NSIDC DAAC: MODIS L2, L3 – 10 TB 887,000 granules 3
- 4. HDF-EOS 5 • Based on HDF5, a complete rewrite of HDF4 with a different interface. – First released in 2000. • Designed to ‘resemble’ HDF-EOS 2 to the maximum extent possible. – – – – Support same data structures Added prefix ‘HE5_’ to HDF-EOS 2 functions. Doesn’t preclude HDF5 functionality. Data Type changes, e.g. INT64 -> 4 H5T_NATIVE_LONG
- 5. HDF-EOS 2 -> HDF-EOS 5 •HDF-EOS 2 SWdefdatafield(swathID, fieldname, dimlist, numtype, merge) •HDF-EOS 5 HE5_SWdefdatafield(swathID, fieldname, dimlist, maxdimlist, numtype, merge, Maxdimlist*) *New HDF5 functionality passed through. This allows the user to set an upper limit to the size of the dataset. 5
- 6. HDF-EOS 5 • Provide a conversion tool: heconvert – HDF-EOS 2 -> HDF-EOS 5 – compliments HDF4 -> HDF5 conversion tool • HDF-EOS 5 will be used by EOS Aura – Standard format for profile data developed • Assume that HDF-EOS 2 producers will convert when PI’s determine that the time is right. 6
- 7. HDF-EOS 5 Functionality • • • • • • • • Basic File I/O Fill Values Compression Chunking/Tiling Swath Interface Grid Interface Point Interface Profile Interface • Global (File), Group & Local Attributes • External Data Files • Subsetting • Unix/Linux Support • Threadsafe Version • FORTRAN, C, C++ • General Table Interface (proposed) 7
- 8. Top Level of HDF-EOS 5 File Root -- “/” HDFEOS INFORMATION HDFEOS STRUC. METADATA ADDITIONAL SWATH GRID POINT Global (File) Attributes The new ADDITIONAL Group has global (file) attributes The new functionality is added to the EH(utility) interface. 8
- 9. Swath Structure Global Attribute <SwathName>:<AttrName> Group Attribute <DataFields>:<AttrName> Local Attribute <FieldName>:<AttrName> SwathName Data Fields Data Data Field.1 Field.n Profile Fields Profile Field.1 Profile Field.n Geolocation Fields Longitude Latitude Time Each Data Field can have Attributes and/or Dimension Scales Group CoLatitude Shaded Objects are implemented in a fixed way so the user doesn’t have direct access via the interface Attribute Data Set 9
- 10. ECS support of HDF5 • ECS toolkit V 5.2.8 supports HDF4 and HDF5 - based applications (eg. Metadata access – HDF-EOS 2.7 (HDF4.1r5) – HDF-EOS 5.1 (hdf5-1.4.1) • HDF5 and HDF4 must be compiled – HDF4 users not affected – HDF5 users must use PGS_MET_SDstart() and PGS_MET_SDend() 10
- 11. ECS support of HDF5 • Both flavors of HDF-EOS (HDF) part of ECS baseline. • ECS will not crack HDF5 - based files for near future. (contents transparent to archive) • Size limit is 2 GBytes 11
- 12. Applications • HE5View (HDF-EOS 5 browser) • Java Earth Browser (HDF-EOS 2 and 5 access) • HDF-EOS to GeoTIFF converter (HDF4 based only) • heconvert (convert HDF-EOS 2 Grid/Point/Swath to HDF-EOS 5 equivalents) • Access libraries and applications at: http://newsroom.gsfc.nasa.gov/sdptoolkit/toolkit .html 12
- 13. Issues • Compound data types in FORTRAN77 • Chunking with variable length and compound data types • Only deflation compression method implemented. • File size limit in ECS - Is two Gbytes enough? 13