HDF-EOS Development Current Status
•
0 likes•299 views
HDF-EOS 5 is a new format based on HDF5 that is designed to resemble the existing HDF-EOS 2 format. It supports the same data structures as HDF-EOS 2 but uses HDF5 functionality like compression and chunking. A conversion tool allows data to be changed from HDF-EOS 2 to HDF-EOS 5 format. While HDF-EOS 2 will still be supported, HDF-EOS 5 will be used by new missions like Aura and provides more options for data storage and filtering.
Report
Share
Report
Share
Download to read offline
Recommended
HDF-EOS Development - Current Status and Schedule
The document discusses the current status and schedule of HDF-EOS development. HDF-EOS 2 is the current storage format for EOS standard products, using HDF4, and is supported by NASA. HDF-EOS 5 is a rewrite using HDF5 that is designed to resemble HDF-EOS 2. It will be used by the EOS Aura mission. The document outlines current archive holdings using HDF-EOS 2 and functionality included in HDF-EOS 5.
What is HDF-EOS?
HDF-EOS defines standard profiles for using HDF to organize Earth observation data. It builds on HDF by adding geolocation information coupled to data for point, swath, and grid collection profiles. HDF-EOS is used to define over 200 products from instruments on Terra to be distributed through EOSDIS. It provides full geolocation support and metadata to allow search and access of Earth science data.
HDF-EOS Development: Current Status and Tools
This document provides a summary of the current status and schedule of HDF-EOS development. It describes what HDF-EOS is, its data structures like Grid, Swath, Point, and Zonal Average, and applications that can be used to access and process HDF-EOS data. HDF-EOS 5, based on HDF5, is now used by EOS Aura producers and users, while HDF-EOS 2 will continue to be supported.
HDF-EOS to GeoTIFF Conversion Tool & HDF-EOS Plug-in for HDFView
HDF-EOS to GeoTIFF Conversion Tool & HDF-EOS Plug-in for HDFViewThe HDF-EOS Tools and Information Center
HDF-EOS to GeoTIFF (HEG) conversion tool is a versatile tool that besides converting HDF-EOS to GeoTIFF it subsets, resamples, convert swath to grid and/or grids to different projections, subsamples, or does mosaicing. HDF-EOS plug-in is a library that extends HDFView, a visual tool for browsing and editing HDF4 and HDF5 files, for EOS applications. These tools will be discussed in detail and a demo will be presented.HDF Tools Tutorial
This tutorial is designed for anyone who needs to work with data stored in HDF and HDF5 files.
The first part of the tutorial will focus on the HDF5 utilities to display the contents of HDF5 files, to extract and to import data from and to HDF5 files, to compare two HDF5 files, and more. Participants will be guided through the hand-on examples and will learn about different tools options. New changes and advanced features will be covered in a separate session (Updates on HDF tools) on Wednesday.
The second part of tutorial includes a hands-on session to learn the HDF (4 & 5) Java browsing tool, HDFView. The tool and special plug-ins will be used to work with the existing HDF, HDF-EOS, and netCDF-4 files, and to create a new HDF5 file. The tutorial will cover basic features of HDFView.
HDF-EOS Status and Developments
This document summarizes the status and development of HDF-EOS software. It notes that maintenance support is funded through July 2008. It describes software releases from May 2005 to the planned December 2005 release. Major developments for HDF-EOS5 since October 2004 are also outlined, including performance improvements and added functionality. Future plans include additional platform support and performance optimizations. The HDF-EOS to GeoTIFF Converter (HEG) is also described, which allows conversion of HDF-EOS data to common GIS formats.
Parallel HDF5 Developments
The document discusses recent and upcoming improvements to parallel HDF5 for improved I/O performance on HPC systems. Recent improvements include reducing file truncations, distributing metadata writes across processes, and improved selection matching. Upcoming work includes a high-level HPC API, funding for Exascale-focused enhancements, and future improvements like asynchronous I/O and auto-tuning to parallel file systems. Performance tips are also provided like passing MPI hints and using collective I/O.
Status of HDF-EOS, Related Software and Tools
This document provides an overview of the status of HDF-EOS software and tools. It describes HDF-EOS5, a rewrite of HDF-EOS2 based on HDF5, which is used operationally by EOS instrument teams. The document also outlines software releases, major developments including bug fixes, and future plans, and provides contact information for support.
Recommended
HDF-EOS Development - Current Status and Schedule
The document discusses the current status and schedule of HDF-EOS development. HDF-EOS 2 is the current storage format for EOS standard products, using HDF4, and is supported by NASA. HDF-EOS 5 is a rewrite using HDF5 that is designed to resemble HDF-EOS 2. It will be used by the EOS Aura mission. The document outlines current archive holdings using HDF-EOS 2 and functionality included in HDF-EOS 5.
What is HDF-EOS?
HDF-EOS defines standard profiles for using HDF to organize Earth observation data. It builds on HDF by adding geolocation information coupled to data for point, swath, and grid collection profiles. HDF-EOS is used to define over 200 products from instruments on Terra to be distributed through EOSDIS. It provides full geolocation support and metadata to allow search and access of Earth science data.
HDF-EOS Development: Current Status and Tools
This document provides a summary of the current status and schedule of HDF-EOS development. It describes what HDF-EOS is, its data structures like Grid, Swath, Point, and Zonal Average, and applications that can be used to access and process HDF-EOS data. HDF-EOS 5, based on HDF5, is now used by EOS Aura producers and users, while HDF-EOS 2 will continue to be supported.
HDF-EOS to GeoTIFF Conversion Tool & HDF-EOS Plug-in for HDFView
HDF-EOS to GeoTIFF Conversion Tool & HDF-EOS Plug-in for HDFViewThe HDF-EOS Tools and Information Center
HDF-EOS to GeoTIFF (HEG) conversion tool is a versatile tool that besides converting HDF-EOS to GeoTIFF it subsets, resamples, convert swath to grid and/or grids to different projections, subsamples, or does mosaicing. HDF-EOS plug-in is a library that extends HDFView, a visual tool for browsing and editing HDF4 and HDF5 files, for EOS applications. These tools will be discussed in detail and a demo will be presented.HDF Tools Tutorial
This tutorial is designed for anyone who needs to work with data stored in HDF and HDF5 files.
The first part of the tutorial will focus on the HDF5 utilities to display the contents of HDF5 files, to extract and to import data from and to HDF5 files, to compare two HDF5 files, and more. Participants will be guided through the hand-on examples and will learn about different tools options. New changes and advanced features will be covered in a separate session (Updates on HDF tools) on Wednesday.
The second part of tutorial includes a hands-on session to learn the HDF (4 & 5) Java browsing tool, HDFView. The tool and special plug-ins will be used to work with the existing HDF, HDF-EOS, and netCDF-4 files, and to create a new HDF5 file. The tutorial will cover basic features of HDFView.
HDF-EOS Status and Developments
This document summarizes the status and development of HDF-EOS software. It notes that maintenance support is funded through July 2008. It describes software releases from May 2005 to the planned December 2005 release. Major developments for HDF-EOS5 since October 2004 are also outlined, including performance improvements and added functionality. Future plans include additional platform support and performance optimizations. The HDF-EOS to GeoTIFF Converter (HEG) is also described, which allows conversion of HDF-EOS data to common GIS formats.
Parallel HDF5 Developments
The document discusses recent and upcoming improvements to parallel HDF5 for improved I/O performance on HPC systems. Recent improvements include reducing file truncations, distributing metadata writes across processes, and improved selection matching. Upcoming work includes a high-level HPC API, funding for Exascale-focused enhancements, and future improvements like asynchronous I/O and auto-tuning to parallel file systems. Performance tips are also provided like passing MPI hints and using collective I/O.
Status of HDF-EOS, Related Software and Tools
This document provides an overview of the status of HDF-EOS software and tools. It describes HDF-EOS5, a rewrite of HDF-EOS2 based on HDF5, which is used operationally by EOS instrument teams. The document also outlines software releases, major developments including bug fixes, and future plans, and provides contact information for support.
hadoop architecture -Big data hadoop
This document provides an overview of the Hadoop architecture, including its two main components: the Hadoop Distributed File System (HDFS) and the MapReduce engine. It describes HDFS' use of blocks and replication for reliability. It explains how files are written to HDFS and accessed via command line tools. It also outlines the different node types in Hadoop including the NameNode, DataNode, JobTracker and TaskTracker. Finally, it discusses topology awareness and how bandwidth decreases between processes on different nodes, racks and data centers.
Easy Access of NASA HDF data via OPeNDAP
This slide will demonstrate how to use OPeNDAP Java clients such as IDV and Panoply via HDF OPeNDAP data handlers to access various NASA HDF products such as AIRS, OMI, MLS, MODIS, TRMM, CERES, SeaWIFS etc. Various features of these tools that can help users easy access the HDF data will also be explored.
Big Data
This document provides an overview of big data and Hadoop. It defines big data and lists some technical and business drivers for implementing big data solutions. It then discusses the steps to implement big data and introduces Hadoop as the industry standard for batch processing of big data. The main components of Hadoop - HDFS and MapReduce - are described. Example use cases of Hive and Pig for analyzing big data using SQL-like and dataflow languages are also provided.
Easy Remote Access Via OPeNDAP
The document discusses updates to OPeNDAP handlers for HDF4 and HDF5 data formats. The HDF4 handler was improved to allow more NASA HDF and HDF-EOS data to be visualized by translating the data structure to follow climate and forecast metadata conventions. The HDF5 handler was updated to support additional HDF-EOS5 data from NASA missions. The handlers address issues that previously prevented data visualization and make the data more interoperable. Limitations include unsupported additional HDF4 objects and untested HDF4 products.
ESDIS Status (2002)
The document provides an overview and status update of the Earth Science Data and Information System (ESDIS). ESDIS has successfully supported numerous Earth science satellite missions and currently manages over 2 petabytes of science data. In fiscal year 2002, ESDIS delivered over 16 million data products to more than 1.8 million users. ESDIS is working to enhance its capabilities through initiatives like Data Pools and the EOS ClearingHOuse (ECHO) metadata broker. HDF-EOS 5 development is nearly complete and the workshop will discuss next steps for HDF-EOS tools and community adoption of HDF-EOS 5.
NetCDF and HDF5
NetCDF and HDF5 are data formats and software libraries used for scientific data. NetCDF began in 1989 and allows for array-oriented data with dimensions, variables, and attributes. NetCDF-4 introduced new features while maintaining backward compatibility. It uses HDF5 for data storage and can read HDF4/HDF5 files. NetCDF provides APIs for C, Fortran, Java, and is widely used for earth science and climate data. It supports conventions, parallel I/O, and reading many data formats.
Analyzing Real-World Data with Apache Drill
The document describes a demo of analyzing real-world data using Apache Drill. The demo involves running Drill, configuring storage plugins for HDFS and MongoDB, and exploring sample data from Yelp including reviews, users, and business data stored in JSON files and MongoDB collections. Queries are run against this data using SQL to analyze basics as well as complex data structures.
Reference Architecture: Architecting Ceph Storage Solutions
This document discusses reference architectures for Ceph storage solutions. It provides guidance on key design considerations for Ceph clusters, including workload profiling, storage access methods, capacity planning, fault tolerance, and data protection schemes. Example hardware configurations are also presented for different performance and cost optimization targets.
HDFEOS.org User Analsys, Updates, and Future
This document discusses user analysis of the HDFEOS.org website and plans for future improvements. It finds that the majority of the site's 100 daily users are "quiet", not posting on forums or other interactive elements. The main user types are locators, who search for examples or data; mergers, who combine or mosaic datasets; and converters, who change file formats. The document outlines recent updates focused on these user types, like adding Python examples for subsetting and calculating latitude and longitude. It proposes future work on artificial intelligence/machine learning uses of HDF files and examples for processing HDF data in the cloud.
Data Are from Mars, Tools Are from Venus
This document discusses challenges with data and tools coming from different "planets" (Mars and Venus), frustrating users. It identifies gaps in file formats, libraries, tools, and services for working with Earth science data. The author provides solutions through the HDFEOS.org project, including file format conversion and examples, and suggestions for data producers, tool developers, and end users to help address interoperability issues. Emerging topics like big data, machine learning, and voice assistants are also discussed.
How to Increase Performance of Your Hadoop Cluster
Three identical Apache Hadoop clusters were provisioned on Joyent infrastructure using different operating systems: SmartOS, Ubuntu, and KVM virtual machines. Monitoring showed the Ubuntu and KVM clusters spent more time in the OS kernel during I/O operations compared to the SmartOS cluster. The SmartOS cluster was able to utilize CPU resources more efficiently and scale to more mappers and reducers. Basic cluster configuration and tuning the number of map and reduce tasks are important to optimize Hadoop performance.
Apache Spark on Supercomputers: A Tale of the Storage Hierarchy with Costin I...
In this session, the speakers will discuss their experiences porting Apache Spark to the Cray XC family of supercomputers. One scalability bottleneck is in handling the global file system present in all large-scale HPC installations. Using two techniques (file open pooling, and mounting the Spark file hierarchy in a specific manner), they were able to improve scalability from O(100) cores to O(10,000) cores. This is the first result at such a large scale on HPC systems, and it had a transformative impact on research, enabling their colleagues to run on 50,000 cores.
With this baseline performance fixed, they will then discuss the impact of the storage hierarchy and of the network on Spark performance. They will contrast a Cray system with two levels of storage with a “data intensive” system with fast local SSDs. The Cray contains a back-end global file system and a mid-tier fast SSD storage. One conclusion is that local SSDs are not needed for good performance on a very broad workload, including spark-perf, TeraSort, genomics, etc.
They will also provide a detailed analysis of the impact of latency of file and network I/O operations on Spark scalability. This analysis is very useful to both system procurements and Spark core developers. By examining the mean/median value in conjunction with variability, one can infer the expected scalability on a given system. For example, the Cray mid-tier storage has been marketed as the magic bullet for data intensive applications. Initially, it did improve scalability and end-to-end performance. After understanding and eliminating variability in I/O operations, they were able to outperform any configurations involving mid-tier storage by using the back-end file system directly. They will also discuss the impact of network performance and contrast results on the Cray Aries HPC network with results on InfiniBand.
Democratizing Memory Storage
The document discusses improvements to HDFS that allow it to leverage memory as a storage medium. Key points include:
- HDFS 2.3 introduced memory as a storage medium, with RAM disks providing persistence across restarts.
- HDFS 2.6 introduced storage policies that allow applications to target different storage media like SSD or memory.
- The Centralized Cache Management feature loads hot data into memory pools to enable zero-copy reads.
- The Lazy Persist Writes feature allows applications to write to memory and have HDFS asynchronously write to persistent storage, reducing latency.
- Future work includes improving caching, short-circuit writes, and the Memfs layered file system to provide more flexible
IBM SONAS and the Cloud Storage Taxonomy
This document discusses IBM's Scale Out Network Attached Storage (SONAS) solution. SONAS provides a global namespace and can scale to support large amounts of unstructured file data across various cloud environments. The document outlines how SONAS utilizes IBM's General Parallel File System (GPFS) and provides features such as high performance, data replication, backups, antivirus integration, and information lifecycle management through migration to tape storage.
Caching and Buffering in HDF5
In this talk we will discuss caching and buffering strategies in HDF5. The information presented will help developers write more efficient applications and avoid performance bottlenecks.
Apache hadoop technology : Beginners
This presentation is about apache hadoop technology. It may be helpful for the beginners to know some terminologies of hadoop.
Google Colaboratory for HDF-EOS
Google Colaboratory allows users to write and execute Python code in the cloud using Jupyter notebooks. It provides a free GPU and TPU for accelerating code. The document discusses how HDF-EOS is a standard format for satellite data and provides many examples for converting and processing HDF-EOS data. It then demonstrates how to install necessary packages and run an example zoo code in Colab to plot HDF-EOS data in the cloud without installing anything locally.
2.introduction to hdfs
This document provides an introduction to HDFS (Hadoop Distributed File System). It discusses what HDFS is, its core components, architecture, and key elements like the NameNode, metadata, and blocks. HDFS is designed for storing very large files across commodity hardware in a fault-tolerant manner and allows for streaming access. While HDFS can handle small datasets, its real power is with large and distributed data.
Aziksa hadoop architecture santosh jha
Hadoop is an open-source framework for distributed storage and processing of large datasets across clusters of commodity hardware. It has several core components including HDFS for distributed file storage and MapReduce for distributed processing. HDFS stores data across clusters of machines with replication for fault tolerance. MapReduce allows parallel processing of large datasets in a distributed manner. Hadoop was designed with goals of using commodity hardware, easy recovery from failures, large distributed file systems, and fast processing of large datasets.
HBase with MapR
- MapR makes it easy to deploy and run HBase in production by providing high availability, data protection, and disaster recovery features out of the box. This includes setting up hourly snapshots and cross-datacenter mirroring with just a few commands.
- With MapR, all HBase data should be stored in a single volume called "hbase.volume" for easy management. Snapshots are taken of this volume to enable consistent backups with no downtime. The snapshots can be mirrored to other clusters for disaster recovery.
- Memory usage on RegionServers is automatically tuned by MapR's Warden service based on configuration. The defaults generally work well but guidelines are provided for tuning in large deployments.
Earth Science Markup Language (ESML) - A Tutorial
This document provides an overview of the Earth Science Markup Language (ESML). ESML is an XML-based interchange format that allows applications and services to access heterogeneous earth science data regardless of the underlying data format. It provides syntactic, semantic, and content metadata that describe data structures, meanings, and contents in a machine-readable way. The document outlines the need for such an interchange format, describes the components of ESML including the schema, libraries, and tools, and provides examples of writing ESML descriptions for different types of data files.
Status of HDF-EOS, Related Software, and Tools
Current status of HDF-EOS and access tools will be summarized. Update on HDF-EOS, HDFView plug-in and The HDF-EOS to GeoTIFF (HEG) conversion tool, including recent changes to the software, ongoing maintenance, upcoming releases, future plans, and issues will be discussed.
More Related Content
What's hot
hadoop architecture -Big data hadoop
This document provides an overview of the Hadoop architecture, including its two main components: the Hadoop Distributed File System (HDFS) and the MapReduce engine. It describes HDFS' use of blocks and replication for reliability. It explains how files are written to HDFS and accessed via command line tools. It also outlines the different node types in Hadoop including the NameNode, DataNode, JobTracker and TaskTracker. Finally, it discusses topology awareness and how bandwidth decreases between processes on different nodes, racks and data centers.
Easy Access of NASA HDF data via OPeNDAP
This slide will demonstrate how to use OPeNDAP Java clients such as IDV and Panoply via HDF OPeNDAP data handlers to access various NASA HDF products such as AIRS, OMI, MLS, MODIS, TRMM, CERES, SeaWIFS etc. Various features of these tools that can help users easy access the HDF data will also be explored.
Big Data
This document provides an overview of big data and Hadoop. It defines big data and lists some technical and business drivers for implementing big data solutions. It then discusses the steps to implement big data and introduces Hadoop as the industry standard for batch processing of big data. The main components of Hadoop - HDFS and MapReduce - are described. Example use cases of Hive and Pig for analyzing big data using SQL-like and dataflow languages are also provided.
Easy Remote Access Via OPeNDAP
The document discusses updates to OPeNDAP handlers for HDF4 and HDF5 data formats. The HDF4 handler was improved to allow more NASA HDF and HDF-EOS data to be visualized by translating the data structure to follow climate and forecast metadata conventions. The HDF5 handler was updated to support additional HDF-EOS5 data from NASA missions. The handlers address issues that previously prevented data visualization and make the data more interoperable. Limitations include unsupported additional HDF4 objects and untested HDF4 products.
ESDIS Status (2002)
The document provides an overview and status update of the Earth Science Data and Information System (ESDIS). ESDIS has successfully supported numerous Earth science satellite missions and currently manages over 2 petabytes of science data. In fiscal year 2002, ESDIS delivered over 16 million data products to more than 1.8 million users. ESDIS is working to enhance its capabilities through initiatives like Data Pools and the EOS ClearingHOuse (ECHO) metadata broker. HDF-EOS 5 development is nearly complete and the workshop will discuss next steps for HDF-EOS tools and community adoption of HDF-EOS 5.
NetCDF and HDF5
NetCDF and HDF5 are data formats and software libraries used for scientific data. NetCDF began in 1989 and allows for array-oriented data with dimensions, variables, and attributes. NetCDF-4 introduced new features while maintaining backward compatibility. It uses HDF5 for data storage and can read HDF4/HDF5 files. NetCDF provides APIs for C, Fortran, Java, and is widely used for earth science and climate data. It supports conventions, parallel I/O, and reading many data formats.
Analyzing Real-World Data with Apache Drill
The document describes a demo of analyzing real-world data using Apache Drill. The demo involves running Drill, configuring storage plugins for HDFS and MongoDB, and exploring sample data from Yelp including reviews, users, and business data stored in JSON files and MongoDB collections. Queries are run against this data using SQL to analyze basics as well as complex data structures.
Reference Architecture: Architecting Ceph Storage Solutions
This document discusses reference architectures for Ceph storage solutions. It provides guidance on key design considerations for Ceph clusters, including workload profiling, storage access methods, capacity planning, fault tolerance, and data protection schemes. Example hardware configurations are also presented for different performance and cost optimization targets.
HDFEOS.org User Analsys, Updates, and Future
This document discusses user analysis of the HDFEOS.org website and plans for future improvements. It finds that the majority of the site's 100 daily users are "quiet", not posting on forums or other interactive elements. The main user types are locators, who search for examples or data; mergers, who combine or mosaic datasets; and converters, who change file formats. The document outlines recent updates focused on these user types, like adding Python examples for subsetting and calculating latitude and longitude. It proposes future work on artificial intelligence/machine learning uses of HDF files and examples for processing HDF data in the cloud.
Data Are from Mars, Tools Are from Venus
This document discusses challenges with data and tools coming from different "planets" (Mars and Venus), frustrating users. It identifies gaps in file formats, libraries, tools, and services for working with Earth science data. The author provides solutions through the HDFEOS.org project, including file format conversion and examples, and suggestions for data producers, tool developers, and end users to help address interoperability issues. Emerging topics like big data, machine learning, and voice assistants are also discussed.
How to Increase Performance of Your Hadoop Cluster
Three identical Apache Hadoop clusters were provisioned on Joyent infrastructure using different operating systems: SmartOS, Ubuntu, and KVM virtual machines. Monitoring showed the Ubuntu and KVM clusters spent more time in the OS kernel during I/O operations compared to the SmartOS cluster. The SmartOS cluster was able to utilize CPU resources more efficiently and scale to more mappers and reducers. Basic cluster configuration and tuning the number of map and reduce tasks are important to optimize Hadoop performance.
Apache Spark on Supercomputers: A Tale of the Storage Hierarchy with Costin I...
In this session, the speakers will discuss their experiences porting Apache Spark to the Cray XC family of supercomputers. One scalability bottleneck is in handling the global file system present in all large-scale HPC installations. Using two techniques (file open pooling, and mounting the Spark file hierarchy in a specific manner), they were able to improve scalability from O(100) cores to O(10,000) cores. This is the first result at such a large scale on HPC systems, and it had a transformative impact on research, enabling their colleagues to run on 50,000 cores.
With this baseline performance fixed, they will then discuss the impact of the storage hierarchy and of the network on Spark performance. They will contrast a Cray system with two levels of storage with a “data intensive” system with fast local SSDs. The Cray contains a back-end global file system and a mid-tier fast SSD storage. One conclusion is that local SSDs are not needed for good performance on a very broad workload, including spark-perf, TeraSort, genomics, etc.
They will also provide a detailed analysis of the impact of latency of file and network I/O operations on Spark scalability. This analysis is very useful to both system procurements and Spark core developers. By examining the mean/median value in conjunction with variability, one can infer the expected scalability on a given system. For example, the Cray mid-tier storage has been marketed as the magic bullet for data intensive applications. Initially, it did improve scalability and end-to-end performance. After understanding and eliminating variability in I/O operations, they were able to outperform any configurations involving mid-tier storage by using the back-end file system directly. They will also discuss the impact of network performance and contrast results on the Cray Aries HPC network with results on InfiniBand.
Democratizing Memory Storage
The document discusses improvements to HDFS that allow it to leverage memory as a storage medium. Key points include:
- HDFS 2.3 introduced memory as a storage medium, with RAM disks providing persistence across restarts.
- HDFS 2.6 introduced storage policies that allow applications to target different storage media like SSD or memory.
- The Centralized Cache Management feature loads hot data into memory pools to enable zero-copy reads.
- The Lazy Persist Writes feature allows applications to write to memory and have HDFS asynchronously write to persistent storage, reducing latency.
- Future work includes improving caching, short-circuit writes, and the Memfs layered file system to provide more flexible
IBM SONAS and the Cloud Storage Taxonomy
This document discusses IBM's Scale Out Network Attached Storage (SONAS) solution. SONAS provides a global namespace and can scale to support large amounts of unstructured file data across various cloud environments. The document outlines how SONAS utilizes IBM's General Parallel File System (GPFS) and provides features such as high performance, data replication, backups, antivirus integration, and information lifecycle management through migration to tape storage.
Caching and Buffering in HDF5
In this talk we will discuss caching and buffering strategies in HDF5. The information presented will help developers write more efficient applications and avoid performance bottlenecks.
Apache hadoop technology : Beginners
This presentation is about apache hadoop technology. It may be helpful for the beginners to know some terminologies of hadoop.
Google Colaboratory for HDF-EOS
Google Colaboratory allows users to write and execute Python code in the cloud using Jupyter notebooks. It provides a free GPU and TPU for accelerating code. The document discusses how HDF-EOS is a standard format for satellite data and provides many examples for converting and processing HDF-EOS data. It then demonstrates how to install necessary packages and run an example zoo code in Colab to plot HDF-EOS data in the cloud without installing anything locally.
2.introduction to hdfs
This document provides an introduction to HDFS (Hadoop Distributed File System). It discusses what HDFS is, its core components, architecture, and key elements like the NameNode, metadata, and blocks. HDFS is designed for storing very large files across commodity hardware in a fault-tolerant manner and allows for streaming access. While HDFS can handle small datasets, its real power is with large and distributed data.
Aziksa hadoop architecture santosh jha
Hadoop is an open-source framework for distributed storage and processing of large datasets across clusters of commodity hardware. It has several core components including HDFS for distributed file storage and MapReduce for distributed processing. HDFS stores data across clusters of machines with replication for fault tolerance. MapReduce allows parallel processing of large datasets in a distributed manner. Hadoop was designed with goals of using commodity hardware, easy recovery from failures, large distributed file systems, and fast processing of large datasets.
HBase with MapR
- MapR makes it easy to deploy and run HBase in production by providing high availability, data protection, and disaster recovery features out of the box. This includes setting up hourly snapshots and cross-datacenter mirroring with just a few commands.
- With MapR, all HBase data should be stored in a single volume called "hbase.volume" for easy management. Snapshots are taken of this volume to enable consistent backups with no downtime. The snapshots can be mirrored to other clusters for disaster recovery.
- Memory usage on RegionServers is automatically tuned by MapR's Warden service based on configuration. The defaults generally work well but guidelines are provided for tuning in large deployments.
What's hot (20)
Reference Architecture: Architecting Ceph Storage Solutions
Reference Architecture: Architecting Ceph Storage Solutions
How to Increase Performance of Your Hadoop Cluster
How to Increase Performance of Your Hadoop Cluster
Apache Spark on Supercomputers: A Tale of the Storage Hierarchy with Costin I...
Apache Spark on Supercomputers: A Tale of the Storage Hierarchy with Costin I...
Viewers also liked
Earth Science Markup Language (ESML) - A Tutorial
This document provides an overview of the Earth Science Markup Language (ESML). ESML is an XML-based interchange format that allows applications and services to access heterogeneous earth science data regardless of the underlying data format. It provides syntactic, semantic, and content metadata that describe data structures, meanings, and contents in a machine-readable way. The document outlines the need for such an interchange format, describes the components of ESML including the schema, libraries, and tools, and provides examples of writing ESML descriptions for different types of data files.
Status of HDF-EOS, Related Software, and Tools
Current status of HDF-EOS and access tools will be summarized. Update on HDF-EOS, HDFView plug-in and The HDF-EOS to GeoTIFF (HEG) conversion tool, including recent changes to the software, ongoing maintenance, upcoming releases, future plans, and issues will be discussed.
McIDAS Lite
This document discusses McIDAS-Lite, a software tool developed by the Space Science and Engineering Center (SSEC) for browsing and analyzing satellite data. McIDAS-Lite allows users to view local or remote data from satellites like GOES and MODIS in formats like JPEG and netCDF. It provides basic geolocation, analysis functions and image enhancement capabilities. Since its initial release in 2002, McIDAS-Lite has gained popularity with over 1300 data requests in the last 3 months and is available across various Unix and Windows platforms.
Aggregation and Subsetting in ERDDAP
This document discusses how the ERDDAP data server can aggregate and subset gridded and in-situ data. It aggregates multiple files containing time series or variable data into single virtual datasets. This allows users to access all data through a single query. It also supports subsetting aggregated data spatially and temporally. For gridded data, subsetting uses indices while for in-situ/tabular data it uses selection constraints on variables. The advantages are users only have to find and query one aggregated dataset instead of many files, saving significant time and effort.
NASA HDF and HDF-EOS Status - Use in EOSDIS
The document summarizes HDF and HDF-EOS, data formats used by NASA's Earth Observing System Data and Information System (EOSDIS). EOSDIS acquires, archives, manages, and distributes Earth observation data from NASA satellites and other sources. It uses HDF and the EOS-specific profile HDF-EOS as its standard data formats. HDF-EOS facilitates data storage, access, analysis and discovery. The document provides background on HDF-EOS, its history and goals, experience using it, and the goals of the HDF/HDF-EOS workshop being summarized.
The Web-based Hierarchical Ordering Mechanism (WHOM) - a tool for ordering HD...
The Web-based Hierarchical Ordering Mechanism (WHOM) - a tool for ordering HD...The HDF-EOS Tools and Information Center
This document describes the Web Hierarchical Ordering Mechanism (WHOM), a tool for ordering HDF and HDF-EOS data from the Goddard Earth Sciences Distributed Active Archive Center (GES DAAC). WHOM provides a web-based interface for navigating, searching, subsetting, and ordering NASA Earth science data held by the GES DAAC. It allows users to easily identify, subset, and order data of interest through a hierarchical browse of data products, calendar and map search tools, and a shopping cart system. The document provides an overview of WHOM's capabilities and interface, using MODIS and ocean data products as examples, and describes other HDF data processing and visualization tools developed by the GES DAAC.Earth Science Markup Language
ESML is a specialized markup language for Earth Science metadata based on XML.
Source:http://hdfeos.org/workshops/ws07/presentations/Smith1/Smith_ESML_HDF-EOS.ppt
HDF-EOS Subsetting: HEW and other tools
This document summarizes subsetting tools and methods for Earth science data in HDF-EOS format. It discusses the HDF-EOS Web-based Subsetter (HEW) tool, the HEW Subsetting Engine (HSE) library, other customized subsetting applications for datasets like MODIS, and plans to integrate subsetting into the EOSDIS Core System. The document provides an overview of subsetting goals, steps, currently available tools, and a demonstration of the HEW tool.
Generalized EOS Data Converter: Making Data Products Accessible to GIS Tools
Generalized EOS Data Converter: Making Data Products Accessible to GIS ToolsThe HDF-EOS Tools and Information Center
The document describes a tool called the HDF-EOS to GeoTIFF conversion tool (HEG) that was developed to convert science data files in HDF-EOS format to GeoTIFF files. GeoTIFF files can be directly accessed by many GIS tools, unlike HDF-EOS files. HEG allows individual users to convert a wide variety of EOS data products, including MODIS, MISR, and ASTER data. It provides functionality like mosaicking, subsetting, reprojection, and metadata creation. HEG has both a graphical and command line interface and runs on Windows, Linux, Sun, and SGI platforms. It has been integrated into online NASA data archives to help make earth science data more accessible.Integrating HDF5 with SRB
Fast partial access to objects from very large files in the SDSC Storage Resource Broker (SRB5) can be extremely challenging, even when those objects are small. The HDF-SRB project integrates the SRB and NCSA Hierarchical Data Format (HDF5), to create an access mechanism within the SRB that is can be orders of magnitude more efficient than current methods for accessing object-based file formats.
The project provides interactive and efficient access to datasets or subsets of datasets in large files without bringing entire files into local machines. A new set of data structures and APIs have been implemented to the SRB support such object-level data access. A working prototype of the HDF5-SRB data system has been developed and tested. The SRB support is implemented in HDFView as a client application.
Introduction to HDFLook_MODIS
HDFLook is a software tool developed jointly by NASA GSFC and LOA USTL, France that allows users to view and analyze Earth science datasets. It provides capabilities to access, visualize, remap, reproject, subset, mosaic and convert files for MODIS, AIRS, and CERES products. HDFLook can be run in interactive, operational and batch modes and supports accessing geophysical and ancillary data for scientific analysis and operational uses. Over 3000 copies of HDFLook have been distributed worldwide to support the user community.
Access HDF-EOS data with OGC Web Coverage Service - Earth Observation Applica...
Access HDF-EOS data with OGC Web Coverage Service - Earth Observation Applica...The HDF-EOS Tools and Information Center
This document discusses accessing Earth observation data through the OGC Web Coverage Service (WCS) 2.0 with an Earth Observation Application Profile (EO AP). It describes how the WCS EO AP maps Earth observation terminology to the WCS model, outlines the implementation of the WCS EO AP including supported data formats and products, and discusses future work such as adding more data support and integrating with other OGC services.Subsetting at UAH
The presentation will provide an overview of subsetting software development activity at UAH. Updates have been made to all packages, reflecting the latest versions of HDF5 and HE5. The library of tools (HSE) for subsetting HDF-EOS data is up-to-date for SGI, Sun, and Linux platforms. Subsetting software is operational at NSIDC DAAC and GDAAC, in testing at LPDAAC. Ongoing work and plans will also be described, including row/column subsetting and index subsampling.
ORNL DAAC MODIS Land Product Subsets
MODIS (Moderate Resolution Imaging Spectroradiometer) sensor data are highly useful for field research. However, the volume of MODIS data and the complexity in data format makes MODIS data less usable for some communities. To expand the use of MODIS data beyond traditional remote sensing specialists, the Oak Ridge National Laboratory (ORNL) Distributed Active Archive Center (DAAC) prepares and distributes subsets of selected Land Products in a scale and format useful for undergraduate students and field researchers. MODIS subsets are provided for more than 1,000 sites across the globe. The subsets are offered in tabular ASCII format and in GIS compatible GeoTIFF format. Time series plots and grid visualizations to help characterize field sites are also provided. In addition to offering subsets for fixed sites, the ORNL DAAC also offers the capability to create user-defined subsets for any location worldwide. The MODIS Global subsetting tool provides subsets from a single pixel up to 201 x 201 km for user-defined time range. Statistics, time series plots and GIS compatible files for the customized subsets are also distributed through this tool. Users can also programmatically retrieve the subsets through a SOAP based Web Service.
Identity, Location, and Citation at NEON
Latest version of an oft-given talk/discussion about data citation and related issues, presented to data scientists at HQ of the National Ecological Observatory Network
HDF5 iRODS
Numerous scientific teams use the HDF5 format to store very large datasets. Efficient use of this data in a distributed environment depends on client applications being able to read any subset of the data without transferring the entire file to the local machine. The goal of the HDF5-iRODS Project was to develop an HDF5-iRODS module for the iRODS datagrid server that supported this capability, and to apply the technology to an NCSA/SDSC Strategic Applications Program (SAP) project, FLASH.
A joint team from The HDF Group (representing NCSA) and the SDSC SRB group collaborated to accomplish the project goal. The team implemented five HDF5 microservices functions on the iRODS server, and developed an iRODS FLASH slice client application. The client implementation also includes a JNI interface that allows HDFView, a standard tool for browsing HDF5 files, to access HDF5 files stored remotely in iRODS. Finally, three new collection client/server calls were added to the iRODS APIs, making it easier for users to query the content of an iRODS collection.
HDF5 Advanced Topics - Datatypes and Partial I/O
It will cover features of the HDF5 library for achieving better I/O performance and efficient storage. The following HDF5 features will be discussed: datatype and partial I/O
This tutorial is for persons who are already familiar with HDF5 and wish to take advantage is some of its advanced features.
Access HDF5 Datasets via OPeNDAP's Data Access Protocol (DAP)
Access HDF5 Datasets via OPeNDAP's Data Access Protocol (DAP)The HDF-EOS Tools and Information Center
The Data Access Protocol(DAP) is a protocol for requesting and transporting data across the web. By using OPeNDAP servers data can be made accessible to remote locations regardless of local storage format. A prototyped HDF5-OPeNDAP server was implemented several years ago. With the HDF5 being selected as the primary data format of NASA Aura satellite, a robust, fully supported HDF5-OPeNDAP server is necessary.
Recently NASA Headquarters Science Mission Directorate Advancing Collaborative Connections For Earth-Sun System Science (ACCESS) program has awarded a joint proposal to implement a fully supported HDF5-OPeNDAP server. This poster will discuss the design and implementation plan for this project. HDF Update
This document summarizes HDF software activities in 2002, including support and funding sources for HDF, recent and upcoming releases of HDF4 and HDF5 libraries and tools, and other HDF-related projects. The HDF5 library saw improvements to performance, compilers supported, and tools like HDFView and converters. The next major HDF5 release in 2003 will focus on new features, performance enhancements, and special platform support. High level APIs and the parallel HDF5 programming model were also under development.
HDF Update
An update on HDF, including a status report on the HDF Group, an overview of recent changes to the HDF4 and HDF5 libraries and tools, plans for future releases, HDF Group projects and collaborations, and future plans.
Viewers also liked (20)
The Web-based Hierarchical Ordering Mechanism (WHOM) - a tool for ordering HD...
The Web-based Hierarchical Ordering Mechanism (WHOM) - a tool for ordering HD...
Generalized EOS Data Converter: Making Data Products Accessible to GIS Tools
Generalized EOS Data Converter: Making Data Products Accessible to GIS Tools
Access HDF-EOS data with OGC Web Coverage Service - Earth Observation Applica...
Access HDF-EOS data with OGC Web Coverage Service - Earth Observation Applica...
Access HDF5 Datasets via OPeNDAP's Data Access Protocol (DAP)
Access HDF5 Datasets via OPeNDAP's Data Access Protocol (DAP)
Similar to HDF-EOS Development Current Status
HDF-EOS APIs, tools, etc.
HDF-EOS is a standard developed by NASA for storing Earth science data in the Hierarchical Data Format (HDF). It defines structures for gridded data, swaths, points, and other arrays. The HDF-EOS APIs provide functions for reading, writing, and manipulating these data structures and attributes in both HDF4 and HDF5 formats. Tools like HDFView and the HDF-EOS to GeoTIFF Converter (HEG) allow users to browse, subset, and convert HDF-EOS data to other formats.
HDF-EOS 3.0 Functional and Structural Design
This document discusses the design of HDF-EOS 3.0, which is based on HDF5 and improves on HDF-EOS 2.x. It overviews the functionality, including I/O, attributes, compression, chunking, and interfaces for swaths, grids, profiles, and points. The file structure is shown, with the root containing HDFEOS, HDFEOS INFORMATION, SWATHS, GRIDS, and POINTS groups. Swath, grid, profile and point structures each allow global, group and local attributes on fields. Some issues around data types, chunking and compression are also noted.
HDF-EOS Tools
The document describes HDF-EOS tools developed to help users work with Earth Observing System (EOS) data stored in HDF-EOS format. It outlines libraries and tools created, including the HDF-EOS to GeoTIFF Converter (HEG) tool, which can convert HDF-EOS files to GeoTIFF and perform subsetting, stitching, and reprojection. It also describes the HDF-EOS plugin for HDFView, which allows browsing of HDF-EOS data and objects. The tools were created to make HDF-EOS data more accessible to users and reduce the need to learn HDF/HDF-EOS libraries.
HDF-EOS Maintenance, Current Development and Tools
We will summarize current status of HDF-EOS and associated tools. Update on HDF-EOS, HDFView plug-in and The HDF-EOS to GeoTIFF (HEG) conversion tool, including recent changes to the software, ongoing maintenance, upcoming releases, future plans, and issues will be discussed.
We will also summarize the status of HDF-EOS RFC. The HDF-EOS plug-in for the THG-developed tool, HDFView, has been enhanced. The plug-in offers browse capability for both HDF4 and HDF5 - based HDF-EOS files. HDFView can also process vanilla HDF4 and HDF5 files. New features including support for Point and Zonal Average objects have been added. A port to Mac OS X version will be available in next release.
The HDF-EOS to GeoTIFF (HEG) conversion tool has been augmented to include new projections, and support for additional AMSR-E and AIRS products. Subsetting features have also been augmented. The tool is available in both stand-alone and EOS DAAC online versions.
Hdf eos status-workshp_xi_nov_2007
The document discusses the current status and future plans for HDF-EOS, a software toolkit for working with Earth science data files. It notes that maintenance support is funded through May 2009, with one full-time and one half-time developer. Major software components like the SDP Toolkit, HDF-EOS2, HDF-EOS5, and the HDF-EOS to GeoTIFF Converter are maintained and new releases planned. Future development focuses on auto-configuration, 64-bit support, and improved tools for viewing and converting HDF-EOS files.
HDF-EOS Workshop II Introduction
This document introduces an HDF-EOS workshop for data producers, users, and tool developers. HDF-EOS is the baseline standard data format for EOS data, based on HDF but adding features for EOS. The workshop aims to provide in-depth information about HDF and HDF-EOS through hands-on tutorials for tools, utilities, and programming. It is sponsored by NASA and the ECS team, and provides individual consultation. HDF-EOS supports a large community that will need various analysis tools, and resources like documentation, sample data, and experts are available to support the HDF-EOS format.
HDF-EOS to GeoTIFF Conversion Tool and HDF-EOS Plug-in for HDFView
HDF-EOS to GeoTIFF Conversion Tool and HDF-EOS Plug-in for HDFViewThe HDF-EOS Tools and Information Center
HDF-EOS to GeoTIFF (HEG) conversion tool is a versatile tool that besides converting HDF-EOS to GeoTIFF it subsets, resamples, convert swath to grid or grids to different projections, subsamples, or does mosaicing. HDF-EOS plug-in is a library that extends HDFView, a visual tool for browsing and editing HDF4 and HDF5 files, for EOS applications. These tools will be discussed in detail and a demo will be presented.HDF-EOS Overview and Status
This document provides an overview of HDF-EOS, which is an extension to HDF that defines standard data structures for remote sensing and in-situ data with tightly coupled geolocation information. It describes the core components of HDF-EOS files, including Grid, Swath, and Point structures, and provides examples. It also outlines the development of an HDF5-based version to overcome limitations of the HDF4-based library and allow for larger files.
The Landsat 7 Processing System (LPS) Level Zero-R Science Products
The Landsat 7 Processing System (LPS) Level Zero-R Science ProductsThe HDF-EOS Tools and Information Center
The Landsat 7 Processing System (LPS) processes Landsat 7 data into science data products. It receives raw telemetry from the Landsat 7 satellite and constructs images from the data without applying corrections. The LPS divides the data into subintervals and produces multi-band files and metadata for each. It also generates browse images and assesses cloud cover for each subinterval. The LPS data products are separated into Format 1 and Format 2 files representing the two types of raw data received by the LPS. The Level Zero-R science products created by the LPS support further processing but are not yet calibrated or rectified.Hadoop security
This document provides an overview of securing Hadoop applications and clusters. It discusses authentication using Kerberos, authorization using POSIX permissions and HDFS ACLs, encrypting HDFS data at rest, and configuring secure communication between Hadoop services and clients. The principles of least privilege and separating duties are important to apply for a secure Hadoop deployment. Application code may need changes to use Kerberos authentication when accessing Hadoop services.
DAOS Middleware overview
DAOS (Distributed Application Object Storage) is a high-performance storage architecture and software stack that delivers scalable object storage capabilities. It uses Intel Optane memory and NVMe SSDs to provide high IOPS, bandwidth, and low latency storage. DAOS supports various data models and interfaces like POSIX, HDF5, Spark, and Python. It allows applications to access storage with library calls instead of system calls for high performance.
HDF-EOS to GeoTIFF Conversion Tool & HDF-EOS Plug-in for HDFView
HDF-EOS to GeoTIFF Conversion Tool & HDF-EOS Plug-in for HDFViewThe HDF-EOS Tools and Information Center
This document describes tools for working with HDF-EOS data formats: the HDF-EOS to GeoTIFF Conversion Tool (HEG) and an HDF-EOS plugin for the HDFView browser. HEG allows conversion between HDF-EOS and GeoTIFF formats and supports subsetting, reprojection, stitching and other functions. The HDFView plugin allows browsing and conversion between HDF-EOS2 and HDF-EOS5 formats within HDFView. Both tools are available for download and support a variety of Earth science datasets.Introduction to HDF5
This tutorial is designed for new HDF5 users. We will go over a brief history of HDF and HDF5 software, and will cover basic HDF5 Data Model objects and their properties; we will give an overview of the HDF5 Libraries and APIs, and discuss the HDF5 programming model. Simple C and Fortran examples, and Java tool HDFView will be used to illustrate HDF5 concepts.
Status of HDF-EOS, Related Software, and Tools
HDF-EOS is a software library designed to support NASA Earth Observing System (EOS) science data, built on top of HDF (Hierarchical Data Format) developed by The HDF Group. Current status of HDF-EOS and access tools will be summarized and updates on TOOLKIT, HDFView plug-in and the HDF-EOS to GeoTIFF (HEG) conversion tool, including recent changes to the software, ongoing maintenance, upcoming releases, and issues will be discussed.
Global Persistence for Docker
The document discusses using EMC ECS for geo-replicated Docker registries and multi-protocol access for microservice applications. It describes how ECS provides an active/active geo-distributed architecture for object storage that can be used to replicate Docker registries across multiple sites for high availability. It also explains how ECS supports multiple protocols like S3, Swift, HDFS, and NFS that can be used by different microservices in an application to access data stored in ECS. There is a demo of a geo-replicated Docker registry using ECS and an example of a MosaicMe application that uses ECS for multi-protocol data access across microservices.
Generalized Conversion of HDF-EOS Products to GIS-Compatible Formats
Generalized Conversion of HDF-EOS Products to GIS-Compatible FormatsThe HDF-EOS Tools and Information Center
This document discusses the development of a tool to convert Earth Observing System (EOS) data products stored in HDF-EOS format to more widely used Geographic Information System (GIS) compatible formats like GeoTIFF. The tool aims to make EOS data more accessible to GIS users by handling various issues with HDF-EOS like different geolocation standards, allowing mosaic-ing of products across files, and integrating with common analysis programs. It will convert HDF-EOS swath and grid products to GeoTIFF, allow for reprojection, subsetting, stitching, and preservation of metadata. The tool is being developed with graphical and command line interfaces using Java and C to work across platforms and without proprietary software dependencies.EAS Data Flow lessons learnt
1. The document describes the data flow and file handling in a mock implementation of the Infrastructure Abstraction Layer (IAL) for the Euclid Archive System (EAS). It notes two key lessons learned: (1) file URLs need to be abstracted to reduce errors, and (2) reducing unnecessary file copies between systems.
2. It proposes a new Euclid File Access Service (EuFAS) that would provide a standardized way to look up, replicate, and manage file access across the EAS components through an abstract interface. This would replace direct file URLs and simplify data flow.
3. The service would also reduce unnecessary file copies by allowing processing to occur directly from cached copies on execution
Aura HDF-EOS File Format Guidelines: Overview and Status
The instrument teams on Aura developed a data format standard for their data files using HDF-EOS5 as the storage mechanism. Now that data files are being shared between instrument teams, the success of this endeavor can be studied. Details of this standard, the process used for creating it, what we've learned and suggestions for future missions will be presented.
Status of HDF-EOS, Related Software and Tools
Status of HDF-EOS and access tools will be summarized. Updates on HDF-EOS, TOOLKIT, HDFView plug-in and The HDF-EOS to GeoTIFF (HEG) conversion tool, including recent changes to the software, ongoing maintenance, upcoming releases, future plans, and issues will be discussed.
An IDL-Based Validation Toolkit: Extensions to use the HDF-EOS Swath Format
An IDL-Based Validation Toolkit: Extensions to use the HDF-EOS Swath FormatThe HDF-EOS Tools and Information Center
This document discusses extensions made to an IDL-based validation toolkit to allow it to work with HDF-EOS swath format files. The extensions addressed problems with the toolkit's previous native format by making it faster and easier to work with satellite data files. While the HDF-EOS format presented some challenges, it provides a standard way of storing atmospheric profile data. Lessons learned include developing generic swath reading tools and working to easily select variables from swaths. Future work may include expanding the data mapping capabilities and developing interfaces for other HDF formats.Similar to HDF-EOS Development Current Status (20)
HDF-EOS Maintenance, Current Development and Tools
HDF-EOS Maintenance, Current Development and Tools
HDF-EOS to GeoTIFF Conversion Tool and HDF-EOS Plug-in for HDFView
HDF-EOS to GeoTIFF Conversion Tool and HDF-EOS Plug-in for HDFView
The Landsat 7 Processing System (LPS) Level Zero-R Science Products
The Landsat 7 Processing System (LPS) Level Zero-R Science Products
HDF-EOS to GeoTIFF Conversion Tool & HDF-EOS Plug-in for HDFView
HDF-EOS to GeoTIFF Conversion Tool & HDF-EOS Plug-in for HDFView
Generalized Conversion of HDF-EOS Products to GIS-Compatible Formats
Generalized Conversion of HDF-EOS Products to GIS-Compatible Formats
Aura HDF-EOS File Format Guidelines: Overview and Status
Aura HDF-EOS File Format Guidelines: Overview and Status
An IDL-Based Validation Toolkit: Extensions to use the HDF-EOS Swath Format
An IDL-Based Validation Toolkit: Extensions to use the HDF-EOS Swath Format
More from The HDF-EOS Tools and Information Center
Cloud-Optimized HDF5 Files
This document discusses how to optimize HDF5 files for efficient access in cloud object stores. Key optimizations include using large dataset chunk sizes of 1-4 MiB, consolidating internal file metadata, and minimizing variable-length datatypes. The document recommends creating files with paged aggregation and storing file content information in the user block to enable fast discovery of file contents when stored in object stores.
Accessing HDF5 data in the cloud with HSDS
This document provides an overview of HSDS (Highly Scalable Data Service), which is a REST-based service that allows accessing HDF5 data stored in the cloud. It discusses how HSDS maps HDF5 objects like datasets and groups to individual cloud storage objects to optimize performance. The document also describes how HSDS was used to improve access performance for NASA ICESat-2 HDF5 data on AWS S3 by hyper-chunking datasets into larger chunks spanning multiple original HDF5 chunks. Benchmark results showed that accessing the data through HSDS provided over 2x faster performance than other methods like ROS3 or S3FS that directly access the cloud storage.
The State of HDF
This document summarizes the current status and focus of the HDF Group. It discusses that the HDF Group is located in Champaign, IL and is a non-profit organization focused on developing and maintaining HDF software and data formats. It provides an overview of recent HDF5, HDF4 and HDFView releases and notes areas of focus for software quality improvements, increased transparency, strengthening the community, and modernizing HDF products. It invites support and participation in upcoming user group meetings.
Highly Scalable Data Service (HSDS) Performance Features
This document provides an overview of HSDS (HDF Server and Data Service), which allows HDF5 files to be stored and accessed from the cloud. Key points include:
- HSDS maps HDF5 objects like datasets and groups to individual cloud storage objects for scalability and parallelism.
- Features include streaming support, fancy indexing for complex queries, and caching for improved performance.
- HSDS can be deployed on Docker, Kubernetes, or AWS Lambda depending on needs.
- Case studies show HSDS is used by organizations like NREL and NSF to make petabytes of scientific data publicly accessible in the cloud.
Creating Cloud-Optimized HDF5 Files
This document discusses creating cloud-optimized HDF5 files by rearranging internal structures for more efficient data access in cloud object stores. It describes cloud-native and cloud-optimized storage formats, with the latter involving storing the entire HDF5 file as a single object. The benefits of cloud-optimized HDF5 include fast scanning and using the HDF5 library. Key aspects covered include using optimal chunk sizes, compression, and minimizing variable-length datatypes.
HDF5 OPeNDAP Handler Updates, and Performance Discussion
This document discusses updates and performance improvements to the HDF5 OPeNDAP data handler. It provides a history of the handler since 2001 and describes recent updates including supporting DAP4, new data types, and NetCDF data models. A performance study showed that passing compressed HDF5 data through the handler without decompressing/recompressing led to speedups of around 17-30x by leveraging HDF5 direct I/O APIs. This allows outputting HDF5 files as NetCDF files much faster through the handler.
Hyrax: Serving Data from S3
This document provides instructions for using the Hyrax software to serve scientific data files stored on Amazon S3 using the OPeNDAP data access protocol. It describes how to generate ancillary metadata files called DMR++ files using the get_dmrpp tool that provide information about the data file structure and locations. The document explains how to run get_dmrpp inside a Docker container to process data files on S3 and generate customized DMR++ files that the Hyrax server can use to serve the files to clients.
Accessing Cloud Data and Services Using EDL, Pydap, MATLAB
This document provides an overview and examples of accessing cloud data and services using the Earthdata Login (EDL), Pydap, and MATLAB. It discusses some common problems users encounter, such as being unable to access HDF5 data on AWS S3 using MATLAB or read data from OPeNDAP servers using Pydap. Solutions presented include using EDL to get temporary AWS tokens for S3 access in MATLAB and providing code examples on the HDFEOS website to help users access S3 data and OPeNDAP services. The document also notes some limitations, such as tokens being valid for only 1 hour, and workarounds like requesting new tokens or using the MATLAB HDF5 API instead of the netCDF API.
HDF - Current status and Future Directions
The HDF5 Roadmap and New Features document outlines upcoming changes and improvements to the HDF5 library. Key points include:
- HDF5 1.13.x releases will include new features like selection I/O, the Onion VFD for versioned files, improved VFD SWMR for single-writer multiple-reader access, and subfiling for parallel I/O.
- The Virtual Object Layer allows customizing HDF5 object storage and introduces terminal and pass-through connectors.
- The Onion VFD stores versions of HDF5 files in a separate onion file for versioned access.
- VFD SWMR improves on legacy SWMR by implementing single-writer multiple-reader capabilities
HDF - Current status and Future Directions
This document summarizes a presentation about the current status and future directions of the Hierarchical Data Format (HDF) software. It provides updates on recent HDF5 releases, development efforts including new compression methods and ways to access HDF5 data, and outreach resources. It concludes by inviting the audience to share wishes for future HDF development.
H5Coro: The Cloud-Optimized Read-Only Library
The document describes H5Coro, a new C++ library for reading HDF5 files from cloud storage. H5Coro was created to optimize HDF5 reading for cloud environments by minimizing I/O operations through caching and efficient HTTP requests. Performance tests showed H5Coro was 77-132x faster than the previous HDF5 library at reading HDF5 data from Amazon S3 for NASA's SlideRule project. H5Coro supports common HDF5 elements but does not support writing or some complex HDF5 data types and messages to focus on optimized read-only performance for time series data stored sequentially in memory.
MATLAB Modernization on HDF5 1.10
This document summarizes MathWorks' work to modernize MATLAB's support for HDF5. Key points include:
1) MATLAB now supports HDF5 1.10.7 features like single-writer/multiple-reader access and virtual datasets through new and updated low-level functions.
2) Performance benchmarks show some improvements but also regressions compared to the previous HDF5 version, and work continues to optimize code and support future versions.
3) There are compatibility considerations for Linux filter plugins, but interim solutions are provided until MathWorks can ship a single HDF5 version.
HDF for the Cloud - Serverless HDF
HSDS provides HDF as a service through a REST API that can scale across nodes. New releases will enable serverless operation using AWS Lambda or direct client access without a server. This allows HDF data to be accessed remotely without managing servers. HSDS stores each HDF object separately, making it compatible with cloud object storage. Performance on AWS Lambda is slower than a dedicated server but has no management overhead. Direct client access has better performance but limits collaboration between clients.
HDF5 <-> Zarr
HDF5 and Zarr are data formats that can be used to store and access scientific data. This presentation discusses approaches to translating between the two formats. It describes how HDF5 files were translated to the Zarr format by creating a separate Zarr store to hold HDF5 file chunks, and storing chunk location metadata. It also discusses an implementation that translates Zarr data to the HDF5 format by using a special chunking layout and storing chunk information in an HDF5 compound dataset. Limitations of the translations include lack of support for some HDF5 dataset properties in Zarr, and lack of support for some Zarr compression methods in the HDF5 implementation.
HDF for the Cloud - New HDF Server Features
The document discusses HDF for the cloud, including new features of the HDF Server and what's next. Key points:
- HDF Server uses a "sharded schema" that maps HDF5 objects to individual storage objects, allowing parallel access and updates without transferring entire files.
- Implementations include HSDS software that uses the sharded schema with an API and SDKs for different languages like h5pyd for Python.
- New features of HSDS 0.6 include support for POSIX, Azure, AWS Lambda, and role-based access control.
- Future work includes direct access to storage without a server intermediary for some use cases.
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3The HDF-EOS Tools and Information Center
This document compares different methods for accessing HDF and netCDF files stored on Amazon S3, including Apache Drill, THREDDS Data Server (TDS), and HDF5 Virtual File Driver (VFD). A benchmark test of accessing a 24GB HDF5/netCDF-4 file on S3 from Amazon EC2 found that TDS performed the best, responding within 2 minutes, while Apache Drill failed after 7 minutes. The document concludes that TDS 5.0 is the clear winner based on performance and support for role-based access control and HDF4 files, but the best solution depends on use case and software.STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...
STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...The HDF-EOS Tools and Information Center
This document discusses STARE-PODS, a proposal to NASA/ACCESS-19 to develop a scalable data store for earth science data using the SpatioTemporal Adaptive Resolution Encoding (STARE) indexing scheme. STARE allows diverse earth science data to be unified and indexed, enabling the data to be partitioned and stored in a Parallel Optimized Data Store (PODS) for efficient analysis. The HDF Virtual Object Layer and Virtual Data Set technologies can then provide interfaces to access the data in STARE-PODS in a familiar way. The goal is for STARE-PODS to organize diverse data for alignment and parallel/distributed storage and processing to enable integrative analysis at scale.HDF5 and Ecosystem: What Is New?
This document provides an overview and update on HDF5 and its ecosystem. Key points include:
- HDF5 1.12.0 was recently released with new features like the Virtual Object Layer and external references.
- The HDF5 library now supports accessing data in the cloud using connectors like S3 VFD and REST VOL without needing to modify applications.
- Projects like HDFql and H5CPP provide additional interfaces for querying and working with HDF5 files from languages like SQL, C++, and Python.
- The HDF5 community is moving development to GitHub and improving documentation resources on the HDF wiki site.
HDF5 Roadmap 2019-2020
This document summarizes new features in HDF5 1.12.0, including support for storing references to objects and attributes across files, new storage backends using a virtual object layer (VOL), and virtual file drivers (VFDs) for Amazon S3 and HDFS. It outlines the HDF5 roadmap for 2019-2022, which includes continued support for HDF5 1.8 and 1.10, and new features in future 1.12.x releases like querying, indexing, and provenance tracking.
Leveraging the Cloud for HDF Software Testing
The document discusses leveraging cloud resources like Amazon Web Services to improve software testing for the HDF group. Currently HDF software is tested on various in-house systems, but moving more testing to the cloud could provide better coverage of operating systems and distributions at a lower cost. AWS spot instances are being used to run HDF5 build and regression tests across different Linux distributions in around 30 minutes for approximately $0.02 per hour.
More from The HDF-EOS Tools and Information Center (20)
Highly Scalable Data Service (HSDS) Performance Features
Highly Scalable Data Service (HSDS) Performance Features
HDF5 OPeNDAP Handler Updates, and Performance Discussion
HDF5 OPeNDAP Handler Updates, and Performance Discussion
Accessing Cloud Data and Services Using EDL, Pydap, MATLAB
Accessing Cloud Data and Services Using EDL, Pydap, MATLAB
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3
Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3
STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...
STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...
Recently uploaded
Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays, June 2024 with Maria Copot 20
Choosing The Best AWS Service For Your Website + API.pptx
Have you ever been confused by the myriad of choices offered by AWS for hosting a website or an API?
Lambda, Elastic Beanstalk, Lightsail, Amplify, S3 (and more!) can each host websites + APIs. But which one should we choose?
Which one is cheapest? Which one is fastest? Which one will scale to meet our needs?
Join me in this session as we dive into each AWS hosting service to determine which one is best for your scenario and explain why!
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Best 20 SEO Techniques To Improve Website Visibility In SERP
Boost your website's visibility with proven SEO techniques! Our latest blog dives into essential strategies to enhance your online presence, increase traffic, and rank higher on search engines. From keyword optimization to quality content creation, learn how to make your site stand out in the crowded digital landscape. Discover actionable tips and expert insights to elevate your SEO game.
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift
Overview
Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices.
Key Topics Covered
1. Introduction to Anomaly Detection
- Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems.
2. Understanding Edge (IoT)
- Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source.
3. What is ArgoCD?
- Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices.
4. Deployment Using ArgoCD for Edge Devices
- Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD.
5. Introduction to Apache Kafka and S3
- Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions.
6. Viewing Kafka Messages in the Data Lake
- Learn how to view and analyze Kafka messages stored in a data lake for better insights.
7. What is Prometheus?
- Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices.
8. Monitoring Application Metrics with Prometheus
- Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system.
9. What is Camel K?
- Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes.
10. Configuring Camel K Integrations for Data Pipelines
- Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow.
11. What is a Jupyter Notebook?
- Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text.
12. Jupyter Notebooks with Code Examples
- Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.
Main news related to the CCS TSI 2023 (2023/1695)
An English 🇬🇧 translation of a presentation to the speech I gave about the main changes brought by CCS TSI 2023 at the biggest Czech conference on Communications and signalling systems on Railways, which was held in Clarion Hotel Olomouc from 7th to 9th November 2023 (konferenceszt.cz). Attended by around 500 participants and 200 on-line followers.
The original Czech 🇨🇿 version of the presentation can be found here: https://www.slideshare.net/slideshow/hlavni-novinky-souvisejici-s-ccs-tsi-2023-2023-1695/269688092 .
The videorecording (in Czech) from the presentation is available here: https://youtu.be/WzjJWm4IyPk?si=SImb06tuXGb30BEH .
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
Imagine a world where machines not only perform tasks but also learn, adapt, and make decisions. This is the promise of Artificial Intelligence (AI), a technology that's not just enhancing our lives but revolutionizing entire industries.
OpenID AuthZEN Interop Read Out - Authorization
During Identiverse 2024 and EIC 2024, members of the OpenID AuthZEN WG got together and demoed their authorization endpoints conforming to the AuthZEN API
Skybuffer SAM4U tool for SAP license adoption
Manage and optimize your license adoption and consumption with SAM4U, an SAP free customer software asset management tool.
SAM4U, an SAP complimentary software asset management tool for customers, delivers a detailed and well-structured overview of license inventory and usage with a user-friendly interface. We offer a hosted, cost-effective, and performance-optimized SAM4U setup in the Skybuffer Cloud environment. You retain ownership of the system and data, while we manage the ABAP 7.58 infrastructure, ensuring fixed Total Cost of Ownership (TCO) and exceptional services through the SAP Fiori interface.
Generating privacy-protected synthetic data using Secludy and Milvus
During this demo, the founders of Secludy will demonstrate how their system utilizes Milvus to store and manipulate embeddings for generating privacy-protected synthetic data. Their approach not only maintains the confidentiality of the original data but also enhances the utility and scalability of LLMs under privacy constraints. Attendees, including machine learning engineers, data scientists, and data managers, will witness first-hand how Secludy's integration with Milvus empowers organizations to harness the power of LLMs securely and efficiently.
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
ABSTRACT: A prima vista, un mattoncino Lego e la backdoor XZ potrebbero avere in comune il fatto di essere entrambi blocchi di costruzione, o dipendenze di progetti creativi e software. La realtà è che un mattoncino Lego e il caso della backdoor XZ hanno molto di più di tutto ciò in comune.
Partecipate alla presentazione per immergervi in una storia di interoperabilità, standard e formati aperti, per poi discutere del ruolo importante che i contributori hanno in una comunità open source sostenibile.
BIO: Sostenitrice del software libero e dei formati standard e aperti. È stata un membro attivo dei progetti Fedora e openSUSE e ha co-fondato l'Associazione LibreItalia dove è stata coinvolta in diversi eventi, migrazioni e formazione relativi a LibreOffice. In precedenza ha lavorato a migrazioni e corsi di formazione su LibreOffice per diverse amministrazioni pubbliche e privati. Da gennaio 2020 lavora in SUSE come Software Release Engineer per Uyuni e SUSE Manager e quando non segue la sua passione per i computer e per Geeko coltiva la sua curiosità per l'astronomia (da cui deriva il suo nickname deneb_alpha).
Presentation of the OECD Artificial Intelligence Review of Germany
Consult the full report at https://www.oecd.org/digital/oecd-artificial-intelligence-review-of-germany-609808d6-en.htm
20240609 QFM020 Irresponsible AI Reading List May 2024
Everything I found interesting about the irresponsible use of machine intelligence in May 2024
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on integration of Salesforce with Bonterra Impact Management.
Interested in deploying an integration with Salesforce for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
Project Management Semester Long Project - Acuity
Acuity is an innovative learning app designed to transform the way you engage with knowledge. Powered by AI technology, Acuity takes complex topics and distills them into concise, interactive summaries that are easy to read & understand. Whether you're exploring the depths of quantum mechanics or seeking insight into historical events, Acuity provides the key information you need without the burden of lengthy texts.
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...Edge AI and Vision Alliance
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/building-and-scaling-ai-applications-with-the-nx-ai-manager-a-presentation-from-network-optix/
Robin van Emden, Senior Director of Data Science at Network Optix, presents the “Building and Scaling AI Applications with the Nx AI Manager,” tutorial at the May 2024 Embedded Vision Summit.
In this presentation, van Emden covers the basics of scaling edge AI solutions using the Nx tool kit. He emphasizes the process of developing AI models and deploying them globally. He also showcases the conversion of AI models and the creation of effective edge AI pipelines, with a focus on pre-processing, model conversion, selecting the appropriate inference engine for the target hardware and post-processing.
van Emden shows how Nx can simplify the developer’s life and facilitate a rapid transition from concept to production-ready applications.He provides valuable insights into developing scalable and efficient edge AI solutions, with a strong focus on practical implementation.Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean Lotus cyber threat actors represent a sophisticated, persistent, and politically motivated group that poses a significant risk to organizations and individuals in the Southeast Asian region. Their continuous evolution and adaptability underscore the need for robust cybersecurity measures and international cooperation to identify and mitigate the threats posed by such advanced persistent threat groups.
Recently uploaded (20)
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
WeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation Techniques
HDF-EOS Development Current Status
- 1. HDF-EOS Development Current Status Larry Klein, Shen Zhao and Ray Milburn L-3 Communications EER Systems, Inc. December 5, 2002 Larry.Klein@L-3com.com, szhao@eos.east.hitc.com 1
- 2. HDF-EOS 2 • HDF4 based, storage format for EOS standard products. • Used operationally by MODIS, MISR, ASTER, Landsat, AIRS and other EOS instruments • Support for Grid/Point/Swath structures • HDF4 and HDF-EOS 2 will be supported by NCSA and NASA for the fore-seeable future. 2
- 3. Current Archive Holdings • GSFC DAAC: AIRS, MODIS L1, L2 Atmos./Ocean – 900 TB • EDC DAAC: Landsat/ASTER/MODIS Land – 500 TB • Langely DAAC: MISR – 300 TB • NSIDC DAAC: MODIS L2, L3 – 20 TB 3
- 4. HDF-EOS 5 • Based on HDF5, a complete rewrite of HDF4 with a different interface. – First released in 2000. • Designed to ‘resemble’ HDF-EOS 2 to the maximum extent possible. – – – – Supports same data structures Added prefix ‘HE5_’ to HDF-EOS 2 functions. Doesn’t preclude HDF5 functionality. Data Type changes, e.g. INT64 -> H5T_NATIVE_LONG 4
- 5. HDF-EOS 2 -> HDF-EOS 5 •HDF-EOS 2 SWdefdatafield(swathID, fieldname, dimlist, numtype, merge) •HDF-EOS 5 HE5_SWdefdatafield(swathID, fieldname, dimlist, maxdimlist, numtype, merge, Maxdimlist*) *New HDF5 functionality passed through. This allows the user to set an upper limit to the size of the dataset. 5
- 6. HDF-EOS 5 • Provide a conversion tool: heconvert – HDF-EOS 2 -> HDF-EOS 5 – compliments HDF4 -> HDF5 conversion tool • HDF-EOS 5 will be used by EOS Aura – Standard format for profile data developed • One GLAS summary product in HDF-EOS 5 • Assume that HDF-EOS 2 producers will convert when PI’s determine that the time is right. 6
- 7. Top Level of HDF-EOS 5 File Root -- “/” HDFEOS INFORMATION HDFEOS STRUC. METADATA ADDITIONAL ARRAY SWATH GRID POINT Global (File) Attributes The new ADDITIONAL Group has global (file) attributes The new functionality is added to the EH(utility) interface. 7
- 8. Swath Structure SwathName Global Attribute <SwathName>:<AttrName> Group Attribute <DataFields>:<AttrName> Local Attribute <FieldName>:<AttrName> Data Fields Data Data Field.1 Field.n Profile Fields Profile Field.1 Profile Field.n Geolocation Fields Longitude Latitude Time Each Data Field can have Attributes and/or Dimension Scales Group CoLatitude Shaded Objects are implemented in a fixed way so the user doesn’t have direct access via the interface Attribute Data Set 8
- 9. HDF-EOS 5 Functionality • • • • • • • • • Basic File I/O Fill Values Compression Chunking/Tiling Swath Interface Grid Interface Point Interface Profile Interface Generalized Array Interface (Zonal Avg.) • Global (File), Group & Local Attributes • External Data Files • Subsetting • Unix/Linux/Windows • Threadsafe Version • FORTRAN, C, C++ 9
- 10. ECS support of HDF5 • ECS toolkit V 5.2.8 supports HDF4 and HDF5 - based applications – HDF-EOS 2.8 (hdf4.1r5) – HDF-EOS 5.3 (hdf5-1.4.3) • HDF5 and HDF4 must both be compiled – HDF4 users not affected – HDF5 users must use PGS_MET_SDstart() and PGS_MET_SDend() 10
- 11. ECS support of HDF5 • Both flavors of HDF-EOS (HDF) are part of the ECS baseline. • OS’s: Sun, SGI, IBM, HP, DEC, Windows NT/98, Linux • Compilers: FORTRAN 77/90, C, C++. • ECS will not crack HDF5 - based files for near future. (contents transparent to archive) • Size limit is 2 GBytes 11
- 12. Applications • HE5View (HDF-EOS 5 browser) • Java EOS Browser (HDF-EOS 2 and 5 access) • HDF-EOS processor: convert to GeoTIFF, subset, mosaic, subsample. In Data Pool interface (HDF4 based only) • heconvert (convert HDF-EOS 2 Grid/Point/Swath to HDF-EOS 5 equivalents) • Access libraries and applications at: http://newsroom.gsfc.nasa.gov/sdptoolkit/toolkit.html 12
- 13. Issues • Only gzip (deflate) compression method implemented. • Metadata not updated for swath stitching in unlimited dimension. • File size limit in ECS - Is two Gbytes enough? • Currently in maintenance mode, but user requests are always welcome. 13