Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3

•Download as PPSX, PDF•

0 likes•331 views

The HDF-EOS Tools and Information Center

This document compares different methods for accessing HDF and netCDF files stored on Amazon S3, including Apache Drill, THREDDS Data Server (TDS), and HDF5 Virtual File Driver (VFD). A benchmark test of accessing a 24GB HDF5/netCDF-4 file on S3 from Amazon EC2 found that TDS performed the best, responding within 2 minutes, while Apache Drill failed after 7 minutes. The document concludes that TDS 5.0 is the clear winner based on performance and support for role-based access control and HDF4 files, but the best solution depends on use case and software.

SESIP-0720-JL
Using Apache Drill and Unidata
TDS* for NASA HDF-EOS on S3
ESIP 2020 Summer / HDF-EOS Workshop XXIII
This work was supported by NASA/GSFC under Raytheon Technologies contract number NNG15HZ39C.
This document does not contain technology or Technical Data controlled under either the U.S. International Traffic
in Arms Regulations or the U.S. Export Administration Regulations.
H. Joe Lee
EED-2 / The HDF Group / Software Engineer
hyoklee@hdfgroup.org
*THREDDS Data Server

SESIP-0720-JL
2
• HDF4
– HDF-EOS2
• HDF5
– HDF-EOS5
– netCDF-4
Hierarchical Data Format-Earth Observing
System

SESIP-0720-JL
3
HDF-EOS on S3
•HDF4?
• No elegant solution other than GDAL*
• Not so elegant: h4mapwriter / s3fs
•HDF5?
• Many OK solutions exist
• HDF5 VFD**/ HSDS*** / GDAL / Hyrax
DMR****++ / etc.
• But “Just OK is not OK.”
*Geospatial Data Abstraction Library
** Virtual File Driver
***Highly Scalable Data Service
****Dataset Metadata Response

SESIP-0720-JL
4
Apache Drill
• Supports Variety of storage - Amazon S3,
Azure Blob Storage, Google Cloud
Storage, Swift, NAS and local files.
• Data agility - query the raw data in-situ.
• Table - in-memory shredded columnar
representation for complex data
• BI Tools and REST API

SESIP-0720-JL
5
Apache Drill 1.18 (beta)
• Collection of HDF5 files on S3
• ANSI SQL
• Geoprocessing?

SESIP-0720-JL
6
THREDDS Data Server 5.0
(beta)
It supports S3!
• both HDF4 and HDF5
• NcML?
• Catalog for collection of files?

SESIP-0720-JL
7
netCDF-Java
• This is core library.
• THREDDS / Panoply / IDV shares this.
• toolsUI is a generic GUI tool based on
netCDF-Java.
• Like GDAL, if netCDF-Java works with
S3, the rest are trivial.

SESIP-0720-JL
9
Benchmark: TerraFusion on S3
• Test file size: 24G
• Format: HDF5/netCDF-4 CF
• One orbit data from 5 sensors on Terra
• S3 access from EC2 (m4.xlarge)

SESIP-0720-JL
10
Apache Drill fails after 7 minute.
read on
s3a://basicterrafusion/TERRA_BF_L1B_O535
57_20100112014327_F000_V001.h5:
com.amazonaws.AbortedException:
org.apache.drill.common.exceptions.UserE
xception$Builder.build(UserException.jav
a:657)
org.apache.drill.exec.store.hdf5.HDF5Bat
chReader.convertInputStreamToFile(HDF5Ba
tchReader.java:356)

SESIP-0720-JL
11
TDS responds within 2 minutes.
Float32
/MOPITT/granule_20100112/Geolocation/Latitude[ntr
ack_1 = 46][nstare = 29][npixels = 4];
Float32
/MOPITT/granule_20100112/Geolocation/Longitude[nt
rack_1 = 36][nstare = 29][npixels = 4];
Float64
/MOPITT/granule_20100112/Geolocation/Time[ntrack_
1 = 436];
} s3-
test/TERRA_BF_L1B_O53557_20100112014327_F000_V001
.h5;
real 1m47.065s

SESIP-0720-JL
12
h5ls responds in 2.5 minutes.
• HDF5 Virtual File Driver (VFD)
• --enable-ros3-vfd configuration option
It takes 2X longer (5 minutes) outside AWS.

SESIP-0720-JL
13
Role-based Access Control
(RBAC)
Drill THREDDS H5 VFD
Always Yes No
• RBAC eliminates access key and token.
• Access with s3://bucket/key.h5 (no https://)
• S3 buckets and objects can be private.

SESIP-0720-JL
14
THREDDS 5.0 is a Clear Winner
Based on our Benchmark Results.
• Performance is good.
• It supports HDF4.
• RBAC is supported.
• Existing netcdf-Java / OPeNDAP based
software works seamlessly.

SESIP-0720-JL
15
However, Use Case Still Matters
• SQL user? Try Drill after sanitization.
• Good for Collection of HDF5 files with 2D Grid.
• Use AWS Lambda (w/ CUMULUS) for sanitization.
• Java user? Try netCDF-Java.
• Python user? Try GDAL vsis3/ driver for HDF5 and viscurl/
for HDF4.
• OPeNDAP user? Try THREDDS 5.0 beta.
• HDF5 C/Fortran user? Try HDF5 VFD.
There are many (read-only) solutions for HDF-EOS on S3:

SESIP-0720-JL
16
This work was supported by NASA/GSFC under
Raytheon Technologies contract number
NNG15HZ39C.
in partnership with

What's hot

Leveraging the Cloud for HDF Software Testing

The HDF-EOS Tools and Information Center

HDF for the Cloud - New HDF Server Features

The HDF-EOS Tools and Information Center

HDF for the Cloud - Serverless HDF

The HDF-EOS Tools and Information Center

Caching and Buffering in HDF5

The HDF-EOS Tools and Information Center

Google Colaboratory for HDF-EOS

The HDF-EOS Tools and Information Center

STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...

The HDF-EOS Tools and Information Center

Parallel HDF5 DevelopmentsThe HDF-EOS Tools and Information Center

Easy Access of NASA HDF data via OPeNDAP

The HDF-EOS Tools and Information Center

HDF-EOS 2/5 to netCDF Converter

The HDF-EOS Tools and Information Center

The tool takes HDF-EOS 5 data as input, and generates COARDS-compatible output - if the input file has enough metadata to be COARDS-compliant, the output file will be COARDS-compliant. The tool is written in portable C, and ought to run on any platform where the HDF-EOS and netCDF libraries are available. This year, we have made two major enhancements to the converter: It now automatically detects whether its input is HDF-EOS2 or HDF-EOS5 format, and handles either one. The previous tool worked with HDF-EOS5 only. Its netCDF output attempts to conform to the new CF conventions (a superset of the COARDS conventions). This is primarily an improvement in its translation of Swath datasets, which CF handles much better than COARDS.

HDF Update 2016

The HDF-EOS Tools and Information Center

NetCDF and HDF5

The HDF-EOS Tools and Information Center

HDF Product Designer

The HDF-EOS Tools and Information Center

Status of HDF-EOS, Related Software and ToolsThe HDF-EOS Tools and Information Center

HDF Product Designer: Using Templates to Achieve Interoperability

The HDF-EOS Tools and Information Center

Open-source Scientific Computing and Data Analytics using HDF

The HDF-EOS Tools and Information Center

HDF & HDF-EOS Data & Support at NSIDCThe HDF-EOS Tools and Information Center

HDF Group Support for NPP/NPOESS/JPSSThe HDF-EOS Tools and Information Center

Easy Remote Access Via OPeNDAPThe HDF-EOS Tools and Information Center

Using IDL with Suomi NPP VIIRS DataThe HDF-EOS Tools and Information Center

Moving form HDF4 to HDF5/netCDF-4

The HDF-EOS Tools and Information Center

What's hot (20)

Leveraging the Cloud for HDF Software Testing

HDF for the Cloud - New HDF Server Features

HDF for the Cloud - Serverless HDF

Caching and Buffering in HDF5

Google Colaboratory for HDF-EOS

STARE-PODS: A Versatile Data Store Leveraging the HDF Virtual Object Layer fo...

Parallel HDF5 Developments

Easy Access of NASA HDF data via OPeNDAP

HDF-EOS 2/5 to netCDF Converter

HDF Update 2016

NetCDF and HDF5

HDF Product Designer

Status of HDF-EOS, Related Software and Tools

HDF Product Designer: Using Templates to Achieve Interoperability

Open-source Scientific Computing and Data Analytics using HDF

HDF & HDF-EOS Data & Support at NSIDC

HDF Group Support for NPP/NPOESS/JPSS

Easy Remote Access Via OPeNDAP

Using IDL with Suomi NPP VIIRS Data

Moving form HDF4 to HDF5/netCDF-4

Similar to Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3

Accessing Cloud Data and Services Using EDL, Pydap, MATLAB

The HDF-EOS Tools and Information Center

HDF5 Tools Updates

The HDF-EOS Tools and Information Center

HDF5 OPeNDAP Handler Updates, and Performance Discussion

The HDF-EOS Tools and Information Center

HDF5 iRODS

The HDF-EOS Tools and Information Center

Numerous scientific teams use the HDF5 format to store very large datasets. Efficient use of this data in a distributed environment depends on client applications being able to read any subset of the data without transferring the entire file to the local machine. The goal of the HDF5-iRODS Project was to develop an HDF5-iRODS module for the iRODS datagrid server that supported this capability, and to apply the technology to an NCSA/SDSC Strategic Applications Program (SAP) project, FLASH. A joint team from The HDF Group (representing NCSA) and the SDSC SRB group collaborated to accomplish the project goal. The team implemented five HDF5 microservices functions on the iRODS server, and developed an iRODS FLASH slice client application. The client implementation also includes a JNI interface that allows HDFView, a standard tool for browsing HDF5 files, to access HDF5 files stored remotely in iRODS. Finally, three new collection client/server calls were added to the iRODS APIs, making it easier for users to query the content of an iRODS collection.

HDF5 Advanced Topics

The HDF-EOS Tools and Information Center

This tutorial is designed for users with some HDF5 experience. It will cover advanced features of the HDF5 library that can be used to achieve better I/O performance and more efficient storage. The following HDF5 features will be discussed: partial I/O; compression and other filters, including new n-bit and scale+offset filters and data storage options. Significant time will be devoted to the discussion of complex HDF5 datatypes such as strings, variable-length datatypes, array datatypes, and compound datatypes.

Tools to improve the usability of NASA HDF DataThe HDF-EOS Tools and Information Center

HDF5 Tools Update

The HDF-EOS Tools and Information Center

We will introduce the basic features and recent updates on the HDF5 tools. The intention of the talk is to keep users updated on the HDF5 tool development such as new features and new tools. It also helps new users to get familiar with the HDF5 tools. Three new tools h5check, h5copy and h5stat will be introduced. It is important that HDF5 files created and modified by the HDF5 library are fully compliant with the defined HDF5 File Format Specification (File Format) to ensure the data model integrity and long term compatibility between evolving versions of the HDF5 library. The h5check tool verifies that the content of an HDF5 file is encoded according to the File Format. The verification role also makes h5check act as a watchdog for the implementation correctness of the HDF5 library. This presentation explains the features of the tool and time permitting, shows a demonstration of the tool. h5copy is a command line tool that allows to make a copy of an HDF5 object (group, dataset or named datatype) from one location to another location within a file or across files. h5copy uses the HDF5 API function H5Gcopy to copy objects, which provides users with several options on how data object is copied, and high efficiency in data copy. h5stat is a new utility for viewing different statistics about an HDF5 file. Tool is still under development. The goal of the talk will be to introduce the tool and get HDF5 users feedback.

The Data Center and Hadoop

DataWorks Summit

What do data center operators need to know when deploying Hadoop in the Data Center? Multi-tenancy, network topology, workload types, and myriad other factors affect the way applications run and perform in the data center. Understanding performance characteristics of the distributed system is key to not only optimize for Hadoop, but allows Hadoop to seamlessly operate side-by-side existing applications.

HDF OPeNDAP project update and demo

The HDF-EOS Tools and Information Center

In this presentation, we will give an update on the HDF OPeNDAP project. We will update the new features inside the HDF5 OPeNDAP data handler. We will also introduce the enhanced HDF4 OPeNDAP data handler and demonstrate how it can help users to view and analyze remote HDF-EOS2 data. A demo that uses OPeNDAP client tools to handle AIRS and MODIS Grid/Swath data with the enhanced handler will be presented.

Creating Cloud-Optimized HDF5 Files

The HDF-EOS Tools and Information Center

HDF4 and HDF5 Performance Preliminary Results

The HDF-EOS Tools and Information Center

Efficiently serving HDF5 via OPeNDAP

The HDF-EOS Tools and Information Center

ESDIS Status (2002)

The HDF-EOS Tools and Information Center

Bridging ICESat and ICESat-2 Standard Data ProductsThe HDF-EOS Tools and Information Center

Status of HDF-EOS, Related Software, and Tools

The HDF-EOS Tools and Information Center

HDF Status Update

The HDF-EOS Tools and Information Center

HDF Tools Tutorial

The HDF-EOS Tools and Information Center

This tutorial is designed for anyone who needs to work with data stored in HDF and HDF5 files. The first part of the tutorial will focus on the HDF5 utilities to display the contents of HDF5 files, to extract and to import data from and to HDF5 files, to compare two HDF5 files, and more. Participants will be guided through the hand-on examples and will learn about different tools options. New changes and advanced features will be covered in a separate session (Updates on HDF tools) on Wednesday. The second part of tutorial includes a hands-on session to learn the HDF (4 & 5) Java browsing tool, HDFView. The tool and special plug-ins will be used to work with the existing HDF, HDF-EOS, and netCDF-4 files, and to create a new HDF5 file. The tutorial will cover basic features of HDFView.

HDFView and HDF Java Products

The HDF-EOS Tools and Information Center

This tutorial will introduce the three levels of the HDF-Java products: the HDF-Java wrapper (or Java Native Interfaces to the standard HDF libraries), the HDF-Java object package, and the HDFView. The Java wrapper provides standard Java APIs that allow applications to call the C HDF libraries from Java. The HDF-Java object package implements HDF data objects, e.g. Groups and Datasets, in an object-oriented form and makes it easy for applications to use the libraries. The HDFView is a visual tool for browsing and editing HDF4 and HDF5 files.

Usage of NCL, IDL, and MATLAB to access NASA HDF4/HDF-EOS2/HDF-EOS5 data

The HDF-EOS Tools and Information Center

Hyrax: Serving Data from S3

The HDF-EOS Tools and Information Center

Similar to Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3 (20)

Accessing Cloud Data and Services Using EDL, Pydap, MATLAB

HDF5 Tools Updates

HDF5 OPeNDAP Handler Updates, and Performance Discussion

HDF5 iRODS

HDF5 Advanced Topics

Tools to improve the usability of NASA HDF Data

HDF5 Tools Update

The Data Center and Hadoop

HDF OPeNDAP project update and demo

Creating Cloud-Optimized HDF5 Files

HDF4 and HDF5 Performance Preliminary Results

Efficiently serving HDF5 via OPeNDAP

ESDIS Status (2002)

Bridging ICESat and ICESat-2 Standard Data Products

Status of HDF-EOS, Related Software, and Tools

HDF Status Update

HDF Tools Tutorial

HDFView and HDF Java Products

Usage of NCL, IDL, and MATLAB to access NASA HDF4/HDF-EOS2/HDF-EOS5 data

Hyrax: Serving Data from S3

Recently uploaded

2024 RoOUG Security model for the cloud.pptx

Georgi Kodinov

Custom Healthcare Software for Managing Chronic Conditions and Remote Patient...

Mind IT Systems

APIs for Browser Automation (MoT Meetup 2024)

Boni García

How Recreation Management Software Can Streamline Your Operations.pptx

wottaspaceseo

Recreation management software streamlines operations by automating key tasks such as scheduling, registration, and payment processing, reducing manual workload and errors. It provides centralized management of facilities, classes, and events, ensuring efficient resource allocation and facility usage. The software offers user-friendly online portals for easy access to bookings and program information, enhancing customer experience. Real-time reporting and data analytics deliver insights into attendance and preferences, aiding in strategic decision-making. Additionally, effective communication tools keep participants and staff informed with timely updates. Overall, recreation management software enhances efficiency, improves service delivery, and boosts customer satisfaction.

Globus Compute wth IRI Workflows - GlobusWorld 2024

Globus

As part of the DOE Integrated Research Infrastructure (IRI) program, NERSC at Lawrence Berkeley National Lab and ALCF at Argonne National Lab are working closely with General Atomics on accelerating the computing requirements of the DIII-D experiment. As part of the work the team is investigating ways to speedup the time to solution for many different parts of the DIII-D workflow including how they run jobs on HPC systems. One of these routes is looking at Globus Compute as a way to replace the current method for managing tasks and we describe a brief proof of concept showing how Globus Compute could help to schedule jobs and be a tool to connect compute at different facilities.

Utilocate provides Smarter, Better, Faster, Safer Locate Ticket Management

Utilocate

Utilocate offers a comprehensive solution for locate ticket management by automating and streamlining the entire process. By integrating with Geospatial Information Systems (GIS), it provides accurate mapping and visualization of utility locations, enhancing decision-making and reducing the risk of errors. The system's advanced data analytics tools help identify trends, predict potential issues, and optimize resource allocation, making the locate ticket management process smarter and more efficient. Additionally, automated ticket management ensures consistency and reduces human error, while real-time notifications keep all relevant personnel informed and ready to respond promptly. The system's ability to streamline workflows and automate ticket routing significantly reduces the time taken to process each ticket, making the process faster and more efficient. Mobile access allows field technicians to update ticket information on the go, ensuring that the latest information is always available and accelerating the locate process. Overall, Utilocate not only enhances the efficiency and accuracy of locate ticket management but also improves safety by minimizing the risk of utility damage through precise and timely locates.

May Marketo Masterclass, London MUG May 22 2024.pdf

Adele Miller

Top Features to Include in Your Winzo Clone App for Business Growth (4).pptx

rickgrimesss22

Vitthal Shirke Java Microservices Resume.pdf

Vitthal Shirke

Cyaniclab : Software Development Agency Portfolio.pdf

Cyanic lab

CyanicLab, an offshore custom software development company based in Sweden,India, Finland, is your go-to partner for startup development and innovative web design solutions. Our expert team specializes in crafting cutting-edge software tailored to meet the unique needs of startups and established enterprises alike. From conceptualization to execution, we offer comprehensive services including web and mobile app development, UI/UX design, and ongoing software maintenance. Ready to elevate your business? Contact CyanicLab today and let us propel your vision to success with our top-notch IT solutions.

Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...

Globus

The U.S. Geological Survey (USGS) has made substantial investments in meeting evolving scientific, technical, and policy driven demands on storing, managing, and delivering data. As these demands continue to grow in complexity and scale, the USGS must continue to explore innovative solutions to improve its management, curation, sharing, delivering, and preservation approaches for large-scale research data. Supporting these needs, the USGS has partnered with the University of Chicago-Globus to research and develop advanced repository components and workflows leveraging its current investment in Globus. The primary outcome of this partnership includes the development of a prototype enterprise repository, driven by USGS Data Release requirements, through exploration and implementation of the entire suite of the Globus platform offerings, including Globus Flow, Globus Auth, Globus Transfer, and Globus Search. This presentation will provide insights into this research partnership, introduce the unique requirements and challenges being addressed and provide relevant project progress.

Enterprise Software Development with No Code Solutions.pptx

QuickwayInfoSystems3

In the ever-evolving landscape of technology, enterprise software development is undergoing a significant transformation. Traditional coding methods are being challenged by innovative no-code solutions, which promise to streamline and democratize the software development process. This shift is particularly impactful for enterprises, which require robust, scalable, and efficient software to manage their operations. In this article, we will explore the various facets of enterprise software development with no-code solutions, examining their benefits, challenges, and the future potential they hold.

Graphic Design Crash Course for beginners

e20449

BoxLang: Review our Visionary Licenses of 2024

Ortus Solutions, Corp

Prosigns: Transforming Business with Tailored Technology Solutions

Prosigns

Unlocking Business Potential: Tailored Technology Solutions by Prosigns Discover how Prosigns, a leading technology solutions provider, partners with businesses to drive innovation and success. Our presentation showcases our comprehensive range of services, including custom software development, web and mobile app development, AI & ML solutions, blockchain integration, DevOps services, and Microsoft Dynamics 365 support. Custom Software Development: Prosigns specializes in creating bespoke software solutions that cater to your unique business needs. Our team of experts works closely with you to understand your requirements and deliver tailor-made software that enhances efficiency and drives growth. Web and Mobile App Development: From responsive websites to intuitive mobile applications, Prosigns develops cutting-edge solutions that engage users and deliver seamless experiences across devices. AI & ML Solutions: Harnessing the power of Artificial Intelligence and Machine Learning, Prosigns provides smart solutions that automate processes, provide valuable insights, and drive informed decision-making. Blockchain Integration: Prosigns offers comprehensive blockchain solutions, including development, integration, and consulting services, enabling businesses to leverage blockchain technology for enhanced security, transparency, and efficiency. DevOps Services: Prosigns' DevOps services streamline development and operations processes, ensuring faster and more reliable software delivery through automation and continuous integration. Microsoft Dynamics 365 Support: Prosigns provides comprehensive support and maintenance services for Microsoft Dynamics 365, ensuring your system is always up-to-date, secure, and running smoothly. Learn how our collaborative approach and dedication to excellence help businesses achieve their goals and stay ahead in today's digital landscape. From concept to deployment, Prosigns is your trusted partner for transforming ideas into reality and unlocking the full potential of your business. Join us on a journey of innovation and growth. Let's partner for success with Prosigns.

Globus Connect Server Deep Dive - GlobusWorld 2024

Globus

Introduction to Pygame (Lecture 7 Python Game Development)

abdulrafaychaudhry

GraphSummit Paris - The art of the possible with Graph Technology

Neo4j

Enterprise Resource Planning System in Telangana

NYGGS Automation Suite

Enterprise Resource Planning System includes various modules that reduce any business's workload. Additionally, it organizes the workflows, which drives towards enhancing productivity. Here are a detailed explanation of the ERP modules. Going through the points will help you understand how the software is changing the work dynamics. To know more details here: https://blogs.nyggs.com/nyggs/enterprise-resource-planning-erp-system-modules/

Vitthal Shirke Microservices Resume Montevideo

Vitthal Shirke

Recently uploaded (20)

2024 RoOUG Security model for the cloud.pptx

Custom Healthcare Software for Managing Chronic Conditions and Remote Patient...

APIs for Browser Automation (MoT Meetup 2024)

How Recreation Management Software Can Streamline Your Operations.pptx

Globus Compute wth IRI Workflows - GlobusWorld 2024

Utilocate provides Smarter, Better, Faster, Safer Locate Ticket Management

May Marketo Masterclass, London MUG May 22 2024.pdf

Top Features to Include in Your Winzo Clone App for Business Growth (4).pptx

Vitthal Shirke Java Microservices Resume.pdf

Cyaniclab : Software Development Agency Portfolio.pdf

Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...

Enterprise Software Development with No Code Solutions.pptx

Graphic Design Crash Course for beginners

BoxLang: Review our Visionary Licenses of 2024

Prosigns: Transforming Business with Tailored Technology Solutions

Globus Connect Server Deep Dive - GlobusWorld 2024

Introduction to Pygame (Lecture 7 Python Game Development)

GraphSummit Paris - The art of the possible with Graph Technology

Enterprise Resource Planning System in Telangana

Vitthal Shirke Microservices Resume Montevideo

Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3

1. SESIP-0720-JL Using Apache Drill and Unidata TDS* for NASA HDF-EOS on S3 ESIP 2020 Summer / HDF-EOS Workshop XXIII This work was supported by NASA/GSFC under Raytheon Technologies contract number NNG15HZ39C. This document does not contain technology or Technical Data controlled under either the U.S. International Traffic in Arms Regulations or the U.S. Export Administration Regulations. H. Joe Lee EED-2 / The HDF Group / Software Engineer hyoklee@hdfgroup.org *THREDDS Data Server

2. SESIP-0720-JL 2 • HDF4 – HDF-EOS2 • HDF5 – HDF-EOS5 – netCDF-4 Hierarchical Data Format-Earth Observing System

3. SESIP-0720-JL 3 HDF-EOS on S3 •HDF4? • No elegant solution other than GDAL* • Not so elegant: h4mapwriter / s3fs •HDF5? • Many OK solutions exist • HDF5 VFD**/ HSDS*** / GDAL / Hyrax DMR****++ / etc. • But “Just OK is not OK.” *Geospatial Data Abstraction Library ** Virtual File Driver ***Highly Scalable Data Service ****Dataset Metadata Response

4. SESIP-0720-JL 4 Apache Drill • Supports Variety of storage - Amazon S3, Azure Blob Storage, Google Cloud Storage, Swift, NAS and local files. • Data agility - query the raw data in-situ. • Table - in-memory shredded columnar representation for complex data • BI Tools and REST API

5. SESIP-0720-JL 5 Apache Drill 1.18 (beta) • Collection of HDF5 files on S3 • ANSI SQL • Geoprocessing?

6. SESIP-0720-JL 6 THREDDS Data Server 5.0 (beta) It supports S3! • both HDF4 and HDF5 • NcML? • Catalog for collection of files?

7. SESIP-0720-JL 7 netCDF-Java • This is core library. • THREDDS / Panoply / IDV shares this. • toolsUI is a generic GUI tool based on netCDF-Java. • Like GDAL, if netCDF-Java works with S3, the rest are trivial.

8. SESIP-0720-JL 8 toolsUI - HDF4 on S3

9. SESIP-0720-JL 9 Benchmark: TerraFusion on S3 • Test file size: 24G • Format: HDF5/netCDF-4 CF • One orbit data from 5 sensors on Terra • S3 access from EC2 (m4.xlarge)

10. SESIP-0720-JL 10 Apache Drill fails after 7 minute. read on s3a://basicterrafusion/TERRA_BF_L1B_O535 57_20100112014327_F000_V001.h5: com.amazonaws.AbortedException: org.apache.drill.common.exceptions.UserE xception$Builder.build(UserException.jav a:657) org.apache.drill.exec.store.hdf5.HDF5Bat chReader.convertInputStreamToFile(HDF5Ba tchReader.java:356)

11. SESIP-0720-JL 11 TDS responds within 2 minutes. Float32 /MOPITT/granule_20100112/Geolocation/Latitude[ntr ack_1 = 46][nstare = 29][npixels = 4]; Float32 /MOPITT/granule_20100112/Geolocation/Longitude[nt rack_1 = 36][nstare = 29][npixels = 4]; Float64 /MOPITT/granule_20100112/Geolocation/Time[ntrack_ 1 = 436]; } s3- test/TERRA_BF_L1B_O53557_20100112014327_F000_V001 .h5; real 1m47.065s

12. SESIP-0720-JL 12 h5ls responds in 2.5 minutes. • HDF5 Virtual File Driver (VFD) • --enable-ros3-vfd configuration option It takes 2X longer (5 minutes) outside AWS.

13. SESIP-0720-JL 13 Role-based Access Control (RBAC) Drill THREDDS H5 VFD Always Yes No • RBAC eliminates access key and token. • Access with s3://bucket/key.h5 (no https://) • S3 buckets and objects can be private.

14. SESIP-0720-JL 14 THREDDS 5.0 is a Clear Winner Based on our Benchmark Results. • Performance is good. • It supports HDF4. • RBAC is supported. • Existing netcdf-Java / OPeNDAP based software works seamlessly.

15. SESIP-0720-JL 15 However, Use Case Still Matters • SQL user? Try Drill after sanitization. • Good for Collection of HDF5 files with 2D Grid. • Use AWS Lambda (w/ CUMULUS) for sanitization. • Java user? Try netCDF-Java. • Python user? Try GDAL vsis3/ driver for HDF5 and viscurl/ for HDF4. • OPeNDAP user? Try THREDDS 5.0 beta. • HDF5 C/Fortran user? Try HDF5 VFD. There are many (read-only) solutions for HDF-EOS on S3:

16. SESIP-0720-JL 16 This work was supported by NASA/GSFC under Raytheon Technologies contract number NNG15HZ39C. in partnership with

Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3

Similar to Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3 (20)

More from The HDF-EOS Tools and Information Center

More from The HDF-EOS Tools and Information Center (12)

Recently uploaded

Recently uploaded (20)

Apache Drill and Unidata THREDDS Data Server for NASA HDF-EOS on S3