Your SlideShare is downloading. ×
0
Subsetting at UAH

Bruce Beaumont, Matt Smith,
Helen Conover, Sara Graves

HDF & HDF-EOS Workshop VIII
2004 October 26-28 ...
Why Subset?
• Goal: to provide a science data user with only the data
they need as quickly as possible.
• Benefits science...
What is HSE?
HEW Subsetting Engine
A new packaging option for the HDFEOS Subsetter designed for users that
want subsetting...
What is HEW?
• HDF-EOS Web-based Subsetter
– Prototype software designed to be datasetindependent (HDF-EOS)
– Funded by NA...
What are HSE’s capabilities?
• Versions available for HDF-EOS 2 and HDF-EOS 5.
• Subsets multiple files in one call.
• Sub...
Stride Subsampling on a Dimension
•STRIDE = 2
0

1

2

3

4

5

6

7

8

9

7

8

9

Selected

Indexed Subsampling on a Di...
Existing HEW Back-End
Architecture
Subset t ing
crit eria
( ODL)

St at us file

End
-user

E-mail

HEW_ DISS

Input
file(...
New HSE
Architecture
All subsetting functionality is contained
within a callable function
•
User Applicat ion Code
Subset ...
SPOT
• Subsettability “checker”
– Displays content/structure
of HDF-EOS4 (or HE5) files
– Examines files for subsettabilit...
HEW integration with ECS
EDG System

EDG
Order
submission
(HTML)

End
user

ECS

2

ECS

1
7
3

Output data
(Reingested)

...
Product Availability Matrix
HDF-EOS 2
(HDF4)

HDF-EOS 5
(HDF5)

SGI • Sun • Linux

N/A

N/A

SGI • Sun • Linux

HEWBE

SGI...
Currently Available/Planned
Subsetting Applications
•

HDF-EOS Subsetting Engine (HDF-EOS, HE5)Status Deployments
–
–
–
–
...
http://subset.org
Upcoming SlideShare
Loading in...5
×

Subsetting at UAH

59

Published on

The presentation will provide an overview of subsetting software development activity at UAH. Updates have been made to all packages, reflecting the latest versions of HDF5 and HE5. The library of tools (HSE) for subsetting HDF-EOS data is up-to-date for SGI, Sun, and Linux platforms. Subsetting software is operational at NSIDC DAAC and GDAAC, in testing at LPDAAC. Ongoing work and plans will also be described, including row/column subsetting and index subsampling.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
59
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Subsetting at UAH"

  1. 1. Subsetting at UAH Bruce Beaumont, Matt Smith, Helen Conover, Sara Graves HDF & HDF-EOS Workshop VIII 2004 October 26-28 Aurora, CO
  2. 2. Why Subset? • Goal: to provide a science data user with only the data they need as quickly as possible. • Benefits science data users and data centers: - Reduces analysis time by reducing amount of data - Reduces time for data delivery - Reduces resources (network, personnel, media, etc.) • Steps: - Locate spatial / temporal / spectral area of interest - Extract - Re-assemble for distribution/use
  3. 3. What is HSE? HEW Subsetting Engine A new packaging option for the HDFEOS Subsetter designed for users that want subsetting, but do not want to host the full HEW installation or even the standalone HEW back-end.
  4. 4. What is HEW? • HDF-EOS Web-based Subsetter – Prototype software designed to be datasetindependent (HDF-EOS) – Funded by NASA/ESDIS for EOS-DIS – To be used within ECS (EOS-DIS Core System) – Original Front-end/GUI (optional) • Uses HTML forms and JavaScript – Original Back-end • Needed subset criteria and HDF-EOS data • Performed subsetting as a “batch” job
  5. 5. What are HSE’s capabilities? • Versions available for HDF-EOS 2 and HDF-EOS 5. • Subsets multiple files in one call. • Subsets properly-formatted HDF-EOS grid and swath objects. • Subsets spatially by latitude/longitude or row/column. • Subsets swaths temporally by date/time range. • Subsets swaths by full or partial (subscan) lines. • Subsets spectrally by HDF-EOS field. • Subsamples along any dimension by stride (repeating interval) or by discrete index list. (see chart) • Copies all file and field attributes to the output file and updates HDF-EOS “core” metadata when possible.
  6. 6. Stride Subsampling on a Dimension •STRIDE = 2 0 1 2 3 4 5 6 7 8 9 7 8 9 Selected Indexed Subsampling on a Dimension •INDEXES = (1, 3, 4, 5, 7) 0 1 2 3 4 5 Selected 6
  7. 7. Existing HEW Back-End Architecture Subset t ing crit eria ( ODL) St at us file End -user E-mail HEW_ DISS Input file( s) Out put file( s) Log file • Front end supplies subsetting criteria file • Subsetter writes messages to status file • Subsetter writes messages to log file • Subsetter sends e-mail to end-user Many sites do not want to create ODL files or the hassle of dealing with message or log files or e-mail.
  8. 8. New HSE Architecture All subsetting functionality is contained within a callable function • User Applicat ion Code Subset t ing crit eria Ret urn code • Input file( s) HEW Subset t ing Engine funct ion HSE_ LogMsg callback HSE_ St at usMsg callback Out put file( s) • User application code builds subsetting criteria structure Subsetting engine function calls user’s functions for status and log messages No e-mail is sent
  9. 9. SPOT • Subsettability “checker” – Displays content/structure of HDF-EOS4 (or HE5) files – Examines files for subsettability by HSE – Simple command-line interface – Stand-alone operation – Available at subset.org
  10. 10. HEW integration with ECS EDG System EDG Order submission (HTML) End user ECS 2 ECS 1 7 3 Output data (Reingested) 4 Data order and reply Subset ODL and reply Output data 6 Subsetter Subsetting System 5 Input data
  11. 11. Product Availability Matrix HDF-EOS 2 (HDF4) HDF-EOS 5 (HDF5) SGI • Sun • Linux N/A N/A SGI • Sun • Linux HEWBE SGI • Sun • Linux Planned HEW SGI • Sun • Linux N/A Product HSE HSE-5 SPOT SGI • Sun • Linux
  12. 12. Currently Available/Planned Subsetting Applications • HDF-EOS Subsetting Engine (HDF-EOS, HE5)Status Deployments – – – – – – • Complete System Subsetting Engine Only SPOT - Subsettability Checker HSE Integrated with ECS Data Order System HSE Integrated with AMSR-E Processing Subsetting as a Web Service GHRC GSFC many NSIDC, EDC AMSR-E SIPS (ECHO) available available Science teams GHRC in work (various) Customized Subsetting – MODIS tools – Coarse-grain SSM/I Subsetter • available available available available available planned General Purpose Customizable Subsetting – Subsetting Tool using ESML
  13. 13. http://subset.org
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×