Aggregation - What's it to The HDF Group

360 views
298 views

Published on

Aggregation is ubiquitous and data is no exception. This slide presents data aggregation concept and The HDF Group's approach to the data aggregation problem in Earth Science. A n JPSS data aggregation tool called "nagg" is explained as a showcase example.

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
360
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Aggregation in HDF The H in HDF means hierarchy, which in practice is an aggregation.A raster image is an aggregationRaster image groups were the first aggregation in HDF.A raster is an aggregation of scan lines, which are aggregations of pixels.Grouping: Vgroups were the next logical step - a general grouping structure.Vdatas aggregating different datatypes together in a single datatype.HDF groups enable us to express more than one aggregation, or views, of the same set of objects in a file.chunkingexternal storageHDF5 groups, datasets and attributes
  • Two independent operations which can be combined
  • Two independent operations which can be combined
  • Two independent operations which can be combined
  • Aggregation - What's it to The HDF Group

    1. 1. Aggregation – What’s it to The HDF Group? ESIP Summer Meeting 2013 Mike Folk & Larry Knox The HDF Group 7/11/2013 Aggregations, What's it to you? 1
    2. 2. 1. Why do we aggregate? 2. Aggregation and HDF 3. Types of aggregation in remote sensing 4. nagg 5. Aggregations needs and solutions we would like to see 7/11/2013 Aggregations, What's it to you? 2
    3. 3. caterpillar 7/11/2013 Aggregations, What's it to you? 3
    4. 4. To see a bigger picture 7/11/2013 Aggregations, What's it to you? 4
    5. 5. 7/11/2013 Aggregations, What's it to you? 5
    6. 6. 7/11/2013 Aggregations, What's it to you? 6
    7. 7. “The whole is more than the sum of its parts.” 7/11/2013 Aggregations, What's it to you? 7
    8. 8. Baphuon Temple, Angkor Thom, Cambodia 7/11/2013 Aggregations, What's it to you? 8
    9. 9. Jerusalem 7/11/2013 Aggregations, What's it to you? 9
    10. 10. Seas and lakes of Titan, from Cassini mosaic 7/11/2013 Aggregations, What's it to you? 10
    11. 11. Greater efficiency in storage and transport. 7/11/2013 Aggregations, What's it to you? 11
    12. 12. Greater efficiency in storage and transport. 7/11/2013 Aggregations, What's it to you? 12
    13. 13. If a tool can only work with a single object, aggregation can combine together into a single object all the information we want the tool to use. 7/11/2013 Aggregations, What's it to you? 13
    14. 14. 7/11/2013 Aggregations, What's it to you? 14
    15. 15. 7/11/2013 Aggregations, What's it to you? 15
    16. 16. The LEGO effect • If we store items in smaller and simpler packages, this can enable use to aggregate objects in a greater variety of ways. 7/11/2013 Aggregations, What's it to you? 16
    17. 17. 7/11/2013 Aggregations, What's it to you? 17
    18. 18. 7/11/2013 Aggregations, What's it to you? 18
    19. 19. 7/11/2013 Aggregations, What's it to you? 19
    20. 20. 2. Aggregation and HDF 7/11/2013 Aggregations, What's it to you? 20
    21. 21. HDF5 groups, datasets and attributes / SimOut Viz Parameters 10;100;1000 lat | lon | temp ----|-----|----12 | 23 | 3.1 15 | 24 | 4.2 17 | 21 | 3.6 Timestep 36,000 22
    22. 22. Using HDF for aggregation • It's everywhere • Perhaps the most common reason for using HDF is its ability to support aggregation in a very flexible way. 7/11/2013 Aggregations, What's it to you? 23
    23. 23. Swath Structure SWATHS SwathName: <name> … Swath_1 Swath_2 DataFields: <name> Data fields Profile fields Geolocation fields FieldName: <name> Data field.1 … Data field.n Profile field.1 … Profile field.n Longitude Latitude Time
    24. 24. 3. Types of aggregation for remote sensing 7/11/2013 Aggregations, What's it to you? 25
    25. 25. Types of aggregation for remote sensing • Temporal: Arranging according to time. • Spatial: Arranging according to space. • Packaging: Grouping a variety of related objects. • An aggregation may consist all instances of an object over the dimensional extent. Or it may be a sampling of instances of an object over the dimensional extent. 7/11/2013 Aggregations, What's it to you? 26
    26. 26. 4. nagg 7/11/2013 Aggregations, What's it to you? 27
    27. 27. What is nagg? Nagg is a tool for rearranging NPP data granules from existing files to create new files with a different aggregation number or a different packaging arrangement. Aggregations, What's it to you? 7/11/2013 28
    28. 28. Definitions • Granule – A grouping of measurements or derived data spanning a defined period (e.g., 28.6 seconds) and integer number of sensor scans. • Geolocation products – Geolocation information is stored in the same manner as other data. – Geolocation products may be packaged with data files, or they may be in separate files. • Aggregation1 – A collection of temporally ordered granules within a JPSS HDF5 file. – Compatible NPP data products together or with corresponding geolocation product in common files. 1 JPSS Common Data Format Control Book – External Volume I, p 76 7/11/2013 Aggregations, What's it to you? 29
    29. 29. Nagg operations Aggregation Packaging • Aggregate data granules • De-aggregate data granules • Re-aggregate data granules • Package granules of multiple compatible products in common files • Un-package products into separate files for each product • -g no or –g <product> 7/11/2013 Aggregations, What's it to you? 30
    30. 30. Nagg operations Aggregation Packaging • Aggregate data granules • De-aggregate data granules • Re-aggregate data granules • Package granules of multiple compatible products in common files • Un-package products into separate files for each product • -g no or –g <product> 7/11/2013 Aggregations, What's it to you? 31
    31. 31. Aggregation Increase number of granules per aggregation from 1 to 4 Input files (8 + 8 geo) 0:31:12 0 0 0:31:44 0 0 0:32:16 0 0 0:32:48 0 0 0:33:20 0 0 0:33:52 0 0 0:34:24 0 0 0:34:56 0 0 SATMS Geolocation product is processed automatically and packaged with sensor data product by default. Command: nagg –n4 –t SATMS SATMS*.h5 Input files: 8 SATMS*.h5 files & 8 GATMO*.h5 files Output: Produced 4 granules in GATMOSATMS_npp_d20120404_t0031123_e0033199_b02251_c2 0120920193004057328_XXXX_XXX.h5 Produced 4 granules in GATMOSATMS_npp_d20120404_t0033203_e0035279_b02251_c2 0120920193004110634_XXXX_XXX.h5 GATMO Aggregations, What's it to you? 7/11/2013 32
    32. 32. Aggregation Increase number of granules per aggregation from 1 to 4 Input files (16) 0:31:12 0 0:31:44 0 0:32:16 0 0:32:48 0 0:33:20 0 0:33:52 0 0:34:24 0 0:34:56 0 SATMS 0 0 0 0 0 0 0 0 Output files (2) 0:31:12 0 0:31:44 1 0:32:16 2 0:32:48 3 0:33:20 0 0:33:52 1 0:34:24 2 0:34:56 3 0 1 2 3 GATMO Aggregations, What's it to you? 7/11/2013 0 1 2 3 33
    33. 33. Nagg operations • Aggregation • Packaging • Aggregate data granules • De-aggregate data granules • Re-aggregate data granules • Package granules of multiple compatible products in common files • Un-package products into separate files for each product • -g no or –g <product> 7/11/2013 Aggregations, What's it to you? 34
    34. 34. Packaging Package SATMS,TATMS,GATMO products Input files (22) 0:31:12 0 0:31:44 0 0:32:16 0 0:32:48 0 0:33:20 0 0:33:52 0 0:34:24 0 0:34:56 0 SATMS 7/11/2013 TATMS 0 0 0 0 0 0 0 0 0 0 0 0 0 0 GATMO Fill granules will be created for missing granules from missing files. Command: ../nagg –t SATMS,TATMS ../testfiles/SATMS*.h5 ../testfiles/TATMS*.h5 Output (8 files): Produced 1 granules in GATMO-SATMSTATMS_npp_d20120404_t0031123_e0031370 _b02251_c20120921043859559810_XXXX_XX X.h5 Produced 1 granules in GATMO-SATMSTATMS_npp_d20120404_t0031443_e0032159 _b02251_c20120921043859591107_XXXX_XX X.h5 … Produced 1 granules in GATMO-SATMSTATMS_npp_d20120404_t0034563_e0035279 _b02251_c20120921043859765891_XXXX_XX X.h5 Aggregations, What's it to you? 35
    35. 35. Packaging Package SATMS,TATMS,GATMO products Input files (22) 0:31:12 0 0:31:44 0 0:32:16 0 0:32:48 0 0:33:20 0 0:33:52 0 0:34:24 0 0:34:56 0 SATMS 7/11/2013 TATMS 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Output files (8) 0:31:12 0 0:31:44 0 0:32:16 0 0:32:48 0 0:33:20 0 0:33:52 0 0:34:24 0 0:34:56 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 GATMO Aggregations, What's it to you? 36
    36. 36. 5. Aggregation needs and solutions we would like to see 7/11/2013 Aggregations, What's it to you? 37
    37. 37. Types of aggregation for remote sensing • Temporal: Arranging according to time. • Spatial: Arranging according to space. • Packaging: Grouping a variety of related objects. • What else? • What is a granule? • Could there be common vocabulary and model that spans the wide variety of products and types of aggregation? 7/11/2013 Aggregations, What's it to you? 38
    38. 38. An aggregation/de-aggregation reference model? 7/11/2013 Aggregations, What's it to you? 39
    39. 39. Questions/comments? 7/11/2013 Aggregations, What's it to you? 40
    40. 40. M.C. Escher 7/11/2013 Aggregations, What's it to you? 41

    ×