Metadata For Humans and Machines

675 views
531 views

Published on

New data access paradigms support a variety of human and machine access paths with data servers (THREDDS, https://www.unidata.ucar.edu/software/thredds/current/tds/ and Hyrax, http://opendap.org) that support multiple services for a given dataset. We need metadata that can describe those services and unambiguously differentiate between access paths for humans and for machines. The ISO 19115 metadata standard includes service metadata and allows data and services for that data to be described in the same record. I propose that we use the service metadata for machine access and the more traditional distribution information for human access. This talk was presented at the ESIP (espied.org) meeting during January 2014.

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
675
On SlideShare
0
From Embeds
0
Number of Embeds
8
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Historic access patterns: download entire file do analysis on desktop.
  • FGDC metadata included URLs without much supporting information
  • Those links pointed to html pages that needed to be interpreted by humans in order to access the data.
  • Today we have more diverse users and serve them with multiple services connected to the data with THREDDS Data Server or Hyrax
  • The THREDDS Data Server connects data sets to multiple services
  • The close-up of the Access options shows links to nine different services.
  • Some are data services
  • Some are metadata services implemented as part of the ncISO and ncWMS packages.
  • Links to tools / viewers are also supported (with data download)
  • Multiple data services for a single dataset are a good thing – like motherhood and apple pie.
  • The service metadata is added to the dataset metadataibNcML by ncISO. This dataset has five services.
  • The FGDC online resource included just a URL. This worked well for simple URLs, but became less useful as web usage and diversity increasde.
  • The ISO 19115 standard added more elements to the URL’s to make them more useful for humans.
  • This made humans happy!
  • The ISO metadata standard allows multiple identificationInfo objects in the same record. The ISO metadata can include both data and service identification. The distribution information section includes two transfer options.
  • The identificationInfo section can be implemented as MD_DataIdentification or SV_ServiceIdentification. Note the XML ids used so that users can easily identify the services.
  • This service metadata is for machines
  • The distributionInfo section has a transfer option (onlineResource) for the human OPenDAP interface and for the viewer.
  • These distribution mechanisms are for humans
  • Metadata For Humans and Machines

    1. 1. Metadata For Humans and Machines Ted Habermann Director of Earth Science The HDF Group thabermann@hdfgroup.org January 8-10, 2014 ESIP Winter 2014 1
    2. 2. Evolving Data Access HDF HDF FTP HTTP Data Center January 8-10, 2014 ESIP Winter 2014 2
    3. 3. Evolving Data Access HDF Data Center January 8-10, 2014 HDF FTP HTTP <metadata> <distinfo> <stdorder> <digform> <digtinfo> <formname> ASCII </formname> <transize> 208.000000 </transize> </digtinfo> <digtopt> <onlinopt> <computer> <networka> <networkr> http://www.ngdc.noaa.gov/geomag/aromag.shtml </networkr> </networka> </computer> </onlinopt> </digtopt> </digform> <fees> Depends on the Data Set </fees> <ordering> Price information is available upon request. </ordering> <turnarnd> 4 Days </turnarnd> </stdorder> </distinfo> </metadata> ESIP Winter 2014 3
    4. 4. Evolving Data Access HDF Data Center January 8-10, 2014 HDF FTP HTTP <metadata> <distinfo> <stdorder> <digform> <digtinfo> <formname> ASCII </formname> <transize> 208.000000 </transize> </digtinfo> <digtopt> <onlinopt> <computer> <networka> <networkr> http://www.ngdc.noaa.gov/geomag/aromag.shtml </networkr> </networka> </computer> </onlinopt> </digtopt> </digform> <fees> Depends on the Data Set </fees> <ordering> Price information is available upon request. </ordering> <turnarnd> 4 Days </turnarnd> </stdorder> </distinfo> </metadata> ESIP Winter 2014 4
    5. 5. Evolving Data Access T H H R Y E or R D A D X S OPeNDAP HTTP WMS/WCS Metadata Data Center January 10, 2014 5
    6. 6. THREDDS = Data + Services January 8-10, 2014
    7. 7. THREDDS = Data + Services January 8-10, 2014
    8. 8. THREDDS = Data + Services Data Services ESIP Winter 2014 January 8-10, 2014 8
    9. 9. THREDDS = Data + Services Metadata Services ESIP Winter 2014 January 8-10, 2014 9
    10. 10. THREDDS = Data + Services Tools January 8-10, 2014
    11. 11. Multiple Services = Good Thing January 8-10, 2014 ESIP Winter 2014 11
    12. 12. Service Metadata (nciSO NcML) <group name="THREDDSMetadata"> <attribute name="id" value="hdf4DatasetScan/NASAHDF/SW_S3E_2003100.20053531923.hdf" /> <attribute name="full_name" value="NASAHDF/SW_S3E_2003100.20053531923.hdf" /> <attribute name="data_type" value="GRID" /> <group name="services"> <attribute name="opendap_service" value="http://eosdap.hdfgroup.uiuc.edu:8887/thredds/dodsC/mnt/ftp/ pub/outgoing/NASAHDF/SW_S3E_2003100.20053531923.hdf" /> <attribute name="httpserver_service" value="http://eosdap.hdfgroup.uiuc.edu:8887/thredds/fileServer/ mnt/ftp/pub/outgoing/NASAHDF/SW_S3E_2003100.20053531923.hdf" /> <attribute name="wcs_service" value="http://eosdap.hdfgroup.uiuc.edu:8887/thredds/wcs/mnt/ftp/pub/ outgoing/NASAHDF/SW_S3E_2003100.20053531923.hdf? service=WCS&amp;version=1.0.0&amp;request=GetCapabilities" /> <attribute name="wms_service" value="http://eosdap.hdfgroup.uiuc.edu:8887/thredds/wms/mnt/ftp/pub/ outgoing/NASAHDF/SW_S3E_2003100.20053531923.hdf? service=WMS&amp;version=1.3.0&amp;request=GetCapabilities" /> <attribute name="nccs_service" value="http://eosdap.hdfgroup.uiuc.edu:8887/thredds/ncss/grid/mnt/ftp/ pub/outgoing/NASAHDF/SW_S3E_2003100.20053531923.hdf/dataset.html" /> </group> January 8-10, 2014 ESIP Winter 2014 12
    13. 13. ISO Online Resource January 8-10, 2014 ESIP Winter 2014 13
    14. 14. ISO Online Resource January 8-10, 2014 ESIP Winter 2014 14
    15. 15. ISO Online Resource January 8-10, 2014 ESIP Winter 2014 15
    16. 16. Metadata (ISO) 1 data identification 4 service identification 2 transfer options January 8-10, 2014 ESIP Winter 2014 16
    17. 17. Service Information January 8-10, 2014 ESIP Winter 2014 17
    18. 18. Service Information January 8-10, 2014 ESIP Winter 2014 18
    19. 19. Distribution Information January 8-10, 2014 ESIP Winter 2014 19
    20. 20. Distribution Information January 8-10, 2014 ESIP Winter 2014 20
    21. 21. Questions? tedhabermann@hdfgroup.org 21
    22. 22. Acknowledgements This work was partially supported by contract number NNG10HP02C from NASA. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author and do not necessarily reflect the views of NASA or The HDF Group. July 24, 2013 BESSIG 22

    ×