How to use NCI's national repository of big spatial data collections

How to use NCI’s national repository of
big spatial data collections
Jingbo Wang & Ben Evans
Drones and Big Spatial data
16 June 2017
Canberra

nci.org.au© NCI Australia 2017
Overview of datasets at NCI
• climate and weather models
• satellite images
• bathymetry and elevation
• hydrology
• geophysics
• Also: optical astro, genomic
and social sciences
NCI makes available national reference datasets – especially those produced
by the government agencies. It is brought together at NCI and organised for
both high performance computation & high performance data analysis, as
well as making available more broadly to the research community.

NCI provides user with Data-as-a-Service
User
generate/transfer
data
Fast data storage Data
Management
Portal
Web-time analytics
softwareHPC
Data Curation,
Publish, Citation
Data
Manager fill
DMP and
create
catalogue
Super
computer
users
Paper and Data
are published
Data
visualization
Visualisation
tools
Data share and
re-use
What do we do?

Data Collections At-a-glance
NCI’s main website on data collection
http://nci.org.au/services/data-collections/
Briefly shows the data collections at NCI. There are some
links to click for “services”, and “data collections” on the
left column.

Summary table of collections:
http://nci.org.au/services/data-
collections/

Where the datasets are stored?
Datasets are stored on the NCI global filesystems. These are called /g/data1, /g/data2,
/g/data3, …
The data stored on these filesystems are available to NCI’s raijin, the VDI, and data services.

How to find information about the datasets?
The easiest places to find information of NCI’s data collection are through the NCI
Data Catalogue:
• https://datacatalogue.nci.org.au
The data at NCI is deeply integrated with activities with the national science orgs
and so are the trusted datasets. e.g. GA, BoM, internationally replicated.

Data Catalogue Front page: https://datacatalogue.nci.org.au

Data Browsing Exercise
Exercise: Browse data catalogue https://datacatalogue.nci.org.au
Try to find “eMAST” collection. How many subcollections does
”eMAST” have? Can you find the datasets under each subcollection?
Hint: you can find “eMAST TERN” from the left column index, or click
the thumbnail image of “eMAST”. There are 6 subcollections, and
each subcollection has a number of datasets.

Exercise : https://datacatalogue.nci.org.au
Try to search “CMIP5” collection using the search utility.
Hint: On the right top corner, click “search”, type “CMIP5” into the
title field.
Data Searching Exercise

Data catalogue offers search function:

CMIP5
Sub-Collections
Datasets
Collection
CSIRO-BOM
ACCESS 1.0
CSIRO-Mk3.6.0
CSIRO-BOM
ACCESS 1.3
piControl
amip
historical
historicalExt
rcp26
rcp45
rcp85
abrupt4xCO2
1pctCO2
piControl
amip
historical
historicalExt
historicalGHG
historicalNat
rcp26
rcp45
rcp85
abrupt4xCO2
1pctCO2
sstClim
sstClim4xCO2
sstClimAerosol
sstClimSulfate
piControl
amip
historical
historicalGHG
historicalMisc
historicalNat
rcp26
rcp45
rcp60
rcp85
sstClim
sstClim4xCO2
sstClimAerosol
sstClimSulfate
MidHolocene
Datasets are stored within data subcollections and collections as
parent-child relationships.

What Data Catalogue offers:
• Tells you what the data is about
• Provide links and references to data
• Provide link to the license files
• Show child-parent relationship between records
• Search capability

Brief Overview
This session will provide an overview of NCI’s web based THREDDS data services:
OPeNDAP
NetCDF Subset Service (NCSS)
OGC Web Map Service (WMS)
OGC Web Coverage Service (WCS)
Godiva Data Viewer
Direct download (if applicable)
Key Objectives
You should leave this session understanding:
The range of data services available with NCI’s Data Collections
Different remote data access methods and which situations they are most applicable
Data services overview

Data services at NCI
What data servers does NCI support?
NCI’s primary data services supported at NCI are:
Production data services, largely based on OPeNDAP, OGC WMS, WCS and WFS
• THREDDS (TDS)
• NCI GSKY
• ESGF (services used for international Climate Model Intercomparison Projects (MIPS))
• (plus other domains like Astronomy IVOA and Social Science unit-record services)
Other NCI data servers used for specifically targeted activities
• ERDDAP, GeoServer, Rasdaman
Additional services in development (with capabilities beyond above)
• WPS, WCPS, Point-cloud
These data services are used by a range of both well-known web sites, and a large range of standard
packages. We will walk-through some examples in this training.
The most broad-scale, general purpose server has become TDS because of the range of data
services and protocols supported.

Accessing data
When searching and browsing data,
published collections will have a direct
link to NCI’s Data Services.
But what do we do from there?

THREDDS data services
What is THREDDS?
THREDDS (Thematic Realtime Environmental Distributed Data Services) data server (TDS) developed by
Unidata (UCAR) Allows for browsing and accessing of data (as well as metadata)
Name Description
OPeNDAP (DAP2) Protocol enabling data access and subsetting through the web
NetCDF Subset Service (NCSS) Web service for subsetting files that can be read by the netCDF
java library
Web Map Service (WMS) OGC web service for requesting static images of data
Web Coverage Service (WCS) OGC web service for requesting data in some output format
Godiva Data Viewer Tool for simple visualisation of data
HTTP File Download Direct downloading

Brief overview of these services:
Name Description
OPeNDAP (DAP2) Protocol enabling data access and subsetting through the
web
NetCDF Subset Service (NCSS) Web service for subsetting files that can be read by the
netCDF java library
Web Coverage Service (WCS) OGC web service for requesting data in some output
format
Data subsetting options

Allows for:
Remote access and subsetting. In many
tools, the OPeNDAP data URL can also be
used in the same manner as in-situ files.
Ideal for:
Remote use with tools and software when
working with small subsets of large
datasets.
When there is no pressing need to
download data.
Possible cons:
Everything is index based.
OPeNDAP
OPeNDAP

Allows for:
Data subsetting of large files based on
spatial and/or temporal queries.
Ideal when:
Needing to extract data for specific
locations or time. Unlike OPeNDAP,
which is all index based, NCSS will do
the work to convert the spatial/temporal
query into index locations.
Possible cons:
Downloading data locally (when using
through the THREDDS interface).
NetCDF Subset Service (NCSS)
Netcdf Subset Service (NCSS)

Name Description
web
NetCDF Subset Service (NCSS) Web service for subsetting files that can be read by the
netCDF java library
Web Coverage Service (WCS) OGC web service for requesting data in some output
format
OGC Web Services

OGC Web Services
Web Map Service (WMS)
Web Coverage Service (WCS)
Allows for:
Allows for generation of static images for data previewing
Ideal when:
Ideal when further analysis is not required, or when data are for display on web-
based mapping systems (e.g., web portals)
Allows for:
Allows for requesting raw data from remote sources as TIFF or NetCDF files
Ideal when:
Ideal when collecting useable subsets of remote datasets for further analysis

Name Description
web
Netcdf Subset Service (NCSS) Web service for subsetting files that can be read by the
netCDF java library
Visualising data

Godiva: ncWMS data viewer
Uses the ncWMS service developed
by the Reading e-Science Centre
http://www.resc.rdg.ac.uk/trac/ncWMS/
ncWMS is a Web Map Service
(WMS) for data stored in CF-
compliant netCDF files

Godiva Data Viewer
Godiva: ncWMS data viewer
Ability to browse through time and
space
Make animations
Export KMZ format
Allows for:
Quick visualisation from your web
browser.
Ideal when:
Inspecting data before accessing
through other services.
Con:
Not all coordinate projections are
supported.

Name Description
web
Netcdf Subset Service (NCSS) Web service for subsetting files that can be read by the
netCDF java library
Downloading full file

Data Download
Allows:
Users to directly download files to
individual workstations.
Ideal when:
Direct access or other data services are
not sufficient in particular workflows.
Possible cons:
Downloading data, particularly if you
only need to work with small subset of
larger collection. Volumes may be very
large. Creating copies can introduce
versioning problems.
HTTP Download
Image:
https://www.google.com.au/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&ved=0
ahUKEwjQrJi50onQAhVJopQKHab7CwsQjRwIBw&url=http%3A%2F%2Ff4bg.com.au%2Fdownload
-f4bg-documents-
tools%2F&psig=AFQjCNHjE8Dk8RwN_uG2NHnf6dsGS3e2tA&ust=1478161379040807

A roadmap for OGC Web Services at NCI
What is NCI’s approach to WFS, WPS, WCPS?
- Web Feature Service (WFS) offers a WCS-like capability for vector and point data - users can
request subsets of vector and point data for download. NCI’s production GeoServer implements
WFS, and It will become available as users request the service.
- Web Processing Service (WPS) is an OGC standard for transactions between a web client and
server-based processing power. NCI’s production GeoServer implements a set of WPS
operations on data published via Geoserver. Additionally, NCI are developing WPS services for
specific tasks. Some of these will be demonstrated on day 2.
- Web Coverage Processing Service (WCPS) is being implemented using the Rasdaman raster
data manager, as part of the EarthServer 2 initiative.

A side note on data that might have data access restrictions
1. Access data by logging into
raijin.nci.org.au
2. Access data through VDI
If you have NCI account and you are
the member of a data project:
1. Access open data through data
services
or apply for an account and join the
data project (need approval from
project lead CI)
If you do not have NCI account:
If the data has access restrictions (the terms and conditions) then the user needs to
be granted read-only access to the data. This can be done through Mancini
http://my.nci.org.au

Examples Search Data and Use Data Services

THREDDS embedded “Godiva” Viewer
1. open either https://geonetwork.nci.org.au or
https://datacatalogue.nci.org.au from a web browser
2. Search project code “fx3”, you will find the record “CSIRO Coastal
Modelling Products (eReefs) ”
3. click the THREDDS link

Godiva Viewer
4. You will see the THREDDS link.
You can also go to this page directly by clicking
http://dapds00.nci.org.au/thredds/catalogs/fx3/catalog.html

Godiva Viewer
5. Randomly choose a file
and open it.
You will find the Godiva2
at the bottom of the page,
click it.

Godiva Viewer
6. Expanding the
“+”, where a list of
variables are shown.
choose one that you
like to view.
The image will be
shown on the right.
Adjust colour scale if
needed.

View TDS data on National Map
1. Find a WMS URL that you
want to view from the TDS page.

2. open http://nationalmap.gov.au/
3. click “Add data”, it will
pop up a window ---------->
4. click “My Data”
5. choose “Add Web Data”

6. right-click the WMS link, choose
“Copy Link Location”

7. paste the URL
of the WMS link (copied from
the TDS page)
8. click “Add” button

9. A list of variables will
show up if clicking the
dataset name
10. select one random
variable, like “depth”,
the image will appear

11. click “Add to the map”

Take away messages:
• search datasets at https://datacatalogue.nci.org.au
– From the record - find file path on the /g/data filesystem and/or
http://dapds00.nci.org.au
– access data through various data services
• Please Let us know if you have trouble. e.g, could not easily find the data or if you
feel the data or service is not functioning for your application.
• more training are available upon request:
– VDI
– advanced features of data formats
– Standard various packages, such as QGIS, paraview
Contact us: help@nci.org.au or Jingbo.Wang@anu.edu.au

For portal site developers
Our data catalogues are also made available through geonetwork https://geonetwork.nci.org.au
Various communities harvest our catalogue information for broader community access. These are
”aggregator services” and examples include Research Data Australia, AuScope, TERN. For these data
discovery portal programmers, our catalogue information is also provided through GeoNetwork
catalogue data services:
Essential GeoNetwork catalogue features are :
• Provide links and references to data
• Provide link to the license files
• Show child-parent relationship between records
• Search capability
Additional useful functions in geonetwork
• Basic web map services
• Allow harvesting by CSW (Catalogue Service for the Web) end points

GeoNetwork catalogue: https://geonetwork.nci.org.au

Exercise : Browse data catalogue https://geonetwork.nci.org.au
Try to find “CMIP5” dataset.
Hint: type “CMIP5” in the search window.

Exercise : https://geonetwork.nci.org.au
Try to find the license file of collection: “3D Geologic Models of
Australia”.
Hint: search the title of “3D Geologic Models of Australia” first. The
license file is on the default view.

Exercise : https://geonetwork.nci.org.au
Try to find the DOI and the local file path of collection: “LOVECLIM
Last Glacial Maximum oceanic d13C and D14C v 1.0”.
Hint: search the title of “LOVECLIM Last Glacial Maximum oceanic
d13C and D14C v 1.0” first. Click the full view on the top right corner,
then click “complete” tab.

Click full view of a record on the right top corner:
You will find the dataset URI on
the top.
Keep scrolling down until
almost the bottom of the page,
you will find the local file path.

How to use NCI's national repository of big spatial data collections

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to How to use NCI's national repository of big spatial data collections

Similar to How to use NCI's national repository of big spatial data collections (20)

More from ARDC

More from ARDC (20)

Recently uploaded

Recently uploaded (20)

How to use NCI's national repository of big spatial data collections