ACEF: Australian Coastal Ecosystems Facility
•Download as PPTX, PDF•
1 like•698 views
The presentation provides an overview of services offered by ACEF. The presentation was part of the Workshop on Approaches to Terrestrial Ecosystem Data Management : from collection to synthesis and beyond which was held on 9th of March 2016 in University of Queensland.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (15)
Similar to ACEF: Australian Coastal Ecosystems Facility
Similar to ACEF: Australian Coastal Ecosystems Facility (20)
More from TERN Australia
More from TERN Australia (20)
Recently uploaded
Recently uploaded (20)
ACEF: Australian Coastal Ecosystems Facility
- 1. ACEF Australian Coastal Ecosystems Facility Sharon Tickell | Senior Software Engineer, CSIRO 9 March 2016
- 2. National Plan for Environmental Information data providers metadata services monitoring sites information models vocabularies environmental features observing procedures Instrumentsdb Featuresregister SISSVoc ‘SolidGround’modelregistry NEMonitoringSitesRegister NEINodes HOW WHAT WHERE Discovery Access Use NEII components - currentNEII standards explore analyse integrate correlate query download search filter NEICatalogue WHEN WHO NEII components - future discovery, download, view web services gauging stations boreholes soil profiles ecological survey plots WaterML GroundWaterML GeoSciML stratigraphy lithology geochemistry water quality coal basins catchments aquifers ANZLIC ISO 2146 ISO 19115 pump meter field guide
- 3. Australian Coastal Ecosystems Facility | Sharon Tickell3 | Part 1: Foundation Services • Coastal Research Portal • Geospatial Data Hosting • Metadata Aggregation • Coastal Data Portal • Continuous Water Quality Monitoring • National Marine Turtle Database • Compute and Storage Support
- 4. • What research is happening in the coastal space? • Who is doing it? Where have they published their data? • Anyone can register a coastal project. Australian Coastal Ecosystems Facility | Sharon Tickell Coastal Research Portal http://coastalresearch.csiro.au 4 |
- 5. • Descriptive / Raster Data: • Land-use Maps • Bathymetry • Ocean colour images • ... • Boundaries / Vector Data: • Catchment contours • River networks • Survey transects • Management zones • ... Linked metadata, full styled feature support, projection translation, regular backups Australian Coastal Ecosystems Facility | Sharon Tickell Geospatial Data Hosting http://acef.tern.org.au/geoserver 5 |
- 6. • Local metadata records for ACEF-hosted data. • Harvested coastal dataset metadata from open services: • AODN • BlueNet • Bureau of Meteorology • CoastalCOMs • CSIRO • eReefs NCI THREDDS • environment.data.gov.au • GeoScience Australia • .. State and Federal Catalogues Australian Coastal Ecosystems Facility | Sharon Tickell Metadata Aggregation http://acef.tern.org.au/geonetwork 6 |
- 7. • Browse for ACEF- hosted datasets • Metadata-supported search • Visualisation for spatial data • Download data for offline use Australian Coastal Ecosystems Facility | Sharon Tickell Coastal Data Portal http://acef.tern.org.au/portal 7 |
- 8. Continuous Water Quality Monitoring Australian Coastal Ecosystems Facility | Sharon Tickell8 | 0 10 20 30 40 50 60 70 13 Sep 2011 03 Oct2011 23 Oct2011 12 Nov 2011 02 Dec 2011 Nitrate(µM) EHMP Site 211 (upstream) EHMP Site 212 (downstream) In-situmeasurments(SUNA) 12 Oct2011 Dep Logan River: Real-time Nitrate Observations vs Monthly Monitoring Data
- 9. Photographic and Video Data Australian Coastal Ecosystems Facility | Sharon Tickell9 |
- 10. Australian Coastal Ecosystems Facility | Sharon Tickell10 | Compute and Storage Support
- 11. Australian Coastal Ecosystems Facility | Sharon Tickell11 | Part 2: Derived Projects • SEQuITOR • Logan Flood Vis • Slacks Creek River Recovery • eReefs • … & more
- 12. • Experimental visualisation. • Combines ACEF Maps, water quality monitoring data and a near-real-time hydrodynamic model of Moreton Bay • Layered map, time series charts, transect slices. Australian Coastal Ecosystems Facility | Sharon Tickell SEQuITOR Data Portal http://www.sequitor.org.au 12 |
- 13. • Mash-up of water-quality monitoring data with gauging-station data, archive photography from Flikr, Instagram. Australian Coastal Ecosystems Facility | Sharon Tickell Logan Flood-Vis http://loganvis.meteor.com/ 13 |
- 14. • Logan City Council regeneration project • Embed local, real-time water quality data in a normal website. Australian Coastal Ecosystems Facility | Sharon Tickell Slacks Creek River Recovery https://riverrecovery.com.au 14 |
- 15. Australian Coastal Ecosystems Facility | Sharon Tickell eReefs http://www.ereefs.info 15 | Funding Bodies: Collaborating Agencies:
- 16. The Data Discovery Problem Australian Coastal Ecosystems Facility | Sharon Tickell16 |
- 17. eReefs Data Brokering Layer Australian Coastal Ecosystems Facility | Sharon Tickell17 | Analysis dashboards Visualisation portals Other client tools DPN 1 DPN 2 DPN 3 Data Brokering Layer Linked Data Vocabularies Harvest Job DPN Definitions Harvest Cache Data Broker API
- 18. • Not yet public! Due for release in the next month or so • Uses vocabulary-based searches for data discovery Australian Coastal Ecosystems Facility | Sharon Tickell eReefs Visualisation http://portal.ereefs.info 18 |
- 19. Presenter: Sharon Tickell Senior Software Engineer CSIRO Oceans and Atmosphere t +61 7 3214 2806 e Sharon.Tickell@csiro.au ACEF Project Team: e team@acef.tern.org.au w http://acef.tern.org.au Australian Coastal Ecosystems Facility | Sharon Tickell19 | Thankyou Referenced Websites and Services: ACEF Website http://acef.tern.org.au Metadata Service http://acef.tern.org.au/geonetwork Geospatial Data http://acef.tern.org.au/geoserver Coastal Data Portal http://acef.tern.org.au/portal Coastal Research Portal http://coastalresearch.csiro.au National Marine Turtle Database http://root-uat.ala.org.au/bdrs- core/acef/home.htm CoastalCOMS https://portal.coastalcoms.com SEQuITOR Portal http://www.sequitor.org.au/ Logan Flood Vis http://loganvis.meteor.com/ Slacks Creek River Recovery https://riverrecovery.com.au/ eReefs http://www.ereefs.info
Editor's Notes
- Australia is a Coastal Country: > 85% of Australians live less than 50km from the coast All but two river catchments in Australia flow to the coast We have significant and internationally important reef, seagrass, mangrove and saltmarsh habitats. There is no specific agency or organisation tasked with managing the coast Many organisations monitor the coastal environment and collect data on water quality, seagrass habitats, mangrove extent, fisheries and many other topics. The amount of data is only increasing as cheap computing power, solar-powered sensors, and ubiquitous networking drive investment in automated monitoring. This has resulted in valuable collections of data, although there has been limited integration of this information : there’s just too much , and it’s too complex to assimilate and analyse using desktop tools. Through several projects, the CSIRO Coastal Research program has been working to bring coastal information together to build national information archives, deliver interoperable data services and fill some of the gaps in coastal environmental monitoring. We have delivered, and continue to deliver several of these in collaboration with TERN, under the banner of the Australian Coastal Ecosystems facility.
- The structure of the ACEF systems is broadly compatible with the National Plan for Environmental Information (NEII => National Environmental Information Infrastructure), which was put together by the Bureau of and is based around a series of standards, linked services and interoperable components. The diagram is a few years old, so some things have different names now, but the idea is still useful. It is intended as a guideline for how to deliver environmental information in Australia so that they can: Deliver multiple types of datasets in ways which users can interact with easily and quickly Include Monitoring data, model data, spatial data Minimise the level of manipulation required to combine datasets Allow end users to extract data from multiple sources We are not attempting to build the one true data portal – we can never know exactly what everyone needs! (Nor could we support it if we did), but we aim to produce a set of stable, reliable data services that are easy to access, and an appropriate set of data discovery tools and usage examples.
- The core ACEF services are strongly geared towards interoperability: Open data Transport and format standards Able to handle any data type Well-built and managed (IT support) Connected to data discovery and visualisation tools. I’m going to give you a brief overview of each of the core ACEF services, and then a glimpse of what CSIRO has been building on top of these services, which should give you and idea of what might be possible.
- What research is happening in the coastal space? Who is doing it? Where have they published their data? Anyone can register a coastal project – several hundred have. Link to your data, metadata, publications. Very good Google-fu.
- Hosts many types of spatial data: Descriptive / Raster Data: Land-use Maps, Bathymetry, Ocean colour images, ... Boundaries / Vector Data: Catchment contours, River networks, Survey transects, ... Many formats: GeoTiff, ESRI-proprietary, Shapefiles, KML, CSV. Often very large files (Tile Servers!) We have sourced data (with permission!) from many agencies including state and federal governments. Political issues around marine reserve boundaries. Working on per-dataset access tracking via google analytics: will continue that back to GeoServer code when we get it working (depends on funding).
- Local metadata records for ACEF-hosted data. Harvested coastal dataset metadata from open services: AODN, BlueNet, Bureau of Meteorology, CoastalCOMs, CSIRO’s Marlin catalogue, eReefs NCI THREDDS, environment.data.gov.au, GeoScience Australia, .. State and Federal Catalogues Serves as a search engine for several ACEF applications. Is harvested in turn by the TERN catalogues
- Browse for ACEF-hosted datasets Metadata-supported search Visualisation for spatial data Download data for offline use
- Observational data collected by instruments coupled with operational and analytical methodologies that enable acquisition of temporally and (or) spatially dense datasets without the need for routine collection and analysis of physical samples other than for periodic calibration. Almost entirely automated, collection many be manual if no phone or satellite network in range. TERN’s SEQ peri-urban supersite in the Logan estuary included the installation three networked, real-time monitoring sensors in the Logan Estuary: Upper, Lower, Albert tributary. A prototype installation was actually live during the 2011 floods, and managed to capture the impact of the flood peak on Turbidity and Salinity, clearly showing the cancellation of normal tidal flushing. We have since added additional monitoring sites in SEQ, including Terranora Creek, Slacks Creek, and a mobile trailer that travels up and down the coast, and spends months at a time setup in interesting places (e.g. post-cyclone or post-flood zones). We handle IMOS buoy data, glider, ship-transects etc. New Methods for Continuous Monitoring: Physico-chemical parameters Nutrients and biogeochemistry Sediments River flow and water movement Multiple options for installation and operation Manual monitoring can be more customised: collect multiple depths, change locations. This is not as flexible, but gives a valuable insight into what happens between manual samples: the two programs complement each other.
- We have partnered with other agencies to handle collection and hosting of image-based data. Opportunistic Data: One-off observations, no planned collection, no method standardisation Difficult to integrate with almost anything. Difficult to verify Often already online. Others already handle this: no need to reinvent the wheel. Collaboration with the Atlas of Living Australia: “Biological Data Recording System” subsite for the National Marine Turtle database (crowd-sourced turtle sightings) and also seagrass. Submitted data is automatically contributed to the ALA We have worked with CoastalCOMS to harvest metadata for their beach cameras (surf, beach morphology etc), so that can be included in search results.
- The ACEF Services and Websites that CSIRO maintains directly are hosted on a collection of 11 Virtual NeCTAR Servers, mostly in the QLD Node which is part of the QRIS Cloud managed by QCIF. Data is backed up to NeCTAR S3 Storage. As usage increases, we have plans to add in geographic redundancy and high-availability clustering to the core data services. Implementation of those is currently constrained by Sysdamin allocation time.
- CSIRO is now using the ACEF services to provide data for its own applications. I’m going to give a quick overview of a few of these, to show what is already possible, and what will be possible once we’ve worked the bugs out of the next generation of tools.
- SEQuITOR was developed alongside the core ACEF services, and was the original test-harness for using those services in a separate product. It was supported by the Australian National Data Service (ANDS), and the Federal Government through the National Collaborative Research Infrastructure Strategy Program (NCRIS) and the Education Investment Fund (EIF) Super Science Initiative The acronym expands to “South-East Queensland Integrated Terrestrial to Ocean Research Network” The SEQuITOR data services were intended to combine model results with observational data across different geographical scales in order to support whole-of-landscape management. It mashes together NetCDF data from a high-ersulution near-real-time model of Moreton bay with the Logan supersite sensor data, streamed in via SensorCloud. That SC server will be re-badged as ACEF in the furture: the original was not quite general enough, but future ones will be. The portal is designed to let users explore, select, download and share the hosted data. Data can be viewed as 2D map, Transect of map, Time Series at multiple points. All the datasets are have proper Data Licenses and metadata, so are citeable. Software is a customisation of the ACEF portal code, which is itself version 2 of the AODN portal. It’s open-source, and code is available via bitbucket.csiro.au
- The next application we build aimed to beyond presenting scientific data to build a narrative of use to the general public. The Logan Floods Mashup: Base map, catchment boundary shapes, Moreton Bay NRT SHOC model, Logan WQ Sensor Data, QLD Govt Gauging Stations, Flikr, YouTube Re-used only existing data services – only needed to build the user interface; took about a month. The driver for this project was the Brisbane floods in January 2013, due to a combination of ex-Tropical Cyclone Oswald and an associated monsoon trough. In South-east Queensland significant flooding occurred over the period January 26 to 30 January, inundating the Logan Albert and catchments, and occurring *just* after the hydro model of the bay was fully up and running. Se we had both modelled and observational data for the time of the flood. Dam releases discharged more than 1819 gigalitres of freshwater into Moreton Bay and the peak flow on 28th January released 391 gigalitres in a day. This lead to some fascinating live timeseries of what was happening in the estuary at a time when it was not possible to perform manual sampling.
- Logan City Council regeneration project Embeds local, real-time water quality data in a normal website: plans to link that with social media, regeneration progress photos etc.
- eReefs is a multi-year collaborative project focused on the Great Barrier Reef Collaborating agencies include GBRF, BOM, CSIRO, AIMS, QLD Government Funding from Caring for our Country, SIEF, Industry Deliverables included improved monitoring, modelling, interoperability and visualization across all contributing agencies. Most are big agencies with existing data services, all of which are incompatible with each other. eReefs integration focus turned to building mechanisms to connect services together.
- Having lots of data is only helpful if you know where it is. Google is not much help with scientific data searches: too many non-scientific documents use the same terms. Even if you search Scholar, you get a lot of papers, very few of which are relevant, and the data they link to is rarely queryable. What we’ve done for eReefs, is combine existing searchable metadata records with semantic-web techniques like linked data URIs. A formal ontology for water quality and marine data was developed – uses environment.data.gov.au namespace, controlled by the Australian linked data working group. Complements NASA and other vocabularies from institutions around the world. The metadata has embedded linked-data URIs as keywords. These can be resolved to understand relationships between concepts: generalization-of sub-category-of, same-as. This lets us build a search engine that understands that ‘plankton’ should also include results for zooplankton species and chlorophyll a remote sensing colourmaps.
- How does this work in practice: The Data Provider Node concept was codified: (formal ontology: DPN-O). DPNs usually combine a data product (model results, observational data etc), a dataset metadata record, a data delivery service (THREDDS, GeoServer, SensorCloud etc) and a ‘data provider node description’ which lists the addresses for all of these. This provides a framework for defining data services which belong to any given provider; ACEF is one, we also have CSIRO Hydrodynamics, CSIRO Remote sensing, BOM remote sensing, QLD Gauging Stations, and more on the way. (click) DPN end-points are aggregated in a Data Brokering Layer (DBL) This uses the DPN description to harvest the provider’s metadata in order to work out what datasets are available. The harvested resolves the metdata URIs to work out how the datasets related to each other, and builds a volatile no-sql cache of possible search results. (click) This content is exposed via the Data Broker API, which allows client applications to search for data of interest by dataset name, vocabulary categories, service type, owner institution, spatial range etc. Queries are much closer to end user needs: e.g. “show me datasets which contain the term ‘temperature’ and are within a particular spatial and temporal extent on a map”
- Discovery and Visualisation Tools: Discovery and visualisation tools only need to know the Data Broker API’s endpoint in order to be able to search for the actual service endpoints that it needs to retrieve data from: no need to bake URIs for 15 different GeoServer instances into your portal. This is the eReefs portal. It’s not public yet – due for release very soon, and I can demo it in the break for anyone who wants a sneak peek.