Download to read offline
Uploaded byGlobus
Gateways 2020 Tutorial - Automated Data Ingest and Search with Globus
The document outlines automated data ingestion processes for science gateways using Globus, emphasizing challenges such as transferring large datasets and managing metadata effectively. Key components include the use of Globus Connect Server for data storage, persistent identifiers through a fair research identifier service, and the implementation of a RESTful API for metadata management. It also details the Globus Search API features that facilitate scalable indexing and search capabilities for indexed metadata.
More Related Content
Similar to Gateways 2020 Tutorial - Automated Data Ingest and Search with Globus
Gateways 2020 Tutorial - Automated Data Ingest and Search with Globus
- 1. Simplifying Science GatewayData Management with Globus Part IV – Automated Data Ingests October 2020, Gateways 2020
- 2. Phase 1 -Gather data Gathering datasets from research partners • Your project is gathering datasets from partners. Each dataset is several TBs and takes ~a day to transfer over the network. • For the data to be useful, it needs descriptive metadata. • Ultimately, the team needs to find datasets that match specific criteria.
- 3. What are thedataset ingest challenges? • Getting very large datasets transferred from gateway users’ systems to the central repository – (This is Scenario I - large-scale data transfer.) • Generating persistent identifiers for the data in the central repository so we can link metadata to data • Storing the metadata • Indexing the metadata to enable searching
- 4. Demonstration Data ingests ina web application https://petraldata.net/
- 5. What needs tobe in place for it to work? • Data storage – Globus Connect Server on Petrel • Persistent identifiers – FAIR Research Identifier Service – Hosted by https://fair-research.org/ • Metadata storage, indexing, search – Globus Search API – Hosted by Globus
- 6. Globus Connect Serveron Petrel • Configured for self-service projects – Researchers do not receive local (Linux) accounts! – Uses Globus for authorization & management • Guest collections and groups – Project PIs request access by applying to join the “Petrel Project Owners” group (using the Globus web app) – Admin creates Globus group, makes PI a group manager – Admin creates guest collection, makes PI an access manager – Admin sets a quota of 100TB for the guest collection
- 7. • RESTful webservice, written in Python, that stores identifier metadata • Mints (creates) identifiers from external service providers using a unified service provider interface (SPI) • Different identifiers supported through namespaces • Client requests served as HTML landing pages or other machine-readable formats (e.g., JSON, JSON-LD) FAIR Research Identifiers AWS-RDS AWS-EC2 Postgres Registration SPI (Python) Web Server - REST API (Apache, Flask, Python) RDBMS ORM (SQLAlchemy) AuthN/AuthZ (Globus Auth, Globus Groups) Web Browser Client APIs HTML JSON, JSON-LD, other extensible renderings DataCite (DOI) EZID (ARK) Minid (Handle) https://minid.readthedocs.io/en/develop/
- 8. • REST APIprovides a simple CRUD interface • Has other capabilities, like finding identifiers by checksum • JSON is used for request and response • Namespaces may also have their own handlers, landing pages, and other customizations. FAIR Research Identifiers
- 9. Globus Search API •RESTful API for indexing & search – Hosted by Globus (including the metadata & index storage!) – Each project gets an “index” object (private tenancy) – REST API, Python client package, Python CLI • https://docs.globus.org/api/search/
- 10. Globus Search APIfeatures • Scalable: to billions of entries • Schema agnostic: can use standard (e.g., DataCite) or custom metadata • Fine-grain access control: only returns results that are visible to user • Plain text search: ranked results • Faceted search: for data discovery • Rich query language: ranges, expressions, regex, fuzzy, stemming, etc.
- 11. Key ingredients 1. UUIDand base path for the guest collection where data is gathered 2. Minid Python client 3. UUID for Globus Search index 4. Your choice of appropriate metadata schema for your project’s datasets
- 12. Code Data ingest ina web application
Editor's Notes
- #3 You’re working on a project with partners at other institutions, each of whom is analyzing unique samples and generating big datasets from them. You need to gather 100s of TBs of data on your campus’s HPC storage system. How can you make it easy for your partners to get the data from their labs to your server? And once it’s there, how are the partners going to understand each others’ datasets? First, they need to be able to see, in general, what’s been uploaded. Then, they need to find datasets that have specific features. NOTE: We’re presenting this as a single project, but at Globus, we see this happening for dozens-to-hundreds of research projects on a continuous basis. Our end goal is to enable research teams to do this routinely, without special planning or extraordinary measures by individual projects.
- #4 Examples of “analysis on a community dataset”: Examples of ”analyze user’s data”: Examples of “download simulation results”: Examples of “submit data to a repository”:
- #5 Petrel Data: https://petreldata.net/ Data storage is provided by the Advanced Leadership Computing Facility at Argonne National Laboratory. Petrel offers 100TB allocations to approved projects, with a total of 3PB of storage. Goal is to enable projects to self-manage themselves, including ingest, metadata management, index & search, and sharing permissions.
- #6 PIs request access by applying to join a Globus group Petrel admin creates a project group for the PI and makes the PI a group manager Petrel admin creates a Globus guest collection with access managed by the PI Petrel admin also sets a quota of 100TB for the guest collection’s directory.
- #13 https://github.com/globus/globus-jupyter-notebooks/blob/master/Data_Publication_Flow.ipynb