This document provides an overview of Globus Data Portals and the Django Globus Portal Framework. Globus Data Portals allow users to securely discover, access, and act on research data through a customizable web interface. The Django Globus Portal Framework enables researchers to quickly build portals by leveraging Globus services like Search, Transfer, and Auth for features like authenticated access, data discovery, and integration with custom applications. It discusses key portal capabilities, architecture, and provides a link to demo building a new portal using the framework and example repo.

1. Globus Portal Framework
An introduction to Globus Data Portals
October 15, 2021

2. Agenda
• Globus Data Portals
– FAIR data
– Data discovery
– Custom use-cases
• Django Globus Portal Framework
– Globus Auth
– Globus Services
– Integration with custom resource servers
• Portal Architecture
– Lifecycle of a request
• Live Demo: Build your own portal

3. What is a Globus Data Portal
A Web interface for a Globus Search index with
integrated Globus capabilities to find, access, and act
on data regardless of location

4. Why we use portals
• Different experiments (beamlines, electron
microscopes, biology, etc) generate data with
different types, size and experimental information
• Processing, curation, and cataloging need to
happen as soon as possible so data are not lost
• Standardize secure access between users
• Work toward FAIR datasets to enable more science

5. Advantages of a portal
• Make data FAIRer
• Track lots of data
– Tens of thousands of datasets
• Discovery
– Free text search in Globus Search
– Filtering on specific values
– User Friendly GUI
• Authorized Access
– Public/private, group-based, subject-level ACLs
• Integration with Globus Services
• Customizability

6. Portal Features
• Watch curated data appear in your portal!
– A Web interface to data collection
– Data acquisition in real time
o Synchrotron serial crystallography
• HTTPS Access to data
– Show images in a portal from a Globus Endpoint
– Add links to specific files for user download
• Easy customizability
– Add your own templates
– Rewrite existing views
– Create and call out to custom services

7. ALCF Community Data Co-Op: acdc.alcf.anl.gov
• A collection of data portals in collaboration with ALCF
• Leverage ALCF resources
– Eagle: 100PB store, 650GB/s transfer, Globus interfaces

8. CosmoFlow Search

9. Serial Crystallography

10. Leveraging the Globus
data platform…

11. Globus Search
Globus Search
• Scalable, secure search
for research data
• Metadata store with fine-
grained access controls
• Schema agnostic
• Free text, faceted search
• Integrated with core
platform services, e.g.
Auth, Groups
Input form Extract
Metadata
Ingest metadata, set
visibility policies
Discovery
POST /index/123
{"filters": [
{"field_name": "record_year",
"values": ["2020"],
"type": "match_all" },
{"field_name": "temp_farenheit",
"values": [{"from": 90, "to": "*"}],
"type": "range" ]}
]}
Query
Bulk
ingest

12. Globus Portal Framework
• Django framework for rapidly creating portals
• Python Package for bootstrapping a portal
– Install from PyPi using pip
– github.com/globus/django-globus-portal-framework
• Globus Portal Framework Deployment
– ALCF Community Data Co-Op (acdc.alcf.anl.gov)
• Example Repo
– github.com/globus/django-globus-portal-framework-example

13. Globus Portal Framework Features
• Authorized Access with Globus Auth
• Plugins for Globus Services
– Globus Search
– Globus Transfer
– Globus Flows
– Third-Party Globus Services
• Customizability
– Redesign how your portal talks to these services
– Extend Globus Portal Framework with new features

14. APS XPCS: secure data discovery
14
Globus Auth
Globus Groups
Globus Search

15. APS XPCS: data access & preview
15
Globus Transfer
HTTPS access

16. APS XPCS: automated processing & indexing
16
Globus Flows:
Transfer,
analysis, and
ingest to
search index

17. Portal Architecture

18. Portal Architecture
• User Authentication
• Globus Portal Framework
– Portal URL Mappings
– Token Loading
• Service Calls to Globus Search
• Lifecycle of a Request
• Post Processing search requests

19. User authentication
• Scopes are configured in the portal
• Users authenticate with Globus using 3-legged flow
– Python Social Auth used for Authentication backend
• User tokens are saved in the database
• Future requests authorized with user access tokens
– New searches use Search bearer token.

20. Service Calls
• Most Portal Service calls use the Globus SDK
– Globus Portal Framework loads tokens from DB
– Globus SDK object instantiated with token
– Call to Globus Service
– Portal renders result in templates

21. Globus Portal Framework URLs
• URLs are laid out in three major categories
– Index Selection
– Index Search page
– Search Subject detail page
• Many Globus Search Indices are supported
• Index Search page links to many result subjects
• Each subject has a unique URL

22. Example: Cosmology
22

23. Format of a URL

24. An index is Configuration Driven
• A Search Index can be setup through configuration
– Add Globus Search index uuid
– Add a name
– Add facets
– Add fields
• Start Searching!

25. Lifecycle of a request
• User makes a query
• Portal sends request to Globus Search
– Request contains user bearer token
• Portal receives response
• Portal does processing on response
– Parse Dates, build URL for Globus webapp, etc.
• Portal renders data into templates
• User receives a search page

26. Pilot
• Tooling to simplify ingesting metadata
• Create valid GMeta entries from JSON
• Upload files to an associated endpoints and track
HTTPS URLs to add to GMeta

27. Demo: Creating your own portal
• Django Globus Portal Framework
– github.com/globus/django-globus-portal-framework
– django-globus-portal-framework.readthedocs.io/en/stable/
• Example Repo
– github.com/globus/django-globus-portal-framework-example