SlideShare a Scribd company logo
1 of 27
Download to read offline
Globus Portal Framework
An introduction to Globus Data Portals
October 15, 2021
Agenda
• Globus Data Portals
– FAIR data
– Data discovery
– Custom use-cases
• Django Globus Portal Framework
– Globus Auth
– Globus Services
– Integration with custom resource servers
• Portal Architecture
– Lifecycle of a request
• Live Demo: Build your own portal
What is a Globus Data Portal
A Web interface for a Globus Search index with
integrated Globus capabilities to find, access, and act
on data regardless of location
Why we use portals
• Different experiments (beamlines, electron
microscopes, biology, etc) generate data with
different types, size and experimental information
• Processing, curation, and cataloging need to
happen as soon as possible so data are not lost
• Standardize secure access between users
• Work toward FAIR datasets to enable more science
Advantages of a portal
• Make data FAIRer
• Track lots of data
– Tens of thousands of datasets
• Discovery
– Free text search in Globus Search
– Filtering on specific values
– User Friendly GUI
• Authorized Access
– Public/private, group-based, subject-level ACLs
• Integration with Globus Services
• Customizability
Portal Features
• Watch curated data appear in your portal!
– A Web interface to data collection
– Data acquisition in real time
o Synchrotron serial crystallography
• HTTPS Access to data
– Show images in a portal from a Globus Endpoint
– Add links to specific files for user download
• Easy customizability
– Add your own templates
– Rewrite existing views
– Create and call out to custom services
ALCF Community Data Co-Op: acdc.alcf.anl.gov
• A collection of data portals in collaboration with ALCF
• Leverage ALCF resources
– Eagle: 100PB store, 650GB/s transfer, Globus interfaces
CosmoFlow Search
Serial Crystallography
Leveraging the Globus
data platform…
Globus Search
Globus Search
• Scalable, secure search
for research data
• Metadata store with fine-
grained access controls
• Schema agnostic
• Free text, faceted search
• Integrated with core
platform services, e.g.
Auth, Groups
Input form Extract
Metadata
Ingest metadata, set
visibility policies
Discovery
POST /index/123
{"filters": [
{"field_name": "record_year",
"values": ["2020"],
"type": "match_all" },
{"field_name": "temp_farenheit",
"values": [{"from": 90, "to": "*"}],
"type": "range" ]}
]}
Query
Bulk
ingest
Globus Portal Framework
• Django framework for rapidly creating portals
• Python Package for bootstrapping a portal
– Install from PyPi using pip
– github.com/globus/django-globus-portal-framework
• Globus Portal Framework Deployment
– ALCF Community Data Co-Op (acdc.alcf.anl.gov)
• Example Repo
– github.com/globus/django-globus-portal-framework-example
Globus Portal Framework Features
• Authorized Access with Globus Auth
• Plugins for Globus Services
– Globus Search
– Globus Transfer
– Globus Flows
– Third-Party Globus Services
• Customizability
– Redesign how your portal talks to these services
– Extend Globus Portal Framework with new features
APS XPCS: secure data discovery
14
Globus Auth
Globus Groups
Globus Search
APS XPCS: data access & preview
15
Globus Transfer
HTTPS access
APS XPCS: automated processing & indexing
16
Globus Flows:
Transfer,
analysis, and
ingest to
search index
Portal Architecture
Portal Architecture
• User Authentication
• Globus Portal Framework
– Portal URL Mappings
– Token Loading
• Service Calls to Globus Search
• Lifecycle of a Request
• Post Processing search requests
User authentication
• Scopes are configured in the portal
• Users authenticate with Globus using 3-legged flow
– Python Social Auth used for Authentication backend
• User tokens are saved in the database
• Future requests authorized with user access tokens
– New searches use Search bearer token.
Service Calls
• Most Portal Service calls use the Globus SDK
– Globus Portal Framework loads tokens from DB
– Globus SDK object instantiated with token
– Call to Globus Service
– Portal renders result in templates
Globus Portal Framework URLs
• URLs are laid out in three major categories
– Index Selection
– Index Search page
– Search Subject detail page
• Many Globus Search Indices are supported
• Index Search page links to many result subjects
• Each subject has a unique URL
Example: Cosmology
22
Format of a URL
An index is Configuration Driven
• A Search Index can be setup through configuration
– Add Globus Search index uuid
– Add a name
– Add facets
– Add fields
• Start Searching!
Lifecycle of a request
• User makes a query
• Portal sends request to Globus Search
– Request contains user bearer token
• Portal receives response
• Portal does processing on response
– Parse Dates, build URL for Globus webapp, etc.
• Portal renders data into templates
• User receives a search page
Pilot
• Tooling to simplify ingesting metadata
• Create valid GMeta entries from JSON
• Upload files to an associated endpoints and track
HTTPS URLs to add to GMeta
Demo: Creating your own portal
• Django Globus Portal Framework
– github.com/globus/django-globus-portal-framework
– django-globus-portal-framework.readthedocs.io/en/stable/
• Example Repo
– github.com/globus/django-globus-portal-framework-example

More Related Content

What's hot

Automating Research Data Management at Scale with Globus
Automating Research Data Management at Scale with GlobusAutomating Research Data Management at Scale with Globus
Automating Research Data Management at Scale with Globus
Globus
 

What's hot (20)

Globus Command Line Interface (APS Workshop)
Globus Command Line Interface (APS Workshop)Globus Command Line Interface (APS Workshop)
Globus Command Line Interface (APS Workshop)
 
GlobusWorld 2021 Tutorial: Introduction to Globus
GlobusWorld 2021 Tutorial: Introduction to GlobusGlobusWorld 2021 Tutorial: Introduction to Globus
GlobusWorld 2021 Tutorial: Introduction to Globus
 
Introduction to Globus (GlobusWorld Tour West)
Introduction to Globus (GlobusWorld Tour West)Introduction to Globus (GlobusWorld Tour West)
Introduction to Globus (GlobusWorld Tour West)
 
What's New in Globus - Internet2 TechEXtra
What's New in Globus - Internet2 TechEXtraWhat's New in Globus - Internet2 TechEXtra
What's New in Globus - Internet2 TechEXtra
 
Introduction to Globus for New Users (GlobusWorld Tour - UCSD)
Introduction to Globus for New Users (GlobusWorld Tour - UCSD)Introduction to Globus for New Users (GlobusWorld Tour - UCSD)
Introduction to Globus for New Users (GlobusWorld Tour - UCSD)
 
Data Orchestration at Scale (GlobusWorld Tour West)
Data Orchestration at Scale (GlobusWorld Tour West)Data Orchestration at Scale (GlobusWorld Tour West)
Data Orchestration at Scale (GlobusWorld Tour West)
 
Best Practices for Data Sharing (GlobusWorld Tour - UCSD)
Best Practices for Data Sharing (GlobusWorld Tour - UCSD)Best Practices for Data Sharing (GlobusWorld Tour - UCSD)
Best Practices for Data Sharing (GlobusWorld Tour - UCSD)
 
Tutorial: Leveraging Globus in your Research Applications
Tutorial: Leveraging Globus in your Research ApplicationsTutorial: Leveraging Globus in your Research Applications
Tutorial: Leveraging Globus in your Research Applications
 
Making Storage Systems Accessible via Globus (GlobusWorld Tour West)
Making Storage Systems Accessible via Globus (GlobusWorld Tour West)Making Storage Systems Accessible via Globus (GlobusWorld Tour West)
Making Storage Systems Accessible via Globus (GlobusWorld Tour West)
 
Automating Research Data Management at Scale with Globus
Automating Research Data Management at Scale with GlobusAutomating Research Data Management at Scale with Globus
Automating Research Data Management at Scale with Globus
 
Globus for System Administrators (GlobusWorld Tour - UCSD)
Globus for System Administrators (GlobusWorld Tour - UCSD)Globus for System Administrators (GlobusWorld Tour - UCSD)
Globus for System Administrators (GlobusWorld Tour - UCSD)
 
Simple Data Automation with Globus (GlobusWorld Tour West)
Simple Data Automation with Globus (GlobusWorld Tour West)Simple Data Automation with Globus (GlobusWorld Tour West)
Simple Data Automation with Globus (GlobusWorld Tour West)
 
Introduction to the Globus Platform (GlobusWorld Tour - UMich)
Introduction to the Globus Platform (GlobusWorld Tour - UMich)Introduction to the Globus Platform (GlobusWorld Tour - UMich)
Introduction to the Globus Platform (GlobusWorld Tour - UMich)
 
GlobusWorld 2020 Keynote
GlobusWorld 2020 KeynoteGlobusWorld 2020 Keynote
GlobusWorld 2020 Keynote
 
Gateways 2020 Tutorial - Automated Data Ingest and Search with Globus
Gateways 2020 Tutorial - Automated Data Ingest and Search with GlobusGateways 2020 Tutorial - Automated Data Ingest and Search with Globus
Gateways 2020 Tutorial - Automated Data Ingest and Search with Globus
 
Tutorial: Managing Protected Data with Globus Connect Server v5
Tutorial: Managing Protected Data with Globus Connect Server v5Tutorial: Managing Protected Data with Globus Connect Server v5
Tutorial: Managing Protected Data with Globus Connect Server v5
 
Gateways 2020 Tutorial - Large Scale Data Transfer with Globus
Gateways 2020 Tutorial - Large Scale Data Transfer with GlobusGateways 2020 Tutorial - Large Scale Data Transfer with Globus
Gateways 2020 Tutorial - Large Scale Data Transfer with Globus
 
Globus and Dataverse: Towards big Data Publication
Globus and Dataverse: Towards big Data PublicationGlobus and Dataverse: Towards big Data Publication
Globus and Dataverse: Towards big Data Publication
 
Gateways 2020 Tutorial - Instrument Data Distribution with Globus
Gateways 2020 Tutorial - Instrument Data Distribution with GlobusGateways 2020 Tutorial - Instrument Data Distribution with Globus
Gateways 2020 Tutorial - Instrument Data Distribution with Globus
 
Enabling Secure Data Discoverability (SC21 Tutorial)
Enabling Secure Data Discoverability (SC21 Tutorial)Enabling Secure Data Discoverability (SC21 Tutorial)
Enabling Secure Data Discoverability (SC21 Tutorial)
 

Similar to Globus Portal Framework (APS Workshop)

Advanced Computing Meets Data FAIRness
Advanced Computing Meets Data FAIRnessAdvanced Computing Meets Data FAIRness
Advanced Computing Meets Data FAIRness
Globus
 
Introduction to Apache Lucene/Solr
Introduction to Apache Lucene/SolrIntroduction to Apache Lucene/Solr
Introduction to Apache Lucene/Solr
Rahul Jain
 

Similar to Globus Portal Framework (APS Workshop) (20)

Advanced Computing Meets Data FAIRness
Advanced Computing Meets Data FAIRnessAdvanced Computing Meets Data FAIRness
Advanced Computing Meets Data FAIRness
 
Building Data Portals and Science Gateways with Globus
Building Data Portals and Science Gateways with GlobusBuilding Data Portals and Science Gateways with Globus
Building Data Portals and Science Gateways with Globus
 
Building Research Applications with Globus PaaS
Building Research Applications with Globus PaaSBuilding Research Applications with Globus PaaS
Building Research Applications with Globus PaaS
 
Working with Globus Platform Services
Working with Globus Platform ServicesWorking with Globus Platform Services
Working with Globus Platform Services
 
Introduction to the Globus SaaS (GlobusWorld Tour - STFC)
Introduction to the Globus SaaS (GlobusWorld Tour - STFC)Introduction to the Globus SaaS (GlobusWorld Tour - STFC)
Introduction to the Globus SaaS (GlobusWorld Tour - STFC)
 
Working with Globus Platform Services and Portals
Working with Globus Platform Services and PortalsWorking with Globus Platform Services and Portals
Working with Globus Platform Services and Portals
 
Facilitating Collaboration with Globus (GlobusWorld Tour - STFC)
Facilitating Collaboration with Globus (GlobusWorld Tour - STFC)Facilitating Collaboration with Globus (GlobusWorld Tour - STFC)
Facilitating Collaboration with Globus (GlobusWorld Tour - STFC)
 
DOXLON November 2016 - Data Democratization Using Splunk
DOXLON November 2016 - Data Democratization Using SplunkDOXLON November 2016 - Data Democratization Using Splunk
DOXLON November 2016 - Data Democratization Using Splunk
 
SRV206 Getting Started with Amazon CloudFront Content Delivery Network
SRV206 Getting Started with Amazon CloudFront Content Delivery NetworkSRV206 Getting Started with Amazon CloudFront Content Delivery Network
SRV206 Getting Started with Amazon CloudFront Content Delivery Network
 
Aws Solution Architecture Associate - summary
Aws Solution Architecture Associate - summaryAws Solution Architecture Associate - summary
Aws Solution Architecture Associate - summary
 
Swoogle
SwoogleSwoogle
Swoogle
 
Internet content as research data
Internet content as research dataInternet content as research data
Internet content as research data
 
AWS re:Invent 2016: Introduction to Amazon CloudFront (CTD205)
AWS re:Invent 2016: Introduction to Amazon CloudFront (CTD205)AWS re:Invent 2016: Introduction to Amazon CloudFront (CTD205)
AWS re:Invent 2016: Introduction to Amazon CloudFront (CTD205)
 
Globus at the U.S. Geological Survey (USGS)
Globus at the U.S. Geological Survey (USGS)Globus at the U.S. Geological Survey (USGS)
Globus at the U.S. Geological Survey (USGS)
 
(ATS6-PLAT04) Query service
(ATS6-PLAT04) Query service (ATS6-PLAT04) Query service
(ATS6-PLAT04) Query service
 
"What's New With Globus" Webinar: Spring 2018
"What's New With Globus" Webinar: Spring 2018"What's New With Globus" Webinar: Spring 2018
"What's New With Globus" Webinar: Spring 2018
 
Echoes Project
Echoes ProjectEchoes Project
Echoes Project
 
Data Publication and Discovery with Globus
Data Publication and Discovery with GlobusData Publication and Discovery with Globus
Data Publication and Discovery with Globus
 
Introduction to Apache Lucene/Solr
Introduction to Apache Lucene/SolrIntroduction to Apache Lucene/Solr
Introduction to Apache Lucene/Solr
 
Experiences In Building Globus Genomics Using Galaxy, Globus Online and AWS
Experiences In Building Globus Genomics Using Galaxy, Globus Online and AWSExperiences In Building Globus Genomics Using Galaxy, Globus Online and AWS
Experiences In Building Globus Genomics Using Galaxy, Globus Online and AWS
 

More from Globus

Providing Globus Services to Users Of JASMIN for Environmental Data Analysis
Providing Globus Services to Users Of JASMIN for Environmental Data AnalysisProviding Globus Services to Users Of JASMIN for Environmental Data Analysis
Providing Globus Services to Users Of JASMIN for Environmental Data Analysis
Globus
 
How to Position Your Globus Data Portal for Success Ten Good Practices
How to Position Your Globus Data Portal for Success Ten Good PracticesHow to Position Your Globus Data Portal for Success Ten Good Practices
How to Position Your Globus Data Portal for Success Ten Good Practices
Globus
 
Extending Globus into a Site-wide Automated Data Infrastructure
Extending Globus into a Site-wide Automated Data InfrastructureExtending Globus into a Site-wide Automated Data Infrastructure
Extending Globus into a Site-wide Automated Data Infrastructure
Globus
 

More from Globus (20)

The Department of Energy's Integrated Research Infrastructure (IRI).pdf
The Department of Energy's Integrated Research Infrastructure (IRI).pdfThe Department of Energy's Integrated Research Infrastructure (IRI).pdf
The Department of Energy's Integrated Research Infrastructure (IRI).pdf
 
Research Automation with Globus Flows.pptx
Research Automation with Globus Flows.pptxResearch Automation with Globus Flows.pptx
Research Automation with Globus Flows.pptx
 
Reactive Documents and Computational Pipelines
Reactive Documents and Computational PipelinesReactive Documents and Computational Pipelines
Reactive Documents and Computational Pipelines
 
Providing Globus Services to Users Of JASMIN for Environmental Data Analysis
Providing Globus Services to Users Of JASMIN for Environmental Data AnalysisProviding Globus Services to Users Of JASMIN for Environmental Data Analysis
Providing Globus Services to Users Of JASMIN for Environmental Data Analysis
 
Understanding Globus Data Transfers with NetSage
Understanding Globus Data Transfers with NetSageUnderstanding Globus Data Transfers with NetSage
Understanding Globus Data Transfers with NetSage
 
Innovating Inference: Remote Triggering of Large Language Models on HPC Clust...
Innovating Inference: Remote Triggering of Large Language Models on HPC Clust...Innovating Inference: Remote Triggering of Large Language Models on HPC Clust...
Innovating Inference: Remote Triggering of Large Language Models on HPC Clust...
 
How to Position Your Globus Data Portal for Success Ten Good Practices
How to Position Your Globus Data Portal for Success Ten Good PracticesHow to Position Your Globus Data Portal for Success Ten Good Practices
How to Position Your Globus Data Portal for Success Ten Good Practices
 
GlobusWorld 2024: Opening Keynote Address
GlobusWorld 2024: Opening Keynote AddressGlobusWorld 2024: Opening Keynote Address
GlobusWorld 2024: Opening Keynote Address
 
Globus Connect Server Deep Dive - Advanced Configuration Options and Use Cases
Globus Connect Server Deep Dive - Advanced Configuration Options and Use CasesGlobus Connect Server Deep Dive - Advanced Configuration Options and Use Cases
Globus Connect Server Deep Dive - Advanced Configuration Options and Use Cases
 
Globus Compute with Integrated Research Infrastructure (IRI) Workflows
Globus Compute with Integrated Research Infrastructure (IRI) WorkflowsGlobus Compute with Integrated Research Infrastructure (IRI) Workflows
Globus Compute with Integrated Research Infrastructure (IRI) Workflows
 
Exploring Innovations in Data Repository Solutions Insights from the U.S. Geo...
Exploring Innovations in Data Repository Solutions Insights from the U.S. Geo...Exploring Innovations in Data Repository Solutions Insights from the U.S. Geo...
Exploring Innovations in Data Repository Solutions Insights from the U.S. Geo...
 
Globus and the Integrated Research Infrastructure (IRI)
Globus and the Integrated Research Infrastructure (IRI)Globus and the Integrated Research Infrastructure (IRI)
Globus and the Integrated Research Infrastructure (IRI)
 
First Steps with Globus Compute Multi-User Endpoints
First Steps with Globus Compute Multi-User EndpointsFirst Steps with Globus Compute Multi-User Endpoints
First Steps with Globus Compute Multi-User Endpoints
 
Extending Globus into a Site-wide Automated Data Infrastructure
Extending Globus into a Site-wide Automated Data InfrastructureExtending Globus into a Site-wide Automated Data Infrastructure
Extending Globus into a Site-wide Automated Data Infrastructure
 
Enhancing Research Orchestration Capabilities at ORNL.pptx
Enhancing Research Orchestration Capabilities at ORNL.pptxEnhancing Research Orchestration Capabilities at ORNL.pptx
Enhancing Research Orchestration Capabilities at ORNL.pptx
 
Enhancing Performance with Globus and the Science DMZ.pdf
Enhancing Performance with Globus and the Science DMZ.pdfEnhancing Performance with Globus and the Science DMZ.pdf
Enhancing Performance with Globus and the Science DMZ.pdf
 
Developing Distributed High-performance Computing Capabilities of an Open Sci...
Developing Distributed High-performance Computing Capabilities of an Open Sci...Developing Distributed High-performance Computing Capabilities of an Open Sci...
Developing Distributed High-performance Computing Capabilities of an Open Sci...
 
Climate Science Flows Enabling Petabyte-Scale Climate Analysis with the Earth...
Climate Science Flows Enabling Petabyte-Scale Climate Analysis with the Earth...Climate Science Flows Enabling Petabyte-Scale Climate Analysis with the Earth...
Climate Science Flows Enabling Petabyte-Scale Climate Analysis with the Earth...
 
Introduction to Globus Compute - GlobusWorld 2024
Introduction to Globus Compute - GlobusWorld 2024Introduction to Globus Compute - GlobusWorld 2024
Introduction to Globus Compute - GlobusWorld 2024
 
Advanced Globus System Administration Topics
Advanced Globus System Administration TopicsAdvanced Globus System Administration Topics
Advanced Globus System Administration Topics
 

Recently uploaded

AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...
AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...
AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...
Alluxio, Inc.
 

Recently uploaded (20)

SQL Injection Introduction and Prevention
SQL Injection Introduction and PreventionSQL Injection Introduction and Prevention
SQL Injection Introduction and Prevention
 
Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...
Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...
Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...
 
AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...
AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...
AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...
 
IT Software Development Resume, Vaibhav jha 2024
IT Software Development Resume, Vaibhav jha 2024IT Software Development Resume, Vaibhav jha 2024
IT Software Development Resume, Vaibhav jha 2024
 
architecting-ai-in-the-enterprise-apis-and-applications.pdf
architecting-ai-in-the-enterprise-apis-and-applications.pdfarchitecting-ai-in-the-enterprise-apis-and-applications.pdf
architecting-ai-in-the-enterprise-apis-and-applications.pdf
 
KLARNA - Language Models and Knowledge Graphs: A Systems Approach
KLARNA -  Language Models and Knowledge Graphs: A Systems ApproachKLARNA -  Language Models and Knowledge Graphs: A Systems Approach
KLARNA - Language Models and Knowledge Graphs: A Systems Approach
 
What need to be mastered as AI-Powered Java Developers
What need to be mastered as AI-Powered Java DevelopersWhat need to be mastered as AI-Powered Java Developers
What need to be mastered as AI-Powered Java Developers
 
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAGAI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG
 
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...
Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...
 
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
 
AI/ML Infra Meetup | Perspective on Deep Learning Framework
AI/ML Infra Meetup | Perspective on Deep Learning FrameworkAI/ML Infra Meetup | Perspective on Deep Learning Framework
AI/ML Infra Meetup | Perspective on Deep Learning Framework
 
OpenChain Webinar: AboutCode and Beyond - End-to-End SCA
OpenChain Webinar: AboutCode and Beyond - End-to-End SCAOpenChain Webinar: AboutCode and Beyond - End-to-End SCA
OpenChain Webinar: AboutCode and Beyond - End-to-End SCA
 
How to install and activate eGrabber JobGrabber
How to install and activate eGrabber JobGrabberHow to install and activate eGrabber JobGrabber
How to install and activate eGrabber JobGrabber
 
The Impact of PLM Software on Fashion Production
The Impact of PLM Software on Fashion ProductionThe Impact of PLM Software on Fashion Production
The Impact of PLM Software on Fashion Production
 
Workforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdfWorkforce Efficiency with Employee Time Tracking Software.pdf
Workforce Efficiency with Employee Time Tracking Software.pdf
 
Optimizing Operations by Aligning Resources with Strategic Objectives Using O...
Optimizing Operations by Aligning Resources with Strategic Objectives Using O...Optimizing Operations by Aligning Resources with Strategic Objectives Using O...
Optimizing Operations by Aligning Resources with Strategic Objectives Using O...
 
Modern binary build systems - PyCon 2024
Modern binary build systems - PyCon 2024Modern binary build systems - PyCon 2024
Modern binary build systems - PyCon 2024
 
CompTIA Security+ (Study Notes) for cs.pdf
CompTIA Security+ (Study Notes) for cs.pdfCompTIA Security+ (Study Notes) for cs.pdf
CompTIA Security+ (Study Notes) for cs.pdf
 
Crafting the Perfect Measurement Sheet with PLM Integration
Crafting the Perfect Measurement Sheet with PLM IntegrationCrafting the Perfect Measurement Sheet with PLM Integration
Crafting the Perfect Measurement Sheet with PLM Integration
 
Lessons Learned from Building a Serverless Notifications System.pdf
Lessons Learned from Building a Serverless Notifications System.pdfLessons Learned from Building a Serverless Notifications System.pdf
Lessons Learned from Building a Serverless Notifications System.pdf
 

Globus Portal Framework (APS Workshop)

  • 1. Globus Portal Framework An introduction to Globus Data Portals October 15, 2021
  • 2. Agenda • Globus Data Portals – FAIR data – Data discovery – Custom use-cases • Django Globus Portal Framework – Globus Auth – Globus Services – Integration with custom resource servers • Portal Architecture – Lifecycle of a request • Live Demo: Build your own portal
  • 3. What is a Globus Data Portal A Web interface for a Globus Search index with integrated Globus capabilities to find, access, and act on data regardless of location
  • 4. Why we use portals • Different experiments (beamlines, electron microscopes, biology, etc) generate data with different types, size and experimental information • Processing, curation, and cataloging need to happen as soon as possible so data are not lost • Standardize secure access between users • Work toward FAIR datasets to enable more science
  • 5. Advantages of a portal • Make data FAIRer • Track lots of data – Tens of thousands of datasets • Discovery – Free text search in Globus Search – Filtering on specific values – User Friendly GUI • Authorized Access – Public/private, group-based, subject-level ACLs • Integration with Globus Services • Customizability
  • 6. Portal Features • Watch curated data appear in your portal! – A Web interface to data collection – Data acquisition in real time o Synchrotron serial crystallography • HTTPS Access to data – Show images in a portal from a Globus Endpoint – Add links to specific files for user download • Easy customizability – Add your own templates – Rewrite existing views – Create and call out to custom services
  • 7. ALCF Community Data Co-Op: acdc.alcf.anl.gov • A collection of data portals in collaboration with ALCF • Leverage ALCF resources – Eagle: 100PB store, 650GB/s transfer, Globus interfaces
  • 11. Globus Search Globus Search • Scalable, secure search for research data • Metadata store with fine- grained access controls • Schema agnostic • Free text, faceted search • Integrated with core platform services, e.g. Auth, Groups Input form Extract Metadata Ingest metadata, set visibility policies Discovery POST /index/123 {"filters": [ {"field_name": "record_year", "values": ["2020"], "type": "match_all" }, {"field_name": "temp_farenheit", "values": [{"from": 90, "to": "*"}], "type": "range" ]} ]} Query Bulk ingest
  • 12. Globus Portal Framework • Django framework for rapidly creating portals • Python Package for bootstrapping a portal – Install from PyPi using pip – github.com/globus/django-globus-portal-framework • Globus Portal Framework Deployment – ALCF Community Data Co-Op (acdc.alcf.anl.gov) • Example Repo – github.com/globus/django-globus-portal-framework-example
  • 13. Globus Portal Framework Features • Authorized Access with Globus Auth • Plugins for Globus Services – Globus Search – Globus Transfer – Globus Flows – Third-Party Globus Services • Customizability – Redesign how your portal talks to these services – Extend Globus Portal Framework with new features
  • 14. APS XPCS: secure data discovery 14 Globus Auth Globus Groups Globus Search
  • 15. APS XPCS: data access & preview 15 Globus Transfer HTTPS access
  • 16. APS XPCS: automated processing & indexing 16 Globus Flows: Transfer, analysis, and ingest to search index
  • 18. Portal Architecture • User Authentication • Globus Portal Framework – Portal URL Mappings – Token Loading • Service Calls to Globus Search • Lifecycle of a Request • Post Processing search requests
  • 19. User authentication • Scopes are configured in the portal • Users authenticate with Globus using 3-legged flow – Python Social Auth used for Authentication backend • User tokens are saved in the database • Future requests authorized with user access tokens – New searches use Search bearer token.
  • 20. Service Calls • Most Portal Service calls use the Globus SDK – Globus Portal Framework loads tokens from DB – Globus SDK object instantiated with token – Call to Globus Service – Portal renders result in templates
  • 21. Globus Portal Framework URLs • URLs are laid out in three major categories – Index Selection – Index Search page – Search Subject detail page • Many Globus Search Indices are supported • Index Search page links to many result subjects • Each subject has a unique URL
  • 23. Format of a URL
  • 24. An index is Configuration Driven • A Search Index can be setup through configuration – Add Globus Search index uuid – Add a name – Add facets – Add fields • Start Searching!
  • 25. Lifecycle of a request • User makes a query • Portal sends request to Globus Search – Request contains user bearer token • Portal receives response • Portal does processing on response – Parse Dates, build URL for Globus webapp, etc. • Portal renders data into templates • User receives a search page
  • 26. Pilot • Tooling to simplify ingesting metadata • Create valid GMeta entries from JSON • Upload files to an associated endpoints and track HTTPS URLs to add to GMeta
  • 27. Demo: Creating your own portal • Django Globus Portal Framework – github.com/globus/django-globus-portal-framework – django-globus-portal-framework.readthedocs.io/en/stable/ • Example Repo – github.com/globus/django-globus-portal-framework-example