SlideShare a Scribd company logo
1 of 25
Download to read offline
Vas Vasiliadis
vas@uchicago.edu
February 28, 2024
Reliable, Remote Computation at all Scales
Why do we need remote computing?
Workloads
• Interactive, real-time
workloads
• Machine learning training
and inference
• Components may best be
executed in different places
Users
• Diverse backgrounds
and expertise
• Different user interfaces
(e.g., notebooks)
Resources
• Hardware specialization
• Specialization leads to
distribution
Why do researchers “hate” remote computing?
• Remote computing is notoriously complicated
– Authentication
– Network connections
– Configuring/managing jobs
– Interacting with resources (waiting in queues, scaling nodes)
– Configuring execution environment
– Getting results back again
• Researchers need to overcome the same obstacles
every time they move to a new resource
3
How can we bridge this gap?
Borrow page from data management playbook
è“Fire-and-forget” computation
èUniform access interface
èFederated access control
èFast networks making compute resource “local”
How can we bridge this gap?
Move closer to researchers’ environments
èResearchers primarily work in high level languages
èFunctions are a natural unit of computation
Function-as-a-Service (FaaS)!
• Researchers work in familiar language, e.g., Python…
• …using familiar interfaces, e.g., Jupyter
FaaS as offered by cloud providers
6
• Single provider, single
location to submit and
manage tasks
• Homogenous execution
environment
• Transparent and elastic
execution (of even very
small tasks)
• Integrated with
cloud provider data
management
Cloud
Provider 1
Cloud
Provider 2
FaaS as an interface to the advanced computing
ecosystem
7
• Multiple provider,
multiple location, single
interface
• Homogenous execution
environment
• Transparent and elastic
execution
• Integrated with
institution-wide data
management
Globus Compute helps bridge the gap
Managed, federated
Function-as-a-Service for
reliably, scaleably and
securely executing functions
on remote endpoints from
laptops to supercomputers
The Globus Compute model
• Compute service — Highly available cloud-hosted
service for managed, fire-and-forget function execution
• Compute endpoint — Abstracts access to compute
resources, from edge device to supercomputer
• Compute SDK — Python interface for interacting with the
service, suing the familiar (to many) Globus look and feel
• Security — Leverages Globus Auth; Compute endpoints
are resource servers, authN and access via OAuth tokens
9
User interaction with Globus Compute
10
A B
You request a function be
executed on endpoints A and B
1
2 Globus Compute manages
the reliable and secure
execution on these endpoints
3
Globus Compute returns results or
stores them until requested
Compute
Service
You register your code
(as a Python function)
0
A compute
resource
Another
compute
resource
Globus Compute transforms any computing
resource into a function serving endpoint
• Python pip installable agent
• Elastic resource provisioning
from local, cluster, or cloud
system (via Parsl)
• Parallel execution using local
fork or via common schedulers
– Slurm, PBS, LSF, Cobalt, K8s
11
Compute
Service
Executing functions with Globus Compute
• Users invoke functions as tasks
– Register Python function
– Pass input arguments
– Select endpoint(s)
• Service stores tasks in the cloud
• Endpoints fetch waiting tasks (when
online), run tasks, and return results
• Results stored in the cloud and on
Globus storage endpoints
• Users retrieve results asynchronously
12
Compute
Service
Common Use Cases
13
Use case 1: fire-and-forget execution
Executing a bag of tasks, e.g., running simulations with
different parameters, executing ML inferences, on one
or more remote computers directly from your
environment, e.g., Jupyter on your laptop
• Fire-and-forget execution managed by Globus Compute;
tasks/results cached until endpoint/client are online
• Portability across different systems—optionally making use of
specialized hardware
• Elastic scaling to provision resources as needed (from HPC and
cloud systems)
ML-based drug screening
15
National Virtual Biotechnology Laboratory, arXiv:2006.02431
Use case 2: automated analysis of data
Constructing and running automated analysis
pipelines with data processing steps that need to be
executed in different locations
• Automatically process data as they are acquired (event- and
workflow-based)
• Integrate with data movement and other actions—human and
machine
• Execute functions across the computing continuum e.g., near
instrument, in data center, on specialized hardware
Enabling serial crystallography at scale
• Serially image chips with
000’s of embedded crystals
• Quality control first 1,000 to
report failures
• Analyze batches of images as
they are collected
• Report statistics and images
during experiment
• Return structure to scientist
Darren Sherrell, Gyorgy Babnigg, Andrzej Joachimiak
Globus
Flows
How remote computing enables this use case
Image
processing
Local workstation
Carbon!
Check
threshold
Data publication
Transfer
Transfer
raw files
Transfer
Move results
to repo
Compute
Analyze
images
Compute
Visualize
Compute
Gather
metadata
Share
Set access
controls
Compute
Launch QA
job
Search
Ingest to
index
Supercomputer
GPU Cluster
Globus enables “smart” instruments
19
aining of Deep Neural Networks
ources
7sec
19sec
5sec
31sec.
cycle time
Seems too easy? Let’s take a look…
• Run function on my laptop …test, debug
• Scale (a little) on a cloud VM (or K8s cluster)
• Scale (a bit more) on my campus cluster
• …run complete model on DOE supercomputer :--(
20
Common pattern of computing in a flow
Transfer raw
instrument
images
Run a compute
job to process
image files
Transfer Compute
Move processed
images to sharing
system/repository
Set access
controls for
sharing data
Share
Transfer
1 2 3 4
EC2
Instance
“Instrument”
EC2
Instance
Compute
Resource
Our environment Registered
Function
Monitor
script
transfer control
access
result files
0 trigger
flow run
transfer
raw files
1
invoke image
processing function 2
set
permissions
4
transfer
result files
3
Data
Endpoint
Data
Endpoint
Data
Endpoint
ALCF
Eagle
Compute
Endpoint
Globus Compute current state
• Service is running in production
– Tried and tested by 10MMs of invocations on 100Ks endpoints
• Globus Computer Agent supports single user
– Akin to Globus Connect Personal
• Endpoints visible via Globus web app
• Function registration and invocation via SDK
23
Compute service will evolve rapidly
• Multi-user compute endpoints
– Akin to Globus Connect Server
• Native integration with transfer for stage in and stage
out of data for compute tasks
– Can be done today via Globus Flows
• Expanding compute service interfaces in the webapp
for administrators and users
Support resources
• Globus documentation: docs.globus.org
• YouTube channel: youtube.com/GlobusOnline
• Helpdesk: support@globus.org
• Mailing Lists: globus.org/mailing-lists
• Customer engagement team (office hours)
• Professional services team (advisory, custom work)

More Related Content

Similar to Reliable, Remote Computation at All Scales

Openstack For Beginners
Openstack For BeginnersOpenstack For Beginners
Openstack For Beginners
cpallares
 
Other distributed systems
Other distributed systemsOther distributed systems
Other distributed systems
Sri Prasanna
 

Similar to Reliable, Remote Computation at All Scales (20)

Brad stack - Digital Health and Well-Being Festival
Brad stack - Digital Health and Well-Being Festival Brad stack - Digital Health and Well-Being Festival
Brad stack - Digital Health and Well-Being Festival
 
Monitoring kubernetes across data center and cloud
Monitoring kubernetes across data center and cloudMonitoring kubernetes across data center and cloud
Monitoring kubernetes across data center and cloud
 
re:Invent 2013-foster-madduri
re:Invent 2013-foster-maddurire:Invent 2013-foster-madduri
re:Invent 2013-foster-madduri
 
Scaling collaborative data science with Globus and Jupyter
Scaling collaborative data science with Globus and JupyterScaling collaborative data science with Globus and Jupyter
Scaling collaborative data science with Globus and Jupyter
 
Sanger, upcoming Openstack for Bio-informaticians
Sanger, upcoming Openstack for Bio-informaticiansSanger, upcoming Openstack for Bio-informaticians
Sanger, upcoming Openstack for Bio-informaticians
 
Flexible compute
Flexible computeFlexible compute
Flexible compute
 
Big Process for Big Data @ NASA
Big Process for Big Data @ NASABig Process for Big Data @ NASA
Big Process for Big Data @ NASA
 
Взгляд на облака с точки зрения HPC
Взгляд на облака с точки зрения HPCВзгляд на облака с точки зрения HPC
Взгляд на облака с точки зрения HPC
 
Taming Big Data!
Taming Big Data!Taming Big Data!
Taming Big Data!
 
A Maturing Role of Workflows in the Presence of Heterogenous Computing Archit...
A Maturing Role of Workflows in the Presence of Heterogenous Computing Archit...A Maturing Role of Workflows in the Presence of Heterogenous Computing Archit...
A Maturing Role of Workflows in the Presence of Heterogenous Computing Archit...
 
Openstack For Beginners
Openstack For BeginnersOpenstack For Beginners
Openstack For Beginners
 
StarlingX - Project Onboarding
StarlingX - Project OnboardingStarlingX - Project Onboarding
StarlingX - Project Onboarding
 
Genomic Computation at Scale with Serverless, StackStorm and Docker Swarm
Genomic Computation at Scale with Serverless, StackStorm and Docker SwarmGenomic Computation at Scale with Serverless, StackStorm and Docker Swarm
Genomic Computation at Scale with Serverless, StackStorm and Docker Swarm
 
RAMSES: Robust Analytic Models for Science at Extreme Scales
RAMSES: Robust Analytic Models for Science at Extreme ScalesRAMSES: Robust Analytic Models for Science at Extreme Scales
RAMSES: Robust Analytic Models for Science at Extreme Scales
 
Other distributed systems
Other distributed systemsOther distributed systems
Other distributed systems
 
DOE Magellan OpenStack user story
DOE Magellan OpenStack user storyDOE Magellan OpenStack user story
DOE Magellan OpenStack user story
 
Overview of Scientific Workflows - Why Use Them?
Overview of Scientific Workflows - Why Use Them?Overview of Scientific Workflows - Why Use Them?
Overview of Scientific Workflows - Why Use Them?
 
Globus Labs: Forging the Next Frontier
Globus Labs: Forging the Next FrontierGlobus Labs: Forging the Next Frontier
Globus Labs: Forging the Next Frontier
 
Alluxio 2.0 & Near Real-time Big Data Platform w/ Spark & Alluxio
Alluxio 2.0 & Near Real-time Big Data Platform w/ Spark & AlluxioAlluxio 2.0 & Near Real-time Big Data Platform w/ Spark & Alluxio
Alluxio 2.0 & Near Real-time Big Data Platform w/ Spark & Alluxio
 
Dsdt meetup 2017 11-21
Dsdt meetup 2017 11-21Dsdt meetup 2017 11-21
Dsdt meetup 2017 11-21
 

More from Globus

Providing Globus Services to Users Of JASMIN for Environmental Data Analysis
Providing Globus Services to Users Of JASMIN for Environmental Data AnalysisProviding Globus Services to Users Of JASMIN for Environmental Data Analysis
Providing Globus Services to Users Of JASMIN for Environmental Data Analysis
Globus
 
How to Position Your Globus Data Portal for Success Ten Good Practices
How to Position Your Globus Data Portal for Success Ten Good PracticesHow to Position Your Globus Data Portal for Success Ten Good Practices
How to Position Your Globus Data Portal for Success Ten Good Practices
Globus
 
Extending Globus into a Site-wide Automated Data Infrastructure
Extending Globus into a Site-wide Automated Data InfrastructureExtending Globus into a Site-wide Automated Data Infrastructure
Extending Globus into a Site-wide Automated Data Infrastructure
Globus
 

More from Globus (20)

Reactive Documents and Computational Pipelines
Reactive Documents and Computational PipelinesReactive Documents and Computational Pipelines
Reactive Documents and Computational Pipelines
 
Providing Globus Services to Users Of JASMIN for Environmental Data Analysis
Providing Globus Services to Users Of JASMIN for Environmental Data AnalysisProviding Globus Services to Users Of JASMIN for Environmental Data Analysis
Providing Globus Services to Users Of JASMIN for Environmental Data Analysis
 
Understanding Globus Data Transfers with NetSage
Understanding Globus Data Transfers with NetSageUnderstanding Globus Data Transfers with NetSage
Understanding Globus Data Transfers with NetSage
 
Innovating Inference: Remote Triggering of Large Language Models on HPC Clust...
Innovating Inference: Remote Triggering of Large Language Models on HPC Clust...Innovating Inference: Remote Triggering of Large Language Models on HPC Clust...
Innovating Inference: Remote Triggering of Large Language Models on HPC Clust...
 
How to Position Your Globus Data Portal for Success Ten Good Practices
How to Position Your Globus Data Portal for Success Ten Good PracticesHow to Position Your Globus Data Portal for Success Ten Good Practices
How to Position Your Globus Data Portal for Success Ten Good Practices
 
GlobusWorld 2024: Opening Keynote Address
GlobusWorld 2024: Opening Keynote AddressGlobusWorld 2024: Opening Keynote Address
GlobusWorld 2024: Opening Keynote Address
 
Globus Connect Server Deep Dive - Advanced Configuration Options and Use Cases
Globus Connect Server Deep Dive - Advanced Configuration Options and Use CasesGlobus Connect Server Deep Dive - Advanced Configuration Options and Use Cases
Globus Connect Server Deep Dive - Advanced Configuration Options and Use Cases
 
Globus Compute with Integrated Research Infrastructure (IRI) Workflows
Globus Compute with Integrated Research Infrastructure (IRI) WorkflowsGlobus Compute with Integrated Research Infrastructure (IRI) Workflows
Globus Compute with Integrated Research Infrastructure (IRI) Workflows
 
Exploring Innovations in Data Repository Solutions Insights from the U.S. Geo...
Exploring Innovations in Data Repository Solutions Insights from the U.S. Geo...Exploring Innovations in Data Repository Solutions Insights from the U.S. Geo...
Exploring Innovations in Data Repository Solutions Insights from the U.S. Geo...
 
Globus at the U.S. Geological Survey (USGS)
Globus at the U.S. Geological Survey (USGS)Globus at the U.S. Geological Survey (USGS)
Globus at the U.S. Geological Survey (USGS)
 
Globus and the Integrated Research Infrastructure (IRI)
Globus and the Integrated Research Infrastructure (IRI)Globus and the Integrated Research Infrastructure (IRI)
Globus and the Integrated Research Infrastructure (IRI)
 
First Steps with Globus Compute Multi-User Endpoints
First Steps with Globus Compute Multi-User EndpointsFirst Steps with Globus Compute Multi-User Endpoints
First Steps with Globus Compute Multi-User Endpoints
 
Extending Globus into a Site-wide Automated Data Infrastructure
Extending Globus into a Site-wide Automated Data InfrastructureExtending Globus into a Site-wide Automated Data Infrastructure
Extending Globus into a Site-wide Automated Data Infrastructure
 
Enhancing Research Orchestration Capabilities at ORNL.pptx
Enhancing Research Orchestration Capabilities at ORNL.pptxEnhancing Research Orchestration Capabilities at ORNL.pptx
Enhancing Research Orchestration Capabilities at ORNL.pptx
 
Enhancing Performance with Globus and the Science DMZ.pdf
Enhancing Performance with Globus and the Science DMZ.pdfEnhancing Performance with Globus and the Science DMZ.pdf
Enhancing Performance with Globus and the Science DMZ.pdf
 
Developing Distributed High-performance Computing Capabilities of an Open Sci...
Developing Distributed High-performance Computing Capabilities of an Open Sci...Developing Distributed High-performance Computing Capabilities of an Open Sci...
Developing Distributed High-performance Computing Capabilities of an Open Sci...
 
Climate Science Flows Enabling Petabyte-Scale Climate Analysis with the Earth...
Climate Science Flows Enabling Petabyte-Scale Climate Analysis with the Earth...Climate Science Flows Enabling Petabyte-Scale Climate Analysis with the Earth...
Climate Science Flows Enabling Petabyte-Scale Climate Analysis with the Earth...
 
Introduction to Globus Compute - GlobusWorld 2024
Introduction to Globus Compute - GlobusWorld 2024Introduction to Globus Compute - GlobusWorld 2024
Introduction to Globus Compute - GlobusWorld 2024
 
Advanced Globus System Administration Topics
Advanced Globus System Administration TopicsAdvanced Globus System Administration Topics
Advanced Globus System Administration Topics
 
Instrument Data Automation: The Life of a Flow
Instrument Data Automation: The Life of a FlowInstrument Data Automation: The Life of a Flow
Instrument Data Automation: The Life of a Flow
 

Recently uploaded

Abortion Pill Prices Boksburg [(+27832195400*)] 🏥 Women's Abortion Clinic in ...
Abortion Pill Prices Boksburg [(+27832195400*)] 🏥 Women's Abortion Clinic in ...Abortion Pill Prices Boksburg [(+27832195400*)] 🏥 Women's Abortion Clinic in ...
Abortion Pill Prices Boksburg [(+27832195400*)] 🏥 Women's Abortion Clinic in ...
Medical / Health Care (+971588192166) Mifepristone and Misoprostol tablets 200mg
 

Recently uploaded (20)

Devoxx UK 2024 - Going serverless with Quarkus, GraalVM native images and AWS...
Devoxx UK 2024 - Going serverless with Quarkus, GraalVM native images and AWS...Devoxx UK 2024 - Going serverless with Quarkus, GraalVM native images and AWS...
Devoxx UK 2024 - Going serverless with Quarkus, GraalVM native images and AWS...
 
WSO2Con2024 - Navigating the Digital Landscape: Transforming Healthcare with ...
WSO2Con2024 - Navigating the Digital Landscape: Transforming Healthcare with ...WSO2Con2024 - Navigating the Digital Landscape: Transforming Healthcare with ...
WSO2Con2024 - Navigating the Digital Landscape: Transforming Healthcare with ...
 
WSO2Con2024 - Simplified Integration: Unveiling the Latest Features in WSO2 L...
WSO2Con2024 - Simplified Integration: Unveiling the Latest Features in WSO2 L...WSO2Con2024 - Simplified Integration: Unveiling the Latest Features in WSO2 L...
WSO2Con2024 - Simplified Integration: Unveiling the Latest Features in WSO2 L...
 
WSO2CON 2024 - Building the API First Enterprise – Running an API Program, fr...
WSO2CON 2024 - Building the API First Enterprise – Running an API Program, fr...WSO2CON 2024 - Building the API First Enterprise – Running an API Program, fr...
WSO2CON 2024 - Building the API First Enterprise – Running an API Program, fr...
 
WSO2CON 2024 - Designing Event-Driven Enterprises: Stories of Transformation
WSO2CON 2024 - Designing Event-Driven Enterprises: Stories of TransformationWSO2CON 2024 - Designing Event-Driven Enterprises: Stories of Transformation
WSO2CON 2024 - Designing Event-Driven Enterprises: Stories of Transformation
 
AzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdf
AzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdfAzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdf
AzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdf
 
WSO2Con2024 - Low-Code Integration Tooling
WSO2Con2024 - Low-Code Integration ToolingWSO2Con2024 - Low-Code Integration Tooling
WSO2Con2024 - Low-Code Integration Tooling
 
WSO2CON 2024 - Building a Digital Government in Uganda
WSO2CON 2024 - Building a Digital Government in UgandaWSO2CON 2024 - Building a Digital Government in Uganda
WSO2CON 2024 - Building a Digital Government in Uganda
 
Evolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI EraEvolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI Era
 
WSO2Con2024 - Facilitating Broadband Switching Services for UK Telecoms Provi...
WSO2Con2024 - Facilitating Broadband Switching Services for UK Telecoms Provi...WSO2Con2024 - Facilitating Broadband Switching Services for UK Telecoms Provi...
WSO2Con2024 - Facilitating Broadband Switching Services for UK Telecoms Provi...
 
WSO2CON 2024 - Lessons from the Field: Legacy Platforms – It's Time to Let Go...
WSO2CON 2024 - Lessons from the Field: Legacy Platforms – It's Time to Let Go...WSO2CON 2024 - Lessons from the Field: Legacy Platforms – It's Time to Let Go...
WSO2CON 2024 - Lessons from the Field: Legacy Platforms – It's Time to Let Go...
 
WSO2CON 2024 - OSU & WSO2: A Decade Journey in Integration & Innovation
WSO2CON 2024 - OSU & WSO2: A Decade Journey in Integration & InnovationWSO2CON 2024 - OSU & WSO2: A Decade Journey in Integration & Innovation
WSO2CON 2024 - OSU & WSO2: A Decade Journey in Integration & Innovation
 
Driving Innovation: Scania's API Revolution with WSO2
Driving Innovation: Scania's API Revolution with WSO2Driving Innovation: Scania's API Revolution with WSO2
Driving Innovation: Scania's API Revolution with WSO2
 
WSO2CON2024 - It's time to go Platformless
WSO2CON2024 - It's time to go PlatformlessWSO2CON2024 - It's time to go Platformless
WSO2CON2024 - It's time to go Platformless
 
WSO2CON 2024 - How to Run a Security Program
WSO2CON 2024 - How to Run a Security ProgramWSO2CON 2024 - How to Run a Security Program
WSO2CON 2024 - How to Run a Security Program
 
Abortion Pill Prices Boksburg [(+27832195400*)] 🏥 Women's Abortion Clinic in ...
Abortion Pill Prices Boksburg [(+27832195400*)] 🏥 Women's Abortion Clinic in ...Abortion Pill Prices Boksburg [(+27832195400*)] 🏥 Women's Abortion Clinic in ...
Abortion Pill Prices Boksburg [(+27832195400*)] 🏥 Women's Abortion Clinic in ...
 
WSO2Con2024 - Organization Management: The Revolution in B2B CIAM
WSO2Con2024 - Organization Management: The Revolution in B2B CIAMWSO2Con2024 - Organization Management: The Revolution in B2B CIAM
WSO2Con2024 - Organization Management: The Revolution in B2B CIAM
 
WSO2Con204 - Hard Rock Presentation - Keynote
WSO2Con204 - Hard Rock Presentation - KeynoteWSO2Con204 - Hard Rock Presentation - Keynote
WSO2Con204 - Hard Rock Presentation - Keynote
 
WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...
WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...
WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...
 
WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...
WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...
WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...
 

Reliable, Remote Computation at All Scales

  • 1. Vas Vasiliadis vas@uchicago.edu February 28, 2024 Reliable, Remote Computation at all Scales
  • 2. Why do we need remote computing? Workloads • Interactive, real-time workloads • Machine learning training and inference • Components may best be executed in different places Users • Diverse backgrounds and expertise • Different user interfaces (e.g., notebooks) Resources • Hardware specialization • Specialization leads to distribution
  • 3. Why do researchers “hate” remote computing? • Remote computing is notoriously complicated – Authentication – Network connections – Configuring/managing jobs – Interacting with resources (waiting in queues, scaling nodes) – Configuring execution environment – Getting results back again • Researchers need to overcome the same obstacles every time they move to a new resource 3
  • 4. How can we bridge this gap? Borrow page from data management playbook è“Fire-and-forget” computation èUniform access interface èFederated access control èFast networks making compute resource “local”
  • 5. How can we bridge this gap? Move closer to researchers’ environments èResearchers primarily work in high level languages èFunctions are a natural unit of computation Function-as-a-Service (FaaS)! • Researchers work in familiar language, e.g., Python… • …using familiar interfaces, e.g., Jupyter
  • 6. FaaS as offered by cloud providers 6 • Single provider, single location to submit and manage tasks • Homogenous execution environment • Transparent and elastic execution (of even very small tasks) • Integrated with cloud provider data management Cloud Provider 1 Cloud Provider 2
  • 7. FaaS as an interface to the advanced computing ecosystem 7 • Multiple provider, multiple location, single interface • Homogenous execution environment • Transparent and elastic execution • Integrated with institution-wide data management
  • 8. Globus Compute helps bridge the gap Managed, federated Function-as-a-Service for reliably, scaleably and securely executing functions on remote endpoints from laptops to supercomputers
  • 9. The Globus Compute model • Compute service — Highly available cloud-hosted service for managed, fire-and-forget function execution • Compute endpoint — Abstracts access to compute resources, from edge device to supercomputer • Compute SDK — Python interface for interacting with the service, suing the familiar (to many) Globus look and feel • Security — Leverages Globus Auth; Compute endpoints are resource servers, authN and access via OAuth tokens 9
  • 10. User interaction with Globus Compute 10 A B You request a function be executed on endpoints A and B 1 2 Globus Compute manages the reliable and secure execution on these endpoints 3 Globus Compute returns results or stores them until requested Compute Service You register your code (as a Python function) 0 A compute resource Another compute resource
  • 11. Globus Compute transforms any computing resource into a function serving endpoint • Python pip installable agent • Elastic resource provisioning from local, cluster, or cloud system (via Parsl) • Parallel execution using local fork or via common schedulers – Slurm, PBS, LSF, Cobalt, K8s 11 Compute Service
  • 12. Executing functions with Globus Compute • Users invoke functions as tasks – Register Python function – Pass input arguments – Select endpoint(s) • Service stores tasks in the cloud • Endpoints fetch waiting tasks (when online), run tasks, and return results • Results stored in the cloud and on Globus storage endpoints • Users retrieve results asynchronously 12 Compute Service
  • 14. Use case 1: fire-and-forget execution Executing a bag of tasks, e.g., running simulations with different parameters, executing ML inferences, on one or more remote computers directly from your environment, e.g., Jupyter on your laptop • Fire-and-forget execution managed by Globus Compute; tasks/results cached until endpoint/client are online • Portability across different systems—optionally making use of specialized hardware • Elastic scaling to provision resources as needed (from HPC and cloud systems)
  • 15. ML-based drug screening 15 National Virtual Biotechnology Laboratory, arXiv:2006.02431
  • 16. Use case 2: automated analysis of data Constructing and running automated analysis pipelines with data processing steps that need to be executed in different locations • Automatically process data as they are acquired (event- and workflow-based) • Integrate with data movement and other actions—human and machine • Execute functions across the computing continuum e.g., near instrument, in data center, on specialized hardware
  • 17. Enabling serial crystallography at scale • Serially image chips with 000’s of embedded crystals • Quality control first 1,000 to report failures • Analyze batches of images as they are collected • Report statistics and images during experiment • Return structure to scientist Darren Sherrell, Gyorgy Babnigg, Andrzej Joachimiak
  • 18. Globus Flows How remote computing enables this use case Image processing Local workstation Carbon! Check threshold Data publication Transfer Transfer raw files Transfer Move results to repo Compute Analyze images Compute Visualize Compute Gather metadata Share Set access controls Compute Launch QA job Search Ingest to index Supercomputer GPU Cluster
  • 19. Globus enables “smart” instruments 19 aining of Deep Neural Networks ources 7sec 19sec 5sec 31sec. cycle time
  • 20. Seems too easy? Let’s take a look… • Run function on my laptop …test, debug • Scale (a little) on a cloud VM (or K8s cluster) • Scale (a bit more) on my campus cluster • …run complete model on DOE supercomputer :--( 20
  • 21. Common pattern of computing in a flow Transfer raw instrument images Run a compute job to process image files Transfer Compute Move processed images to sharing system/repository Set access controls for sharing data Share Transfer 1 2 3 4
  • 22. EC2 Instance “Instrument” EC2 Instance Compute Resource Our environment Registered Function Monitor script transfer control access result files 0 trigger flow run transfer raw files 1 invoke image processing function 2 set permissions 4 transfer result files 3 Data Endpoint Data Endpoint Data Endpoint ALCF Eagle Compute Endpoint
  • 23. Globus Compute current state • Service is running in production – Tried and tested by 10MMs of invocations on 100Ks endpoints • Globus Computer Agent supports single user – Akin to Globus Connect Personal • Endpoints visible via Globus web app • Function registration and invocation via SDK 23
  • 24. Compute service will evolve rapidly • Multi-user compute endpoints – Akin to Globus Connect Server • Native integration with transfer for stage in and stage out of data for compute tasks – Can be done today via Globus Flows • Expanding compute service interfaces in the webapp for administrators and users
  • 25. Support resources • Globus documentation: docs.globus.org • YouTube channel: youtube.com/GlobusOnline • Helpdesk: support@globus.org • Mailing Lists: globus.org/mailing-lists • Customer engagement team (office hours) • Professional services team (advisory, custom work)