What's New in Globus - Internet2 TechEXtra

•

0 likes•226 views

An overview of developments in the Globus platform during 2020-2021, presented at a webinar hosted by Internet2. Includes an overview of Globus Connect Server v5, cloud storage connectors, and platform services for developers (e.g., Globus Search and Globus Flows).

What’s New
in Globus
Vas Vasiliadis – vas@uchicago.edu
Internet2 TechEXtra – February 23, 2022

What a difference three years make!
We’ve been busy…
• …Rebuilding the technical foundation
• …Growing the connected storage ecosystem
• …Expanding the data management platform
• …Automating data management at scale
3

Rebuilding the technical foundation: GCSv5
• Modern security model
• Support compliance requirements
• Deployment flexibility
• Enhanced sharing policies
• Simplified scaling/availability
4

Data-appropriate storage
Google Drive for project admin files
Google Cloud Storage for core research files
Reliably manage API constraints

Support for all major cloud
storage providers

Partnership with the community
to develop new connectors
Community Connector Program

Easy egress and ingress of data
Data sharing with collaborators
via unified interface

POSIX Staging Connector
• Example:
– IBM Spectrum Archive plugin, Brock Palen at University of
Michigan - github.com/brockpalen/ltfsee-globus
12
• For POSIX file systems
that cache from tertiary
storage
• Custom plug-in for
staging files

Streamlined GCP Installation
• Log in from the
application during
installation
• Use setup key method
for automation

New platform services
• Groups
• Search
• Flows
14

Globus Groups
docs.globus.org/api/groups
groups.api.globus.org/redoc
15

Globus Search: Data description/discovery
• Metadata store with fine-
grained visibility controls
• Schema agnostic
à dynamic schemas
• Simple search using URL
query parameters
• Complex search using
search request document
• Example:
16
docs.globus.org/api/search
Search
Index
github.com/globus/searchable-files-demo

Globus Search
Terra Fusion on OSN
17
osn.globusdemo.org

3 Degrees of Automation
Timer Service
Scheduled and recurring transfers
(a.k.a. Globus cron)
Command Line Interface
Ad hoc scripting and integration
Globus Flows service
Comprehensive task (data and
compute) orchestration with human in
the loop interactions

CLI v3
• Collection support
• Group support
• Search support
• Multiple profiles
• Python SDK updates
20

Globus Flows: Managed task automation
• Flows: A platform service for defining, applying, and
sharing distributed research automation flows
• Flows comprise Actions
• Action Providers: Called by Flows to perform tasks
• Triggers*: Start flows based on events
* In development

Serial Synchrotron Crystallography of SARS-CoV-2 proteins
www.alcf.anl.gov/news/argonne-researchers-use-theta-real-time-analysis-covid-19-proteins
The pipeline generates large image batches at a high rate, with data transfers achieving speeds of
700 megabytes per second thanks to Globus, a University of Chicago-run data management service.

Gladier: The Globus Architecture for Data-Intensive
Experimental Research
• Accelerate and simplify flow
development and deployment
• Combine tools into reliable,
flexible, secure, distributed
flows
• Bridge instruments and
computing facilities
• Automate data collection and
publication to create FAIR data

funcX: managed and federated FaaS
• Cloud-hosted service for managing compute
• Register and share compute endpoints
• Register and share Python functions
• Reliably, scalable, securely execute functions on
remote endpoints
• Integrated with Globus Auth and data ecosystem
25
Try funcx on Binder
https://funcx.org

Our Mission
Increase the efficiency and
effectiveness of researchers
engaged in data-driven
science and scholarship
through sustainable software

Google’s software-defined global network, underlying network virtualization stack (Andromeda) and network services along with open innovations provide the foundation for securely delivering a diversity of workloads and services on Google Cloud Platform (GCP). Google Cloud Virtual Private Cloud (VPC) provides you with a secure and flexible sandbox to run your cloud workloads. Global Load Balancing and Cloud CDN deliver global reach, scale and high availability and secure your apps against DDoS attacks. Cloud Interconnect enables seamless connectivity options for hybrid/multi-cloud app delivery. This talk provides an in-depth look at Google Cloud Networking, what's under the hood, the benefits it delivers along with real world GCP customer stories.

Event-based API Patterns and Practices - AsyncAPI Online Conference

LaunchAny

Benchmarking Oracle I/O Performance with Orion by Alex GorbachevAlex Gorbachev

[UC4] Version and Automate Everything

Perforce

OSMC 2022 | Ignite: Observability with Grafana & Prometheus for Kafka on Kube...

NETWAYS

Self-managing a highly scalable distributed system with Apache Kafka® at its core is not an easy feat. That’s why operators prefer tooling such as Confluent Control Center for administering and monitoring their deployments. However, sometimes, you might also like to import monitoring data into a third-party metrics aggregation platform for service correlations, consolidated dashboards, root cause analysis, or more fine-grained alerts. If you’ve ever asked a question along these lines: Can I export JMX data from Confluent clusters to my monitoring system with minimal configuration? What if I could correlate this service’s data spike with metrics from Confluent clusters in a single UI pane? Can I configure some Grafana dashboards for Confluent clusters? This talk will enable you on achieving the below: Monitoring Your Event Streams: Integrating Confluent with Prometheus and Grafana (this article) Monitoring Your Event Streams: Tutorial for Observability Into Apache Kafka Clients

Location Analytics - Real-Time Geofencing using Apache Kafka

Guido Schmutz

An important underlying concept behind location-based applications is called geofencing. Geofencing is a process that allows acting on users and/or devices who enter/exit a specific geographical area, known as a geo-fence. A geo-fence can be dynamically generated—as in a radius around a point location, or a geo-fence can be a predefined set of boundaries (such as secured areas, buildings, boarders of counties, states or countries). Geofencing lays the foundation for realizing use cases around fleet monitoring, asset tracking, phone tracking across cell sites, connected manufacturing, ride-sharing solutions and many others. GPS tracking tells constantly and in real time where a device is located and forms the stream of events which needs to be analyzed against the much more static set of geo-fences. Many of the use cases mentioned above require low-latency actions taken place, if either a device enters or leaves a geo-fence or when it is approaching such a geo-fence. That’s where streaming data ingestion and streaming analytics and therefore the Kafka ecosystem comes into play. This session will present how location analytics applications can be implemented using Kafka and KSQL & Kafka Streams. It highlights the exiting features available out-of-the-box and then shows how easy it is to extend it by custom defined functions (UDFs). The design of such solution so that it can scale with both an increasing amount of position events as well as geo-fences will be discussed as well.

Machine Learning is separated into model training and model inference. ML frameworks typically load historical data from a data store like HDFS or S3 to train models. This talk shows how you can completely avoid such a data store by ingesting streaming data directly via Apache Kafka from any source system into TensorFlow for model training and model inference using the capabilities of “TensorFlow I/O” add-on. The talk compares this modern streaming architecture to traditional batch and big data alternatives and explains benefits like the simplified architecture, the ability of reprocessing events in the same order for training different models, and the possibility to build a scalable, mission-critical, real time ML architecture with muss less headaches and problems. Key takeaways for the audience • Scalable open source Machine Learning infrastructure • Streaming ingestion into TensorFlow without the need for another data store like HDFS or S3 (leveraging TensorFlow I/O and its Kafka plugin) • Stream Processing using analytic models in mission-critical deployments to act in Real Time • Learn how Apache Kafka open source ecosystem including Kafka Connect, Kafka Streams and KSQL help to build, deploy, score and monitor analytic models • Comparison and trade-offs between this modern streaming approach and traditional batch model training infrastructures

TLC303_Walkthrough Setting up a Highly Available Communications Platform on AWS

Amazon Web Services

Come join this workshop to set up a highly available and fault-tolerant real-time communication platform on AWS. We walk you through setting up load balancing and multiple failover mechanisms for the unique requirements of real-time communication. You set up both SBC and PBX servers, use WebRTC and SIP standards, and learn how to optimize this platform on Amazon EC2. The workshop also guides you with an example. The example uses Amazon Polly for lifelike text-to-speech and integrates it with the communication platform that you build. Lastly, you learn how to test this communication platform in a distributed manner with Amazon EC2 Systems Manager.

JHipster overview

Julien Dubois

AWS Batch: Simplifying Batch Computing in the Cloud

Amazon Web Services

Moving Gigantic Files Into and Out of the Alfresco Repository

Jeff Potts

DSpace-CRIS: a CRIS enhanced repository platform

Andrea Bollini

International Conference on Economics and Business Information 19 to 20 April 2016 in Berlin This presentation introduces you to the version 5.5.0 of the DSpace-CRIS extension. With such extension you can capture the full picture of the research activities conduct in your institution and their context. It enables to showcase the experts, the facilities, the services and much more to attract funding, facilitate collaborations and curate the scientific reputation of your Institution.

OPA: The Cloud Native Policy Engine

Torin Sandall

Big Data Analytics (ML, DL, AI) hands-on

Dony Riyanto

Apache Spark and the Hadoop Ecosystem on AWS

Amazon Web Services

DataPower API Gateway Performance Benchmarks

IBM DataPower Gateway

Microsoft Azure Big Data Analytics

Mark Kromer

Unified Big Data Processing with Apache Spark (QCON 2014)

Databricks

While early big data systems, such as MapReduce, focused on batch processing, the demands on these systems have quickly grown. Users quickly needed to run (1) more interactive ad-hoc queries, (2) sophisticated multi-pass algorithms (e.g. machine learning), and (3) real-time stream processing. The result has been an explosion of specialized systems to tackle these new workloads. Unfortunately, this means more systems to learn, manage, and stitch together into pipelines. Spark is unique in taking a step back and trying to provide a *unified* post-MapReduce programming model that tackles all these workloads. By generalizing MapReduce to support fast data sharing and low-latency jobs, we achieve best-in-class performance in a variety of workloads, while providing a simple programming model that lets users easily and efficiently combine them. Today, Spark is the most active open source project in big data, with high activity in both the core engine and a growing array of standard libraries built on top (e.g. machine learning, stream processing, SQL). I'm going to talk about the latest developments in Spark and show examples of how it can combine processing algorithms to build rich data pipelines in just a few lines of code. Talk by Databricks CTO and Apache Spark creator Matei Zaharia at QCON San Francisco 2014.

SSO introduction

Aidy Tificate

Spring Boot & Spring Cloud Apps on Pivotal Application Service - Daniel Lavoie

VMware Tanzu

The Patterns of Distributed Logging and Containers

SATOSHI TAGOMORI

Firebase

Ranjith Alappadan

Open Policy Agent Deep Dive Seattle 2018

Torin Sandall

Topics: * Background on Open Policy Agent project: users, use cases, and stats. * How OPA works (decoupling policy decision-making from enforcement) * Hands-on example: Users can view their own account details and support staff can view accounts they have are assigned to via a ticketing system. * SQL data filtering use case: writing policy in OPA and enforcing policy in SQL. * WebAssembly compiler.

GlobusWorld 2020 Keynote

What's New in Globus - Internet2 TechEXtra

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to What's New in Globus - Internet2 TechEXtra

Similar to What's New in Globus - Internet2 TechEXtra (20)

More from Globus

More from Globus (20)

Recently uploaded

Recently uploaded (20)

What's New in Globus - Internet2 TechEXtra