Presented at GlobusWorld 2022 by the Globus professional services team. Describes the Modern Research Data Portal design pattern and an implementation using the Django framework.
Beyaz Şapkalı Hacker CEH Eğitimi - Pasif Bilgi Toplama (OSINT)PRISMA CSI
Bu sunum, Prisma tarafından verilen “Uygulamalı Beyaz Şapkalı Hacker Eğitimi v1” de anlatılan bir üniteye aittir.
PRISMA CSI • Cyber Security and Intelligence www.prismacsi.com
Bu doküman, alıntı vererek kullanılabilir ya da paylaşılabilir ancak değiştirilemez ve ticari amaçla kullanılamaz. Detaylı bilgiye https://creativecommons.org/licenses/by-nc-nd/4.0/legalcode.tr bağlantısından erişebilirsiniz.
Aryaka is an industry leader for Managed SD-WAN, delivered as-a-Service! With a global private backbone and patented software stack, we deliver services to more than 850+ global enterprises.
To learn more, visit us at www.aryaka.com or write to me at nitish.khanna@aryaka.com
Beyaz Şapkalı Hacker CEH Eğitimi - Pasif Bilgi Toplama (OSINT)PRISMA CSI
Bu sunum, Prisma tarafından verilen “Uygulamalı Beyaz Şapkalı Hacker Eğitimi v1” de anlatılan bir üniteye aittir.
PRISMA CSI • Cyber Security and Intelligence www.prismacsi.com
Bu doküman, alıntı vererek kullanılabilir ya da paylaşılabilir ancak değiştirilemez ve ticari amaçla kullanılamaz. Detaylı bilgiye https://creativecommons.org/licenses/by-nc-nd/4.0/legalcode.tr bağlantısından erişebilirsiniz.
Aryaka is an industry leader for Managed SD-WAN, delivered as-a-Service! With a global private backbone and patented software stack, we deliver services to more than 850+ global enterprises.
To learn more, visit us at www.aryaka.com or write to me at nitish.khanna@aryaka.com
With the big delays in the time it takes until an iOS jailbreak is public and stable, it is often not possible to test mobile apps in the latest iOS version. Occasionally customers might also provide builds that only work in iOS versions for which no jailbreak is available. On Android the situation is better, but there can also be problems to root certain phone models. These trends make security testing of mobile apps difficult. This talk will cover approaches to defeat common security mechanisms that must be bypassed in the absence of root/jailbreak.
This session explains how the combination of IEEE 802.1AE (data link encryption) with the power of Session Group Tags achieves trusted security in a network. It covers the protocols details as well as use case and more importantly how CTS can be deployed in a network. This session is targeted mainly to enterprise customers.
Innovation in logistics and supply chain management blockchainsAinsley Brown
In the simplest of terms, blockchain is just a new form of decentralized database. But what is its connection to logistics and supply chain management? The very function of blockchain technology tells the story.
It's a shared ledger for recording the history of transactions - that cannot be altered. Blockchains are ledgers (like Excel spreadsheets), but they accept inputs from lots of different parties. The ledger can only be changed when there is a consensus among the group. That makes them more secure, and it means there's no need for a central authority to approve transactions.
You can think of blockchain as the “internet of value.” It’s a real-time ledger of anything that can be recorded (financial transactions, contracts, physical assets, supply chain info, etc.) but there is no one person or organization in charge of the entire chain. In fact, it’s open and everyone in the chain can see the detail of each record—what’s known as a block.
Since blockchains can be effectively tracked through all processes, companies using blockchain technology will be able to more readily produce detailed information about a product’s lifecycle, including supplier information, manufacturing details and logistics information.
With the dominance of Mobile Apps, Single Page Apps for the Web, and Micro-Services, we are all building more APIs than ever before. Like many other developers, I had struggled with finding the right mix of security and simplicity for securing APIs. Some standards from the IETF have made it possible to accomplish both. Let me show you how to utilize existing libraries to lock down you API without writing a ton of code.
In this tutorial, you will learn how to write a secure API with future proof security utilizing JOSE. JOSE is a collection of complimentary standards: JWT, JWE, JWS, JWA, and JWK. JOSE is used by OAuth, OpenID, and others to secure communications between APIs and consumers. Now you can use it to secure your API.
SSRF vs. Business-critical applications. XXE tunneling in SAPERPScan
Any information an attacker might want is stored in a company’s ERP. This information can include financial, customer or public relations, intellectual property, personally identifiable information and more. Industrial espionage, sabotage and fraud or insider embezzlement may be very effective if targeted at the victim’s ERP system and cause significant damage to the business.
The presentation describes the history of SSRF attack, or Server Side Request Forgery, its types and different kinds of attacks on SAP.
Tutorial presented at Mini Gateways 2022. Demonstrates how to build data portals and science gateways with the Django Globus Portal Framework.
The broad scope of a typical science gateway—to simplify access to shared data, computing and other resources—makes building such a gateway from scratch a daunting task. Investigators must be able to stage data from instruments (or other sources), submit compute jobs to analyze data, move data to more persistent storage, describe data products, and provide a means for collaborators to search, discover, reuse and augment these data products. Myriad tools are available to enable all these tasks but integrating them in a way that hides the complexity from users, is a challenge.
In this tutorial we will describe an approach that bootstraps science gateway development based on the Modern Research Data Portal[1] design pattern. The solution uses a set of open source tools that build on the established Django web framework, the ubiquitous OAuth2/OpenID connect standards for authentication/authorization, the widely deployed Globus service for research data management, and the nascent funcX functions-as-a-service platform. Attendees will learn how to rapidly deploy a science gateway that enables both automated computation at scale and data enhanced discovery of resulting data products. The emphasis will be on automating many of the required tasks so that gateway developers can focus on building differentiated, discipline-specific functionality rather than low-value—yet critical—supporting infrastructure.
We will use the ALCF Community Data Co-Op as an exemplar to illustrate how these tools have been used to support large-scale collaborative research. We will describe the overall solution architecture and introduce attendees to the individual tools. Attendees will then use these tools to deploy and configure their own science gateway to support image analysis, description, indexing and search.
The tutorial will comprise a mix of lectures, demonstration and hands-on exercises. Virtual machines will be provided for computation and for hosting the science gateway. The objective is for attendees to develop a high-level understanding of the various components and leave with working code that can serve as the starting point for their own science gateway implementation.
Building Research Applications with Globus PaaSGlobus
We provide a brief introduction to the Globus platform-as-a-service for developers, with emphasis on building simple web applications for data distribution and discovery. We describe how to register an application with Globus and access platform APIs using the Globus Python SDK and a Jupyter Notebook. We also introduce the Globus Search service and demonstrate how it is used by an open source web portal framework that can jumpstart research application development.
This material was presented at the Research Computing and Data Management Workshop, hosted by Rensselaer Polytechnic Institute on February 27-28, 2024.
Enabling Secure Data Discoverability (SC21 Tutorial)Globus
Major research instruments are generating orders of magnitude more data in relatively short timeframes. As a result, the research enterprise is increasingly challenged by what should be mundane tasks: describing data for discovery and making data securely accessible to the broader research community. The ad hoc methods currently employed place undue burden on scientists and system administrators alike, and it is clear that a more robust, scalable approach is required.
Bespoke data portals (and science gateways/data commons) are becoming more prominent as a means of enabling access to large datasets. in this tutorial we demonstrate how services for authentication, authorization, metadata management, and search may be integrated with popular web frameworks, and used in combination with fast, well-architected networks to make data discoverable and accessible. Outcomes: build a simple, but functional, data portal that facilitates flexible data description, faceted data search and secure data access.
We describe the various Globus APIs and demonstrate how developers can use them to integrate robust data management capabilties into their research applications. We also provide an overview of advanced services such as Globus Search and tools such as the data portal framework that you can use to simplify data search and discovery.
Presented at a workshop at KU Leuven on July 8, 2022.
Working with Globus Platform Services and PortalsGlobus
We describe how developers can use Globus APIs to integrate robust data management capabilities into their research applications. We also demonstrate the new Globus portal framework that can be used in conjunction with the Globus Search service to simplify data search and discovery.
With the big delays in the time it takes until an iOS jailbreak is public and stable, it is often not possible to test mobile apps in the latest iOS version. Occasionally customers might also provide builds that only work in iOS versions for which no jailbreak is available. On Android the situation is better, but there can also be problems to root certain phone models. These trends make security testing of mobile apps difficult. This talk will cover approaches to defeat common security mechanisms that must be bypassed in the absence of root/jailbreak.
This session explains how the combination of IEEE 802.1AE (data link encryption) with the power of Session Group Tags achieves trusted security in a network. It covers the protocols details as well as use case and more importantly how CTS can be deployed in a network. This session is targeted mainly to enterprise customers.
Innovation in logistics and supply chain management blockchainsAinsley Brown
In the simplest of terms, blockchain is just a new form of decentralized database. But what is its connection to logistics and supply chain management? The very function of blockchain technology tells the story.
It's a shared ledger for recording the history of transactions - that cannot be altered. Blockchains are ledgers (like Excel spreadsheets), but they accept inputs from lots of different parties. The ledger can only be changed when there is a consensus among the group. That makes them more secure, and it means there's no need for a central authority to approve transactions.
You can think of blockchain as the “internet of value.” It’s a real-time ledger of anything that can be recorded (financial transactions, contracts, physical assets, supply chain info, etc.) but there is no one person or organization in charge of the entire chain. In fact, it’s open and everyone in the chain can see the detail of each record—what’s known as a block.
Since blockchains can be effectively tracked through all processes, companies using blockchain technology will be able to more readily produce detailed information about a product’s lifecycle, including supplier information, manufacturing details and logistics information.
With the dominance of Mobile Apps, Single Page Apps for the Web, and Micro-Services, we are all building more APIs than ever before. Like many other developers, I had struggled with finding the right mix of security and simplicity for securing APIs. Some standards from the IETF have made it possible to accomplish both. Let me show you how to utilize existing libraries to lock down you API without writing a ton of code.
In this tutorial, you will learn how to write a secure API with future proof security utilizing JOSE. JOSE is a collection of complimentary standards: JWT, JWE, JWS, JWA, and JWK. JOSE is used by OAuth, OpenID, and others to secure communications between APIs and consumers. Now you can use it to secure your API.
SSRF vs. Business-critical applications. XXE tunneling in SAPERPScan
Any information an attacker might want is stored in a company’s ERP. This information can include financial, customer or public relations, intellectual property, personally identifiable information and more. Industrial espionage, sabotage and fraud or insider embezzlement may be very effective if targeted at the victim’s ERP system and cause significant damage to the business.
The presentation describes the history of SSRF attack, or Server Side Request Forgery, its types and different kinds of attacks on SAP.
Tutorial presented at Mini Gateways 2022. Demonstrates how to build data portals and science gateways with the Django Globus Portal Framework.
The broad scope of a typical science gateway—to simplify access to shared data, computing and other resources—makes building such a gateway from scratch a daunting task. Investigators must be able to stage data from instruments (or other sources), submit compute jobs to analyze data, move data to more persistent storage, describe data products, and provide a means for collaborators to search, discover, reuse and augment these data products. Myriad tools are available to enable all these tasks but integrating them in a way that hides the complexity from users, is a challenge.
In this tutorial we will describe an approach that bootstraps science gateway development based on the Modern Research Data Portal[1] design pattern. The solution uses a set of open source tools that build on the established Django web framework, the ubiquitous OAuth2/OpenID connect standards for authentication/authorization, the widely deployed Globus service for research data management, and the nascent funcX functions-as-a-service platform. Attendees will learn how to rapidly deploy a science gateway that enables both automated computation at scale and data enhanced discovery of resulting data products. The emphasis will be on automating many of the required tasks so that gateway developers can focus on building differentiated, discipline-specific functionality rather than low-value—yet critical—supporting infrastructure.
We will use the ALCF Community Data Co-Op as an exemplar to illustrate how these tools have been used to support large-scale collaborative research. We will describe the overall solution architecture and introduce attendees to the individual tools. Attendees will then use these tools to deploy and configure their own science gateway to support image analysis, description, indexing and search.
The tutorial will comprise a mix of lectures, demonstration and hands-on exercises. Virtual machines will be provided for computation and for hosting the science gateway. The objective is for attendees to develop a high-level understanding of the various components and leave with working code that can serve as the starting point for their own science gateway implementation.
Building Research Applications with Globus PaaSGlobus
We provide a brief introduction to the Globus platform-as-a-service for developers, with emphasis on building simple web applications for data distribution and discovery. We describe how to register an application with Globus and access platform APIs using the Globus Python SDK and a Jupyter Notebook. We also introduce the Globus Search service and demonstrate how it is used by an open source web portal framework that can jumpstart research application development.
This material was presented at the Research Computing and Data Management Workshop, hosted by Rensselaer Polytechnic Institute on February 27-28, 2024.
Enabling Secure Data Discoverability (SC21 Tutorial)Globus
Major research instruments are generating orders of magnitude more data in relatively short timeframes. As a result, the research enterprise is increasingly challenged by what should be mundane tasks: describing data for discovery and making data securely accessible to the broader research community. The ad hoc methods currently employed place undue burden on scientists and system administrators alike, and it is clear that a more robust, scalable approach is required.
Bespoke data portals (and science gateways/data commons) are becoming more prominent as a means of enabling access to large datasets. in this tutorial we demonstrate how services for authentication, authorization, metadata management, and search may be integrated with popular web frameworks, and used in combination with fast, well-architected networks to make data discoverable and accessible. Outcomes: build a simple, but functional, data portal that facilitates flexible data description, faceted data search and secure data access.
We describe the various Globus APIs and demonstrate how developers can use them to integrate robust data management capabilties into their research applications. We also provide an overview of advanced services such as Globus Search and tools such as the data portal framework that you can use to simplify data search and discovery.
Presented at a workshop at KU Leuven on July 8, 2022.
Working with Globus Platform Services and PortalsGlobus
We describe how developers can use Globus APIs to integrate robust data management capabilities into their research applications. We also demonstrate the new Globus portal framework that can be used in conjunction with the Globus Search service to simplify data search and discovery.
GlobusWorld 2021 Tutorial: Building with the Globus PlatformGlobus
Using Globus platform services like Search and Flows to build data portals, science gateways and data commons that facilitate data discovery and collaboration. This tutorial was presented at the GlobusWorld 2021 conference in Chicago, IL by Vas Vasiliadis.
This tutorial from the Gateways 2018 conference in Austin, TX explored the capabilities provided by Globus for assembling, describing, publishing, identifying, searching, and discovering datasets.
Jupyter + Globus: The Foundation for Interactive Data ScienceGlobus
This tutorial from the Gateways 2018 conference in Austin, TX showed participants how Globus may be used in conjunction with the Jupyter platform to open up new avenues—and new data sources--for interactive data science.
Scalable Data Management: Automation and the Modern Research Data PortalGlobus
Globus is an established service from the University of Chicago that is widely used for managing research data in national laboratories, campus computing centers, and HPC facilities. While its interactive web browser interface addresses simple file transfer and sharing scenarios, large scale automation typically requires integration of the research data management platform it provides into bespoke applications.
We will describe one such example, the Petrel data portal (https://petreldata.net), used by researchers to manage data in diverse fields including materials science, cosmology, machine learning, and serial crystallography. The portal facilitates automated ingest of data, extraction and addition of metadata for creating search indexes, assignment of persistent identifiers faceted search for rapid data discovery, and point-and-click downloading of datasets by authorized users. As security and privacy are often critical requirements, the portal employs fine-grained permissions that control both visibility of metadata and access to the datasets themselves. It is based on the Modern Research Data Portal design pattern, jointly developed by the ESnet and Globus teams, and leverages capabilities such as the Science DMZ for enhanced performance and to streamline the user experience.
Introduction to Globus: Research Data Management Software at the ALCFGlobus
These Globus Intro slides were presented in a webinar on June 26, 2019 at the Argonne National Laboratory Leadership Computing Facility (ALCF) by Rick Wagner, Globus Professional Services Manager.
Globus Compute wth IRI Workflows - GlobusWorld 2024Globus
As part of the DOE Integrated Research Infrastructure (IRI) program, NERSC at Lawrence Berkeley National Lab and ALCF at Argonne National Lab are working closely with General Atomics on accelerating the computing requirements of the DIII-D experiment. As part of the work the team is investigating ways to speedup the time to solution for many different parts of the DIII-D workflow including how they run jobs on HPC systems. One of these routes is looking at Globus Compute as a way to replace the current method for managing tasks and we describe a brief proof of concept showing how Globus Compute could help to schedule jobs and be a tool to connect compute at different facilities.
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...Globus
The Earth System Grid Federation (ESGF) is a global network of data servers that archives and distributes the planet’s largest collection of Earth system model output for thousands of climate and environmental scientists worldwide. Many of these petabyte-scale data archives are located in proximity to large high-performance computing (HPC) or cloud computing resources, but the primary workflow for data users consists of transferring data, and applying computations on a different system. As a part of the ESGF 2.0 US project (funded by the United States Department of Energy Office of Science), we developed pre-defined data workflows, which can be run on-demand, capable of applying many data reduction and data analysis to the large ESGF data archives, transferring only the resultant analysis (ex. visualizations, smaller data files). In this talk, we will showcase a few of these workflows, highlighting how Globus Flows can be used for petabyte-scale climate analysis.
We describe the deployment and use of Globus Compute for remote computation. This content is aimed at researchers who wish to compute on remote resources using a unified programming interface, as well as system administrators who will deploy and operate Globus Compute services on their research computing infrastructure.
Globus Connect Server Deep Dive - GlobusWorld 2024Globus
We explore the Globus Connect Server (GCS) architecture and experiment with advanced configuration options and use cases. This content is targeted at system administrators who are familiar with GCS and currently operate—or are planning to operate—broader deployments at their institution.
Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...Globus
Large Language Models (LLMs) are currently the center of attention in the tech world, particularly for their potential to advance research. In this presentation, we'll explore a straightforward and effective method for quickly initiating inference runs on supercomputers using the vLLM tool with Globus Compute, specifically on the Polaris system at ALCF. We'll begin by briefly discussing the popularity and applications of LLMs in various fields. Following this, we will introduce the vLLM tool, and explain how it integrates with Globus Compute to efficiently manage LLM operations on Polaris. Attendees will learn the practical aspects of setting up and remotely triggering LLMs from local machines, focusing on ease of use and efficiency. This talk is ideal for researchers and practitioners looking to leverage the power of LLMs in their work, offering a clear guide to harnessing supercomputing resources for quick and effective LLM inference.
Providing Globus Services to Users of JASMIN for Environmental Data AnalysisGlobus
JASMIN is the UK’s high-performance data analysis platform for environmental science, operated by STFC on behalf of the UK Natural Environment Research Council (NERC). In addition to its role in hosting the CEDA Archive (NERC’s long-term repository for climate, atmospheric science & Earth observation data in the UK), JASMIN provides a collaborative platform to a community of around 2,000 scientists in the UK and beyond, providing nearly 400 environmental science projects with working space, compute resources and tools to facilitate their work. High-performance data transfer into and out of JASMIN has always been a key feature, with many scientists bringing model outputs from supercomputers elsewhere in the UK, to analyse against observational or other model data in the CEDA Archive. A growing number of JASMIN users are now realising the benefits of using the Globus service to provide reliable and efficient data movement and other tasks in this and other contexts. Further use cases involve long-distance (intercontinental) transfers to and from JASMIN, and collecting results from a mobile atmospheric radar system, pushing data to JASMIN via a lightweight Globus deployment. We provide details of how Globus fits into our current infrastructure, our experience of the recent migration to GCSv5.4, and of our interest in developing use of the wider ecosystem of Globus services for the benefit of our user community.
First Steps with Globus Compute Multi-User EndpointsGlobus
In this presentation we will share our experiences around getting started with the Globus Compute multi-user endpoint. Working with the Pharmacology group at the University of Auckland, we have previously written an application using Globus Compute that can offload computationally expensive steps in the researcher's workflows, which they wish to manage from their familiar Windows environments, onto the NeSI (New Zealand eScience Infrastructure) cluster. Some of the challenges we have encountered were that each researcher had to set up and manage their own single-user globus compute endpoint and that the workloads had varying resource requirements (CPUs, memory and wall time) between different runs. We hope that the multi-user endpoint will help to address these challenges and share an update on our progress here.
Enhancing Research Orchestration Capabilities at ORNL.pdfGlobus
Cross-facility research orchestration comes with ever-changing constraints regarding the availability and suitability of various compute and data resources. In short, a flexible data and processing fabric is needed to enable the dynamic redirection of data and compute tasks throughout the lifecycle of an experiment. In this talk, we illustrate how we easily leveraged Globus services to instrument the ACE research testbed at the Oak Ridge Leadership Computing Facility with flexible data and task orchestration capabilities.
Understanding Globus Data Transfers with NetSageGlobus
NetSage is an open privacy-aware network measurement, analysis, and visualization service designed to help end-users visualize and reason about large data transfers. NetSage traditionally has used a combination of passive measurements, including SNMP and flow data, as well as active measurements, mainly perfSONAR, to provide longitudinal network performance data visualization. It has been deployed by dozens of networks world wide, and is supported domestically by the Engagement and Performance Operations Center (EPOC), NSF #2328479. We have recently expanded the NetSage data sources to include logs for Globus data transfers, following the same privacy-preserving approach as for Flow data. Using the logs for the Texas Advanced Computing Center (TACC) as an example, this talk will walk through several different example use cases that NetSage can answer, including: Who is using Globus to share data with my institution, and what kind of performance are they able to achieve? How many transfers has Globus supported for us? Which sites are we sharing the most data with, and how is that changing over time? How is my site using Globus to move data internally, and what kind of performance do we see for those transfers? What percentage of data transfers at my institution used Globus, and how did the overall data transfer performance compare to the Globus users?
How to Position Your Globus Data Portal for Success Ten Good PracticesGlobus
Science gateways allow science and engineering communities to access shared data, software, computing services, and instruments. Science gateways have gained a lot of traction in the last twenty years, as evidenced by projects such as the Science Gateways Community Institute (SGCI) and the Center of Excellence on Science Gateways (SGX3) in the US, The Australian Research Data Commons (ARDC) and its platforms in Australia, and the projects around Virtual Research Environments in Europe. A few mature frameworks have evolved with their different strengths and foci and have been taken up by a larger community such as the Globus Data Portal, Hubzero, Tapis, and Galaxy. However, even when gateways are built on successful frameworks, they continue to face the challenges of ongoing maintenance costs and how to meet the ever-expanding needs of the community they serve with enhanced features. It is not uncommon that gateways with compelling use cases are nonetheless unable to get past the prototype phase and become a full production service, or if they do, they don't survive more than a couple of years. While there is no guaranteed pathway to success, it seems likely that for any gateway there is a need for a strong community and/or solid funding streams to create and sustain its success. With over twenty years of examples to draw from, this presentation goes into detail for ten factors common to successful and enduring gateways that effectively serve as best practices for any new or developing gateway.
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...Globus
The U.S. Geological Survey (USGS) has made substantial investments in meeting evolving scientific, technical, and policy driven demands on storing, managing, and delivering data. As these demands continue to grow in complexity and scale, the USGS must continue to explore innovative solutions to improve its management, curation, sharing, delivering, and preservation approaches for large-scale research data. Supporting these needs, the USGS has partnered with the University of Chicago-Globus to research and develop advanced repository components and workflows leveraging its current investment in Globus. The primary outcome of this partnership includes the development of a prototype enterprise repository, driven by USGS Data Release requirements, through exploration and implementation of the entire suite of the Globus platform offerings, including Globus Flow, Globus Auth, Globus Transfer, and Globus Search. This presentation will provide insights into this research partnership, introduce the unique requirements and challenges being addressed and provide relevant project progress.
Developing Distributed High-performance Computing Capabilities of an Open Sci...Globus
COVID-19 had an unprecedented impact on scientific collaboration. The pandemic and its broad response from the scientific community has forged new relationships among public health practitioners, mathematical modelers, and scientific computing specialists, while revealing critical gaps in exploiting advanced computing systems to support urgent decision making. Informed by our team’s work in applying high-performance computing in support of public health decision makers during the COVID-19 pandemic, we present how Globus technologies are enabling the development of an open science platform for robust epidemic analysis, with the goal of collaborative, secure, distributed, on-demand, and fast time-to-solution analyses to support public health.
The Department of Energy's Integrated Research Infrastructure (IRI)Globus
We will provide an overview of DOE’s IRI initiative as it moves into early implementation, what drives the IRI vision, and the role of DOE in the larger national research ecosystem.
Listen to the keynote address and hear about the latest developments from Rachana Ananthakrishnan and Ian Foster who review the updates to the Globus Platform and Service, and the relevance of Globus to the scientific community as an automation platform to accelerate scientific discovery.
Enhancing Performance with Globus and the Science DMZGlobus
ESnet has led the way in helping national facilities—and many other institutions in the research community—configure Science DMZs and troubleshoot network issues to maximize data transfer performance. In this talk we will present a summary of approaches and tips for getting the most out of your network infrastructure using Globus Connect Server.
Extending Globus into a Site-wide Automated Data Infrastructure.pdfGlobus
The Rosalind Franklin Institute hosts a variety of scientific instruments, which allow us to capture a multifaceted and multilevel view of biological systems, generating around 70 terabytes of data a month. Distributed solutions, such as Globus and Ceph, facilitates storage, access, and transfer of large amount of data. However, we still must deal with the heterogeneity of the file formats and directory structure at acquisition, which is optimised for fast recording, rather than for efficient storage and processing. Our data infrastructure includes local storage at the instruments and workstations, distributed object stores with POSIX and S3 access, remote storage on HPCs, and taped backup. This can pose a challenge in ensuring fast, secure, and efficient data transfer. Globus allows us to handle this heterogeneity, while its Python SDK allows us to automate our data infrastructure using Globus microservices integrated with our data access models. Our data management workflows are becoming increasingly complex and heterogenous, including desktop PCs, virtual machines, and offsite HPCs, as well as several open-source software tools with different computing and data structure requirements. This complexity commands that data is annotated with enough details about the experiments and the analysis to ensure efficient and reproducible workflows. This talk explores how we extend Globus into different parts of our data lifecycle to create a secure, scalable, and high performing automated data infrastructure that can provide FAIR[1,2] data for all our science.
1. https://doi.org/10.1038/sdata.2016.18
2. https://www.go-fair.org/fair-principles
Providing Globus Services to Users of JASMIN for Environmental Data AnalysisGlobus
JASMIN is the UK’s high-performance data analysis platform for environmental science, operated by STFC on behalf of the UK Natural Environment Research Council (NERC). In addition to its role in hosting the CEDA Archive (NERC’s long-term repository for climate, atmospheric science & Earth observation data in the UK), JASMIN provides a collaborative platform to a community of around 2,000 scientists in the UK and beyond, providing nearly 400 environmental science projects with working space, compute resources and tools to facilitate their work. High-performance data transfer into and out of JASMIN has always been a key feature, with many scientists bringing model outputs from supercomputers elsewhere in the UK, to analyse against observational or other model data in the CEDA Archive. A growing number of JASMIN users are now realising the benefits of using the Globus service to provide reliable and efficient data movement and other tasks in this and other contexts. Further use cases involve long-distance (intercontinental) transfers to and from JASMIN, and collecting results from a mobile atmospheric radar system, pushing data to JASMIN via a lightweight Globus deployment. We provide details of how Globus fits into our current infrastructure, our experience of the recent migration to GCSv5.4, and of our interest in developing use of the wider ecosystem of Globus services for the benefit of our user community.
Globus Compute with Integrated Research Infrastructure (IRI) workflowsGlobus
As part of the DOE Integrated Research Infrastructure (IRI) program, NERSC at Lawrence Berkeley National Lab and ALCF at Argonne National Lab are working closely with General Atomics on accelerating the computing requirements of the DIII-D experiment. As part of the work the team is investigating ways to speedup the time to solution for many different parts of the DIII-D workflow including how they run jobs on HPC systems. One of these routes is looking at Globus Compute as a way to replace the current method for managing tasks and I will give a brief proof of concept showing how Globus Compute could help to schedule jobs and be a tool to connect compute at different facilities.
Reactive Documents and Computational Pipelines - Bridging the GapGlobus
As scientific discovery and experimentation become increasingly reliant on computational methods, the static nature of traditional publications renders them progressively fragmented and unreproducible. How can workflow automation tools, such as Globus, be leveraged to address these issues and potentially create a new, higher-value form of publication? LivePublication leverages Globus’s custom Action Provider integrations and Compute nodes to capture semantic and provenance information during distributed flow executions. This information is then embedded within an RO-crate and interfaced with a programmatic document, creating a seamless pipeline from instruments, to computation, to publication.
Enterprise Resource Planning System includes various modules that reduce any business's workload. Additionally, it organizes the workflows, which drives towards enhancing productivity. Here are a detailed explanation of the ERP modules. Going through the points will help you understand how the software is changing the work dynamics.
To know more details here: https://blogs.nyggs.com/nyggs/enterprise-resource-planning-erp-system-modules/
Accelerate Enterprise Software Engineering with PlatformlessWSO2
Key takeaways:
Challenges of building platforms and the benefits of platformless.
Key principles of platformless, including API-first, cloud-native middleware, platform engineering, and developer experience.
How Choreo enables the platformless experience.
How key concepts like application architecture, domain-driven design, zero trust, and cell-based architecture are inherently a part of Choreo.
Demo of an end-to-end app built and deployed on Choreo.
Custom Healthcare Software for Managing Chronic Conditions and Remote Patient...Mind IT Systems
Healthcare providers often struggle with the complexities of chronic conditions and remote patient monitoring, as each patient requires personalized care and ongoing monitoring. Off-the-shelf solutions may not meet these diverse needs, leading to inefficiencies and gaps in care. It’s here, custom healthcare software offers a tailored solution, ensuring improved care and effectiveness.
Prosigns: Transforming Business with Tailored Technology SolutionsProsigns
Unlocking Business Potential: Tailored Technology Solutions by Prosigns
Discover how Prosigns, a leading technology solutions provider, partners with businesses to drive innovation and success. Our presentation showcases our comprehensive range of services, including custom software development, web and mobile app development, AI & ML solutions, blockchain integration, DevOps services, and Microsoft Dynamics 365 support.
Custom Software Development: Prosigns specializes in creating bespoke software solutions that cater to your unique business needs. Our team of experts works closely with you to understand your requirements and deliver tailor-made software that enhances efficiency and drives growth.
Web and Mobile App Development: From responsive websites to intuitive mobile applications, Prosigns develops cutting-edge solutions that engage users and deliver seamless experiences across devices.
AI & ML Solutions: Harnessing the power of Artificial Intelligence and Machine Learning, Prosigns provides smart solutions that automate processes, provide valuable insights, and drive informed decision-making.
Blockchain Integration: Prosigns offers comprehensive blockchain solutions, including development, integration, and consulting services, enabling businesses to leverage blockchain technology for enhanced security, transparency, and efficiency.
DevOps Services: Prosigns' DevOps services streamline development and operations processes, ensuring faster and more reliable software delivery through automation and continuous integration.
Microsoft Dynamics 365 Support: Prosigns provides comprehensive support and maintenance services for Microsoft Dynamics 365, ensuring your system is always up-to-date, secure, and running smoothly.
Learn how our collaborative approach and dedication to excellence help businesses achieve their goals and stay ahead in today's digital landscape. From concept to deployment, Prosigns is your trusted partner for transforming ideas into reality and unlocking the full potential of your business.
Join us on a journey of innovation and growth. Let's partner for success with Prosigns.
Experience our free, in-depth three-part Tendenci Platform Corporate Membership Management workshop series! In Session 1 on May 14th, 2024, we began with an Introduction and Setup, mastering the configuration of your Corporate Membership Module settings to establish membership types, applications, and more. Then, on May 16th, 2024, in Session 2, we focused on binding individual members to a Corporate Membership and Corporate Reps, teaching you how to add individual members and assign Corporate Representatives to manage dues, renewals, and associated members. Finally, on May 28th, 2024, in Session 3, we covered questions and concerns, addressing any queries or issues you may have.
For more Tendenci AMS events, check out www.tendenci.com/events
Cyaniclab : Software Development Agency Portfolio.pdfCyanic lab
CyanicLab, an offshore custom software development company based in Sweden,India, Finland, is your go-to partner for startup development and innovative web design solutions. Our expert team specializes in crafting cutting-edge software tailored to meet the unique needs of startups and established enterprises alike. From conceptualization to execution, we offer comprehensive services including web and mobile app development, UI/UX design, and ongoing software maintenance. Ready to elevate your business? Contact CyanicLab today and let us propel your vision to success with our top-notch IT solutions.
Into the Box Keynote Day 2: Unveiling amazing updates and announcements for modern CFML developers! Get ready for exciting releases and updates on Ortus tools and products. Stay tuned for cutting-edge innovations designed to boost your productivity.
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...Juraj Vysvader
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I didn't get rich from it but it did have 63K downloads (powered possible tens of thousands of websites).
Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...Anthony Dahanne
Les Buildpacks existent depuis plus de 10 ans ! D’abord, ils étaient utilisés pour détecter et construire une application avant de la déployer sur certains PaaS. Ensuite, nous avons pu créer des images Docker (OCI) avec leur dernière génération, les Cloud Native Buildpacks (CNCF en incubation). Sont-ils une bonne alternative au Dockerfile ? Que sont les buildpacks Paketo ? Quelles communautés les soutiennent et comment ?
Venez le découvrir lors de cette session ignite
How Recreation Management Software Can Streamline Your Operations.pptxwottaspaceseo
Recreation management software streamlines operations by automating key tasks such as scheduling, registration, and payment processing, reducing manual workload and errors. It provides centralized management of facilities, classes, and events, ensuring efficient resource allocation and facility usage. The software offers user-friendly online portals for easy access to bookings and program information, enhancing customer experience. Real-time reporting and data analytics deliver insights into attendance and preferences, aiding in strategic decision-making. Additionally, effective communication tools keep participants and staff informed with timely updates. Overall, recreation management software enhances efficiency, improves service delivery, and boosts customer satisfaction.
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoamtakuyayamamoto1800
In this slide, we show the simulation example and the way to compile this solver.
In this solver, the Helmholtz equation can be solved by helmholtzFoam. Also, the Helmholtz equation with uniformly dispersed bubbles can be simulated by helmholtzBubbleFoam.
Code reviews are vital for ensuring good code quality. They serve as one of our last lines of defense against bugs and subpar code reaching production.
Yet, they often turn into annoying tasks riddled with frustration, hostility, unclear feedback and lack of standards. How can we improve this crucial process?
In this session we will cover:
- The Art of Effective Code Reviews
- Streamlining the Review Process
- Elevating Reviews with Automated Tools
By the end of this presentation, you'll have the knowledge on how to organize and improve your code review proces
Quarkus Hidden and Forbidden ExtensionsMax Andersen
Quarkus has a vast extension ecosystem and is known for its subsonic and subatomic feature set. Some of these features are not as well known, and some extensions are less talked about, but that does not make them less interesting - quite the opposite.
Come join this talk to see some tips and tricks for using Quarkus and some of the lesser known features, extensions and development techniques.
Check out the webinar slides to learn more about how XfilesPro transforms Salesforce document management by leveraging its world-class applications. For more details, please connect with sales@xfilespro.com
If you want to watch the on-demand webinar, please click here: https://www.xfilespro.com/webinars/salesforce-document-management-2-0-smarter-faster-better/
top nidhi software solution freedownloadvrstrong314
This presentation emphasizes the importance of data security and legal compliance for Nidhi companies in India. It highlights how online Nidhi software solutions, like Vector Nidhi Software, offer advanced features tailored to these needs. Key aspects include encryption, access controls, and audit trails to ensure data security. The software complies with regulatory guidelines from the MCA and RBI and adheres to Nidhi Rules, 2014. With customizable, user-friendly interfaces and real-time features, these Nidhi software solutions enhance efficiency, support growth, and provide exceptional member services. The presentation concludes with contact information for further inquiries.
Navigating the Metaverse: A Journey into Virtual Evolution"Donna Lenk
Join us for an exploration of the Metaverse's evolution, where innovation meets imagination. Discover new dimensions of virtual events, engage with thought-provoking discussions, and witness the transformative power of digital realms."
Navigating the Metaverse: A Journey into Virtual Evolution"
Building Data Portals and Science Gateways with Globus
1. Lee Liming – lliming@uchicago.edu
Steve Turoscy – sturoscy@globus.org
Vas Vasiliadis – vas@uchicago.edu
May 11, 2022
Building Data Portals and Science
Gateways with Globus
2. Agenda
• Introduction and motivation
• The Modern Research Data Portal design pattern
• Introducing the Django Globus Portal
• Deploying your Django Globus Portal
• Globus data transfer: a range of options
• Making data findable with Globus Search
• Other customization examples
- Hands-on exercise
- Live demonstration
5. Some challenges…
• Increasing data rates, heterogeneity
• Continuum of computing resources
• Differing workflows across instruments
6. Distribution Store
Data Portal
Advanced Computing Facility
Instrument Facility
A common data flow pattern
Image Analysis
3
Search/Discovery
5
Science!
6
Imaging
1 Acquisition
2
Description/Identification
4
v
7. Data gathering mediated by a
web application
A simpler case: import “big data” into a web app
• You provide a web application (data
portal, library service) that allows
researchers to import “big” datasets
• The datasets are too big for normal file
upload interfaces or storage systems
(1000+ files, TB+ data)
• The datasets must be curated (authorized,
reviewed & catalogued, managed)
• You don’t want a lot of code maintenance,
and you don’t want to give a lot of
technical support to researchers
Your Cloud Storage
Example website: NIH Common Fund Data Ecosystem (CFDE) Portal
8. Why we provide portals and science gateways
• Enable a broad audience of researchers to access the
latest research data
• Simplify access to complicated data sources
(beamlines, electron microscopes, sequencers, etc)
• Add curation and cataloguing so data is findable
• Enable researchers to customize their experience
• Enforce (sometimes complex) access policies
9. What does Globus do for portals?
• Federated login
– Globus handles authentication &
identity federation
– Your portal manages profiles
• Rich groups API for access
management
– Public/private, group-, subject-level
ACLs
• Data upload/download at scale
– Call out to Globus Transfer API
• Facilitate discovery
– Free text search in Globus Search
– Filtering on specific values
– User Friendly GUI
• Automation
– Define Flows for data handling steps
(copy, move, add a search record,
create a DOI, change permissions, etc.)
– Run each Flow w/one API call & let
Globus manage everything
– Simplify your curation code
10. Everything can be done using our web app…
Web app Python CLI
Python SDK
Globus Public REST APIs
Transfer API Search API
Auth API Groups API Flows API
…scripted using our Python CLI…
…or built into an app with SDK & REST APIs
11. A whirlwind tour of Globus APIs
• Globus Auth * – authentication & identities
• Globus Groups – groups & membership
• Globus Transfer * – data transfer & guest collections
• Globus Search – metadata & indexing
• Globus Flows * – automation
* covered in previous session
12. Globus Groups: Use groups for authorization
• Globus Connect Server & Transfer API use groups for guest
collection permissions
– Grant membership manager role to your application
– Your web app can add/remove members to grant/remove access
• Use groups for your application’s permissions
– Instead of managing a bunch of ACLs in your application, use group membership
– Lookup membership
o Check membership to determine permissions
– Add/remove members
– Configure policy settings
– Create/delete groups
– Remember: you can also use the web app for any of the above!
docs.globus.org/api/groups
13. Using guest collections in your data portal
• Create a guest collection; requires authentication
– Cannot be completely automated – must ”log in”
– Create once and automate rest of the steps
• Grant the application Access Manager role
– Allows the application to manage permissions on the collection
– Set for application identity: appclientid@clients.auth.globus.org
• Grant roles for management of endpoint and tasks
14. Globus Search - Data description and discovery
• Metadata store with fine-
grained visibility controls
• Schema agnostic
à dynamic schemas
• Simple search using URL
query parameters
• Complex search using
search request document
14
docs.globus.org/api/search
Search
Index
15. Distinct access policies
may be applied to
Data and Metadata
…(ideally) using
permissions on
guest collections
…using
permissions on
metadata elements
16. Globus Search API overview
• Ingest a new record
– POST / index / id / ingest
– Records include visibility field (Individual & Group IDs)
• Simple query
– GET / index / id / search ? q=type%3Ahdf5
• Faceted search
– POST / index / id / search
– Posted doc includes a query string and facet specifiers
docs.globus.org/api/search
18. MRDP: Key elements
Science DMZ
Fast, clean data path
Data Transfer Nodes
Purpose-built data movers
Globus Platform
Secure, reliable data
orchestration
Globus Connect
Storage system enabler
18
Globus Portal
Framework
Data discovery and access
docs.globus.org/mrdp
24. Key features
• Federated login (InCommon campus IDs)
• Big data export using Globus
• Browse datasets w/Globus Search calls
• Template-driven search results & landing pages
• Django-based framework & templating
• Bootstrap your project with Cookiecutter Django
24
Source: github.com/globus/django-globus-portal-framework
Docs: django-globus-portal-framework.readthedocs.io/en/stable/
25. Step 0: Application registration
• Set redirect URLs
• Get client ID and secret
• Consents implement least
privileges principle
25
developers.globus.org
Redirect URLs
https://tutN.globusdemo.org:8443/
https://tutN.globusdemo.org:8443/complete/globus/
26. Portal deployment
• Install dependent libraries
– For production use, add robust WSGI/ASGI server
• Deploy a portal instance using cookiecutter
• Configure settings
• Run and use!
• Future: containers
28. Where’s the data?
• Remember – we’re using Globus Connect, so your
datasets are in a Globus collection
• Three options for enabling transfers from your portal:
1. Link to the collection in the Globus web app (Easy! But not
customizable.)
2. Use the Globus Helper Page (Easy! A bit customizable.)
3. Use a JavaScript interface (Less easy. Very customizable.)
• Let’s see an example of each…