Cyberinfrastructure and Applications Overview: Howard University June22
•Download as PPT, PDF•
0 likes•1,688 views
1) Cyberinfrastructure refers to the combination of computing systems, data storage systems, advanced instruments and data repositories, visualization environments, and people that enable knowledge discovery through integrated multi-scale simulations and analyses. 2) Cloud computing, multicore processors, and Web 2.0 tools are changing the landscape of cyberinfrastructure by providing new approaches to distributed computing and data sharing that emphasize usability, collaboration, and accessibility. 3) Scientific applications are increasingly data-intensive, requiring high-performance computing resources to analyze large datasets from sources like gene sequencers, telescopes, sensors, and web crawlers.
Report
Share
Report
Share
![Overview of Cyberinfrastructure and the Breadth of Its Application Geoffrey Fox Computer Science, Informatics, Physics Chair Informatics Department Director Community Grids Laboratory and Digital Science Center Indiana University Bloomington IN 47404 (Presenter: Marlon Pierce ) [email_address] http://www.infomall.org [email_address]](https://image.slidesharecdn.com/howarduniversityjune22-090622074945-phpapp02/85/Cyberinfrastructure-and-Applications-Overview-Howard-University-June22-1-320.jpg)
Recommended
OGCE TeraGrid 2010 ASTA Support
This document summarizes the Open Grid Computing Environments (OGCE) project. It describes OGCE software tools like the Gadget Container, XBaya workflow composer, and GFAC application wrapper. It focuses on providing these tools to enable running science applications on grids and clouds. The tools can be used individually or together. OGCE outsources security and data services to providers like Globus, Condor, and iRods. It supports workflows like GridChem, UltraScan, and bioinformatics pipelines. The software is open source and available via anonymous SVN checkout.
Sgg crest-presentation-final
The document provides an overview of the Science Gateway Group at Indiana University. It introduces the group members and describes their focus areas as developing open source software for cyberinfrastructure like Apache Rave and Apache Airavata. It discusses the group's work on extending collaborations with application scientists in various domains. The document also outlines possibilities for collaboration with the PTI CREST Lab on topics like scientific workflows and generalized execution frameworks.
OGCE MSI Presentation
OGCE Presentation by Marlon Pierce at University of Minnesota Supercomputing Institute, February 11, 2011
Computing Outside The Box June 2009
Keynote talk at the International Conference on Supercoming 2009, at IBM Yorktown in New York. This is a major update of a talk first given in New Zealand last January. The abstract follows.
The past decade has seen increasingly ambitious and successful methods for outsourcing computing. Approaches such as utility computing, on-demand computing, grid computing, software as a service, and cloud computing all seek to free computer applications from the limiting confines of a single computer. Software that thus runs "outside the box" can be more powerful (think Google, TeraGrid), dynamic (think Animoto, caBIG), and collaborative (think FaceBook, myExperiment). It can also be cheaper, due to economies of scale in hardware and software. The combination of new functionality and new economics inspires new applications, reduces barriers to entry for application providers, and in general disrupts the computing ecosystem. I discuss the new applications that outside-the-box computing enables, in both business and science, and the hardware and software architectures that make these new applications possible.
Data Automation at Light Sources
Plenary talk at the international Synchrotron Radiation Instrumentation conference in Taiwan, on work with great colleagues Ben Blaiszik, Ryan Chard, Logan Ward, and others.
Rapidly growing data volumes at light sources demand increasingly automated data collection, distribution, and analysis processes, in order to enable new scientific discoveries while not overwhelming finite human capabilities. I present here three projects that use cloud-hosted data automation and enrichment services, institutional computing resources, and high- performance computing facilities to provide cost-effective, scalable, and reliable implementations of such processes. In the first, Globus cloud-hosted data automation services are used to implement data capture, distribution, and analysis workflows for Advanced Photon Source and Advanced Light Source beamlines, leveraging institutional storage and computing. In the second, such services are combined with cloud-hosted data indexing and institutional storage to create a collaborative data publication, indexing, and discovery service, the Materials Data Facility (MDF), built to support a host of informatics applications in materials science. The third integrates components of the previous two projects with machine learning capabilities provided by the Data and Learning Hub for science (DLHub) to enable on-demand access to machine learning models from light source data capture and analysis workflows, and provides simplified interfaces to train new models on data from sources such as MDF on leadership scale computing resources. I draw conclusions about best practices for building next-generation data automation systems for future light sources.
Accelerating Discovery via Science Services
[A talk presented at Oak Ridge National Laboratory on October 15, 2015]
We have made much progress over the past decade toward harnessing the collective power of IT resources distributed across the globe. In big-science projects in high-energy physics, astronomy, and climate, thousands work daily within virtual computing systems with global scope. But we now face a far greater challenge: Exploding data volumes and powerful simulation tools mean that many more--ultimately most?--researchers will soon require capabilities not so different from those used by such big-science teams. How are we to meet these needs? Must every lab be filled with computers and every researcher become an IT specialist? Perhaps the solution is rather to move research IT out of the lab entirely: to develop suites of science services to which researchers can dispatch mundane but time-consuming tasks, and thus to achieve economies of scale and reduce cognitive load. I explore the past, current, and potential future of large-scale outsourcing and automation for science, and suggest opportunities and challenges for today’s researchers. I use examples from Globus and other projects to demonstrate what can be achieved.
Science Services and Science Platforms: Using the Cloud to Accelerate and Dem...
Ever more data- and compute-intensive science makes computing increasingly important for research. But for advanced computing infrastructure to benefit more than the scientific 1%, we need new delivery methods that slash access costs, new sustainability models beyond direct research funding, and new platform capabilities to accelerate the development of new, interoperable tools and services.
The Globus team has been working towards these goals since 2010. We have developed software-as-a-service methods that move complex and time-consuming research IT tasks out of the lab and into the cloud, thus greatly reducing the expertise and resources required to use them. We have demonstrated a subscription-based funding model that engages research institutions in supporting service operations. And we are now also showing how the platform services that underpin Globus applications can accelerate the development and use of an integrated ecosystem of advanced science applications, such as NCAR’s Research Data Archive and OSG Connect, thus enabling access to powerful data and compute resources by many more people than is possible today.
In this talk, I introduce Globus services and the underlying Globus platform. I present representative applications and discuss opportunities that this platform presents for both small science and large facilities.
Accelerating Data-driven Discovery in Energy Science
A talk given at the US Department of Energy, covering our work on research data management and analysis. Three themes:
(1) Eliminate data friction (use of SaaS for research data management)
(2) Liberate scientific data (research on data extraction, organization, publication)
(3) Create discovery engines at DOE facilities (services that organize data + computation)
Recommended
OGCE TeraGrid 2010 ASTA Support
This document summarizes the Open Grid Computing Environments (OGCE) project. It describes OGCE software tools like the Gadget Container, XBaya workflow composer, and GFAC application wrapper. It focuses on providing these tools to enable running science applications on grids and clouds. The tools can be used individually or together. OGCE outsources security and data services to providers like Globus, Condor, and iRods. It supports workflows like GridChem, UltraScan, and bioinformatics pipelines. The software is open source and available via anonymous SVN checkout.
Sgg crest-presentation-final
The document provides an overview of the Science Gateway Group at Indiana University. It introduces the group members and describes their focus areas as developing open source software for cyberinfrastructure like Apache Rave and Apache Airavata. It discusses the group's work on extending collaborations with application scientists in various domains. The document also outlines possibilities for collaboration with the PTI CREST Lab on topics like scientific workflows and generalized execution frameworks.
OGCE MSI Presentation
OGCE Presentation by Marlon Pierce at University of Minnesota Supercomputing Institute, February 11, 2011
Computing Outside The Box June 2009
Keynote talk at the International Conference on Supercoming 2009, at IBM Yorktown in New York. This is a major update of a talk first given in New Zealand last January. The abstract follows.
The past decade has seen increasingly ambitious and successful methods for outsourcing computing. Approaches such as utility computing, on-demand computing, grid computing, software as a service, and cloud computing all seek to free computer applications from the limiting confines of a single computer. Software that thus runs "outside the box" can be more powerful (think Google, TeraGrid), dynamic (think Animoto, caBIG), and collaborative (think FaceBook, myExperiment). It can also be cheaper, due to economies of scale in hardware and software. The combination of new functionality and new economics inspires new applications, reduces barriers to entry for application providers, and in general disrupts the computing ecosystem. I discuss the new applications that outside-the-box computing enables, in both business and science, and the hardware and software architectures that make these new applications possible.
Data Automation at Light Sources
Plenary talk at the international Synchrotron Radiation Instrumentation conference in Taiwan, on work with great colleagues Ben Blaiszik, Ryan Chard, Logan Ward, and others.
Rapidly growing data volumes at light sources demand increasingly automated data collection, distribution, and analysis processes, in order to enable new scientific discoveries while not overwhelming finite human capabilities. I present here three projects that use cloud-hosted data automation and enrichment services, institutional computing resources, and high- performance computing facilities to provide cost-effective, scalable, and reliable implementations of such processes. In the first, Globus cloud-hosted data automation services are used to implement data capture, distribution, and analysis workflows for Advanced Photon Source and Advanced Light Source beamlines, leveraging institutional storage and computing. In the second, such services are combined with cloud-hosted data indexing and institutional storage to create a collaborative data publication, indexing, and discovery service, the Materials Data Facility (MDF), built to support a host of informatics applications in materials science. The third integrates components of the previous two projects with machine learning capabilities provided by the Data and Learning Hub for science (DLHub) to enable on-demand access to machine learning models from light source data capture and analysis workflows, and provides simplified interfaces to train new models on data from sources such as MDF on leadership scale computing resources. I draw conclusions about best practices for building next-generation data automation systems for future light sources.
Accelerating Discovery via Science Services
[A talk presented at Oak Ridge National Laboratory on October 15, 2015]
We have made much progress over the past decade toward harnessing the collective power of IT resources distributed across the globe. In big-science projects in high-energy physics, astronomy, and climate, thousands work daily within virtual computing systems with global scope. But we now face a far greater challenge: Exploding data volumes and powerful simulation tools mean that many more--ultimately most?--researchers will soon require capabilities not so different from those used by such big-science teams. How are we to meet these needs? Must every lab be filled with computers and every researcher become an IT specialist? Perhaps the solution is rather to move research IT out of the lab entirely: to develop suites of science services to which researchers can dispatch mundane but time-consuming tasks, and thus to achieve economies of scale and reduce cognitive load. I explore the past, current, and potential future of large-scale outsourcing and automation for science, and suggest opportunities and challenges for today’s researchers. I use examples from Globus and other projects to demonstrate what can be achieved.
Science Services and Science Platforms: Using the Cloud to Accelerate and Dem...
Ever more data- and compute-intensive science makes computing increasingly important for research. But for advanced computing infrastructure to benefit more than the scientific 1%, we need new delivery methods that slash access costs, new sustainability models beyond direct research funding, and new platform capabilities to accelerate the development of new, interoperable tools and services.
The Globus team has been working towards these goals since 2010. We have developed software-as-a-service methods that move complex and time-consuming research IT tasks out of the lab and into the cloud, thus greatly reducing the expertise and resources required to use them. We have demonstrated a subscription-based funding model that engages research institutions in supporting service operations. And we are now also showing how the platform services that underpin Globus applications can accelerate the development and use of an integrated ecosystem of advanced science applications, such as NCAR’s Research Data Archive and OSG Connect, thus enabling access to powerful data and compute resources by many more people than is possible today.
In this talk, I introduce Globus services and the underlying Globus platform. I present representative applications and discuss opportunities that this platform presents for both small science and large facilities.
Accelerating Data-driven Discovery in Energy Science
A talk given at the US Department of Energy, covering our work on research data management and analysis. Three themes:
(1) Eliminate data friction (use of SaaS for research data management)
(2) Liberate scientific data (research on data extraction, organization, publication)
(3) Create discovery engines at DOE facilities (services that organize data + computation)
Taming Big Data!
My talk at the Winter School on Big Data in Tarragona, Spain.
Abstract: We have made much progress over the past decade toward harnessing the collective power of IT resources distributed across the globe. In high-energy physics, astronomy, and climate, thousands work daily within virtual computing systems with global scope. But we now face a far greater challenge: Exploding data volumes and powerful simulation tools mean that many more--ultimately most?--researchers will soon require capabilities not so different from those used by such big-science teams. How are we to meet these needs? Must every lab be filled with computers and every researcher become an IT specialist? Perhaps the solution is rather to move research IT out of the lab entirely: to leverage the “cloud” (whether private or public) to achieve economies of scale and reduce cognitive load. I explore the past, current, and potential future of large-scale outsourcing and automation for science, and suggest opportunities and challenges for today’s researchers.
Virtual Science in the Cloud
This document discusses using cloud computing and virtualization for scientific research. Some key points:
- Scientists can access remote sensors, share data and workflows, and store personal data in the cloud. Beginners can click to code, while experts can build complex workflows.
- Services allow publishing, finding, and binding to distributed resources through registries. Data can be queried through standards like Simple Image Access Protocol.
- Distributed registries from various organizations harvest metadata to enable semantic search across sky regions, identifiers, tags, vocabularies, schemas, and service descriptions.
- Tools provide code/presentation environments and access to distributed data in the cloud. Services include astronomical cross-matching and event notification through Sky
Towards Enabling Mid-Scale Geo-Science Experiments Through Microsoft Trident ...
Towards Enabling Mid-Scale Geo-Science Experiments Through Microsoft Trident ...Eran Chinthaka Withana
This document proposes a framework to enable mid-scale geo-science experiments through Microsoft Trident and Windows Azure. It discusses challenges faced by geo-science applications in utilizing computational resources and opportunities provided by cloud computing. The proposed framework uses Sigiri to schedule jobs on grid and cloud resources and Trident activities to compose scientific workflows. It aims to support applications like WRF and ensembles through optimized use of Azure worker roles, fault tolerance, and data management capabilities.Big data at experimental facilities
1) Scientists at the Advanced Photon Source use the Argonne Leadership Computing Facility for data reconstruction and analysis from experimental facilities in real-time or near real-time. This provides feedback during experiments.
2) Using the Swift parallel scripting language and ALCF supercomputers like Mira, scientists can process terabytes of data from experiments in minutes rather than hours or days. This enables errors to be detected and addressed during experiments.
3) Key applications discussed include near-field high-energy X-ray diffraction microscopy, X-ray nano/microtomography, and determining crystal structures from diffuse scattering images through simulation and optimization. The workflows developed provide significant time savings and improved experimental outcomes.
The Discovery Cloud: Accelerating Science via Outsourcing and Automation
Director's Colloquium at Los Alamos National Laboratory, September 18, 2014.
We have made much progress over the past decade toward harnessing the collective power of IT resources distributed across the globe. In high-energy physics, astronomy, and climate, thousands work daily within virtual computing systems with global scope. But we now face a far greater challenge: Exploding data volumes and powerful simulation tools mean that many more--ultimately most?--researchers will soon require capabilities not so different from those used by such big-science teams. How are we to meet these needs? Must every lab be filled with computers and every researcher become an IT specialist? Perhaps the solution is rather to move research IT out of the lab entirely: to leverage the “cloud” (whether private or public) to achieve economies of scale and reduce cognitive load. In this talk, I explore the past, current, and potential future of large-scale outsourcing and automation for science.
How HPC and large-scale data analytics are transforming experimental science
In this deck from DataTech19, Debbie Bard from NERSC presents: Supercomputing and the scientist: How HPC and large-scale data analytics are transforming experimental science.
"Debbie Bard leads the Data Science Engagement Group NERSC. NERSC is the mission supercomputing center for the USA Department of Energy, and supports over 7000 scientists and 700 projects with supercomputing needs. A native of the UK, her career spans research in particle physics, cosmology and computing on both sides of the Atlantic. She obtained her PhD at Edinburgh University, and has worked at Imperial College London as well as the Stanford Linear Accelerator Center (SLAC) in the USA, before joining the Data Department at NERSC, where she focuses on data-intensive computing and research, including supercomputing for experimental science and machine learning at scale."
Watch the video: https://wp.me/p3RLHQ-kLV
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
TG11 ORPS Poster
Poster given at TG11, Salt Lake City, July 2011 by Xuan Wu. Raminder Singh, Suresh Marru, Suresh Kumar Deivasigamani, and Marlon Pierce
Invited cloud-e-Genome project talk at 2015 NGS Data Congress
The document describes a scalable WES/WGS processing pipeline and variant interpretation tool. The pipeline uses a workflow programming model to process data in parallel across cloud resources for improved scalability and cost efficiency. The variant interpretation tool provides a simple interface for clinicians while maintaining a history of investigations and provenance for accountability and analytics across patient cases.
Streamlined data sharing and analysis to accelerate cancer research
Advances in genomics and data analytics create new opportunities for cancer research and personalized medical treatment via large-scale federation of genomic, clinical, imaging and other data from many thousands of patients across institutions around the world. Despite these opportunities and promising early results, cancer research is often stymied by information technology barriers. One major barrier is a lack of tools for the reliable, secure, rapid, and easy transfer, sharing, and management of large collections of human data. In the absence of such tools, security and performance concerns often prevent sharing altogether or force researchers to resort to slow and error prone shipping of physical media. If data are received, timely analysis is further impeded by the difficulties inherent in verifying data integrity and managing who can access data and for what purpose. I will discuss how the mature Globus data management platform addresses these obstacles to discovery and explain how its intuitive, web-based interfaces enable use by researchers without specialized IT knowledge. I also describe how Globus technologies can be extended to meet the security requirements of human data so as to enable use in data-intensive cancer research.
qconsf 2013: Top 10 Performance Gotchas for scaling in-memory Algorithms - Sr...
Top 10 Performance Gotchas in scaling in-memory Algorithms
Abstract:
Math Algorithms have primarily been the domain of desktop data science. With the success of scalable algorithms at Google, Amazon, and Netflix, there is an ever growing demand for sophisticated algorithms over big data. In this talk, we get a ringside view in the making of the world's most scalable and fastest machine learning framework, H2O, and the performance lessons learnt scaling it over EC2 for Netflix and over commodity hardware for other power users.
Top 10 Performance Gotchas is about the white hot stories of i/o wars, S3 resets, and muxers, as well as the power of primitive byte arrays, non-blocking structures, and fork/join queues. Of good data distribution & fine-grain decomposition of Algorithms to fine-grain blocks of parallel computation. It's a 10-point story of the rage of a network of machines against the tyranny of Amdahl while keeping the statistical properties of the data and accuracy of the algorithm.
Track: Scalability, Availability, and Performance: Putting It All Together
Time: Wednesday, 11:45am - 12:35pm
Discovery Engines for Big Data: Accelerating Discovery in Basic Energy Sciences
Argonne’s Discovery Engines for Big Data project is working to enable new research modalities based on the integration of advanced computing with experiments at facilities such as the Advanced Photon Source (APS). I review science drivers and initial results in diffuse scattering, high energy diffraction microscopy, tomography, and pythography. I also describe the computational methods and infrastructure that we leverage to support such applications, which include the Petrel online data store, ALCF supercomputers, Globus research data management services, and Swift parallel scripting. This work points to a future in which tight integration of DOE’s experimental and computational facilities enables both new science and more efficient and rapid discovery.
Cloud Accelerated Genomics
Dr Allen Day, PHD presentation on Google Cloud Innovations for Genomics & BIO. BigQuery, Google ML, Google Genomics API and more
Genomic Scale Big Data Pipelines
deck from talk at YOW Data in Sydney, covers VariantSpark, custom Apache Spark Machine Learning library and also GT-Scan2 using AWS Lambda architecture for bioinformatics
Dynamic Resource Allocation Algorithm using Containers
1) The document proposes a dynamic resource allocation algorithm using containers to optimize resource utilization in server farms.
2) It uses Docker to deploy applications in lightweight containers instead of virtual machines to reduce overhead. A node selection algorithm uses fuzzy logic to determine the most suitable node for container deployment based on resource availability and workload.
3) The proposed approach is tested on a small cluster using Docker, Hadoop and the node selection algorithm to process queries. Results show increased processing speed and better resource utilization compared to traditional virtualization methods.
grid mining
The document discusses grids and their potential use for data mining applications in Earth science. Some key points:
- Grids can connect distributed computing and data resources to enable large-scale applications and collaboration.
- The Grid Miner application was developed to mine satellite data on NASA's Information Power Grid as a demonstration.
- Grids could help couple satellite data archives to computational resources, allowing users to process large datasets.
- For this to be realized, data archives need to be connected to grids and tools developed to enable scientists to access and analyze data.
Globus Integrations (CHPC 2019 - South Africa)
This document discusses enabling large-scale data-intensive science through integrating JupyterHub with Globus services. It describes how JupyterHub can be secured using Globus authentication to provide users tokens for remote Globus services from within notebooks. Examples shown include downloading data from Globus into a notebook, analyzing the data, and sharing results to an endpoint. The document also discusses extending this approach to more fully automate neuroscience data workflows using additional Globus services like Search, Transfer, and identity management.
Distributed Cyberinfrastructure to Support Big Data Machine Learning
Panel on the Future of Machine Learning
California Institute for Telecommunications and Information Technology
University of California, Irvine
May 24, 2018
Big Data Day LA 2016/ Big Data Track - Twitter Heron @ Scale - Karthik Ramasa...
Twitter generates billions and billions of events per day. Analyzing these events in real time presents a massive challenge. Twitter designed and deployed a new streaming system called Heron. Heron has been in production nearly 2 years and is widely used by several teams for diverse use cases. This talk looks at Twitter's operating experiences and challenges of running Heron at scale and the approaches taken to solve those challenges.
Cloud com foster december 2010
We've all heard about how on-demand computing and storage will transform scientific practice. But by focusing on resources alone, we're missing the real benefit of the large-scale outsourcing and consequent economies of scale that cloud is about. The biggest IT challenge facing science today is not volume but complexity. Sure, terabytes demand new storage and computing solutions. But they're cheap. It is establishing and operating the processes required to collect, manage, analyze, share, archive, etc., that data that is taking all of our time and killing creativity. And that's where outsourcing can be transformative. An entrepreneur can run a small business from a coffee shop, outsourcing essentially every business function to a software-as-a-service provider--accounting, payroll, customer relationship management, the works. Why can't a young researcher run a research lab from a coffee shop? For that to happen, we need to make it easy for providers to develop "apps" that encapsulate useful capabilities and for researchers to discover, customize, and apply these "apps" in their work. The effect, I will argue, will be a dramatic acceleration of discovery.
Overlay Opportunistic Clouds in CMS/ATLAS at CERN: The CMSooooooCloud in Detail
Overlay Opportunistic Clouds in CMS/ATLAS at CERN: The CMSooooooCloud in DetailJose Antonio Coarasa Perez
Overlay opportunistic clouds in CMS/ATLAS at CERN: The CMSooooooCloud in detail
The CMS and ATLAS online clusters consist of more than 3000 computers each. They have been exclusively used for the data acquisition that led to the Higgs particle discovery, handling 100Gbytes/s data flows and archiving 20Tbytes of data per day.
An openstack cloud layer has been deployed on the newest part of the clusters (totalling 1300 hypervisors and more than 13000 cores in CMS alone) as a minimal overlay so as to leave the primary role of the computers untouched while allowing an opportunistic usage of the cluster.
This presentation will show how to share resources with a minimal impact on the existing infrastructure. We will present the architectural choices made to deploy an unusual, as opposed to dedicated, "overlaid cloud infrastructure". These architectural choices ensured a minimal impact on the running cluster configuration while giving a maximal segregation of the overlaid virtual computer infrastructure. The use of openvswitch to avoid changes on the network infrastructure and encapsulate the virtual machines traffic will be illustrated, as well as the networking configuration adopted due to the nature of our private network. The design and performance of the openstack cloud controlling layer will be presented. We will also show the integration carried out to allow the cluster to be used in an opportunistic way while giving full control to the CMS online run control.
云计算及其应用
The document discusses cloud computing technology and applications. It provides an introduction to cloud computing concepts, distributed systems, MapReduce, and technologies like Google File System, BigTable and AppEngine. It then outlines the syllabus for a cloud computing course, including topics on virtualization, data centers, and guest lectures. Project presentations will account for 60% of the grading.
Grid Computing
The document provides an overview of grid computing, including:
1) Grid computing involves sharing distributed computational resources over a network and providing single login access for users. Resources may be owned by different organizations.
2) Examples of current grids discussed include the NSF PACI/NCSA Alliance Grid, the NSF PACI/SDSC NPACI Grid, and the NASA Information Power Grid.
3) The document also discusses various grid middleware tools and projects for using grid resources, such as Globus, Condor, Legion, Harness, and the Internet Backplane Protocol.
More Related Content
What's hot
Taming Big Data!
My talk at the Winter School on Big Data in Tarragona, Spain.
Abstract: We have made much progress over the past decade toward harnessing the collective power of IT resources distributed across the globe. In high-energy physics, astronomy, and climate, thousands work daily within virtual computing systems with global scope. But we now face a far greater challenge: Exploding data volumes and powerful simulation tools mean that many more--ultimately most?--researchers will soon require capabilities not so different from those used by such big-science teams. How are we to meet these needs? Must every lab be filled with computers and every researcher become an IT specialist? Perhaps the solution is rather to move research IT out of the lab entirely: to leverage the “cloud” (whether private or public) to achieve economies of scale and reduce cognitive load. I explore the past, current, and potential future of large-scale outsourcing and automation for science, and suggest opportunities and challenges for today’s researchers.
Virtual Science in the Cloud
This document discusses using cloud computing and virtualization for scientific research. Some key points:
- Scientists can access remote sensors, share data and workflows, and store personal data in the cloud. Beginners can click to code, while experts can build complex workflows.
- Services allow publishing, finding, and binding to distributed resources through registries. Data can be queried through standards like Simple Image Access Protocol.
- Distributed registries from various organizations harvest metadata to enable semantic search across sky regions, identifiers, tags, vocabularies, schemas, and service descriptions.
- Tools provide code/presentation environments and access to distributed data in the cloud. Services include astronomical cross-matching and event notification through Sky
Towards Enabling Mid-Scale Geo-Science Experiments Through Microsoft Trident ...
Towards Enabling Mid-Scale Geo-Science Experiments Through Microsoft Trident ...Eran Chinthaka Withana
This document proposes a framework to enable mid-scale geo-science experiments through Microsoft Trident and Windows Azure. It discusses challenges faced by geo-science applications in utilizing computational resources and opportunities provided by cloud computing. The proposed framework uses Sigiri to schedule jobs on grid and cloud resources and Trident activities to compose scientific workflows. It aims to support applications like WRF and ensembles through optimized use of Azure worker roles, fault tolerance, and data management capabilities.Big data at experimental facilities
1) Scientists at the Advanced Photon Source use the Argonne Leadership Computing Facility for data reconstruction and analysis from experimental facilities in real-time or near real-time. This provides feedback during experiments.
2) Using the Swift parallel scripting language and ALCF supercomputers like Mira, scientists can process terabytes of data from experiments in minutes rather than hours or days. This enables errors to be detected and addressed during experiments.
3) Key applications discussed include near-field high-energy X-ray diffraction microscopy, X-ray nano/microtomography, and determining crystal structures from diffuse scattering images through simulation and optimization. The workflows developed provide significant time savings and improved experimental outcomes.
The Discovery Cloud: Accelerating Science via Outsourcing and Automation
Director's Colloquium at Los Alamos National Laboratory, September 18, 2014.
We have made much progress over the past decade toward harnessing the collective power of IT resources distributed across the globe. In high-energy physics, astronomy, and climate, thousands work daily within virtual computing systems with global scope. But we now face a far greater challenge: Exploding data volumes and powerful simulation tools mean that many more--ultimately most?--researchers will soon require capabilities not so different from those used by such big-science teams. How are we to meet these needs? Must every lab be filled with computers and every researcher become an IT specialist? Perhaps the solution is rather to move research IT out of the lab entirely: to leverage the “cloud” (whether private or public) to achieve economies of scale and reduce cognitive load. In this talk, I explore the past, current, and potential future of large-scale outsourcing and automation for science.
How HPC and large-scale data analytics are transforming experimental science
In this deck from DataTech19, Debbie Bard from NERSC presents: Supercomputing and the scientist: How HPC and large-scale data analytics are transforming experimental science.
"Debbie Bard leads the Data Science Engagement Group NERSC. NERSC is the mission supercomputing center for the USA Department of Energy, and supports over 7000 scientists and 700 projects with supercomputing needs. A native of the UK, her career spans research in particle physics, cosmology and computing on both sides of the Atlantic. She obtained her PhD at Edinburgh University, and has worked at Imperial College London as well as the Stanford Linear Accelerator Center (SLAC) in the USA, before joining the Data Department at NERSC, where she focuses on data-intensive computing and research, including supercomputing for experimental science and machine learning at scale."
Watch the video: https://wp.me/p3RLHQ-kLV
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
TG11 ORPS Poster
Poster given at TG11, Salt Lake City, July 2011 by Xuan Wu. Raminder Singh, Suresh Marru, Suresh Kumar Deivasigamani, and Marlon Pierce
Invited cloud-e-Genome project talk at 2015 NGS Data Congress
The document describes a scalable WES/WGS processing pipeline and variant interpretation tool. The pipeline uses a workflow programming model to process data in parallel across cloud resources for improved scalability and cost efficiency. The variant interpretation tool provides a simple interface for clinicians while maintaining a history of investigations and provenance for accountability and analytics across patient cases.
Streamlined data sharing and analysis to accelerate cancer research
Advances in genomics and data analytics create new opportunities for cancer research and personalized medical treatment via large-scale federation of genomic, clinical, imaging and other data from many thousands of patients across institutions around the world. Despite these opportunities and promising early results, cancer research is often stymied by information technology barriers. One major barrier is a lack of tools for the reliable, secure, rapid, and easy transfer, sharing, and management of large collections of human data. In the absence of such tools, security and performance concerns often prevent sharing altogether or force researchers to resort to slow and error prone shipping of physical media. If data are received, timely analysis is further impeded by the difficulties inherent in verifying data integrity and managing who can access data and for what purpose. I will discuss how the mature Globus data management platform addresses these obstacles to discovery and explain how its intuitive, web-based interfaces enable use by researchers without specialized IT knowledge. I also describe how Globus technologies can be extended to meet the security requirements of human data so as to enable use in data-intensive cancer research.
qconsf 2013: Top 10 Performance Gotchas for scaling in-memory Algorithms - Sr...
Top 10 Performance Gotchas in scaling in-memory Algorithms
Abstract:
Math Algorithms have primarily been the domain of desktop data science. With the success of scalable algorithms at Google, Amazon, and Netflix, there is an ever growing demand for sophisticated algorithms over big data. In this talk, we get a ringside view in the making of the world's most scalable and fastest machine learning framework, H2O, and the performance lessons learnt scaling it over EC2 for Netflix and over commodity hardware for other power users.
Top 10 Performance Gotchas is about the white hot stories of i/o wars, S3 resets, and muxers, as well as the power of primitive byte arrays, non-blocking structures, and fork/join queues. Of good data distribution & fine-grain decomposition of Algorithms to fine-grain blocks of parallel computation. It's a 10-point story of the rage of a network of machines against the tyranny of Amdahl while keeping the statistical properties of the data and accuracy of the algorithm.
Track: Scalability, Availability, and Performance: Putting It All Together
Time: Wednesday, 11:45am - 12:35pm
Discovery Engines for Big Data: Accelerating Discovery in Basic Energy Sciences
Argonne’s Discovery Engines for Big Data project is working to enable new research modalities based on the integration of advanced computing with experiments at facilities such as the Advanced Photon Source (APS). I review science drivers and initial results in diffuse scattering, high energy diffraction microscopy, tomography, and pythography. I also describe the computational methods and infrastructure that we leverage to support such applications, which include the Petrel online data store, ALCF supercomputers, Globus research data management services, and Swift parallel scripting. This work points to a future in which tight integration of DOE’s experimental and computational facilities enables both new science and more efficient and rapid discovery.
Cloud Accelerated Genomics
Dr Allen Day, PHD presentation on Google Cloud Innovations for Genomics & BIO. BigQuery, Google ML, Google Genomics API and more
Genomic Scale Big Data Pipelines
deck from talk at YOW Data in Sydney, covers VariantSpark, custom Apache Spark Machine Learning library and also GT-Scan2 using AWS Lambda architecture for bioinformatics
Dynamic Resource Allocation Algorithm using Containers
1) The document proposes a dynamic resource allocation algorithm using containers to optimize resource utilization in server farms.
2) It uses Docker to deploy applications in lightweight containers instead of virtual machines to reduce overhead. A node selection algorithm uses fuzzy logic to determine the most suitable node for container deployment based on resource availability and workload.
3) The proposed approach is tested on a small cluster using Docker, Hadoop and the node selection algorithm to process queries. Results show increased processing speed and better resource utilization compared to traditional virtualization methods.
grid mining
The document discusses grids and their potential use for data mining applications in Earth science. Some key points:
- Grids can connect distributed computing and data resources to enable large-scale applications and collaboration.
- The Grid Miner application was developed to mine satellite data on NASA's Information Power Grid as a demonstration.
- Grids could help couple satellite data archives to computational resources, allowing users to process large datasets.
- For this to be realized, data archives need to be connected to grids and tools developed to enable scientists to access and analyze data.
Globus Integrations (CHPC 2019 - South Africa)
This document discusses enabling large-scale data-intensive science through integrating JupyterHub with Globus services. It describes how JupyterHub can be secured using Globus authentication to provide users tokens for remote Globus services from within notebooks. Examples shown include downloading data from Globus into a notebook, analyzing the data, and sharing results to an endpoint. The document also discusses extending this approach to more fully automate neuroscience data workflows using additional Globus services like Search, Transfer, and identity management.
Distributed Cyberinfrastructure to Support Big Data Machine Learning
Panel on the Future of Machine Learning
California Institute for Telecommunications and Information Technology
University of California, Irvine
May 24, 2018
Big Data Day LA 2016/ Big Data Track - Twitter Heron @ Scale - Karthik Ramasa...
Twitter generates billions and billions of events per day. Analyzing these events in real time presents a massive challenge. Twitter designed and deployed a new streaming system called Heron. Heron has been in production nearly 2 years and is widely used by several teams for diverse use cases. This talk looks at Twitter's operating experiences and challenges of running Heron at scale and the approaches taken to solve those challenges.
Cloud com foster december 2010
We've all heard about how on-demand computing and storage will transform scientific practice. But by focusing on resources alone, we're missing the real benefit of the large-scale outsourcing and consequent economies of scale that cloud is about. The biggest IT challenge facing science today is not volume but complexity. Sure, terabytes demand new storage and computing solutions. But they're cheap. It is establishing and operating the processes required to collect, manage, analyze, share, archive, etc., that data that is taking all of our time and killing creativity. And that's where outsourcing can be transformative. An entrepreneur can run a small business from a coffee shop, outsourcing essentially every business function to a software-as-a-service provider--accounting, payroll, customer relationship management, the works. Why can't a young researcher run a research lab from a coffee shop? For that to happen, we need to make it easy for providers to develop "apps" that encapsulate useful capabilities and for researchers to discover, customize, and apply these "apps" in their work. The effect, I will argue, will be a dramatic acceleration of discovery.
Overlay Opportunistic Clouds in CMS/ATLAS at CERN: The CMSooooooCloud in Detail
Overlay Opportunistic Clouds in CMS/ATLAS at CERN: The CMSooooooCloud in DetailJose Antonio Coarasa Perez
Overlay opportunistic clouds in CMS/ATLAS at CERN: The CMSooooooCloud in detail
The CMS and ATLAS online clusters consist of more than 3000 computers each. They have been exclusively used for the data acquisition that led to the Higgs particle discovery, handling 100Gbytes/s data flows and archiving 20Tbytes of data per day.
An openstack cloud layer has been deployed on the newest part of the clusters (totalling 1300 hypervisors and more than 13000 cores in CMS alone) as a minimal overlay so as to leave the primary role of the computers untouched while allowing an opportunistic usage of the cluster.
This presentation will show how to share resources with a minimal impact on the existing infrastructure. We will present the architectural choices made to deploy an unusual, as opposed to dedicated, "overlaid cloud infrastructure". These architectural choices ensured a minimal impact on the running cluster configuration while giving a maximal segregation of the overlaid virtual computer infrastructure. The use of openvswitch to avoid changes on the network infrastructure and encapsulate the virtual machines traffic will be illustrated, as well as the networking configuration adopted due to the nature of our private network. The design and performance of the openstack cloud controlling layer will be presented. We will also show the integration carried out to allow the cluster to be used in an opportunistic way while giving full control to the CMS online run control.
What's hot (20)
Towards Enabling Mid-Scale Geo-Science Experiments Through Microsoft Trident ...
Towards Enabling Mid-Scale Geo-Science Experiments Through Microsoft Trident ...
The Discovery Cloud: Accelerating Science via Outsourcing and Automation
The Discovery Cloud: Accelerating Science via Outsourcing and Automation
How HPC and large-scale data analytics are transforming experimental science
How HPC and large-scale data analytics are transforming experimental science
Invited cloud-e-Genome project talk at 2015 NGS Data Congress
Invited cloud-e-Genome project talk at 2015 NGS Data Congress
Streamlined data sharing and analysis to accelerate cancer research
Streamlined data sharing and analysis to accelerate cancer research
qconsf 2013: Top 10 Performance Gotchas for scaling in-memory Algorithms - Sr...
qconsf 2013: Top 10 Performance Gotchas for scaling in-memory Algorithms - Sr...
Discovery Engines for Big Data: Accelerating Discovery in Basic Energy Sciences
Discovery Engines for Big Data: Accelerating Discovery in Basic Energy Sciences
Dynamic Resource Allocation Algorithm using Containers
Dynamic Resource Allocation Algorithm using Containers
Distributed Cyberinfrastructure to Support Big Data Machine Learning
Distributed Cyberinfrastructure to Support Big Data Machine Learning
Big Data Day LA 2016/ Big Data Track - Twitter Heron @ Scale - Karthik Ramasa...
Big Data Day LA 2016/ Big Data Track - Twitter Heron @ Scale - Karthik Ramasa...
Overlay Opportunistic Clouds in CMS/ATLAS at CERN: The CMSooooooCloud in Detail
Overlay Opportunistic Clouds in CMS/ATLAS at CERN: The CMSooooooCloud in Detail
Similar to Cyberinfrastructure and Applications Overview: Howard University June22
云计算及其应用
The document discusses cloud computing technology and applications. It provides an introduction to cloud computing concepts, distributed systems, MapReduce, and technologies like Google File System, BigTable and AppEngine. It then outlines the syllabus for a cloud computing course, including topics on virtualization, data centers, and guest lectures. Project presentations will account for 60% of the grading.
Grid Computing
The document provides an overview of grid computing, including:
1) Grid computing involves sharing distributed computational resources over a network and providing single login access for users. Resources may be owned by different organizations.
2) Examples of current grids discussed include the NSF PACI/NCSA Alliance Grid, the NSF PACI/SDSC NPACI Grid, and the NASA Information Power Grid.
3) The document also discusses various grid middleware tools and projects for using grid resources, such as Globus, Condor, Legion, Harness, and the Internet Backplane Protocol.
grid computing
The document discusses grid computing and the development of computational grids. Key points:
- Grids allow for sharing of computing power and resources across geographic locations through networked supercomputers, databases, and instruments.
- Major organizations like NASA, DOE, and NSF are working to build computational grids for applications like scientific simulations and instrument control.
- Indiana University is involved in grid research through various departments and projects focused on resource sharing, portals, middleware, and more.
Qiu bosc2010
This document summarizes cloud technologies and their applications in life sciences. It discusses how cloud computing can help address challenges posed by big data through cost-effective data centers, hiding complexity, and parallel computing frameworks like MapReduce. Specific applications highlighted include DNA sequence assembly, metagenomics, and correlating health data with environmental factors. Frameworks like Hadoop, DryadLINQ, and Twister are examined for processing large-scale biological data on clouds.
Iaetsd survey on big data analytics for sdn (software defined networks)
This document discusses using software-defined networking and OpenFlow to improve network architectures for scientific data sharing. It proposes exploring a virtual switch network abstraction combined with SDN concepts to provide a simple, adaptable framework for science users. The challenges of current campus networks not being optimized for large data flows are outlined. Leveraging SDN could help build end-to-end network services with traffic isolation to meet the needs of data-intensive science applications and collaborations.
My Other Computer is a Data Center (2010 v21)
This is a talk I gave recently about cloud computing from the perspective of thinking of a data center as your "other computer."
2023comp90024_Spartan.pdf
The document provides information about the Spartan HPC system at the University of Melbourne, including:
- Spartan is the University of Melbourne's general purpose high performance computing system.
- The document outlines logging into Spartan, using environment modules and job submission, and introduces parallel programming with OpenMP and MPI.
- Spartan has been recognized internationally as a model HPC-cloud hybrid system and has supported over 150 research papers.
Cloud computing and grid computing 360 degree compared
Cloud computing builds upon concepts from cluster and grid computing. Cluster computing links multiple computers to share workloads, while grid computing dynamically aggregates distributed resources for tasks. Cloud computing provides scalable resources and services over the internet. It extends concepts from grid computing by offering virtualized, dynamically provisioned resources on-demand. Key differences are that cloud computing has loose coupling between providers and consumers, supports scaling, and offers services under a pay-per-use business model. Common cloud services are SaaS, PaaS, and IaaS. Challenges include dynamic scalability, security, and standardization. Cloud computing shows promise for further research in areas like security, interoperability and dynamic pricing models.
Big Data Session 1.pptx
This document provides an introduction to a course on big data and analytics. It outlines the instructor and teaching assistant contact information. It then lists the main topics to be covered, including data analytics and mining techniques, Hadoop/MapReduce programming, graph databases and analytics. It defines big data and discusses the 3Vs of big data - volume, variety and velocity. It also covers big data technologies like cloud computing, Hadoop, and graph databases. Course requirements and the grading scheme are outlined.
Linking Programming models between Grids, Web 2.0 and Multicore
Distributed Programming Abstractions are discussed using Web Services, Grids, Parallel Computing and Multicore architectures
Computer project
This document provides an outline for a talk on cloud computing. It begins with an introduction to cloud concepts and technologies like virtualization and parallel computing models. It then discusses different cloud models including IaaS, PaaS and SaaS. The outline includes demonstrations of cloud capabilities with Amazon AWS and Microsoft Azure, as well as data and computing models using MapReduce. It concludes with a case study of a real business application of the cloud and a question and answer section.
GridComputing-an introduction.ppt
Grid computing is a form of distributed computing that utilizes a network of loosely coupled computers acting together to perform large tasks. It facilitates large-scale resource sharing and coordinated problem solving among organizations. The key aspects of grid computing covered in the document include grid middleware, methods of grid computing like distributed supercomputing and data-intensive computing, grid architectures like layered grid architecture and data grid architecture, and simulation tools for modeling grid systems.
Big Data to SMART Data : Process Scenario
Big Data to SMART Data : Process scenario
Scenario of an implementation of a transformation process of the Data towards exploitable data and representative with treatments of the streaming, the distributed systems, the messages, the storage in an NoSQL environment, a management with an ecosystem Big Data graphic visualization of the data with the technologies:
Apache Storm, Apache Zookeeper, Apache Kafka, Apache Cassandra, Apache Spark and Data-Driven Document.
So Long Computer Overlords
The title of this talk is a crass attempt to be catchy and topical, by referring to the recent victory of Watson in Jeopardy.
My point (perhaps confusingly) is not that new computer capabilities are a bad thing. On the contrary, these capabilities represent a tremendous opportunity for science. The challenge that I speak to is how we leverage these capabilities without computers and computation overwhelming the research community in terms of both human and financial resources. The solution, I suggest, is to get computation out of the lab—to outsource it to third party providers.
Abstract follows:
We have made much progress over the past decade toward effective distributed cyberinfrastructure. In big-science fields such as high energy physics, astronomy, and climate, thousands benefit daily from tools that enable the distributed management and analysis of vast quantities of data. But we now face a far greater challenge. Exploding data volumes and new research methodologies mean that many more--ultimately most?--researchers will soon require similar capabilities. How can we possible supply information technology (IT) at this scale, given constrained budgets? Must every lab become filled with computers, and every researcher an IT specialist?
I propose that the answer is to take a leaf from industry, which is slashing both the costs and complexity of consumer and business IT by moving it out of homes and offices to so-called cloud providers. I suggest that by similarly moving research IT out of the lab, we can realize comparable economies of scale and reductions in complexity, empowering investigators with new capabilities and freeing them to focus on their research.
I describe work we are doing to realize this approach, focusing initially on research data lifecycle management. I present promising results obtained to date, and suggest a path towards large-scale delivery of these capabilities. I also suggest that these developments are part of a larger "revolution in scientific affairs," as profound in its implications as the much-discussed "revolution in military affairs" resulting from more capable, low-cost IT. I conclude with some thoughts on how researchers, educators, and institutions may want to prepare for this revolution.
Grid computing
In computing, It is the description about Grid Computing.
It gives deep idea about grid, what is grid computing? , why we need it? , why it is so ? etc. History and Architecture of grid computing is also there. Advantages , disadvantages and conclusion is also included.
TeraGrid Communication and Computation
4 TeraGrid Sites Have Focal Points:
SDSC – The Data Place
Large-scale and high-performance data analysis/handling
Every Cluster Node is Directly Attached to SAN
NCSA – The Compute Place
Large-scale, Large Flops computation
Argonne – The Viz place
Scalable Viz walls
Caltech – The Applications place
Data and flops for applications – Especially some of the GriPhyN Apps
Specific machine configurations reflect this
Presentation-1.ppt
This document provides an overview of distributed and cloud computing technologies. It discusses the evolution from centralized computing to distributed models over the Internet. Key points include:
- Computing has shifted from centralized mainframes to distributed systems using networks, grids, and now Internet clouds.
- Multicore CPUs and many-core GPUs enable massive parallelism for high-performance and high-throughput computing.
- Technologies like virtualization and service-oriented architectures helped enable cloud computing as a new paradigm.
1. GRID COMPUTING
OUTLINE
Introduction to Grid Computing
Methods of Grid computing
Grid Middleware
Grid Architecture
Open Cloud Consortium Overview (01-10-10 V6)
The Open Cloud Consortium (OCC) is a non-profit organization that supports cloud computing standards and develops testbeds for interoperability. It has members from companies, universities, and government agencies. The OCC manages the Open Cloud Testbed, Intercloud Testbed, and Open Science Data Cloud. It also has working groups focused on large data clouds, applications, and cloud services. The Intercloud Testbed aims to address gaps in linking infrastructure and platform services. Benchmarks like Gray Sort and MalStone are used to evaluate large data cloud performance. The Open Cloud Testbed provides shared cloud resources through a "condominium cloud" model. The Open Science Data Cloud hosts scientific data sets for research.
The Evolving Landscape of Data Engineering
The document discusses the evolving landscape of data engineering. It provides context on the past, present, and future of data engineering. Specifically, it notes that in the past, data engineering was driven by open source communities and the early histories of AWS and Google Cloud. It describes common present-day patterns like serverless architectures and data locality. Finally, it outlines a future wish list, including data catalogs, monitoring systems, and more intelligent data infrastructure. The document concludes by offering recommendations on where to start with technologies, Google Cloud courses, and developing domain knowledge.
Similar to Cyberinfrastructure and Applications Overview: Howard University June22 (20)
Iaetsd survey on big data analytics for sdn (software defined networks)
Iaetsd survey on big data analytics for sdn (software defined networks)
Cloud computing and grid computing 360 degree compared
Cloud computing and grid computing 360 degree compared
Linking Programming models between Grids, Web 2.0 and Multicore
Linking Programming models between Grids, Web 2.0 and Multicore
More from marpierc
Scientific
Apache Airavata is an open source science gateway software framework that allows users to compose, manage, execute, and monitor distributed computational workflows. It provides tools and services to register applications, schedule jobs on various resources, and manage workflows and generated data. Airavata is used across several domains to support scientific workflows and is largely derived from academic research funded by the NSF.
IWSG2014: Developing Science Gateways Using Apache Airavata
Presentation on using Apache Airavata to build science gateways at the International Workshop on Science Gateways (IWSG) 2014 in Dublin, Ireland.
XSEDE14 SciGaP-Apache Airavata Tutorial
Tutorial Presentation on Apache Airavata and the Science Gateway Platform as a service (SciGaP) project at XSEDE14 (Atlanta, GA, July 14, 2014)
ACES QuakeSim 2011
The document discusses the OpenQuake Infomall, which aims to provide earthquake data, simulations, and analysis tools as cloud-based services, enabling researchers to access and share resources and build workflows linking different services. It notes important trends like data growth, parallel computing on multicore systems and clouds, and the potential for "X as a Service" delivery models to improve collaboration and reproducibility in earthquake science. Key challenges include standardizing interfaces to allow interoperability between different data sources and analysis tools.
GCE11 Apache Rave Presentation
Rave is an Apache incubator project that provides tools for building science gateways using open standards. It allows creation of a downloadable portal using minimal configuration and provides ways for developers to customize and extend the portal. Rave uses a model-view-controller architecture and is implemented in JavaScript and Java with components like user management, widgets, and configuration files that can be modified by developers.
SC11 Science Gateway Group Overview
The IU Science Gateway Group supports the development of web-based scientific research tools and gateways. Led by Marlon Pierce and including several senior staff and interns, the group develops interfaces, workflows, and APIs. They foster sustainability through Apache projects like Airavata and Rave. The group collaborates widely and works to advance gateway computing through the Open Gateway Computing Environments partnership and XSEDE support activities.
Experiences with the Apache Software Foundation
This talk describes the OGCE group's motivations for and early experiences participating in the Apache Software Foundation.
PNNL April 2011 ogce
This document discusses developing cyberinfrastructure to support computational chemistry workflows. It describes the OREChem project which aims to develop infrastructure for scholarly materials in chemistry. It outlines IU's objectives to build pipelines to fetch OREChem data, perform computations on resources like TeraGrid, and store results. It also discusses the GridChem science gateway which supports various chemistry applications and the ParamChem project which automates parameterization of molecular mechanics methods through workflows. Finally, it covers the Open Gateway Computing Environments project and efforts to sustain software through the Apache Software Foundation.
OGCE RT Rroject Review
The document summarizes the Open Grid Computing Environments (OGCE) software and activities. It describes various OGCE software components like the gadget container, XBaya workflow composer, and GFAC application wrapper. It also discusses collaborations with gateways like UltraScan, GridChem, and SimpleGrid to integrate OGCE tools and provide more advanced support for workflows, job management, and other capabilities.
OGCE SC10
The document discusses Open Gateway Computing Environments (OGCE) and its software components. OGCE develops secure web-based science gateways for fields like chemistry, bioinformatics, biophysics, and environmental sciences. It is funded by the NSF. Key OGCE software includes the Gadget Container, GFAC for invoking scientific applications on grids and clouds, and workflow tools. Partners include Indiana University, NCSA, Purdue University, and UTHSCSA. The document provides examples of OGCE components in action, like UltraScan, GridChem, and BioVLAB. It also discusses building simple grid gadgets and computational chemistry workflows with GridChem.
Building Science Gateways with Gadgets and OpenSocial
Talk given remotely at OSE2010, Utrecht, Netherlands, Dec 6-7, 2010. See http://wiki.opensocial.org/index.php?title=OSE2010
OGCE TeraGrid 2010 Science Gateway Tutorial Intro
The document summarizes a tutorial presentation about the Open Grid Computing Environments (OGCE) software tools for building science gateways. The OGCE tools include a gadget container, workflow composer called XBaya, and application factory service called GFac. The presentation demonstrates how these tools can be used to build portals and compose workflows to access resources like the TeraGrid.
OGCE SciDAC2010 Tutorial
The document summarizes the Open Grid Computing Environments (OGCE) software tools for building science gateways. It describes several key components:
1) The OGCE Gadget Container allows building portals out of Google gadgets and supports workflows, registries, and experiments.
2) Tools like XBaya allow composing scientific workflows that can run on resources like the TeraGrid.
3) The software is open source and can be used individually or together to power science gateways and provide interfaces and services to computational resources.
OGCE Review for Indiana University Research Technologies
The document describes the Open Grid Computing Environments (OGCE) software suite and related activities. It provides an overview of various OGCE tools like the gadget container, XBaya workflow composer, and GFAC application wrapper service. It also summarizes collaborations with gateways like UltraScan, GridChem, and SimpleGrid to integrate OGCE tools and develop gateway components.
OREChem Services and Workflows
The document discusses Microsoft Research's ORECHEM project, which aims to integrate chemistry scholarship with web architectures, grid computing, and the semantic web. It involves developing infrastructure to enable new models for research and dissemination of scholarly materials in chemistry. Key aspects include using OAI-ORE standards to describe aggregations of web resources related to crystallography experiments. The objective is to build a pipeline that extracts 3D coordinate data from feeds, performs computations on resources like TeraGrid, and stores resulting RDF triples in a triplestore. RESTful web services are implemented to access different steps in the workflow.
Ogce about-sc10
The OGCE team develops open source software for building secure science gateways in various domains like chemistry, bioinformatics, and environmental sciences. They are funded by the National Science Foundation to support the full lifecycle of gateway software development. Their software components enable web-based access to remote resources and tools.
Indiana University's Advanced Science Gateway Support
Overview of Indiana University's Advanced Science Gateway support activities for drug discovery, computational chemistry, and other Web portals. For a broader overview of the OGCE project, see http://www.collab-ogce.org/ogce/index.php
OGCE TG09 Tech Track Presentation
This document provides an overview of the Open Grid Computing Environments (OGCE) project, including portals, services, workflows, gadgets, and tags they develop. It discusses how OGCE software is used in science gateways and contributes code back to these projects. It also summarizes upcoming and existing OGCE services, strategies for adopting web 2.0 technologies, examples of OGCE gadgets and integration with open social containers, and a plan to integrate these components for demonstration at SC09.
GTLAB Installation Tutorial for SciDAC 2009
GTLAB is a Java Server Faces tag library that wraps Grid and web services to build portal-based and standalone applications. It contains tags for common tasks like job submission, file transfer, credential management. GTLAB applications can be deployed as portlets or converted to Google Gadgets. The document provides instructions for installing GTLAB, examples of tags, and making new custom tags.
OGCE Overview for SciDAC 2009
The document provides an overview of the Open Grid Computing Environments (OGCE) project, which develops and packages software for science gateways and resources. Key components discussed include the OGCE portal for building grid portals, Axis services for resource discovery and prediction, a workflow suite, and JavaScript and tag libraries. The document describes downloading and installing the OGCE software, which can be done with a single command, and discusses some of the portlets, services, and components included in the OGCE toolkit.
More from marpierc (20)
IWSG2014: Developing Science Gateways Using Apache Airavata
IWSG2014: Developing Science Gateways Using Apache Airavata
Building Science Gateways with Gadgets and OpenSocial
Building Science Gateways with Gadgets and OpenSocial
OGCE Review for Indiana University Research Technologies
OGCE Review for Indiana University Research Technologies
Indiana University's Advanced Science Gateway Support
Indiana University's Advanced Science Gateway Support
Recently uploaded
Pushing the limits of ePRTC: 100ns holdover for 100 days
At WSTS 2024, Alon Stern explored the topic of parametric holdover and explained how recent research findings can be implemented in real-world PNT networks to achieve 100 nanoseconds of accuracy for up to 100 days.
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Discover the seamless integration of RPA (Robotic Process Automation), COMPOSER, and APM with AWS IDP enhanced with Slack notifications. Explore how these technologies converge to streamline workflows, optimize performance, and ensure secure access, all while leveraging the power of AWS IDP and real-time communication via Slack notifications.
UiPath Test Automation using UiPath Test Suite series, part 5
Welcome to UiPath Test Automation using UiPath Test Suite series part 5. In this session, we will cover CI/CD with devops.
Topics covered:
CI/CD with in UiPath
End-to-end overview of CI/CD pipeline with Azure devops
Speaker:
Lyndsey Byblow, Test Suite Sales Engineer @ UiPath, Inc.
National Security Agency - NSA mobile device best practices
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
A tale of scale & speed: How the US Navy is enabling software delivery from l...
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
20240609 QFM020 Irresponsible AI Reading List May 2024
Everything I found interesting about the irresponsible use of machine intelligence in May 2024
Microsoft - Power Platform_G.Aspiotis.pdf
Revolutionizing Application Development
with AI-powered low-code, presentation by George Aspiotis, Sr. Partner Development Manager, Microsoft
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
Dr. Sean Tan, Head of Data Science, Changi Airport Group
Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.
Essentials of Automations: The Art of Triggers and Actions in FME
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
ABSTRACT: A prima vista, un mattoncino Lego e la backdoor XZ potrebbero avere in comune il fatto di essere entrambi blocchi di costruzione, o dipendenze di progetti creativi e software. La realtà è che un mattoncino Lego e il caso della backdoor XZ hanno molto di più di tutto ciò in comune.
Partecipate alla presentazione per immergervi in una storia di interoperabilità, standard e formati aperti, per poi discutere del ruolo importante che i contributori hanno in una comunità open source sostenibile.
BIO: Sostenitrice del software libero e dei formati standard e aperti. È stata un membro attivo dei progetti Fedora e openSUSE e ha co-fondato l'Associazione LibreItalia dove è stata coinvolta in diversi eventi, migrazioni e formazione relativi a LibreOffice. In precedenza ha lavorato a migrazioni e corsi di formazione su LibreOffice per diverse amministrazioni pubbliche e privati. Da gennaio 2020 lavora in SUSE come Software Release Engineer per Uyuni e SUSE Manager e quando non segue la sua passione per i computer e per Geeko coltiva la sua curiosità per l'astronomia (da cui deriva il suo nickname deneb_alpha).
Artificial Intelligence for XMLDevelopment
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers:
* What is Vector Search?
* Importance and benefits of vector search
* Practical use cases across various industries
* Step-by-step implementation guide
* Live demos with code snippets
* Enhancing LLM capabilities with vector search
* Best practices and optimization strategies
Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications.
#MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
Leonard Jayamohan, Partner & Generative AI Lead, Deloitte
This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.
20240605 QFM017 Machine Intelligence Reading List May 2024
Everything I found interesting about machines behaving intelligently during May 2024
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AIVladimir Iglovikov, Ph.D.
Presented by Vladimir Iglovikov:
- https://www.linkedin.com/in/iglovikov/
- https://x.com/viglovikov
- https://www.instagram.com/ternaus/
This presentation delves into the journey of Albumentations.ai, a highly successful open-source library for data augmentation.
Created out of a necessity for superior performance in Kaggle competitions, Albumentations has grown to become a widely used tool among data scientists and machine learning practitioners.
This case study covers various aspects, including:
People: The contributors and community that have supported Albumentations.
Metrics: The success indicators such as downloads, daily active users, GitHub stars, and financial contributions.
Challenges: The hurdles in monetizing open-source projects and measuring user engagement.
Development Practices: Best practices for creating, maintaining, and scaling open-source libraries, including code hygiene, CI/CD, and fast iteration.
Community Building: Strategies for making adoption easy, iterating quickly, and fostering a vibrant, engaged community.
Marketing: Both online and offline marketing tactics, focusing on real, impactful interactions and collaborations.
Mental Health: Maintaining balance and not feeling pressured by user demands.
Key insights include the importance of automation, making the adoption process seamless, and leveraging offline interactions for marketing. The presentation also emphasizes the need for continuous small improvements and building a friendly, inclusive community that contributes to the project's growth.
Vladimir Iglovikov brings his extensive experience as a Kaggle Grandmaster, ex-Staff ML Engineer at Lyft, sharing valuable lessons and practical advice for anyone looking to enhance the adoption of their open-source projects.
Explore more about Albumentations and join the community at:
GitHub: https://github.com/albumentations-team/albumentations
Website: https://albumentations.ai/
LinkedIn: https://www.linkedin.com/company/100504475
Twitter: https://x.com/albumentationsGraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
Maruthi Prithivirajan, Head of ASEAN & IN Solution Architecture, Neo4j
Get an inside look at the latest Neo4j innovations that enable relationship-driven intelligence at scale. Learn more about the newest cloud integrations and product enhancements that make Neo4j an essential choice for developers building apps with interconnected data and generative AI.
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
Neha Bajwa, Vice President of Product Marketing, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
Presentation of the OECD Artificial Intelligence Review of Germany
Consult the full report at https://www.oecd.org/digital/oecd-artificial-intelligence-review-of-germany-609808d6-en.htm
Recently uploaded (20)
Pushing the limits of ePRTC: 100ns holdover for 100 days
Pushing the limits of ePRTC: 100ns holdover for 100 days
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
Monitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR Events
A tale of scale & speed: How the US Navy is enabling software delivery from l...
A tale of scale & speed: How the US Navy is enabling software delivery from l...
20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
Cyberinfrastructure and Applications Overview: Howard University June22
- 1. Overview of Cyberinfrastructure and the Breadth of Its Application Geoffrey Fox Computer Science, Informatics, Physics Chair Informatics Department Director Community Grids Laboratory and Digital Science Center Indiana University Bloomington IN 47404 (Presenter: Marlon Pierce ) [email_address] http://www.infomall.org [email_address]
- 2. Time Parallel Computing Grids and Federated Computing Scientific Enterprise Computing Scientific Web 2.0 Cloud Computing Parallel Computing Evolution of Scientific Computing, 1985-2010 Evidence of Intelligent Design? Y-Axis is whatever you want it to be.
- 4. TeraGrid High Performance Computing Systems 2007-8 Computational Resources (size approximate - not to scale) Slide Courtesy Tommy Minyard, TACC SDSC TACC NCSA ORNL PU IU PSC NCAR (504TF) 2008 (~1PF) Tennessee LONI/LSU UC/ANL
- 9. Environmental Monitoring Cyberinfrastructure at Clemson
- 11. Forces on Cyberinfrastructure: Clouds, Multicore, and Web 2.0
- 12. Gartner 2008 Technology Hype Curve Clouds, Microblogs and Green IT appear Basic Web Services, Wikis and SOA becoming mainstream
- 13. Gartner’s 2005 Hype Curve
- 15. Enterprise Approach Web 2.0 Approach JSR 168 Portlets Google Gadgets, Widgets, badges Server-side integration and processing AJAX, client-side integration and processing, JavaScript SOAP RSS, Atom, JSON WSDL REST (GET, PUT, DELETE, POST) Portlet Containers Open Social Containers (Orkut, LinkedIn, Shindig); Facebook; StartPages User Centric Gateways Social Networking Portals Workflow managers (Taverna, Kepler, XBaya, etc) Mash-ups WS-Eventing, WS-Notification, Enterprise Messaging Blogging and Micro-blogging with REST, RSS/Atom, and JSON messages (Blogger, Twitter) Semantic Web: RDF, OWL, ontologies Microformats, folksonomies
- 17. Some Commercial Clouds Bold faced entries have open source equivalents Cloud/ Service Amazon Microsoft Azure Google (and Apache) Data S3, EBS, SimpleDB Blob, Table, SQL Services GFS, BigTable Computing EC2, Elastic Map Reduce (runs Hadoop) Compute Service MapReduce (not public, but Hadoop) Service Hosting Amazon Load Balancing Web Hosting Service AppEngine/AppDrop
- 24. Parallel Clustering and Parallel Multidimensional Scaling MDS 4500 Points : Clustal MSA 3000 Points : Clustal MSA Kimura2 Distance Applied to ~5000 dimensional gene sequences and ~20 dimensional patient record data Very good parallel speedup 4000 Points : Patient Record Data on Obesity and Environment 4500 Points : Pairwise Aligned
- 26. Intel’s Projection Technology might support: 2010: 16—64 cores 200GF—1 TF 2013: 64—256 cores 500GF– 4 TF 2016: 256--1024 cores 2 TF– 20 TF