Bioinformatics Data Pipelines built by CSIRO on AWS

I presented to the Environmental Data Science group at UChicago, with the goal of getting them excited about the opportunities inherent in big data, big computing, and AI--and to think about how to collaborate with Argonne in those areas. We had a great and long conversation about Takuya Kurihana's work on unsupervised learning for cloud classification. I also mentioned our work making NASA and CMIP data accessible on AI supercomputers.

Data Automation at Light Sources

Plenary talk at the international Synchrotron Radiation Instrumentation conference in Taiwan, on work with great colleagues Ben Blaiszik, Ryan Chard, Logan Ward, and others. Rapidly growing data volumes at light sources demand increasingly automated data collection, distribution, and analysis processes, in order to enable new scientific discoveries while not overwhelming finite human capabilities. I present here three projects that use cloud-hosted data automation and enrichment services, institutional computing resources, and high- performance computing facilities to provide cost-effective, scalable, and reliable implementations of such processes. In the first, Globus cloud-hosted data automation services are used to implement data capture, distribution, and analysis workflows for Advanced Photon Source and Advanced Light Source beamlines, leveraging institutional storage and computing. In the second, such services are combined with cloud-hosted data indexing and institutional storage to create a collaborative data publication, indexing, and discovery service, the Materials Data Facility (MDF), built to support a host of informatics applications in materials science. The third integrates components of the previous two projects with machine learning capabilities provided by the Data and Learning Hub for science (DLHub) to enable on-demand access to machine learning models from light source data capture and analysis workflows, and provides simplified interfaces to train new models on data from sources such as MDF on leadership scale computing resources. I draw conclusions about best practices for building next-generation data automation systems for future light sources.

Machine Learning in Healthcare Diagnostics

Foster CRA March 2022.pptx

Daniel Lopresti, Bill Gropp, Mark D. Hill, Katie Schuman, and I put together a white paper on "Building a National Discovery Cloud" for the Computing Community Consortium (http://cra.org/ccc). I presented these slides at a Computing Research Association "Best Practices on using the Cloud for Computing Research Workshop" (https://cra.org/industry/events/cloudworkshop/). Abstract from White Paper: The nature of computation and its role in our lives have been transformed in the past two decades by three remarkable developments: the emergence of public cloud utilities as a new computing platform; the ability to extract information from enormous quantities of data via machine learning; and the emergence of computational simulation as a research method on par with experimental science. Each development has major implications for how societies function and compete; together, they represent a change in technological foundations of society as profound as the telegraph or electrification. Societies that embrace these changes will lead in the 21st Century; those that do not, will decline in prosperity and influence. Nowhere is this stark choice more evident than in research and education, the two sectors that produce the innovations that power the future and prepare a workforce able to exploit those innovations, respectively. In this article, we introduce these developments and suggest steps that the US government might take to prepare the research and education system for its implications.

Coding the Continuum

Microsoft Azure for Research

In 2001, as early high-speed networks were deployed, George Gilder observed that “when the network is as fast as the computer's internal links, the machine disintegrates across the net into a set of special purpose appliances.” Two decades later, our networks are 1,000 times faster, our appliances are increasingly specialized, and our computer systems are indeed disintegrating. As hardware acceleration overcomes speed-of-light delays, time and space merge into a computing continuum. Familiar questions like “where should I compute,” “for what workloads should I design computers,” and "where should I place my computers” seem to allow for a myriad of new answers that are exhilarating but also daunting. Are there concepts that can help guide us as we design applications and computer systems in a world that is untethered from familiar landmarks like center, cloud, edge? I propose some ideas and report on experiments in coding the continuum.

2014 moore-dddc.titus.brown

Living Outside the Comfort Zone - Daron green florianopolis 5-7-2014

High Performance Cyberinfrastructure Enabling Data-Driven Science Supporting ...

NERSC, AI and the Superfacility, Debbie Bard

PacificResearchPlatform

The Pacific Research Platform (PRP) aims to achieve transparent and rapid data access among collaborating scientists at multiple institutions through an integrated implementation of data-focused networking that extends the university campus Science DMZ model to a regional, national, and, eventually, a global scale. PRP researchers are routinely achieving high-performance end-to-end networking from their labs to their collaborators’ labs and data centers, traversing multiple, heterogeneous Science DMZs and wide-area networks connecting multiple campus gateways, enabling researchers across the partnership to transfer data over dedicated optical lightpaths at speeds from 10Gb/s to 100Gb/s.

Big data ecosystem

SlideCentral

Foss4G 2009 Scenz Grid

noho

Cloud Accelerated Genomics

Idan Tohami

Computational Materials Design and Data Dissemination through the Materials P...

Big Data Day LA 2016/ Big Data Track - Twitter Heron @ Scale - Karthik Ramasa...

Data Con LA

Twitter generates billions and billions of events per day. Analyzing these events in real time presents a massive challenge. Twitter designed and deployed a new streaming system called Heron. Heron has been in production nearly 2 years and is widely used by several teams for diverse use cases. This talk looks at Twitter's operating experiences and challenges of running Heron at scale and the approaches taken to solve those challenges.

Accelerating data-intensive science by outsourcing the mundane

Talk at eResearch New Zealand Conference, June 2011 (given remotely from Italy, unfortunately!) Abstract: Whitehead observed that "civilization advances by extending the number of important operations which we can perform without thinking of them." I propose that cloud computing can allow us to accelerate dramatically the pace of discovery by removing a range of mundane but timeconsuming research data management tasks from our consciousness. I describe the Globus Online system that we are developing to explore these possibilities, and propose milestones for evaluating progress towards smarter science.

DuraMat Data Management and Analytics

Research workflow - 4 June 2018

Zachary Labe

Butler - a framework for a large-scale scientific analysis on the cloud - EOS...

ATMOSPHERE .

Sgg crest-presentation-final

marpierc

Time to Science/Time to Results: Transforming Research in the Cloud

Amazon Web Services

This session demonstrates how cloud can accelerate breakthroughs in scientific research by providing on-demand access to powerful computing. You will gain insight into how scientific researchers are using the cloud to solve complex science, engineering, and business problems that require high bandwidth, low latency networking and very high compute capabilities. You will hear how leveraging the cloud reduces the costs and time to conduct large scale, worldwide collaborative research. Researchers can then access computational power, data storage, and supercomputing resources, and data sharing capabilities in a cost-efficient manner without implementation delays. Disease research can be accomplished in a fraction of the time, and innovative researchers in small schools or distant corners of the world have access to the same computing power as those at major research institutions by leveraging Amazon EC2, Amazon S3, optimizing C3 instances and more to increase collaboration. This session will provide best practices and insight from UC Berkeley AMP Lab on the services used to connect disparate sets of data to drive meaningful new insight and impact.

Utility HPC: Right Systems, Right Scale, Right ScienceChef Software, Inc.

What's hot

Accelerating Science with Cloud Technologies in the ABoVE Science Cloud

Globus

Big Data, Big Computing, AI, and Environmental Science

Data Automation at Light Sources

Machine Learning in Healthcare Diagnostics

Foster CRA March 2022.pptx

Coding the Continuum

Microsoft Azure for Research

2014 moore-dddc.titus.brown

Living Outside the Comfort Zone - Daron green florianopolis 5-7-2014

High Performance Cyberinfrastructure Enabling Data-Driven Science Supporting ...

NERSC, AI and the Superfacility, Debbie Bard

PacificResearchPlatform

Big data ecosystem

SlideCentral

Foss4G 2009 Scenz Grid

noho

Cloud Accelerated Genomics

Idan Tohami

Computational Materials Design and Data Dissemination through the Materials P...

Big Data Day LA 2016/ Big Data Track - Twitter Heron @ Scale - Karthik Ramasa...

Data Con LA

Accelerating data-intensive science by outsourcing the mundane

DuraMat Data Management and Analytics

Research workflow - 4 June 2018

Zachary Labe

Butler - a framework for a large-scale scientific analysis on the cloud - EOS...

ATMOSPHERE .

Sgg crest-presentation-final

marpierc

What's hot (20)

Accelerating Science with Cloud Technologies in the ABoVE Science Cloud

Big Data, Big Computing, AI, and Environmental Science

Data Automation at Light Sources

Machine Learning in Healthcare Diagnostics

Foster CRA March 2022.pptx

Coding the Continuum

2014 moore-ddd

Living Outside the Comfort Zone - Daron green florianopolis 5-7-2014

High Performance Cyberinfrastructure Enabling Data-Driven Science Supporting ...

NERSC, AI and the Superfacility, Debbie Bard

Big data ecosystem

Foss4G 2009 Scenz Grid

Cloud Accelerated Genomics

Computational Materials Design and Data Dissemination through the Materials P...

Big Data Day LA 2016/ Big Data Track - Twitter Heron @ Scale - Karthik Ramasa...

Accelerating data-intensive science by outsourcing the mundane

DuraMat Data Management and Analytics

Research workflow - 4 June 2018

Butler - a framework for a large-scale scientific analysis on the cloud - EOS...

Sgg crest-presentation-final

Similar to Bioinformatics Data Pipelines built by CSIRO on AWS

Time to Science/Time to Results: Transforming Research in the Cloud

Amazon Web Services

Utility HPC: Right Systems, Right Scale, Right ScienceChef Software, Inc.

VariantSpark: applying Spark-based machine learning methods to genomic inform...

Denis C. Bauer

Genomic information is increasingly used in medical practice giving rise to the need for efficient analysis methodology able to cope with thousands of individuals and millions of variants. Here we introduce VariantSpark, which utilizes Hadoop/Spark along with its machine learning library, MLlib, providing the means of parallelisation for population-scale bioinformatics tasks. VariantSpark is the interface to the standard variant format (VCF), offers seamless genome-wide sampling of variants and provides a pipeline for visualising results. To demonstrate the capabilities of VariantSpark, we clustered more than 3,000 individuals with 80 Million variants each to determine the population structure in the dataset. VariantSpark is 80% faster than the Spark-based genome clustering approach, ADAM, the comparable implementation using Hadoop/Mahout, as well as Admixture, a commonly used tool for determining individual ancestries. It is over 90% faster than traditional implementations using R and Python. These benefits of speed, resource consumption and scalability enables VariantSpark to open up the usage of advanced, efficient machine learning algorithms to genomic data. The package is written in Scala and available at https://github.com/BauerLab/VariantSpark.

Computation and Knowledge

How novel compute technology transforms life science research

Denis C. Bauer

Unprecedented data volumes and pressure on turnaround time driven by commercial applications require bioinformatics solutions to evolve to meed these new demands. New compute paradigms and cloud-based IT solutions enable this transition. Here I present two solution capable of meeting these demands for genomic variant analysis, VariantSpark, as well as genome engineering applications, GT-Scan2. VariantSpark classifies 3000 individuals with 80 Million genomic variants each in under 30 minutes. This Hadoop/Spark solution for machine learning application on genomic data is hence capable to scale up to population size cohorts. GT-Scan2, identifies CRISPR target sites by minimizing off-target effects and maximizing on-target efficiency. This optimization is powered by AWS Lambda functions, which offer an “always-on” web service that can instantaneously recruit enough compute resources keep runtime stable even for queries with several thousand of potential target sites.

Azure Databricks for Data Scientists

Richard Garris

CS Guest Lecture 2015 10-05 advanced databases

Gabe Rudy

Chemical workflows supporting automated research data collection

Valery Tkachenko

Acquisition of data from public sources is inefficient, time consuming and limited in scope. The NIH has recently posted its intention to financially support data deposition by investigators through the ‘data sharing plan' for each funded proposal. However, this plan also points to a current weakness of the centralized data sharing and acquisition as all laboratories use different data collection and formatting approaches. These inconsistencies in data formatting by individual labs leads to the need to invest significant resources in data curation and interpretation by the technical staff involved in the maintenance of the centralized data collection resource such as CaNanoLab or Nanomaterial Registry. It would be far more efficient and useful if there were a standardized data collection and deposition template with standard key terms (such as Minimal Information about Nanomaterials, MIAN) that could be modified to add new or important additional data or parameters for each investigator. These new features cold be ultimately adopted in the classification scheme and guide the scope of the expanding database. This approach would be a win-win as it would enable structure for the investigators laboratory, consistency in data reporting and a means of transmitting data to the database in parallel to publication to eliminate the acquisition step from the process. In this talk we will outline our experience building Open Science Data Repository, a federated database system for direct acquisition, curation and management of research data, including nanomaterial data capture, transformation, and streamlined submission to nanomaterial knowledgebases. The key part of the system is microservices based architecture which exposes RESTful API suitable for direct integration into Workflow Management Systems as well as built-in modules facilitating and enforcing various lab-specific standard operating procedures.

What’s New in the Berkeley Data Analytics Stack

Turi, Inc.

Big Data at Golden Helix: Scaling to Meet the Demand of Clinical and Research...

Golden Helix Inc

AWS Customer Presentation- University of Maryland

Amazon Web Services

Computing Outside The Box June 2009

Keynote talk at the International Conference on Supercoming 2009, at IBM Yorktown in New York. This is a major update of a talk first given in New Zealand last January. The abstract follows. The past decade has seen increasingly ambitious and successful methods for outsourcing computing. Approaches such as utility computing, on-demand computing, grid computing, software as a service, and cloud computing all seek to free computer applications from the limiting confines of a single computer. Software that thus runs "outside the box" can be more powerful (think Google, TeraGrid), dynamic (think Animoto, caBIG), and collaborative (think FaceBook, myExperiment). It can also be cheaper, due to economies of scale in hardware and software. The combination of new functionality and new economics inspires new applications, reduces barriers to entry for application providers, and in general disrupts the computing ecosystem. I discuss the new applications that outside-the-box computing enables, in both business and science, and the hardware and software architectures that make these new applications possible.

Opportunities for X-Ray science in future computing architectures