Research Computing Storage at the Francis Crick Institute
•Download as PPTX, PDF•
1 like•449 views
In this deck from DDN User Group at SC18, Steve Hindmarsh from the Crick Institute presents: Research Computing Storage at the Francis Crick Institute. "The Francis Crick Institute is dedicated to understanding the fundamental biology underlying health and disease. Our work is helping to understand why disease develops and to translate discoveries into new ways to prevent, diagnose and treat illnesses such as cancer, heart disease, stroke, infections and neurodegenerative diseases. An independent organisation, our founding partners are the Medical Research Council (MRC), Cancer Research UK, Wellcome, UCL, Imperial College London and King's College London. The Crick was formed in 2015, and in 2016 we moved into our brand new state-of-the-art building in central London which brings together 1500 scientists and support staff working collaboratively across disciplines, making it the biggest biomedical research facility under a single roof in Europe." Learn more: https://www.crick.ac.uk/ and http://ddn.com Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Report
Share
Report
Share
Recommended
DA-JPL-final
This document discusses CERN's need for a big data analytics infrastructure to analyze the large volumes of data produced by its particle accelerators and detectors. It notes that CERN operates the largest particle accelerator in the world, the Large Hadron Collider (LHC), which produces terabytes of data per second. CERN aims to use big data analytics to optimize its systems, reduce faults, and increase availability. The desired infrastructure would provide both real-time and batch processing capabilities to analyze the heterogeneous experimental and operational data.
Visualising Research Graph using Neo4j and Gephi
This presentation discusses visualizing research graphs using Neo4j and Gephi. It introduces Research Graph, which connects research datasets, ORCID profiles, and grants to enable cross-platform discovery. The presentation demonstrates loading DSpace data into Neo4j, exploring the graph, and filtering the graph for export to Gephi for visualization. The goal is to automate connecting research information across platforms through authors, funding, and other relationships.
UKRepNet presentation at Pure UK User Group Meeting Dundee
UKRepNet presentation at Pure UK User Group Meeting DundeeeuroCRIS - Current Research Information Systems
The presentation provides a general UKRepNet project overview, plus a description of the highly fragmented landscape for CRIS/IR implementation at UK HEIs. Finally, a brief description is offered of the current work the project is jointly carrying out with the University of St Andrews and the Scottish Digital Library Consortium (SDLC) for enhancing the University's CRIS/IR system.HNSciCloud represented at HUAWEI CONNECT 2017 in Shanghai
HUAWEI CONNECT is an annual conference for the global ICT ecosystem, where industry and research leaders share ideas, explore technologies, and join forces for shared growth. The topic of 2017 conference was "Grow with the Cloud".
Jan van Eldik from CERN made a presentation to promote the achievements of this project and sowed how public procurers can drive innovation in cloud industry from the demand side.
Globus in European Life Science
This presentation was given at the 2019 GlobusWorld Conference in Chicago, IL by Steven Newhouse from European Bioinformatics Institute.
Virtualization for HPC at NCI
In this presentation from the Dell booth at SC13, Joseph Antony from NCI describes how they are using HPC Virtualization to meet user needs.
Watch the video presentation: http://insidehpc.com/2013/12/05/panel-discussion-thought-hpc-virtualization-never-going-happen/
EPCC MSc industry projects
EPCC is a supercomputing centre at the University of Edinburgh that has been self-funded for over 28 years. It has over 110 staff and £5 million in annual turnover. EPCC supports multi-disciplinary research through access to its high performance computing facilities, training courses, and collaborative projects. It houses various supercomputing systems totaling over 150,000 CPU cores for researchers to use. EPCC also works with over 1000 companies through technology transfer and industrial collaborations in areas like simulation, data processing, and cloud computing. One example is its partnership with Rolls-Royce on a £15 million virtual gas turbine engine simulation project.
A National Big Data Cyberinfrastructure Supporting Computational Biomedical R...
Invited Presentation
Symposium on Computational Biology and Bioinformatics:
Remembering John Wooley
National Institutes of Health
Bethesda, MD
July 29, 2016
Recommended
DA-JPL-final
This document discusses CERN's need for a big data analytics infrastructure to analyze the large volumes of data produced by its particle accelerators and detectors. It notes that CERN operates the largest particle accelerator in the world, the Large Hadron Collider (LHC), which produces terabytes of data per second. CERN aims to use big data analytics to optimize its systems, reduce faults, and increase availability. The desired infrastructure would provide both real-time and batch processing capabilities to analyze the heterogeneous experimental and operational data.
Visualising Research Graph using Neo4j and Gephi
This presentation discusses visualizing research graphs using Neo4j and Gephi. It introduces Research Graph, which connects research datasets, ORCID profiles, and grants to enable cross-platform discovery. The presentation demonstrates loading DSpace data into Neo4j, exploring the graph, and filtering the graph for export to Gephi for visualization. The goal is to automate connecting research information across platforms through authors, funding, and other relationships.
UKRepNet presentation at Pure UK User Group Meeting Dundee
UKRepNet presentation at Pure UK User Group Meeting DundeeeuroCRIS - Current Research Information Systems
The presentation provides a general UKRepNet project overview, plus a description of the highly fragmented landscape for CRIS/IR implementation at UK HEIs. Finally, a brief description is offered of the current work the project is jointly carrying out with the University of St Andrews and the Scottish Digital Library Consortium (SDLC) for enhancing the University's CRIS/IR system.HNSciCloud represented at HUAWEI CONNECT 2017 in Shanghai
HUAWEI CONNECT is an annual conference for the global ICT ecosystem, where industry and research leaders share ideas, explore technologies, and join forces for shared growth. The topic of 2017 conference was "Grow with the Cloud".
Jan van Eldik from CERN made a presentation to promote the achievements of this project and sowed how public procurers can drive innovation in cloud industry from the demand side.
Globus in European Life Science
This presentation was given at the 2019 GlobusWorld Conference in Chicago, IL by Steven Newhouse from European Bioinformatics Institute.
Virtualization for HPC at NCI
In this presentation from the Dell booth at SC13, Joseph Antony from NCI describes how they are using HPC Virtualization to meet user needs.
Watch the video presentation: http://insidehpc.com/2013/12/05/panel-discussion-thought-hpc-virtualization-never-going-happen/
EPCC MSc industry projects
EPCC is a supercomputing centre at the University of Edinburgh that has been self-funded for over 28 years. It has over 110 staff and £5 million in annual turnover. EPCC supports multi-disciplinary research through access to its high performance computing facilities, training courses, and collaborative projects. It houses various supercomputing systems totaling over 150,000 CPU cores for researchers to use. EPCC also works with over 1000 companies through technology transfer and industrial collaborations in areas like simulation, data processing, and cloud computing. One example is its partnership with Rolls-Royce on a £15 million virtual gas turbine engine simulation project.
A National Big Data Cyberinfrastructure Supporting Computational Biomedical R...
Invited Presentation
Symposium on Computational Biology and Bioinformatics:
Remembering John Wooley
National Institutes of Health
Bethesda, MD
July 29, 2016
Hybrid Cloud for CERN
CERN operates the Large Hadron Collider (LHC) and other particle physics experiments that generate enormous amounts of data. To handle this data, CERN uses a hybrid cloud model combining on-premise resources with public cloud services. CERN is participating in a joint procurement project called HNSciCloud to procure cloud services that can seamlessly integrate with CERN's internal resources and European research networks. The goal is to establish a hybrid cloud platform to meet the growing computing needs of particle physics and other research domains dealing with large datasets.
The Cambridge Research Computing Service
In this deck from the UK HPC Conference, Paul Calleja from the University of Cambridge presents: The Cambridge Research Computing Service.
"The Cambridge Centre for Data-Driven Discovery (C2D3) brings together researchers and expertise from across the academic departments and industry to drive research into the analysis, understanding and use of data science. The Centre is funded by a series of collaborations with partners in business and industry which have an interest in using data science for the benefit of their customers and their organisations. With unprecedented access to increasing volumes of data, our research ranges from the underlying fundamentals in mathematics and computer science, to data science applications across all six University Schools of Arts and Humanities, Biological Sciences, Clinical Medicine, Humanities and Social Sciences, Physical Sciences, and Technology. In parallel, our research addresses important issues around law, ethics and economics, in order to apply data science to solve challenging problems for society."
Watch the video: https://wp.me/p3RLHQ-kYb
Learn more: https://www.hpc.cam.ac.uk/
and
http://hpcadvisorycouncil.com/events/2019/uk-conference/agenda.php
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
CLIMB talk in the Virtual Laboratories session at the RCUK Cloud Working Grou...
Slides for talk given on the Cloud Infrastructure for Microbial Bioinformatics (CLIMB) for the RCUK Cloud Working Group Workshop at the Crick Institute on 8/1/18
Electron Microscopy Between OPIC, Oxford and eBIC
From Jisc's campus network engineering for data-intensive science workshop on 19 October 2016.
https://www.jisc.ac.uk/events/campus-network-engineering-for-data-intensive-science-workshop-19-oct-2016
Foundations for the Future of Science
Foundations for the future of science discusses using artificial intelligence and machine learning to advance scientific research. Key points discussed include using AI to analyze large datasets, develop scientific models, and automate experimental workflows. The document also outlines several examples of how the Globus data platform is currently enabling AI-powered scientific applications across multiple domains. Overall, the document advocates that embracing "AI for science" has the potential to accelerate scientific discovery by overcoming limitations in human analysis capabilities and computational resources.
Advancing Research at London's Global University
In this video from the DDN User Group Meeting at ISC'13, Dr. Daniel Hanlon from the University College of London presents: Advancing Research at London's Global University.
"As UCL's storage demands grow, the university expects to build a storage foundation that will scale up to 100PB. Looking for a storage solution that was massively scalable yet simple to manage as part of the first phase of the infrastructure build out, UCL will use DDN object storage technology to store up to 600TB of research data. DDN object storage capabilities also will be able to empower UCL researchers to collaborate without having to worry about data reliability, compliance obligations or long-term retention of critical research assets."
Learn more: http://www.ddn.com/press-releases/2013/ucl-selects-ddn-object-storage-for-cloud-infrastructure
Watch the presentation video: http://inside-bigdata.com/video-advancing-research-at-londons-global-university/
De gaynsrdec overview_11jan16
Dutch DeGay's presentation from the MIT Enterprise Forum of Cambridge Connected Soldier event on January 12, 2016
Working with Instrument Data (GlobusWorld Tour - UMich)
This document discusses using Globus to automate the management and analysis of large scientific instrument data. It provides examples of challenges with managing large datasets from the Event Horizon Telescope and applying Globus services and automation to help address these challenges. Specific use cases discussed include building connectomes from microscopy data and applying deep learning to flag bad scanning electron microscope images. The document emphasizes that automation needs transparency, results need to be easily findable, and leveraging specialized services can help.
Hybrid Cloud for CERN
Hybrid Cloud for CERN
Experience with Open Telecom Cloud and OpenStack
1) CERN uses a hybrid cloud approach combining on-premise resources with public clouds like the Open Telecom Cloud to address its increasing computing needs for projects like the LHC.
2) A pilot project using the Open Telecom Cloud was successful but identified some issues around networking and storage.
3) CERN is now participating in the HNSciCloud project to jointly procure innovative hybrid cloud services that fully integrate commercial clouds with in-house and European e-infrastructure resources.
Green Shoots: Research Data Management Pilot at Imperial College London
The document summarizes the results of a research data management (RDM) pilot project at Imperial College London. It describes how £100k in funding was provided for six academic projects to develop exemplars of best practices in RDM. The funded projects developed various tools and frameworks to improve data curation, sharing, and citation. Overall, the pilot demonstrated that innovative RDM is possible but also difficult and expensive to develop sustainably. It helped establish an initial RDM community at Imperial.
Big Data for Big Discoveries
Presentation on Big Data for Big Discoveries by Alberto Di Meglio, CERN openlab Head at HPC and Big Data 2016 in Central London
Data Automation at Light Sources
Plenary talk at the international Synchrotron Radiation Instrumentation conference in Taiwan, on work with great colleagues Ben Blaiszik, Ryan Chard, Logan Ward, and others.
Rapidly growing data volumes at light sources demand increasingly automated data collection, distribution, and analysis processes, in order to enable new scientific discoveries while not overwhelming finite human capabilities. I present here three projects that use cloud-hosted data automation and enrichment services, institutional computing resources, and high- performance computing facilities to provide cost-effective, scalable, and reliable implementations of such processes. In the first, Globus cloud-hosted data automation services are used to implement data capture, distribution, and analysis workflows for Advanced Photon Source and Advanced Light Source beamlines, leveraging institutional storage and computing. In the second, such services are combined with cloud-hosted data indexing and institutional storage to create a collaborative data publication, indexing, and discovery service, the Materials Data Facility (MDF), built to support a host of informatics applications in materials science. The third integrates components of the previous two projects with machine learning capabilities provided by the Data and Learning Hub for science (DLHub) to enable on-demand access to machine learning models from light source data capture and analysis workflows, and provides simplified interfaces to train new models on data from sources such as MDF on leadership scale computing resources. I draw conclusions about best practices for building next-generation data automation systems for future light sources.
Escaping Flatland: Interactive High-Dimensional Data Analysis in Drug Discove...
This document describes a project at Novartis to use Apache Spark for high-dimensional data analysis from drug screening. Large datasets from various screening technologies were analyzed using Spark pipelines for quality control, normalization, and classification. Visualizations were built using WebGL. The goals were to speed up multi-day batch jobs, create a unified analysis workflow, and build an application for scientists. Future work includes elastic infrastructure, supervised learning of cell phenotypes, and contributing methods to open source.
Big Data Europe SC6 WS 3: Ron Dekker, Director CESSDA European Open Science A...
Slides for keynote talk at the Big Data Europe workshop nr 3 on 11.9.2017 in Amsterdam co-located with SEMANTiCS2017 conference by Ron Dekker, Director CESSDA: European Open Science Agenda: where we are and where we are going?
ELIXIR and data grand challenges in life sciences
This document discusses challenges in life sciences data management and services provided by ELIXIR to address these challenges. ELIXIR aims to facilitate life sciences research by building a sustainable infrastructure for biological data in Europe. It coordinates several nodes across member states that provide specialized data services. ELIXIR is also running pilot projects to test integration of services, including providing cloud access to reference data and distributed authentication and access to clinical archives. Future challenges include sustaining funding and scaling to handle exponentially growing data volumes.
PLNOG 18 - Dr Marek Michalewicz - InfiniCortex: Superkomputer wielki jak świat
YouTube: https://www.youtube.com/watch?v=MsXhC-Kb7SI&index=47&list=PLnKL6-WWWE_VNp6tUznu7Ca8hBF8yjKj2
The Pacific Research Platform:a Science-Driven Big-Data Freeway System
The Pacific Research Platform will create a regional "Big Data Freeway System" along the West Coast to support science. It will connect major research institutions with high-speed optical networks, allowing them to share vast amounts of data and computational resources. This will enable new forms of collaborative, data-intensive research for fields like particle physics, astronomy, biomedicine, and earth sciences. The first phase aims to establish a basic networked infrastructure, with later phases advancing capabilities to 100Gbps and beyond with security and distributed technologies.
Climb stateoftheartintro
Presentation/outline introducing the CLIMB system presented at the NERC Cloud Computing Workshop at the CLIMB facility in Warwick, 4th June 2015
Shared services - the future of HPC and big data facilities for UK research
Slides from Jisc panel session at HPC & Big Data 2016 with contributions from the Francis Crick Institute, QMUL and King's College London covering their use of the Jisc shared data centre and the eMedLab project
2015 04 bio it world
The document discusses updates from the Bio-IT World Expo in April 2015 regarding research computing at the Broad Institute. It provides three key takeaways: 1) Federating environments to public clouds will become standard practice; 2) Understanding data lifecycles and storing non-latency sensitive data in object stores is important; 3) Issues of consent, privacy, and data ownership remain challenging and require collaboration on standards. It then provides context about the Broad Institute and an overview of their high performance computing environment and initiatives to incorporate private and public clouds.
Major Market Shifts in IT
The document discusses the top 5 technologies that all organizations must understand: digital transformation, quantum computing, IoT, 5G, and AI/HPC. It provides an overview of each technology including opportunities and threats to organizations. The document emphasizes that understanding these emerging technologies is mandatory as the information revolution changes many aspects of life and business.
Preparing to program Aurora at Exascale - Early experiences and future direct...
In this deck from IWOCL / SYCLcon 2020, Hal Finkel from Argonne National Laboratory presents: Preparing to program Aurora at Exascale - Early experiences and future directions.
"Argonne National Laboratory’s Leadership Computing Facility will be home to Aurora, our first exascale supercomputer. Aurora promises to take scientific computing to a whole new level, and scientists and engineers from many different fields will take advantage of Aurora’s unprecedented computational capabilities to push the boundaries of human knowledge. In addition, Aurora’s support for advanced machine-learning and big-data computations will enable scientific workflows incorporating these techniques along with traditional HPC algorithms. Programming the state-of-the-art hardware in Aurora will be accomplished using state-of-the-art programming models. Some of these models, such as OpenMP, are long-established in the HPC ecosystem. Other models, such as Intel’s oneAPI, based on SYCL, are relatively-new models constructed with the benefit of significant experience. Many applications will not use these models directly, but rather, will use C++ abstraction libraries such as Kokkos or RAJA. Python will also be a common entry point to high-performance capabilities. As we look toward the future, features in the C++ standard itself will become increasingly relevant for accessing the extreme parallelism of exascale platforms.
This presentation will summarize the experiences of our team as we prepare for Aurora, exploring how to port applications to Aurora’s architecture and programming models, and distilling the challenges and best practices we’ve developed to date. oneAPI/SYCL and OpenMP are both critical models in these efforts, and while the ecosystem for Aurora has yet to mature, we’ve already had a great deal of success. Importantly, we are not passive recipients of programming models developed by others. Our team works not only with vendor-provided compilers and tools, but also develops improved open-source LLVM-based technologies that feed both open-source and vendor-provided capabilities. In addition, we actively participate in the standardization of OpenMP, SYCL, and C++. To conclude, I’ll share our thoughts on how these models can best develop in the future to support exascale-class systems."
Watch the video: https://wp.me/p3RLHQ-lPT
Learn more: https://www.iwocl.org/iwocl-2020/conference-program/
and
https://www.anl.gov/topic/aurora
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
More Related Content
Similar to Research Computing Storage at the Francis Crick Institute
Hybrid Cloud for CERN
CERN operates the Large Hadron Collider (LHC) and other particle physics experiments that generate enormous amounts of data. To handle this data, CERN uses a hybrid cloud model combining on-premise resources with public cloud services. CERN is participating in a joint procurement project called HNSciCloud to procure cloud services that can seamlessly integrate with CERN's internal resources and European research networks. The goal is to establish a hybrid cloud platform to meet the growing computing needs of particle physics and other research domains dealing with large datasets.
The Cambridge Research Computing Service
In this deck from the UK HPC Conference, Paul Calleja from the University of Cambridge presents: The Cambridge Research Computing Service.
"The Cambridge Centre for Data-Driven Discovery (C2D3) brings together researchers and expertise from across the academic departments and industry to drive research into the analysis, understanding and use of data science. The Centre is funded by a series of collaborations with partners in business and industry which have an interest in using data science for the benefit of their customers and their organisations. With unprecedented access to increasing volumes of data, our research ranges from the underlying fundamentals in mathematics and computer science, to data science applications across all six University Schools of Arts and Humanities, Biological Sciences, Clinical Medicine, Humanities and Social Sciences, Physical Sciences, and Technology. In parallel, our research addresses important issues around law, ethics and economics, in order to apply data science to solve challenging problems for society."
Watch the video: https://wp.me/p3RLHQ-kYb
Learn more: https://www.hpc.cam.ac.uk/
and
http://hpcadvisorycouncil.com/events/2019/uk-conference/agenda.php
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
CLIMB talk in the Virtual Laboratories session at the RCUK Cloud Working Grou...
Slides for talk given on the Cloud Infrastructure for Microbial Bioinformatics (CLIMB) for the RCUK Cloud Working Group Workshop at the Crick Institute on 8/1/18
Electron Microscopy Between OPIC, Oxford and eBIC
From Jisc's campus network engineering for data-intensive science workshop on 19 October 2016.
https://www.jisc.ac.uk/events/campus-network-engineering-for-data-intensive-science-workshop-19-oct-2016
Foundations for the Future of Science
Foundations for the future of science discusses using artificial intelligence and machine learning to advance scientific research. Key points discussed include using AI to analyze large datasets, develop scientific models, and automate experimental workflows. The document also outlines several examples of how the Globus data platform is currently enabling AI-powered scientific applications across multiple domains. Overall, the document advocates that embracing "AI for science" has the potential to accelerate scientific discovery by overcoming limitations in human analysis capabilities and computational resources.
Advancing Research at London's Global University
In this video from the DDN User Group Meeting at ISC'13, Dr. Daniel Hanlon from the University College of London presents: Advancing Research at London's Global University.
"As UCL's storage demands grow, the university expects to build a storage foundation that will scale up to 100PB. Looking for a storage solution that was massively scalable yet simple to manage as part of the first phase of the infrastructure build out, UCL will use DDN object storage technology to store up to 600TB of research data. DDN object storage capabilities also will be able to empower UCL researchers to collaborate without having to worry about data reliability, compliance obligations or long-term retention of critical research assets."
Learn more: http://www.ddn.com/press-releases/2013/ucl-selects-ddn-object-storage-for-cloud-infrastructure
Watch the presentation video: http://inside-bigdata.com/video-advancing-research-at-londons-global-university/
De gaynsrdec overview_11jan16
Dutch DeGay's presentation from the MIT Enterprise Forum of Cambridge Connected Soldier event on January 12, 2016
Working with Instrument Data (GlobusWorld Tour - UMich)
This document discusses using Globus to automate the management and analysis of large scientific instrument data. It provides examples of challenges with managing large datasets from the Event Horizon Telescope and applying Globus services and automation to help address these challenges. Specific use cases discussed include building connectomes from microscopy data and applying deep learning to flag bad scanning electron microscope images. The document emphasizes that automation needs transparency, results need to be easily findable, and leveraging specialized services can help.
Hybrid Cloud for CERN
Hybrid Cloud for CERN
Experience with Open Telecom Cloud and OpenStack
1) CERN uses a hybrid cloud approach combining on-premise resources with public clouds like the Open Telecom Cloud to address its increasing computing needs for projects like the LHC.
2) A pilot project using the Open Telecom Cloud was successful but identified some issues around networking and storage.
3) CERN is now participating in the HNSciCloud project to jointly procure innovative hybrid cloud services that fully integrate commercial clouds with in-house and European e-infrastructure resources.
Green Shoots: Research Data Management Pilot at Imperial College London
The document summarizes the results of a research data management (RDM) pilot project at Imperial College London. It describes how £100k in funding was provided for six academic projects to develop exemplars of best practices in RDM. The funded projects developed various tools and frameworks to improve data curation, sharing, and citation. Overall, the pilot demonstrated that innovative RDM is possible but also difficult and expensive to develop sustainably. It helped establish an initial RDM community at Imperial.
Big Data for Big Discoveries
Presentation on Big Data for Big Discoveries by Alberto Di Meglio, CERN openlab Head at HPC and Big Data 2016 in Central London
Data Automation at Light Sources
Plenary talk at the international Synchrotron Radiation Instrumentation conference in Taiwan, on work with great colleagues Ben Blaiszik, Ryan Chard, Logan Ward, and others.
Rapidly growing data volumes at light sources demand increasingly automated data collection, distribution, and analysis processes, in order to enable new scientific discoveries while not overwhelming finite human capabilities. I present here three projects that use cloud-hosted data automation and enrichment services, institutional computing resources, and high- performance computing facilities to provide cost-effective, scalable, and reliable implementations of such processes. In the first, Globus cloud-hosted data automation services are used to implement data capture, distribution, and analysis workflows for Advanced Photon Source and Advanced Light Source beamlines, leveraging institutional storage and computing. In the second, such services are combined with cloud-hosted data indexing and institutional storage to create a collaborative data publication, indexing, and discovery service, the Materials Data Facility (MDF), built to support a host of informatics applications in materials science. The third integrates components of the previous two projects with machine learning capabilities provided by the Data and Learning Hub for science (DLHub) to enable on-demand access to machine learning models from light source data capture and analysis workflows, and provides simplified interfaces to train new models on data from sources such as MDF on leadership scale computing resources. I draw conclusions about best practices for building next-generation data automation systems for future light sources.
Escaping Flatland: Interactive High-Dimensional Data Analysis in Drug Discove...
This document describes a project at Novartis to use Apache Spark for high-dimensional data analysis from drug screening. Large datasets from various screening technologies were analyzed using Spark pipelines for quality control, normalization, and classification. Visualizations were built using WebGL. The goals were to speed up multi-day batch jobs, create a unified analysis workflow, and build an application for scientists. Future work includes elastic infrastructure, supervised learning of cell phenotypes, and contributing methods to open source.
Big Data Europe SC6 WS 3: Ron Dekker, Director CESSDA European Open Science A...
Slides for keynote talk at the Big Data Europe workshop nr 3 on 11.9.2017 in Amsterdam co-located with SEMANTiCS2017 conference by Ron Dekker, Director CESSDA: European Open Science Agenda: where we are and where we are going?
ELIXIR and data grand challenges in life sciences
This document discusses challenges in life sciences data management and services provided by ELIXIR to address these challenges. ELIXIR aims to facilitate life sciences research by building a sustainable infrastructure for biological data in Europe. It coordinates several nodes across member states that provide specialized data services. ELIXIR is also running pilot projects to test integration of services, including providing cloud access to reference data and distributed authentication and access to clinical archives. Future challenges include sustaining funding and scaling to handle exponentially growing data volumes.
PLNOG 18 - Dr Marek Michalewicz - InfiniCortex: Superkomputer wielki jak świat
YouTube: https://www.youtube.com/watch?v=MsXhC-Kb7SI&index=47&list=PLnKL6-WWWE_VNp6tUznu7Ca8hBF8yjKj2
The Pacific Research Platform:a Science-Driven Big-Data Freeway System
The Pacific Research Platform will create a regional "Big Data Freeway System" along the West Coast to support science. It will connect major research institutions with high-speed optical networks, allowing them to share vast amounts of data and computational resources. This will enable new forms of collaborative, data-intensive research for fields like particle physics, astronomy, biomedicine, and earth sciences. The first phase aims to establish a basic networked infrastructure, with later phases advancing capabilities to 100Gbps and beyond with security and distributed technologies.
Climb stateoftheartintro
Presentation/outline introducing the CLIMB system presented at the NERC Cloud Computing Workshop at the CLIMB facility in Warwick, 4th June 2015
Shared services - the future of HPC and big data facilities for UK research
Slides from Jisc panel session at HPC & Big Data 2016 with contributions from the Francis Crick Institute, QMUL and King's College London covering their use of the Jisc shared data centre and the eMedLab project
2015 04 bio it world
The document discusses updates from the Bio-IT World Expo in April 2015 regarding research computing at the Broad Institute. It provides three key takeaways: 1) Federating environments to public clouds will become standard practice; 2) Understanding data lifecycles and storing non-latency sensitive data in object stores is important; 3) Issues of consent, privacy, and data ownership remain challenging and require collaboration on standards. It then provides context about the Broad Institute and an overview of their high performance computing environment and initiatives to incorporate private and public clouds.
Similar to Research Computing Storage at the Francis Crick Institute (20)
CLIMB talk in the Virtual Laboratories session at the RCUK Cloud Working Grou...
CLIMB talk in the Virtual Laboratories session at the RCUK Cloud Working Grou...
Working with Instrument Data (GlobusWorld Tour - UMich)
Working with Instrument Data (GlobusWorld Tour - UMich)
Green Shoots: Research Data Management Pilot at Imperial College London
Green Shoots: Research Data Management Pilot at Imperial College London
Escaping Flatland: Interactive High-Dimensional Data Analysis in Drug Discove...
Escaping Flatland: Interactive High-Dimensional Data Analysis in Drug Discove...
Big Data Europe SC6 WS 3: Ron Dekker, Director CESSDA European Open Science A...
Big Data Europe SC6 WS 3: Ron Dekker, Director CESSDA European Open Science A...
PLNOG 18 - Dr Marek Michalewicz - InfiniCortex: Superkomputer wielki jak świat
PLNOG 18 - Dr Marek Michalewicz - InfiniCortex: Superkomputer wielki jak świat
The Pacific Research Platform:a Science-Driven Big-Data Freeway System
The Pacific Research Platform:a Science-Driven Big-Data Freeway System
Shared services - the future of HPC and big data facilities for UK research
Shared services - the future of HPC and big data facilities for UK research
More from inside-BigData.com
Major Market Shifts in IT
The document discusses the top 5 technologies that all organizations must understand: digital transformation, quantum computing, IoT, 5G, and AI/HPC. It provides an overview of each technology including opportunities and threats to organizations. The document emphasizes that understanding these emerging technologies is mandatory as the information revolution changes many aspects of life and business.
Preparing to program Aurora at Exascale - Early experiences and future direct...
In this deck from IWOCL / SYCLcon 2020, Hal Finkel from Argonne National Laboratory presents: Preparing to program Aurora at Exascale - Early experiences and future directions.
"Argonne National Laboratory’s Leadership Computing Facility will be home to Aurora, our first exascale supercomputer. Aurora promises to take scientific computing to a whole new level, and scientists and engineers from many different fields will take advantage of Aurora’s unprecedented computational capabilities to push the boundaries of human knowledge. In addition, Aurora’s support for advanced machine-learning and big-data computations will enable scientific workflows incorporating these techniques along with traditional HPC algorithms. Programming the state-of-the-art hardware in Aurora will be accomplished using state-of-the-art programming models. Some of these models, such as OpenMP, are long-established in the HPC ecosystem. Other models, such as Intel’s oneAPI, based on SYCL, are relatively-new models constructed with the benefit of significant experience. Many applications will not use these models directly, but rather, will use C++ abstraction libraries such as Kokkos or RAJA. Python will also be a common entry point to high-performance capabilities. As we look toward the future, features in the C++ standard itself will become increasingly relevant for accessing the extreme parallelism of exascale platforms.
This presentation will summarize the experiences of our team as we prepare for Aurora, exploring how to port applications to Aurora’s architecture and programming models, and distilling the challenges and best practices we’ve developed to date. oneAPI/SYCL and OpenMP are both critical models in these efforts, and while the ecosystem for Aurora has yet to mature, we’ve already had a great deal of success. Importantly, we are not passive recipients of programming models developed by others. Our team works not only with vendor-provided compilers and tools, but also develops improved open-source LLVM-based technologies that feed both open-source and vendor-provided capabilities. In addition, we actively participate in the standardization of OpenMP, SYCL, and C++. To conclude, I’ll share our thoughts on how these models can best develop in the future to support exascale-class systems."
Watch the video: https://wp.me/p3RLHQ-lPT
Learn more: https://www.iwocl.org/iwocl-2020/conference-program/
and
https://www.anl.gov/topic/aurora
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Transforming Private 5G Networks
In this deck, Greg Wahl from Advantech presents: Transforming Private 5G Networks.
Advantech Networks & Communications Group is driving innovation in next-generation network solutions with their High Performance Servers. We provide business critical hardware to the world's leading telecom and networking equipment manufacturers with both standard and customized products. Our High Performance Servers are highly configurable platforms designed to balance the best in x86 server-class processing performance with maximum I/O and offload density. The systems are cost effective, highly available and optimized to meet next generation networking and media processing needs.
“Advantech’s Networks and Communication Group has been both an innovator and trusted enabling partner in the telecommunications and network security markets for over a decade, designing and manufacturing products for OEMs that accelerate their network platform evolution and time to market.” Said Advantech Vice President of Networks & Communications Group, Ween Niu. “In the new IP Infrastructure era, we will be expanding our expertise in Software Defined Networking (SDN) and Network Function Virtualization (NFV), two of the essential conduits to 5G infrastructure agility making networks easier to install, secure, automate and manage in a cloud-based infrastructure.”
In addition to innovation in air interface technologies and architecture extensions, 5G will also need a new generation of network computing platforms to run the emerging software defined infrastructure, one that provides greater topology flexibility, essential to deliver on the promises of high availability, high coverage, low latency and high bandwidth connections. This will open up new parallel industry opportunities through dedicated 5G network slices reserved for specific industries dedicated to video traffic, augmented reality, IoT, connected cars etc. 5G unlocks many new doors and one of the keys to its enablement lies in the elasticity and flexibility of the underlying infrastructure.
Advantech’s corporate vision is to enable an intelligent planet. The company is a global leader in the fields of IoT intelligent systems and embedded platforms. To embrace the trends of IoT, big data, and artificial intelligence, Advantech promotes IoT hardware and software solutions with the Edge Intelligence WISE-PaaS core to assist business partners and clients in connecting their industrial chains. Advantech is also working with business partners to co-create business ecosystems that accelerate the goal of industrial intelligence."
Watch the video: https://wp.me/p3RLHQ-lPQ
* Company website: https://www.advantech.com/
* Solution page: https://www2.advantech.com/nc/newsletter/NCG/SKY/benefits.html
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
The Incorporation of Machine Learning into Scientific Simulations at Lawrence...
In this deck from the Stanford HPC Conference, Katie Lewis from Lawrence Livermore National Laboratory presents: The Incorporation of Machine Learning into Scientific Simulations at Lawrence Livermore National Laboratory.
"Scientific simulations have driven computing at Lawrence Livermore National Laboratory (LLNL) for decades. During that time, we have seen significant changes in hardware, tools, and algorithms. Today, data science, including machine learning, is one of the fastest growing areas of computing, and LLNL is investing in hardware, applications, and algorithms in this space. While the use of simulations to focus and understand experiments is well accepted in our community, machine learning brings new challenges that need to be addressed. I will explore applications for machine learning in scientific simulations that are showing promising results and further investigation that is needed to better understand its usefulness."
Watch the video: https://youtu.be/NVwmvCWpZ6Y
Learn more: https://computing.llnl.gov/research-area/machine-learning
and
http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
How to Achieve High-Performance, Scalable and Distributed DNN Training on Mod...
In this deck from the Stanford HPC Conference, DK Panda from Ohio State University presents: How to Achieve High-Performance, Scalable and Distributed DNN Training on Modern HPC Systems?
"This talk will start with an overview of challenges being faced by the AI community to achieve high-performance, scalable and distributed DNN training on Modern HPC systems with both scale-up and scale-out strategies. After that, the talk will focus on a range of solutions being carried out in my group to address these challenges. The solutions will include: 1) MPI-driven Deep Learning, 2) Co-designing Deep Learning Stacks with High-Performance MPI, 3) Out-of- core DNN training, and 4) Hybrid (Data and Model) parallelism. Case studies to accelerate DNN training with popular frameworks like TensorFlow, PyTorch, MXNet and Caffe on modern HPC systems will be presented."
Watch the video: https://youtu.be/LeUNoKZVuwQ
Learn more: http://web.cse.ohio-state.edu/~panda.2/
and
http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze ...
In this deck from the Stanford HPC Conference, Nick Nystrom and Paola Buitrago provide an update from the Pittsburgh Supercomputing Center.
Nick Nystrom is Chief Scientist at the Pittsburgh Supercomputing Center (PSC). Nick is architect and PI for Bridges, PSC's flagship system that successfully pioneered the convergence of HPC, AI, and Big Data. He is also PI for the NIH Human Biomolecular Atlas Program’s HIVE Infrastructure Component and co-PI for projects that bring emerging AI technologies to research (Open Compass), apply machine learning to biomedical data for breast and lung cancer (Big Data for Better Health), and identify causal relationships in biomedical big data (the Center for Causal Discovery, an NIH Big Data to Knowledge Center of Excellence). His current research interests include hardware and software architecture, applications of machine learning to multimodal data (particularly for the life sciences) and to enhance simulation, and graph analytics.
Watch the video: https://youtu.be/LWEU1L1o7yY
Learn more: https://www.psc.edu/
and
http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
HPC Impact: EDA Telemetry Neural Networks
The document discusses using systems intelligence and artificial intelligence/neural networks to enhance semiconductor electronic design automation (EDA) workflows by collecting telemetry data from EDA jobs and infrastructure and analyzing it using complex event processing, machine learning models, and messaging substrates to provide insights that could optimize EDA pipelines and infrastructure. The approach aims to allow both internal and external augmentation of EDA processes and environments through unsupervised and incremental learning.
Biohybrid Robotic Jellyfish for Future Applications in Ocean Monitoring
In this deck from the Stanford HPC Conference, Nicole Xu from Stanford University describes how she transformed a common jellyfish into a bionic creature that is part animal and part machine.
"Animal locomotion and bioinspiration have the potential to expand the performance capabilities of robots, but current implementations are limited. Mechanical soft robots leverage engineered materials and are highly controllable, but these biomimetic robots consume more power than corresponding animal counterparts. Biological soft robots from a bottom-up approach offer advantages such as speed and controllability but are limited to survival in cell media. Instead, biohybrid robots that comprise live animals and self- contained microelectronic systems leverage the animals’ own metabolism to reduce power constraints and body as an natural scaffold with damage tolerance. We demonstrate that by integrating onboard microelectronics into live jellyfish, we can enhance propulsion up to threefold, using only 10 mW of external power input to the microelectronics and at only a twofold increase in cost of transport to the animal. This robotic system uses 10 to 1000 times less external power per mass than existing swimming robots in literature and can be used in future applications for ocean monitoring to track environmental changes."
Watch the video: https://youtu.be/HrmJFyvInj8
Learn more: https://sanfrancisco.cbslocal.com/2020/02/05/stanford-research-project-common-jellyfish-bionic-sea-creatures/
and
http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Machine Learning for Weather Forecasts
In this deck from the Stanford HPC Conference, Peter Dueben from the European Centre for Medium-Range Weather Forecasts (ECMWF) presents: Machine Learning for Weather Forecasts.
"I will present recent studies that use deep learning to learn the equations of motion of the atmosphere, to emulate model components of weather forecast models and to enhance usability of weather forecasts. I will than talk about the main challenges for the application of deep learning in cutting-edge weather forecasts and suggest approaches to improve usability in the future."
Peter is contributing to the development and optimization of weather and climate models for modern supercomputers. He is focusing on a better understanding of model error and model uncertainty, on the use of reduced numerical precision that is optimised for a given level of model error, on global cloud- resolving simulations with ECMWF's forecast model, and the use of machine learning, and in particular deep learning, to improve the workflow and predictions. Peter has graduated in Physics and wrote his PhD thesis at the Max Planck Institute for Meteorology in Germany. He worked as Postdoc with Tim Palmer at the University of Oxford and has taken up a position as University Research Fellow of the Royal Society at the European Centre for Medium-Range Weather Forecasts (ECMWF) in 2017.
Watch the video: https://youtu.be/ks3fkRj8Iqc
Learn more: https://www.ecmwf.int/
and
http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
HPC AI Advisory Council Update
In this deck, Gilad Shainer from the HPC AI Advisory Council describes how this organization fosters innovation in the high performance computing community.
"The HPC-AI Advisory Council’s mission is to bridge the gap between high-performance computing (HPC) and Artificial Intelligence (AI) use and its potential, bring the beneficial capabilities of HPC and AI to new users for better research, education, innovation and product manufacturing, bring users the expertise needed to operate HPC and AI systems, provide application designers with the tools needed to enable parallel computing, and to strengthen the qualification and integration of HPC and AI system products."
Watch the video: https://wp.me/p3RLHQ-lNz
Learn more: http://hpcadvisorycouncil.com
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Fugaku Supercomputer joins fight against COVID-19
Today RIKEN in Japan announced that the Fugaku supercomputer will be made available for research projects aimed to combat COVID-19.
"Fugaku is currently being installed and is scheduled to be available to the public in 2021. However, faced with the devastating disaster unfolding before our eyes, RIKEN and MEXT decided to make a portion of the computational resources of Fugaku available for COVID-19-related projects ahead of schedule while continuing the installation process.
Fugaku is being developed not only for the progress in science, but also to help build the society dubbed as the “Society 5.0” by the Japanese government, where all people will live safe and comfortable lives. The current initiative to fight against the novel coronavirus is driven by the philosophy behind the development of Fugaku."
Initial Projects
Exploring new drug candidates for COVID-19 by "Fugaku"
Yasushi Okuno, RIKEN / Kyoto University
Prediction of conformational dynamics of proteins on the surface of SARS-Cov-2 using Fugaku
Yuji Sugita, RIKEN
Simulation analysis of pandemic phenomena
Nobuyasu Ito, RIKEN
Fragment molecular orbital calculations for COVID-19 proteins
Yuji Mochizuki, Rikkyo University
Energy Efficient Computing using Dynamic Tuning
In this deck from the Performance Optimisation and Productivity group, Lubomir Riha from IT4Innovations presents: Energy Efficient Computing using Dynamic Tuning.
"We now live in a world of power-constrained architectures and systems and power consumption represents a significant cost factor in the overall HPC system economy. For these reasons, in recent years researchers, supercomputing centers and major vendors have developed new tools and methodologies to measure and optimize the energy consumption of large-scale high performance system installations. Due to the link between energy consumption, power consumption and execution time of an application executed by the final user, it is important for these tools and the methodology used to consider all these aspects, empowering the final user and the system administrator with the capability of finding the best configuration given different high level objectives.
This webinar focused on tools designed to improve the energy-efficiency of HPC applications using a methodology of dynamic tuning of HPC applications, developed under the H2020 READEX project. The READEX methodology has been designed for exploiting the dynamic behaviour of software. At design time, different runtime situations (RTS) are detected and optimized system configurations are determined. RTSs with the same configuration are grouped into scenarios, forming the tuning model. At runtime, the tuning model is used to switch system configurations dynamically.
The MERIC tool, that implements the READEX methodology, is presented. It supports manual or binary instrumentation of the analysed applications to simplify the analysis. This instrumentation is used to identify and annotate the significant regions in the HPC application. Automatic binary instrumentation annotates regions with significant runtime. Manual instrumentation, which can be combined with automatic, allows code developer to annotate regions of particular interest."
Watch the video: https://wp.me/p3RLHQ-lJP
Learn more: https://pop-coe.eu/blog/14th-pop-webinar-energy-efficient-computing-using-dynamic-tuning
and
https://code.it4i.cz/vys0053/meric
Sign up for our insideHPC Newsletter: http://insidehpc.com/newslett
HPC at Scale Enabled by DDN A3i and NVIDIA SuperPOD
The document discusses how DDN A3I storage solutions and Nvidia's SuperPOD platform can enable HPC at scale. It provides details on DDN's A3I appliances that are optimized for AI and deep learning workloads and validated for Nvidia's DGX-2 SuperPOD reference architecture. The solutions are said to deliver the fastest performance, effortless scaling, reliability and flexibility for data-intensive workloads.
State of ARM-based HPC
In this deck, Paul Isaacs from Linaro presents: State of ARM-based HPC. This talk provides an overview of applications and infrastructure services successfully ported to Aarch64 and benefiting from scale.
"With its debut on the TOP500, the 125,000-core Astra supercomputer at New Mexico’s Sandia Labs uses Cavium ThunderX2 chips to mark Arm’s entry into the petascale world. In Japan, the Fujitsu A64FX Arm-based CPU in the pending Fugaku supercomputer has been optimized to achieve high-level, real-world application performance, anticipating up to one hundred times the application execution performance of the K computer. K was the first computer to top 10 petaflops in 2011."
Watch the video: https://wp.me/p3RLHQ-lIT
Learn more: https://www.linaro.org/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Versal Premium ACAP for Network and Cloud Acceleration
Today Xilinx announced Versal Premium, the third series in the Versal ACAP portfolio. The Versal Premium series features highly integrated, networked and power-optimized cores and the industry’s highest bandwidth and compute density on an adaptable platform. Versal Premium is designed for the highest bandwidth networks operating in thermally and spatially constrained environments, as well as for cloud providers who need scalable, adaptable application acceleration.
Versal is the industry’s first adaptive compute acceleration platform (ACAP), a revolutionary new category of heterogeneous compute devices with capabilities that far exceed those of conventional silicon architectures. Developed on TSMC’s 7-nanometer process technology, Versal Premium combines software programmability with dynamically configurable hardware acceleration and pre-engineered connectivity and security features to enable a faster time-to- market. The Versal Premium series delivers up to 3X higher throughput compared to current generation FPGAs, with built-in Ethernet, Interlaken, and cryptographic engines that enable fast and secure networks. The series doubles the compute density of currently deployed mainstream FPGAs and provides the adaptability to keep pace with increasingly diverse and evolving cloud and networking workloads.
Learn more: https://insidehpc.com/2020/03/xilinx-announces-versal-premium-acap-for-network-and-cloud-acceleration/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Zettar: Moving Massive Amounts of Data across Any Distance Efficiently
In this video from the Rice Oil & Gas Conference, Chin Fang from Zettar presents: Moving Massive Amounts of Data across Any Distance Efficiently.
The objective of this talk is to present two on-going projects aiming at improving and ensuring highly efficient bulk transferring or streaming of massive amounts of data over digital connections across any distance. It examines the current state of the art, a few very common misconceptions, the differences among the three major type of data movement solutions, a current initiative attempting to improve the data movement efficiency from the ground up, and another multi-stage project that shows how to conduct long distance large scale data movement at speed and scale internationally. Both projects have real world motivations, e.g. the ambitious data transfer requirements of Linac Coherent Light Source II (LCLS-II) [1], a premier preparation project of the U.S. DOE Exascale Computing Initiative (ECI) [2]. Their immediate goals are described and explained, together with the solution used for each. Findings and early results are reported. Possible future works are outlined.
Watch the video: https://wp.me/p3RLHQ-lBX
Learn more: https://www.zettar.com/
and
https://rice2020oghpc.rice.edu/program-2/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Scaling TCO in a Post Moore's Era
In this deck from the Rice Oil & Gas Conference, Bradley McCredie from AMD presents: Scaling TCO in a Post Moore's Law Era.
"While foundries bravely drive forward to overcome the technical and economic challenges posed by scaling to 5nm and beyond, Moore’s law alone can provide only a fraction of the performance / watt and performance / dollar gains needed to satisfy the demands of today’s high performance computing and artificial intelligence applications. To close the gap, multiple strategies are required. First, new levels of innovation and design efficiency will supplement technology gains to continue to deliver meaningful improvements in SoC performance. Second, heterogenous compute architectures will create x-factor increases of performance efficiency for the most critical applications. Finally, open software frameworks, APIs, and toolsets will enable broad ecosystems of application level innovation."
Watch the video:
Learn more: http://amd.com
and
https://rice2020oghpc.rice.edu/program-2/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
CUDA-Python and RAPIDS for blazing fast scientific computing
In this deck from the ECSS Symposium, Abe Stern from NVIDIA presents: CUDA-Python and RAPIDS for blazing fast scientific computing.
"We will introduce Numba and RAPIDS for GPU programming in Python. Numba allows us to write just-in-time compiled CUDA code in Python, giving us easy access to the power of GPUs from a powerful high-level language. RAPIDS is a suite of tools with a Python interface for machine learning and dataframe operations. Together, Numba and RAPIDS represent a potent set of tools for rapid prototyping, development, and analysis for scientific computing. We will cover the basics of each library and go over simple examples to get users started. Finally, we will briefly highlight several other relevant libraries for GPU programming."
Watch the video: https://wp.me/p3RLHQ-lvu
Learn more: https://developer.nvidia.com/rapids
and
https://www.xsede.org/for-users/ecss/ecss-symposium
Sign up for our insideHPC Newsletter: http://insidehp.com/newsletter
Introducing HPC with a Raspberry Pi Cluster
In this deck from FOSDEM 2020, Colin Sauze from Aberystwyth University describes the development of a RaspberryPi cluster for teaching an introduction to HPC.
"The motivation for this was to overcome four key problems faced by new HPC users:
* The availability of a real HPC system and the effect running training courses can have on the real system, conversely the availability of spare resources on the real system can cause problems for the training course.
* A fear of using a large and expensive HPC system for the first time and worries that doing something wrong might damage the system.
* That HPC systems are very abstract systems sitting in data centres that users never see, it is difficult for them to understand exactly what it is they are using.
* That new users fail to understand resource limitations, in part because of the vast resources in modern HPC systems a lot of mistakes can be made before running out of resources. A more resource constrained system makes it easier to understand this.
The talk will also discuss some of the technical challenges in deploying an HPC environment to a Raspberry Pi and attempts to keep that environment as close to a "real" HPC as possible. The issue to trying to automate the installation process will also be covered."
Learn more: https://github.com/colinsauze/pi_cluster
and
https://fosdem.org/2020/schedule/events/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Overview of HPC Interconnects
In this deck from ATPESC 2019, Ken Raffenetti from Argonne presents an overview of HPC interconnects.
"The Argonne Training Program on Extreme-Scale Computing (ATPESC) provides intensive, two-week training on the key skills, approaches, and tools to design, implement, and execute computational science and engineering applications on current high-end computing systems and the leadership-class computing systems of the future."
Watch the video: https://wp.me/p3RLHQ-luc
Learn more: https://extremecomputingtraining.anl.gov/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
More from inside-BigData.com (20)
Preparing to program Aurora at Exascale - Early experiences and future direct...
Preparing to program Aurora at Exascale - Early experiences and future direct...
The Incorporation of Machine Learning into Scientific Simulations at Lawrence...
The Incorporation of Machine Learning into Scientific Simulations at Lawrence...
How to Achieve High-Performance, Scalable and Distributed DNN Training on Mod...
How to Achieve High-Performance, Scalable and Distributed DNN Training on Mod...
Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze ...
Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze ...
Biohybrid Robotic Jellyfish for Future Applications in Ocean Monitoring
Biohybrid Robotic Jellyfish for Future Applications in Ocean Monitoring
HPC at Scale Enabled by DDN A3i and NVIDIA SuperPOD
HPC at Scale Enabled by DDN A3i and NVIDIA SuperPOD
Versal Premium ACAP for Network and Cloud Acceleration
Versal Premium ACAP for Network and Cloud Acceleration
Zettar: Moving Massive Amounts of Data across Any Distance Efficiently
Zettar: Moving Massive Amounts of Data across Any Distance Efficiently
CUDA-Python and RAPIDS for blazing fast scientific computing
CUDA-Python and RAPIDS for blazing fast scientific computing
Recently uploaded
TrustArc Webinar - 2024 Global Privacy Survey
How does your privacy program stack up against your peers? What challenges are privacy teams tackling and prioritizing in 2024?
In the fifth annual Global Privacy Benchmarks Survey, we asked over 1,800 global privacy professionals and business executives to share their perspectives on the current state of privacy inside and outside of their organizations. This year’s report focused on emerging areas of importance for privacy and compliance professionals, including considerations and implications of Artificial Intelligence (AI) technologies, building brand trust, and different approaches for achieving higher privacy competence scores.
See how organizational priorities and strategic approaches to data security and privacy are evolving around the globe.
This webinar will review:
- The top 10 privacy insights from the fifth annual Global Privacy Benchmarks Survey
- The top challenges for privacy leaders, practitioners, and organizations in 2024
- Key themes to consider in developing and maintaining your privacy program
Choosing The Best AWS Service For Your Website + API.pptx
Have you ever been confused by the myriad of choices offered by AWS for hosting a website or an API?
Lambda, Elastic Beanstalk, Lightsail, Amplify, S3 (and more!) can each host websites + APIs. But which one should we choose?
Which one is cheapest? Which one is fastest? Which one will scale to meet our needs?
Join me in this session as we dive into each AWS hosting service to determine which one is best for your scenario and explain why!
HCL Notes and Domino License Cost Reduction in the World of DLAU
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-and-domino-license-cost-reduction-in-the-world-of-dlau/
The introduction of DLAU and the CCB & CCX licensing model caused quite a stir in the HCL community. As a Notes and Domino customer, you may have faced challenges with unexpected user counts and license costs. You probably have questions on how this new licensing approach works and how to benefit from it. Most importantly, you likely have budget constraints and want to save money where possible. Don’t worry, we can help with all of this!
We’ll show you how to fix common misconfigurations that cause higher-than-expected user counts, and how to identify accounts which you can deactivate to save money. There are also frequent patterns that can cause unnecessary cost, like using a person document instead of a mail-in for shared mailboxes. We’ll provide examples and solutions for those as well. And naturally we’ll explain the new licensing model.
Join HCL Ambassador Marc Thomas in this webinar with a special guest appearance from Franz Walder. It will give you the tools and know-how to stay on top of what is going on with Domino licensing. You will be able lower your cost through an optimized configuration and keep it low going forward.
These topics will be covered
- Reducing license cost by finding and fixing misconfigurations and superfluous accounts
- How do CCB and CCX licenses really work?
- Understanding the DLAU tool and how to best utilize it
- Tips for common problem areas, like team mailboxes, functional/test users, etc
- Practical examples and best practices to implement right away
5th LF Energy Power Grid Model Meet-up Slides
5th Power Grid Model Meet-up
It is with great pleasure that we extend to you an invitation to the 5th Power Grid Model Meet-up, scheduled for 6th June 2024. This event will adopt a hybrid format, allowing participants to join us either through an online Mircosoft Teams session or in person at TU/e located at Den Dolech 2, Eindhoven, Netherlands. The meet-up will be hosted by Eindhoven University of Technology (TU/e), a research university specializing in engineering science & technology.
Power Grid Model
The global energy transition is placing new and unprecedented demands on Distribution System Operators (DSOs). Alongside upgrades to grid capacity, processes such as digitization, capacity optimization, and congestion management are becoming vital for delivering reliable services.
Power Grid Model is an open source project from Linux Foundation Energy and provides a calculation engine that is increasingly essential for DSOs. It offers a standards-based foundation enabling real-time power systems analysis, simulations of electrical power grids, and sophisticated what-if analysis. In addition, it enables in-depth studies and analysis of the electrical power grid’s behavior and performance. This comprehensive model incorporates essential factors such as power generation capacity, electrical losses, voltage levels, power flows, and system stability.
Power Grid Model is currently being applied in a wide variety of use cases, including grid planning, expansion, reliability, and congestion studies. It can also help in analyzing the impact of renewable energy integration, assessing the effects of disturbances or faults, and developing strategies for grid control and optimization.
What to expect
For the upcoming meetup we are organizing, we have an exciting lineup of activities planned:
-Insightful presentations covering two practical applications of the Power Grid Model.
-An update on the latest advancements in Power Grid -Model technology during the first and second quarters of 2024.
-An interactive brainstorming session to discuss and propose new feature requests.
-An opportunity to connect with fellow Power Grid Model enthusiasts and users.
Azure API Management to expose backend services securely
How to use Azure API Management to expose backend service securely
Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays, June 2024 with Maria Copot 20
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Main news related to the CCS TSI 2023 (2023/1695)
An English 🇬🇧 translation of a presentation to the speech I gave about the main changes brought by CCS TSI 2023 at the biggest Czech conference on Communications and signalling systems on Railways, which was held in Clarion Hotel Olomouc from 7th to 9th November 2023 (konferenceszt.cz). Attended by around 500 participants and 200 on-line followers.
The original Czech 🇨🇿 version of the presentation can be found here: https://www.slideshare.net/slideshow/hlavni-novinky-souvisejici-s-ccs-tsi-2023-2023-1695/269688092 .
The videorecording (in Czech) from the presentation is available here: https://youtu.be/WzjJWm4IyPk?si=SImb06tuXGb30BEH .
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Are you ready to revolutionize how you handle data? Join us for a webinar where we’ll bring you up to speed with the latest advancements in Generative AI technology and discover how leveraging FME with tools from giants like Google Gemini, Amazon, and Microsoft OpenAI can supercharge your workflow efficiency.
During the hour, we’ll take you through:
Guest Speaker Segment with Hannah Barrington: Dive into the world of dynamic real estate marketing with Hannah, the Marketing Manager at Workspace Group. Hear firsthand how their team generates engaging descriptions for thousands of office units by integrating diverse data sources—from PDF floorplans to web pages—using FME transformers, like OpenAIVisionConnector and AnthropicVisionConnector. This use case will show you how GenAI can streamline content creation for marketing across the board.
Ollama Use Case: Learn how Scenario Specialist Dmitri Bagh has utilized Ollama within FME to input data, create custom models, and enhance security protocols. This segment will include demos to illustrate the full capabilities of FME in AI-driven processes.
Custom AI Models: Discover how to leverage FME to build personalized AI models using your data. Whether it’s populating a model with local data for added security or integrating public AI tools, find out how FME facilitates a versatile and secure approach to AI.
We’ll wrap up with a live Q&A session where you can engage with our experts on your specific use cases, and learn more about optimizing your data workflows with AI.
This webinar is ideal for professionals seeking to harness the power of AI within their data management systems while ensuring high levels of customization and security. Whether you're a novice or an expert, gain actionable insights and strategies to elevate your data processes. Join us to see how FME and AI can revolutionize how you work with data!
Taking AI to the Next Level in Manufacturing.pdf
Read Taking AI to the Next Level in Manufacturing to gain insights on AI adoption in the manufacturing industry, such as:
1. How quickly AI is being implemented in manufacturing.
2. Which barriers stand in the way of AI adoption.
3. How data quality and governance form the backbone of AI.
4. Organizational processes and structures that may inhibit effective AI adoption.
6. Ideas and approaches to help build your organization's AI strategy.
Trusted Execution Environment for Decentralized Process Mining
Presentation of the paper "Trusted Execution Environment for Decentralized Process Mining" given during the CAiSE 2024 Conference in Cyprus on June 7, 2024.
Letter and Document Automation for Bonterra Impact Management (fka Social Sol...
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on automated letter generation for Bonterra Impact Management using Google Workspace or Microsoft 365.
Interested in deploying letter generation automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Discover the seamless integration of RPA (Robotic Process Automation), COMPOSER, and APM with AWS IDP enhanced with Slack notifications. Explore how these technologies converge to streamline workflows, optimize performance, and ensure secure access, all while leveraging the power of AWS IDP and real-time communication via Slack notifications.
Building Production Ready Search Pipelines with Spark and Milvus
Spark is the widely used ETL tool for processing, indexing and ingesting data to serving stack for search. Milvus is the production-ready open-source vector database. In this talk we will show how to use Spark to process unstructured data to extract vector representations, and push the vectors to Milvus vector database for search serving.
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
Predictive maintenance is a proactive approach that anticipates equipment failures before they happen. At the forefront of this innovative strategy is Artificial Intelligence (AI), which brings unprecedented precision and efficiency. AI in predictive maintenance is transforming industries by reducing downtime, minimizing costs, and enhancing productivity.
Recently uploaded (20)
Deep Dive: Getting Funded with Jason Jason Lemkin Founder & CEO @ SaaStr
Deep Dive: Getting Funded with Jason Jason Lemkin Founder & CEO @ SaaStr
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
Azure API Management to expose backend services securely
Azure API Management to expose backend services securely
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Trusted Execution Environment for Decentralized Process Mining
Trusted Execution Environment for Decentralized Process Mining
Letter and Document Automation for Bonterra Impact Management (fka Social Sol...
Letter and Document Automation for Bonterra Impact Management (fka Social Sol...
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
Research Computing Storage at the Francis Crick Institute
- 2. Research Computing Storage at The Francis Crick Institute Steve Hindmarsh Head of Scientific Computing steve.hindmarsh@crick.ac.uk Michael Holliday Senior HPC & Research Data Systems Engineer michael.holliday@crick.ac.uk DDN User Group SC18
- 3. The Francis Crick Institute • A biomedical discovery institute dedicated to understanding the fundamental biology underlying health and disease. • Founded in 2015 with the merger of two London research institutes from the Medical Research Council and Cancer Research UK into the Crick • Supported by our founding partners: 3
- 4. “Discovery without boundaries” • Core research areas: • Growth and development • Health and ageing • Human biology • Cancer • Immune system • Infectious disease • Neuroscience Multi-disciplinary approach: biology, physics, chemistry, bioinformatics, maths, engineering… 4
- 5. The Crick in numbers • 1300 Scientists across 130 Labs, Research Groups & Science Technology Platforms • 300 Operations Staff • Over 30,000 pieces of Scientific Equipment • 13 Floors, 1500 Rooms • 2 National / International facilities • 1 Nobel Prize winner! 5
- 6. Scientific instruments • Cryo-Electron microscopes • Electron microscopes • Light microscopes • DNA/RNA sequencers • Mass spectrometers • Nuclear Magnetic Resonance (NMR) spectroscopes • Etc. All produce large volumes of data which needs to be captured, stored and analysed. 6
- 7. DDN at the Crick DDN underpins our research computing platform providing: • Scalability – currently 10 PB and filling up fast! • Performance • Multiple and varied workloads • Resilience • Manageability So far it has stood up well to all the workloads our scientists can throw at it. New challenges include: • Unprecedented data growth • Machine/Deep Learning/AI – Nvidia V100 GPU cluster coming soon 7
- 9. CAMP – Crick Data Analysis and Management Platform – Where did we come from: • 4 Isilon Clusters with around 4PB of data going back 40 years • Numerous NAS Boxes, Hard drives, USBs etc (we’re still finding out about them now…) • Storage from legacy institutes were managed in very different ways, and had their own Domains and Permissions • Data replicated in multiple locations but it wasn’t always clear
- 10. CAMP – Crick Data Analysis and Management Platform–
- 11. Where we started at the Crick: CAMP (3PB) INGEST (1PB) GENERAL (500TB) Mill Hill LIF GARLIC HPC Cluster (200 Nodes) AFM (IW) AFM (IW) AFM (RO) AFM (RO) Remote Mount Remote Mount Remote Mount
- 12. CAMP – Crick Data Analysis and Management Platform – • FDR IB Fabric • 2 Level Fabric • CAMP • 2 DDN GS12K Systems • 20 Enclosures • INGEST, General, LIF, Millhill: • Each has 1 DDN GS7K • 1 or 2 Enclosures
- 13. Problems we encountered… • AFM • Migration of legacy data • Data, More Data & Even More Data…. • Instruments • Labs adapting to the new systems 13
- 14. AFM • INGEST and General were set up to be caches of CAMP storage • At one point INGEST had 130 AFM links to CAMP • Delays to syncing were noticeable to scientists, particularly those using the HPC cluster • Permissions were not syncing correctly between the two • Cache was out of sync with home and unable to recover • Used Independent Writer Mode 14
- 15. Data, More Data & Even More Data…. • Our Labs and STPs are producing more data at faster and faster rates • 1 Titan Cryo-EM Microscope can produce up to 1PB data a year (we have 3!). • Currently ½ Billion Files most only Kb • Covers pretty much every file type • CAMP has been Expanded from 3PB to 10PB • Data needs to be kept for 10 years • Archive facility is in progress! 16
- 16. Where we are working towards: CAMP (9PB) INGEST (1-2PB) Melchett (500TB) BlackAdder (testing) GARLIC HPC Cluster (400 Nodes) Spectrum Protect Backup (14PB) Archive (TBD) AFM (single writer) Remote Mount HPC Workstations AFM (single writer)
- 17. Questions? 20