Past, present, and future of HPC in life sciences
•
1 like•439 views
Traditional access to HPC systems is provided through batch jobs submitted directly by users. Today, an increasing number of biological software is made accessible through web portals. This improved ease of use has led towards a democratization of access to computational resources, allowing scientists without computational backgrounds to carry out data- and compute-intensive analyses. However, these persistent services do not integrate well into a classical batch oriented HPC system such as MENDEL. This shift in requirements for emerging services demands a re-evaluation of infrastructure planning and deployment. We will present our ongoing efforts to fulfil these requirements in our existing system, and how we plan to address them in a next generation data analysis platform.
Report
Share
Report
Share
Download to read offline
Recommended
XSEDE April 2017
XSEDE (Extreme Science and Engineering Discovery Environment) is an NSF-funded project that provides researchers access to multiple supercomputers and computational resources hosted at various institutions, including Stampede 2 at TACC which is undergoing an upgrade to 18 petaflops. The document describes the computational resources available through XSEDE such as Comet at SDSC, Bridges at PSC, XStream at Stanford, and Jetstream at IU/TACC, and outlines the different types of allocation requests that can be made including trial, startup, education, and research allocations.
GraphBLAS and Emus
1) Lucata's Emu architecture features cacheless multicore processors on DRAM with hardware that migrates threads between cores to access data, providing high memory bandwidth.
2) Benchmark results show the Emu architecture achieves near-peak memory bandwidth and flat performance for a pointer-chasing workload, unlike x86 which varies with data access patterns.
3) Implementing BFS on a dynamic graph data structure achieved higher performance and bandwidth on Emu compared to x86, demonstrating Emu's suitability for streaming workloads.
OpenNebulaConf2015 2.06 OpenNebula in the Wild - Ander Astudillo
Where OpenNebula is typically used by system administrators to manage a data centers, SURFsara offers the OpenNebula interface directly to end users. SURFsara’s end users are researchers with diverse needs, and as such, offering custom solutions to all end users is impossible. Although we were hesitant at first, we gave researchers full control over their virtual infrstructure. It turns out that many researchers find the OpenNebula interface easy enough to use, enabling them to build complex compute infrastructures for their analysis workflows. This presentation discusses the features our researchers in the OpenNebula appreciate, and we discuss a selection of succesful projects.
Author Biography
SURFsara is the Dutch national provider of computing and data infrastructure for the research community. As a consultant for our HPC Cloud service, I help scientists (not necessarily tech experts) use our computing facilities to solve their research questions.
GIST AI-X Computing Cluster
The GIST AI-X Computing Cluster provides powerful accelerated computation resources for machine learning using GPUs and other hardware. It includes DGX A100 and DGX-1V nodes with 8 NVIDIA A100 or V100 GPUs each, connected by high-speed networking. The cluster uses Singularity containers, Slurm scheduling, and Ceph storage. It allows researchers to request resources, build container images, and run distributed deep learning jobs across multiple GPUs.
Datasheet - Pivot3 - HCI Family
This document summarizes Pivot3's hyperconverged infrastructure (HCI) solutions. It highlights that Pivot3's distributed scale-out architecture pools compute and storage resources across nodes for maximum utilization. It also notes that Pivot3's patented erasure coding provides efficient high availability that maintains performance during failures. The summary concludes that Pivot3's solutions offer scalable storage and performance to meet application needs.
Integrating openSUSE Ceph Block Device & OpenStack
Utian Ayuba provides information about integrating Ceph block device storage and OpenStack. He discusses Ceph architecture including OSD, monitor, and metadata server nodes. Ceph uses placement groups to store and replicate data across nodes. Ceph supports block, object, and file storage interfaces that can be used by OpenStack services like Glance (images), Cinder (volumes), and Nova (instance disks). Utian then outlines three labs for installing OpenStack on openSUSE, deploying a Ceph cluster, and integrating Ceph block storage with OpenStack. Topologies are provided for the single-node OpenStack and Ceph cluster deployments, as well as the integrated OpenStack and Ceph architecture.
User-space Network Processing
This document discusses optimizations for TCP/IP networking performance on multicore systems. It describes several inefficiencies in the Linux kernel TCP/IP stack related to shared resources between cores, broken data locality, and per-packet processing overhead. It then introduces mTCP, a user-level TCP/IP stack that addresses these issues through a thread model with pairwise threading, batch packet processing from I/O to applications, and a BSD-like socket API. mTCP achieves a 2.35x performance improvement over the kernel TCP/IP stack on a web server workload.
Software Defined storage
GlusterFS is an open-source distributed file system that aggregates various storage servers over a network into one large parallel file system. It does not require a metadata server, ensuring better performance, linear scalability, and reliability compared to traditional distributed file systems. Red Hat Gluster Storage provides a scalable, reliable data management platform using GlusterFS to streamline file access across physical, virtual, and cloud environments. It supports various volume types including distributed replicated volumes which replicate data across multiple servers for high availability.
Recommended
XSEDE April 2017
XSEDE (Extreme Science and Engineering Discovery Environment) is an NSF-funded project that provides researchers access to multiple supercomputers and computational resources hosted at various institutions, including Stampede 2 at TACC which is undergoing an upgrade to 18 petaflops. The document describes the computational resources available through XSEDE such as Comet at SDSC, Bridges at PSC, XStream at Stanford, and Jetstream at IU/TACC, and outlines the different types of allocation requests that can be made including trial, startup, education, and research allocations.
GraphBLAS and Emus
1) Lucata's Emu architecture features cacheless multicore processors on DRAM with hardware that migrates threads between cores to access data, providing high memory bandwidth.
2) Benchmark results show the Emu architecture achieves near-peak memory bandwidth and flat performance for a pointer-chasing workload, unlike x86 which varies with data access patterns.
3) Implementing BFS on a dynamic graph data structure achieved higher performance and bandwidth on Emu compared to x86, demonstrating Emu's suitability for streaming workloads.
OpenNebulaConf2015 2.06 OpenNebula in the Wild - Ander Astudillo
Where OpenNebula is typically used by system administrators to manage a data centers, SURFsara offers the OpenNebula interface directly to end users. SURFsara’s end users are researchers with diverse needs, and as such, offering custom solutions to all end users is impossible. Although we were hesitant at first, we gave researchers full control over their virtual infrstructure. It turns out that many researchers find the OpenNebula interface easy enough to use, enabling them to build complex compute infrastructures for their analysis workflows. This presentation discusses the features our researchers in the OpenNebula appreciate, and we discuss a selection of succesful projects.
Author Biography
SURFsara is the Dutch national provider of computing and data infrastructure for the research community. As a consultant for our HPC Cloud service, I help scientists (not necessarily tech experts) use our computing facilities to solve their research questions.
GIST AI-X Computing Cluster
The GIST AI-X Computing Cluster provides powerful accelerated computation resources for machine learning using GPUs and other hardware. It includes DGX A100 and DGX-1V nodes with 8 NVIDIA A100 or V100 GPUs each, connected by high-speed networking. The cluster uses Singularity containers, Slurm scheduling, and Ceph storage. It allows researchers to request resources, build container images, and run distributed deep learning jobs across multiple GPUs.
Datasheet - Pivot3 - HCI Family
This document summarizes Pivot3's hyperconverged infrastructure (HCI) solutions. It highlights that Pivot3's distributed scale-out architecture pools compute and storage resources across nodes for maximum utilization. It also notes that Pivot3's patented erasure coding provides efficient high availability that maintains performance during failures. The summary concludes that Pivot3's solutions offer scalable storage and performance to meet application needs.
Integrating openSUSE Ceph Block Device & OpenStack
Utian Ayuba provides information about integrating Ceph block device storage and OpenStack. He discusses Ceph architecture including OSD, monitor, and metadata server nodes. Ceph uses placement groups to store and replicate data across nodes. Ceph supports block, object, and file storage interfaces that can be used by OpenStack services like Glance (images), Cinder (volumes), and Nova (instance disks). Utian then outlines three labs for installing OpenStack on openSUSE, deploying a Ceph cluster, and integrating Ceph block storage with OpenStack. Topologies are provided for the single-node OpenStack and Ceph cluster deployments, as well as the integrated OpenStack and Ceph architecture.
User-space Network Processing
This document discusses optimizations for TCP/IP networking performance on multicore systems. It describes several inefficiencies in the Linux kernel TCP/IP stack related to shared resources between cores, broken data locality, and per-packet processing overhead. It then introduces mTCP, a user-level TCP/IP stack that addresses these issues through a thread model with pairwise threading, batch packet processing from I/O to applications, and a BSD-like socket API. mTCP achieves a 2.35x performance improvement over the kernel TCP/IP stack on a web server workload.
Software Defined storage
GlusterFS is an open-source distributed file system that aggregates various storage servers over a network into one large parallel file system. It does not require a metadata server, ensuring better performance, linear scalability, and reliability compared to traditional distributed file systems. Red Hat Gluster Storage provides a scalable, reliable data management platform using GlusterFS to streamline file access across physical, virtual, and cloud environments. It supports various volume types including distributed replicated volumes which replicate data across multiple servers for high availability.
Shooting schedule
The document outlines the filming schedule for shooting a band performing on stage. It includes 21 planned shots from 9:10 to 11:05, with descriptions of the camera setup, focus of each shot, and notes that the band members will be in normal clothes and the instruments and lighting will be centered on the band. It also notes breaks in the schedule for a potential small photo shoot and another break. The purpose is to capture the band's performance from various camera angles and perspectives.
Gospel of hip hop
KRS-One wrote "The Gospel of Hip-Hop", a 600-page book that serves as a philosophical guide to hip-hop culture. As one of hip-hop's most respected emcees, KRS-One details the origins and development of hip-hop and provides teachings on how practitioners of hip-hop culture can preserve its future and create one focused on health, love, awareness and wealth. Known as "The Teacha", KRS-One draws from his own experience and 14 years of research to share the insights on hip-hop contained in this inspiring new work.
Annual-Report-2013
Transportation is provided for workers to agricultural greenhouses in the Souss Massa Drâa area of Morocco. Vans pick up workers from villages and bring them to the greenhouses, where they work long hours in difficult conditions. After the work is done, the vans return the tired workers back to their homes at the end of the day.
Mood board
A digipak typically contains two CDs labeled as disk one and disk two, along with a contents listing providing the song number for each track to help listeners identify and skip to favorite songs. Some digipaks may also include additional items like a poster of the artist, as Beyoncé did for her Flawless album which included a poster for purchasers. Digipaks appeal to music fans who enjoy collecting albums across various music genres.
Important Personalities of Mahabharata
''Mahabharata''!!!!! A famous Hindu epic story needs to be discussed.....Lets see who were the important people of the story who made ''Mahabharata'' an epic.....
Pengolahan Limbah Cair dengan metode Elektrokoagulasi
Proses elektrokoagulasi menggunakan arus listrik untuk mengkoagulasi limbah cair domestik. Reaksi redoks terjadi pada elektroda logam alumunium yang dicelupkan ke dalam larutan, menghasilkan ion Al3+ dan membentuk flokulan untuk menghilangkan kontaminan. Langkah-langkah meliputi persiapan peralatan, pencelupan elektroda ke dalam limbah, dan penyaringan hasil setelah 1,5 jam untuk men
Fast & Furious: building HPC solutions in a nutshell
Victor Haydin presented on high-performance computing (HPC). He began by defining HPC as using supercomputers and computer clusters to solve advanced computation problems, such as those involving teraflops-level performance. He discussed why HPC is needed for applications in areas like finances, healthcare, fluid dynamics, and genetics. Finally, he outlined some approaches to implementing HPC, such as using commodity hardware, GPU-based systems, distributed and load-balanced architectures, and middleware to handle errors and allow the same code to run on CPUs and GPUs.
FIEV report on steel sector
This report basically deals on the impact of various economic and social factors that leads in making investment decision for the Investors
Kathryn Gregg Resume
Kathryn E. Gregg is an experienced teacher and early childhood development professional. She is dedicated to providing comprehensive instruction that fosters child development through stimulating classroom environments accommodating different learning styles. Gregg has demonstrated the ability to teach, motivate, and direct students while maintaining high levels of interest and achievement. She has strong planning, organization, and leadership abilities as well as superb interpersonal skills for establishing positive relationships with students, parents, and faculty.
CASO CLÍNICO ORTO ADRIÁN QUIZHPE
Este documento presenta el caso de un paciente de 24 años que consulta por revisión ortodóncica. Se realiza un examen clínico completo incluyendo análisis dental, cefalométrico y modelos de yeso. Los resultados muestran un patrón de crecimiento vertical hiperdivergente con retrusión maxilar y mandibular, así como una discrepancia hueso-diente positiva.
HPC in healthcare
1) HPC is being used in healthcare to quickly diagnose cancer through genomic analysis and to develop new drug therapies through simulations.
2) Pathwork Diagnostics uses cloud-based HPC to develop models that can diagnose cancer in months rather than years.
3) GNS Healthcare applies HPC to reverse engineer disease models and simulate new drug targets for conditions like cancer and diabetes.
Digital pen
Digital pens capture handwriting and convert it to digital data. They contain electronics like memory, buttons, and Bluetooth connectivity. A digital pen works by using an infrared LED and light detector to detect marks on special paper and record writing as images. It then transmits the stored data wirelessly via Bluetooth to a computer where software converts it to editable text. The document provides an overview of digital pens, including their components and how they function to digitize handwritten notes.
Introduction To Apache Mesos
This document provides an introduction and overview of Apache Mesos. It begins by describing Mesos' origins at companies like Google and how it enables fine-grained resource sharing in data centers. It then discusses concepts like schedulers, executors, frameworks and how Mesos allows building distributed applications and data center infrastructure. The document also covers Mesos concepts such as resources, attributes, roles, constraints and how tools like Marathon interact with Mesos. Finally, it provides examples of Mesos frameworks for technologies like Kafka and Cassandra.
Big Data HPC Convergence
5th Multicore World
15-17 February 2016 – Shed 6, Wellington, New Zealand
http://openparallel.com/multicore-world-2016/
We start by dividing applications into data plus model components and classifying each component (whether from Big Data or Big Simulations) in the same way. These leads to 64 properties divided into 4 views, which are Problem Architecture (Macro pattern); Execution Features (Micro patterns); Data Source and Style; and finally the Processing (runtime) View.
We discuss convergence software built around HPC-ABDS (High Performance Computing enhanced Apache Big Data Stack) http://hpc-abds.org/kaleidoscope/ and show how one can merge Big Data and HPC (Big Simulation) concepts into a single stack.
We give examples of data analytics running on HPC systems including details on persuading Java to run fast.
Some details can be found at http://dsc.soic.indiana.edu/publications/HPCBigDataConvergence.pdf
ODCA Board Best Practice: High Performance Computing at BMW
The document discusses high performance computing (HPC) demands and strategies at BMW. It notes that BMW's HPC usage and simulation models are increasing in complexity, driving the need for more computing power. BMW has established an HPC cluster to meet these demands, and is taking steps to improve efficiency and cost-effectiveness through HPC co-location and exploring public and private cloud options. This includes relocating some workloads to a data center in Iceland and testing moving an elementary HPC cell to a cloud platform.
Big Data and High Performance Computing Solutions in the AWS Cloud
Managing big data and running supercomputing jobs used to be for only well-funded research organizations and large corporations, but not any longer. AWS has democratized supercomputing and big data for the masses! AWS can provide you with the 64th fastest supercomputer in the world, on-demand and pay as you go. Hear from Ben Butler, Head of AWS Big Data Marketing, to learn how our customers are using big data and high performance computing to change the world. Not only is AWS technology available to everyone, but it is self-service and cheaper than ever before, featuring innovative technology and flexible pricing models – our AWS cloud computing platform has disrupted big data and HPC. Learn from customer successes, as Ben shares real-world case studies describing the specific big data and high performance computing challenges being solved on AWS. We will conclude with a discussion around the tutorials, public datasets, test drives, and our grants program - all of the tools needed to get you started quickly.
Intro to High Performance Computing in the AWS Cloud
Visit http:aws.amazon.com/hpc for more information about HPC on AWS.
High Performance Computing (HPC) allows scientists and engineers to solve complex science, engineering, and business problems using applications that require high bandwidth, low latency networking, and very high compute capabilities. AWS allows you to increase the speed of research by running high performance computing in the cloud and to reduce costs by providing Cluster Compute or Cluster GPU servers on-demand without large capital investments. You have access to a full-bisection, high bandwidth network for tightly-coupled, IO-intensive workloads, which enables you to scale out across thousands of cores for throughput-oriented applications.
HPC Market Update from IDC
In this deck from ISC 2015, Earl Joseph and Bob Sorensen from IDC provide an HPC Market Update.
Watch the video presentation: http://wp.me/p3RLHQ-emz
Sign up for our insideHPC Newsletter: http://insideHPC.com/newsletter
AWS re:Invent 2016: High Performance Computing on AWS (CMP207)
High performance computing in the cloud is enabling high scale compute- and graphics-intensive workloads across industries, ranging from aerospace, automotive, and manufacturing to life sciences, financial services, and energy. AWS provides application developers and end users with unprecedented computational power for massively parallel applications, in areas such as large-scale fluid and materials simulations, 3D content rendering, financial computing, and deep learning. This session provides an overview of HPC capabilities on AWS, describes the newest generations of accelerated computing instances (including P2), as well as highlighting customer and partner use-cases across industries.
Attendees learn about best practices for running HPC workflows in the cloud, including graphical pre- and post-processing, workflow automation, and optimization. Attendees also learn about new and emerging HPC use cases: in particular, deep learning training and inference, large-scale simulations, and high performance data analytics.
General Introduction to technologies that will be seen in the school
1. The document discusses principles of distributed computing technologies including service-oriented architectures, high-throughput computing, distributed data management, job submission and execution management, using distributed systems, higher-level APIs like OGSA-DAI and SAGA, and workflows.
2. It provides histories and visions for technologies like UNICORE, Condor, Globus, gLite, ARC, and P-GRADE portal.
3. Key principles covered include Web services, parallelism, interoperability, simplicity, scalability, and hiding complexity through high-level interfaces.
Hungarian ClusterGrid and its applications
This document summarizes the Hungarian ClusterGrid and its applications. ClusterGrid integrates individual clusters and supercomputers with grid middleware. It currently includes 1000 nodes across 32 cluster sites providing 22 TB of distributed storage. A new generation middleware called Grid Underground has been developed which uses a pure web services approach. Core services include security, job management, and storage. One application involved virtual screening of 8 million molecules to identify potential new medicines.
More Related Content
Viewers also liked
Shooting schedule
The document outlines the filming schedule for shooting a band performing on stage. It includes 21 planned shots from 9:10 to 11:05, with descriptions of the camera setup, focus of each shot, and notes that the band members will be in normal clothes and the instruments and lighting will be centered on the band. It also notes breaks in the schedule for a potential small photo shoot and another break. The purpose is to capture the band's performance from various camera angles and perspectives.
Gospel of hip hop
KRS-One wrote "The Gospel of Hip-Hop", a 600-page book that serves as a philosophical guide to hip-hop culture. As one of hip-hop's most respected emcees, KRS-One details the origins and development of hip-hop and provides teachings on how practitioners of hip-hop culture can preserve its future and create one focused on health, love, awareness and wealth. Known as "The Teacha", KRS-One draws from his own experience and 14 years of research to share the insights on hip-hop contained in this inspiring new work.
Annual-Report-2013
Transportation is provided for workers to agricultural greenhouses in the Souss Massa Drâa area of Morocco. Vans pick up workers from villages and bring them to the greenhouses, where they work long hours in difficult conditions. After the work is done, the vans return the tired workers back to their homes at the end of the day.
Mood board
A digipak typically contains two CDs labeled as disk one and disk two, along with a contents listing providing the song number for each track to help listeners identify and skip to favorite songs. Some digipaks may also include additional items like a poster of the artist, as Beyoncé did for her Flawless album which included a poster for purchasers. Digipaks appeal to music fans who enjoy collecting albums across various music genres.
Important Personalities of Mahabharata
''Mahabharata''!!!!! A famous Hindu epic story needs to be discussed.....Lets see who were the important people of the story who made ''Mahabharata'' an epic.....
Pengolahan Limbah Cair dengan metode Elektrokoagulasi
Proses elektrokoagulasi menggunakan arus listrik untuk mengkoagulasi limbah cair domestik. Reaksi redoks terjadi pada elektroda logam alumunium yang dicelupkan ke dalam larutan, menghasilkan ion Al3+ dan membentuk flokulan untuk menghilangkan kontaminan. Langkah-langkah meliputi persiapan peralatan, pencelupan elektroda ke dalam limbah, dan penyaringan hasil setelah 1,5 jam untuk men
Fast & Furious: building HPC solutions in a nutshell
Victor Haydin presented on high-performance computing (HPC). He began by defining HPC as using supercomputers and computer clusters to solve advanced computation problems, such as those involving teraflops-level performance. He discussed why HPC is needed for applications in areas like finances, healthcare, fluid dynamics, and genetics. Finally, he outlined some approaches to implementing HPC, such as using commodity hardware, GPU-based systems, distributed and load-balanced architectures, and middleware to handle errors and allow the same code to run on CPUs and GPUs.
FIEV report on steel sector
This report basically deals on the impact of various economic and social factors that leads in making investment decision for the Investors
Kathryn Gregg Resume
Kathryn E. Gregg is an experienced teacher and early childhood development professional. She is dedicated to providing comprehensive instruction that fosters child development through stimulating classroom environments accommodating different learning styles. Gregg has demonstrated the ability to teach, motivate, and direct students while maintaining high levels of interest and achievement. She has strong planning, organization, and leadership abilities as well as superb interpersonal skills for establishing positive relationships with students, parents, and faculty.
CASO CLÍNICO ORTO ADRIÁN QUIZHPE
Este documento presenta el caso de un paciente de 24 años que consulta por revisión ortodóncica. Se realiza un examen clínico completo incluyendo análisis dental, cefalométrico y modelos de yeso. Los resultados muestran un patrón de crecimiento vertical hiperdivergente con retrusión maxilar y mandibular, así como una discrepancia hueso-diente positiva.
HPC in healthcare
1) HPC is being used in healthcare to quickly diagnose cancer through genomic analysis and to develop new drug therapies through simulations.
2) Pathwork Diagnostics uses cloud-based HPC to develop models that can diagnose cancer in months rather than years.
3) GNS Healthcare applies HPC to reverse engineer disease models and simulate new drug targets for conditions like cancer and diabetes.
Digital pen
Digital pens capture handwriting and convert it to digital data. They contain electronics like memory, buttons, and Bluetooth connectivity. A digital pen works by using an infrared LED and light detector to detect marks on special paper and record writing as images. It then transmits the stored data wirelessly via Bluetooth to a computer where software converts it to editable text. The document provides an overview of digital pens, including their components and how they function to digitize handwritten notes.
Introduction To Apache Mesos
This document provides an introduction and overview of Apache Mesos. It begins by describing Mesos' origins at companies like Google and how it enables fine-grained resource sharing in data centers. It then discusses concepts like schedulers, executors, frameworks and how Mesos allows building distributed applications and data center infrastructure. The document also covers Mesos concepts such as resources, attributes, roles, constraints and how tools like Marathon interact with Mesos. Finally, it provides examples of Mesos frameworks for technologies like Kafka and Cassandra.
Big Data HPC Convergence
5th Multicore World
15-17 February 2016 – Shed 6, Wellington, New Zealand
http://openparallel.com/multicore-world-2016/
We start by dividing applications into data plus model components and classifying each component (whether from Big Data or Big Simulations) in the same way. These leads to 64 properties divided into 4 views, which are Problem Architecture (Macro pattern); Execution Features (Micro patterns); Data Source and Style; and finally the Processing (runtime) View.
We discuss convergence software built around HPC-ABDS (High Performance Computing enhanced Apache Big Data Stack) http://hpc-abds.org/kaleidoscope/ and show how one can merge Big Data and HPC (Big Simulation) concepts into a single stack.
We give examples of data analytics running on HPC systems including details on persuading Java to run fast.
Some details can be found at http://dsc.soic.indiana.edu/publications/HPCBigDataConvergence.pdf
ODCA Board Best Practice: High Performance Computing at BMW
The document discusses high performance computing (HPC) demands and strategies at BMW. It notes that BMW's HPC usage and simulation models are increasing in complexity, driving the need for more computing power. BMW has established an HPC cluster to meet these demands, and is taking steps to improve efficiency and cost-effectiveness through HPC co-location and exploring public and private cloud options. This includes relocating some workloads to a data center in Iceland and testing moving an elementary HPC cell to a cloud platform.
Big Data and High Performance Computing Solutions in the AWS Cloud
Managing big data and running supercomputing jobs used to be for only well-funded research organizations and large corporations, but not any longer. AWS has democratized supercomputing and big data for the masses! AWS can provide you with the 64th fastest supercomputer in the world, on-demand and pay as you go. Hear from Ben Butler, Head of AWS Big Data Marketing, to learn how our customers are using big data and high performance computing to change the world. Not only is AWS technology available to everyone, but it is self-service and cheaper than ever before, featuring innovative technology and flexible pricing models – our AWS cloud computing platform has disrupted big data and HPC. Learn from customer successes, as Ben shares real-world case studies describing the specific big data and high performance computing challenges being solved on AWS. We will conclude with a discussion around the tutorials, public datasets, test drives, and our grants program - all of the tools needed to get you started quickly.
Intro to High Performance Computing in the AWS Cloud
Visit http:aws.amazon.com/hpc for more information about HPC on AWS.
High Performance Computing (HPC) allows scientists and engineers to solve complex science, engineering, and business problems using applications that require high bandwidth, low latency networking, and very high compute capabilities. AWS allows you to increase the speed of research by running high performance computing in the cloud and to reduce costs by providing Cluster Compute or Cluster GPU servers on-demand without large capital investments. You have access to a full-bisection, high bandwidth network for tightly-coupled, IO-intensive workloads, which enables you to scale out across thousands of cores for throughput-oriented applications.
HPC Market Update from IDC
In this deck from ISC 2015, Earl Joseph and Bob Sorensen from IDC provide an HPC Market Update.
Watch the video presentation: http://wp.me/p3RLHQ-emz
Sign up for our insideHPC Newsletter: http://insideHPC.com/newsletter
AWS re:Invent 2016: High Performance Computing on AWS (CMP207)
High performance computing in the cloud is enabling high scale compute- and graphics-intensive workloads across industries, ranging from aerospace, automotive, and manufacturing to life sciences, financial services, and energy. AWS provides application developers and end users with unprecedented computational power for massively parallel applications, in areas such as large-scale fluid and materials simulations, 3D content rendering, financial computing, and deep learning. This session provides an overview of HPC capabilities on AWS, describes the newest generations of accelerated computing instances (including P2), as well as highlighting customer and partner use-cases across industries.
Attendees learn about best practices for running HPC workflows in the cloud, including graphical pre- and post-processing, workflow automation, and optimization. Attendees also learn about new and emerging HPC use cases: in particular, deep learning training and inference, large-scale simulations, and high performance data analytics.
Viewers also liked (20)
Pengolahan Limbah Cair dengan metode Elektrokoagulasi
Pengolahan Limbah Cair dengan metode Elektrokoagulasi
Fast & Furious: building HPC solutions in a nutshell
Fast & Furious: building HPC solutions in a nutshell
ODCA Board Best Practice: High Performance Computing at BMW
ODCA Board Best Practice: High Performance Computing at BMW
Big Data and High Performance Computing Solutions in the AWS Cloud
Big Data and High Performance Computing Solutions in the AWS Cloud
Intro to High Performance Computing in the AWS Cloud
Intro to High Performance Computing in the AWS Cloud
AWS re:Invent 2016: High Performance Computing on AWS (CMP207)
AWS re:Invent 2016: High Performance Computing on AWS (CMP207)
Similar to Past, present, and future of HPC in life sciences
General Introduction to technologies that will be seen in the school
1. The document discusses principles of distributed computing technologies including service-oriented architectures, high-throughput computing, distributed data management, job submission and execution management, using distributed systems, higher-level APIs like OGSA-DAI and SAGA, and workflows.
2. It provides histories and visions for technologies like UNICORE, Condor, Globus, gLite, ARC, and P-GRADE portal.
3. Key principles covered include Web services, parallelism, interoperability, simplicity, scalability, and hiding complexity through high-level interfaces.
Hungarian ClusterGrid and its applications
This document summarizes the Hungarian ClusterGrid and its applications. ClusterGrid integrates individual clusters and supercomputers with grid middleware. It currently includes 1000 nodes across 32 cluster sites providing 22 TB of distributed storage. A new generation middleware called Grid Underground has been developed which uses a pure web services approach. Core services include security, job management, and storage. One application involved virtual screening of 8 million molecules to identify potential new medicines.
Accelerate Big Data Processing with High-Performance Computing Technologies
Learn about opportunities and challenges for accelerating big data middleware on modern high-performance computing (HPC) clusters by exploiting HPC technologies.
NIIF Grid Development portfolio
The document summarizes the NIIF Grid development portfolio, including its ClusterGrid infrastructure with 1000 computational nodes across 32 clusters. It describes the Grid Underground (GUG) middleware project, a new generation framework developed since 2006 using selected grid standards. GUG features a pure web services architecture, core services for information, storage, job management, security and monitoring, and has been tested successfully with real-life applications.
20160201_resume_Vladimir_Chesnokov
Vladimir V. Chesnokov is a software developer located in Novosibirsk, Russia seeking a challenging position. He has over 10 years of experience programming in C++, SQL, Java, Perl, and other languages. He is proficient in Linux, Windows, databases, web development, and more. His objective is to join a team of experienced engineers working on emerging technologies like multi-tier architectures and Internet applications.
Designing HPC, Deep Learning, and Cloud Middleware for Exascale Systems
In this deck from the Stanford HPC Conference, DK Panda from Ohio State University presents: Designing HPC, Deep Learning, and Cloud Middleware for Exascale Systems.
"This talk will focus on challenges in designing HPC, Deep Learning, and HPC Cloud middleware for Exascale systems with millions of processors and accelerators. For the HPC domain, we will discuss the challenges in designing runtime environments for MPI+X (PGAS-OpenSHMEM/UPC/CAF/UPC++, OpenMP and Cuda) programming models by taking into account support for multi-core systems (KNL and OpenPower), high networks, GPGPUs (including GPUDirect RDMA) and energy awareness. Features and sample performance numbers from MVAPICH2 libraries will be presented. For the Deep Learning domain, we will focus on popular Deep Learning framewords (Caffe, CNTK, and TensorFlow) to extract performance and scalability with MVAPICH2-GDR MPI library and RDMA-enabled Big Data stacks. Finally, we will outline the challenges in moving these middleware to the Cloud environments."
Watch the video: https://youtu.be/i2I6XqOAh_I
Learn more: http://web.cse.ohio-state.edu/~panda.2/
and
http://hpcadvisorycouncil.com
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Scallable Distributed Deep Learning on OpenPOWER systems
Overview of challenges being faced by the AI community to achieve high-performance, scalable and distributed DNN training on Modern HPC systems with both scale-up and scale-out strategies. After that, the talk will focus on a range of solutions being carried out in my group to address these challenges. The solutions will include: 1) MPI-driven Deep Learning, 2) Co-designing Deep Learning Stacks with High-Performance MPI, 3) Out-of-core DNN training, and 4) Hybrid (Data and Model) parallelism. Case studies to accelerate DNN training with popular frameworks like TensorFlow, PyTorch, MXNet and Caffe on modern HPC systems will be presented.
Pathways for EOSC-hub and MaX collaboration
1) The MaX Centre of Excellence aims to enable high-throughput materials design through automated simulations and tracking of provenance using the AiiDA platform.
2) AiiDA and the Materials Cloud platform allow over 10,000 simulations per day by automating workflows, tracking provenance to ensure reproducibility, and sharing data according to FAIR principles.
3) Potential areas for collaboration with EOSC include integrating AiiDA Lab and the Materials Cloud Archive, developing standardized workflows as services, and providing authentication and authorization through B2ACCESS and EGI Check-In.
NWU and HPC
The document discusses plans to establish an institutional high performance computing (HPC) facility at North-West University. It outlines the technical goals of building a Beowulf cluster to link existing departmental clusters and integrate with national and international computational grids. It also discusses management principles for the new HPC facility to ensure sustainability, efficiency, reliability, availability and high performance.
Best pratices at BGI for the Challenges in the Era of Big Genomics Data
BGI is the world's largest genome sequencing center, with over 150 sequencers and a sequencing throughput of 6 TB per day. It also has the largest computing and storage center for genomics in China, with over 20,000 CPU cores, 19 GPUs, 220+ teraflops of peak performance, and 17 petabytes of data storage. BGI faces challenges from the exponential growth of genomic data, complex data analysis processes, and widely distributed data. It addresses these challenges through solutions like high-speed data transfer, cloud computing platforms like EasyGenomics, and distributed algorithms and infrastructure using Hadoop and GPU acceleration.
H2020-AHTOOLS Use Case 3 Functional Design
Status of development of UC3 within the H2020 ECSEL AHTOOLS project. Traceability, Systems Engineering, Complex Systems.
optimizing_ceph_flash
Vijayendra Shamanna from SanDisk presented on optimizing the Ceph distributed storage system for all-flash architectures. Some key points:
1) Ceph is an open-source distributed storage system that provides file, block, and object storage interfaces. It operates by spreading data across multiple commodity servers and disks for high performance and reliability.
2) SanDisk has optimized various aspects of Ceph's software architecture and components like the messenger layer, OSD request processing, and filestore to improve performance on all-flash systems.
3) Testing showed the optimized Ceph configuration delivering over 200,000 IOPS and low latency with random 8K reads on an all-flash setup.
EGI Cloud Compute service for EOSC-hub
EGI Cloud Compute provides users with on-demand access to virtual machines and computing resources across multiple cloud providers through a unified interface. It offers various virtual machine configurations, integrated monitoring and accounting, and the ability to execute batch jobs and host long-running services. Documentation and tutorials are available to help users access and manage their resources through the service's APIs, command line tools, and graphical user interface.
Available HPC Resources at CSUC
The document provides information about available HPC resources at CSUC. It summarizes the hardware facilities which include the Canigó and Pirineus II clusters totaling 3,888 cores and 391 TFlops of computing power. It describes the working environment including the Slurm workload manager, storage units, and development tools available. It also outlines how users can access the services through RES projects, pricing, and the EuroCC Spain testbed for national HPC competence.
Designing High-Performance and Scalable Middleware for HPC, AI and Data Science
This document discusses designing high-performance middleware for HPC, AI, and data science applications. It provides an overview of the MVAPICH2 project, which develops an open-source MPI library supporting modern HPC architectures and networking technologies. MVAPICH2 aims to provide a converged software stack for HPC, deep learning, and data science through libraries like MVAPICH2, HiDL, and HiBD. The document outlines challenges in communication library design for exascale systems and MVAPICH2's architecture supporting programming models across domains.
Designing HPC & Deep Learning Middleware for Exascale Systems
DK Panda from Ohio State University presented this deck at the 2017 HPC Advisory Council Stanford Conference.
"This talk will focus on challenges in designing runtime environments for exascale systems with millions of processors and accelerators to support various programming models. We will focus on MPI, PGAS (OpenSHMEM, CAF, UPC and UPC++) and Hybrid MPI+PGAS programming models by taking into account support for multi-core, high-performance networks, accelerators (GPGPUs and Intel MIC), virtualization technologies (KVM, Docker, and Singularity), and energy-awareness. Features and sample performance numbers from the MVAPICH2 libraries will be presented."
Watch the video: http://wp.me/p3RLHQ-glW
Learn more: http://hpcadvisorycouncil.com
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
NSCC Training Introductory Class
The document provides an introduction to the National Supercomputing Centre (NSCC) high performance computing cluster. It describes the 1 petaflop system consisting of 1300 nodes and 13 petabytes of storage. The system uses PBS Pro for job scheduling and includes compilers, libraries, developer tools, and applications for engineering, science, and industry users from organizations such as A*STAR, NUS, and NTU.
Designing High performance & Scalable Middleware for HPC
The document discusses the design of high-performance middleware for HPC, AI, and data science applications. It describes challenges including supporting various programming models, applications, network technologies, and architectures. The MVAPICH project is presented as an open-source MPI library that supports these domains and has been downloaded over 1.5 million times. It provides optimized communication through features like GPU-direct support and improved nested datatype transfers.
Opentelemetry - From frontend to backend
This presentation has been presented at the "Vienna DevOps & Security Meetup" in 2021.
It discusses the state of monitoring, what Opentelemetry is and why should you care about it.
Concepts and basics are discussed and presented in a full example extracting traces, metrics and logs.
Demo: https://github.com/secustor/opentelemetry-meetup
Exploring emerging technologies in the HPC co-design space
This document discusses emerging technologies for high performance computing (HPC), focusing on heterogeneous computing and non-volatile memory. It provides an overview of HPC architectures past and present, highlighting the trend toward more heterogeneous systems using GPUs and other accelerators. The document discusses challenges for applications to adapt to these changing architectures. It also explores potential future technologies like 3D memory and discusses the Department of Energy's efforts in codesign centers to facilitate collaboration between application developers and emerging hardware.
Similar to Past, present, and future of HPC in life sciences (20)
General Introduction to technologies that will be seen in the school
General Introduction to technologies that will be seen in the school
Accelerate Big Data Processing with High-Performance Computing Technologies
Accelerate Big Data Processing with High-Performance Computing Technologies
Designing HPC, Deep Learning, and Cloud Middleware for Exascale Systems
Designing HPC, Deep Learning, and Cloud Middleware for Exascale Systems
Scallable Distributed Deep Learning on OpenPOWER systems
Scallable Distributed Deep Learning on OpenPOWER systems
Best pratices at BGI for the Challenges in the Era of Big Genomics Data
Best pratices at BGI for the Challenges in the Era of Big Genomics Data
Designing High-Performance and Scalable Middleware for HPC, AI and Data Science
Designing High-Performance and Scalable Middleware for HPC, AI and Data Science
Designing HPC & Deep Learning Middleware for Exascale Systems
Designing HPC & Deep Learning Middleware for Exascale Systems
Designing High performance & Scalable Middleware for HPC
Designing High performance & Scalable Middleware for HPC
Exploring emerging technologies in the HPC co-design space
Exploring emerging technologies in the HPC co-design space
Recently uploaded
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for life science domain, where you retriever information from biomedical knowledge graphs using LLMs to increase the accuracy and performance of generated answers
20240609 QFM020 Irresponsible AI Reading List May 2024
Everything I found interesting about the irresponsible use of machine intelligence in May 2024
Artificial Intelligence for XMLDevelopment
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
20240607 QFM018 Elixir Reading List May 2024
Everything I found interesting about the Elixir programming ecosystem in May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
Everything I found interesting about machines behaving intelligently during May 2024
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers:
* What is Vector Search?
* Importance and benefits of vector search
* Practical use cases across various industries
* Step-by-step implementation guide
* Live demos with code snippets
* Enhancing LLM capabilities with vector search
* Best practices and optimization strategies
Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications.
#MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology
AI-Powered Food Delivery Transforming App Development in Saudi Arabia.pdf
In this blog post, we'll delve into the intersection of AI and app development in Saudi Arabia, focusing on the food delivery sector. We'll explore how AI is revolutionizing the way Saudi consumers order food, how restaurants manage their operations, and how delivery partners navigate the bustling streets of cities like Riyadh, Jeddah, and Dammam. Through real-world case studies, we'll showcase how leading Saudi food delivery apps are leveraging AI to redefine convenience, personalization, and efficiency.
Choosing The Best AWS Service For Your Website + API.pptx
Have you ever been confused by the myriad of choices offered by AWS for hosting a website or an API?
Lambda, Elastic Beanstalk, Lightsail, Amplify, S3 (and more!) can each host websites + APIs. But which one should we choose?
Which one is cheapest? Which one is fastest? Which one will scale to meet our needs?
Join me in this session as we dive into each AWS hosting service to determine which one is best for your scenario and explain why!
Infrastructure Challenges in Scaling RAG with Custom AI models
Building Retrieval-Augmented Generation (RAG) systems with open-source and custom AI models is a complex task. This talk explores the challenges in productionizing RAG systems, including retrieval performance, response synthesis, and evaluation. We’ll discuss how to leverage open-source models like text embeddings, language models, and custom fine-tuned models to enhance RAG performance. Additionally, we’ll cover how BentoML can help orchestrate and scale these AI components efficiently, ensuring seamless deployment and management of RAG systems in the cloud.
Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays, June 2024 with Maria Copot 20
Building Production Ready Search Pipelines with Spark and Milvus
Spark is the widely used ETL tool for processing, indexing and ingesting data to serving stack for search. Milvus is the production-ready open-source vector database. In this talk we will show how to use Spark to process unstructured data to extract vector representations, and push the vectors to Milvus vector database for search serving.
UiPath Test Automation using UiPath Test Suite series, part 6
Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI.
UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities.
Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes.
What will you get from this session?
1. Insights into integrating generative AI.
2. Understanding how this integration enhances test automation within the UiPath platform
3. Practical demonstrations
4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath
Topics covered:
What is generative AI
Test Automation with generative AI and Open AI.
UiPath integration with generative AI
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Things to Consider When Choosing a Website Developer for your Website | FODUU
Choosing the right website developer is crucial for your business. This article covers essential factors to consider, including experience, portfolio, technical skills, communication, pricing, reputation & reviews, cost and budget considerations and post-launch support. Make an informed decision to ensure your website meets your business goals.
OpenID AuthZEN Interop Read Out - Authorization
During Identiverse 2024 and EIC 2024, members of the OpenID AuthZEN WG got together and demoed their authorization endpoints conforming to the AuthZEN API
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift
Overview
Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices.
Key Topics Covered
1. Introduction to Anomaly Detection
- Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems.
2. Understanding Edge (IoT)
- Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source.
3. What is ArgoCD?
- Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices.
4. Deployment Using ArgoCD for Edge Devices
- Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD.
5. Introduction to Apache Kafka and S3
- Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions.
6. Viewing Kafka Messages in the Data Lake
- Learn how to view and analyze Kafka messages stored in a data lake for better insights.
7. What is Prometheus?
- Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices.
8. Monitoring Application Metrics with Prometheus
- Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system.
9. What is Camel K?
- Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes.
10. Configuring Camel K Integrations for Data Pipelines
- Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow.
11. What is a Jupyter Notebook?
- Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text.
12. Jupyter Notebooks with Code Examples
- Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.
Microsoft - Power Platform_G.Aspiotis.pdf
Revolutionizing Application Development
with AI-powered low-code, presentation by George Aspiotis, Sr. Partner Development Manager, Microsoft
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
Imagine a world where machines not only perform tasks but also learn, adapt, and make decisions. This is the promise of Artificial Intelligence (AI), a technology that's not just enhancing our lives but revolutionizing entire industries.
Essentials of Automations: The Art of Triggers and Actions in FME
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
Recently uploaded (20)
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
AI-Powered Food Delivery Transforming App Development in Saudi Arabia.pdf
AI-Powered Food Delivery Transforming App Development in Saudi Arabia.pdf
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
Infrastructure Challenges in Scaling RAG with Custom AI models
Infrastructure Challenges in Scaling RAG with Custom AI models
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
Things to Consider When Choosing a Website Developer for your Website | FODUU
Things to Consider When Choosing a Website Developer for your Website | FODUU
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
Past, present, and future of HPC in life sciences
- 1. The past, present, and future of HPC in life sciences Erich Birngruber, Ümit Seren Gregor Mendel Institute for Molecular Plant Biology (GMI) AHPC17
- 2. Who we are - Basic research institute in plant sciences - 9 independent research groups - Employees 100 + 20 (scientific + admin) - HPC Operations Team: 2 + 1 (engineer + lead)
- 3. Past: Beginnings as traditional HPC Scientific computing at GMI - Started in 2010 - SGI ICE-X since 2013 (MENDEL) (72 nodes, 144 today) - SGI UV2000 - Rich software environment (EasyBuild, lmod) - Keeping up with current developments Machine specs 3 generations of nodes: - 72x 16c E5-2609, 192gb mem - 18x 20c E5-2680, 256gb mem - 54x 24c E5-2650, 256gb mem, 230gb ssd UV2000: 96c E5-4617, 2tb mem IB FDR interconnect (1 fabric) Storage: Lustre 300tb, NetApp >1pb
- 5. Present: GMI site specifics - Services: customers are biologists - On campus initial training - Consulting and support (w/ ticket system, intranet wiki) - Software installations - Provided as modules: different versions, repeatability - This is getting harder with the demand for more complex software - Monitoring software usage
- 6. Present: Monitoring software usage - Software in env modules - 460 software packages in 1297 versions - Monitoring module usage (load, unload) - Reporting by user, job, project
- 7. Present: Monitoring system activity Monitoring and metrics The foundation for all future decisions - Resource consumption - Capacity planning - Software, technology usage - Auditing Alerting
- 8. Nodestatus
- 9. Jobresources
- 10. Present: Applications & Appliances Phenobox (in development) - Web-interface, API - MySQL (DB) - DSLR, RaspberryPi - HPC (computer vision, storage) GWA-Portal (https://gwas.gmi.oeaw.ac.at) - Web-interface, API - Elasticsearch (fulltext search) - PostgreSQL (DB) - Docker (Python microservices) - HPC (analysis, storage) Galaxy (https://galaxyproject.org/) - Web-interface, API - MySQL (DB) - Visualization - HPC (analysis, storage) PacBio SMRT Link (https://github.com/PacificBiosciences/SMRT-Link) - Web-interface, API - MySQL (DB) - HPC (analysis, storage) Own developments: 3rd party software:
- 11. Present: new developments Deployment of OpenStack (IaaS): - Cross-vendor open source project - On-premises cloud - Provision VMs and containers - Deploy classic application services - Enables self-service for customers Consequences: - More heterogeneous use-cases - Customer base is increasing - Non-human “customers” of HPC - Services are more complex and distributed over subsystems
- 14. Present: Problem 1: maintenance - VMs are difficult to maintain - Wrong abstraction for the use-case - What is the next step? - Containers? - Container Orchestration Engines? - Provide Software as a Service (SaaS)? Fact is: the field is evolving
- 16. Future: Problem 2: integration Applications sit on different islands: HPC vs. Cloud Drawbacks: - Hard to maintain (infra) - Hard to debug (app) Vision: converged compute platform. Unified infrastructure to schedule all types of tasks New challenges: - Networking - Storage - IDM - Accounting What do others do? Container Orchestration Engine (Google Kubernetes, Docker Swarm, Apache Mesos) First steps: - Containers for HPC - Biocontainers http://biocontainers.pro - Singularity http://singularity.lbl.gov - Current status: test deployment
- 17. Contact / References: Erich Birngruber <erich.birngruber@gmi.oeaw.ac.at>, @ebirn Ümit Seren <uemit.seren@gmi.oeaw.ac.at>, @timeu_s GMI on Github: https://github.com/Gregor-Mendel-Institute Total recall: holistic metrics for broad systems performance and user experience visibility in a data-intensive computing environment https://dl.acm.org/citation.cfm?id=2835001
- 18. Acknowledgements Gregor Mendel Institute of Molecular Plant Biology Dr Bohr-Gasse 3 1030 Vienna, Austria
- 19. EOF