Singularity: The Inner Workings of Securely Running User Containers on HPC Systems
•
1 like•1,411 views
“Singularity is an open source container solution being developed specifically for HPC environments. With Singularity, HPC users can safely bring their own execution environments to the cluster. Unlike other container solutions,Singularity does not require root level permissions to run containers, which allows users to freely control what software stack they wish to use.Provisioning of a container image can be done locally on the user’s machine or on Singularity Hub. The resulting image can then be securely executed on any machine with Singularity installed. Reproduction of results has never been easier: a user can now share a single Singularity image file that will ensure a consistent execution environment wherever it is run. This presentation will provide an in-depth look at how Singularity is able to securely run user containers on HPC systems. After a brief introduction to Singularity and its relationship to other container solutions, the details of Singularity’s runtime will be explored. The way that Singularity leverages Linux features such as namespaces, bind mounts, and SUID binaries will be discussed in further detail as well.”
Report
Share
Report
Share
Download to read offline
Recommended
Introduction to Docker Containers - Docker Captain
This document provides an introduction to Docker. It begins with an overview of the shift from monolithic to microservices architecture and how Docker addresses the complexity problems that arise. Docker is described as a tool that packages applications and dependencies into standardized units called containers that can run on any Linux server. Key differences between Docker containers and traditional virtual machines are outlined. The document then covers Docker concepts like images, containers, and the Docker Engine. It demonstrates the Docker build, ship, and run workflow and introduces common Docker commands and tools.
Docker 101 : Introduction to Docker and Containers
This document provides an introduction to Docker and containers. It discusses why containers are useful for software deployment given changes in the industry. Containers provide lightweight isolation of applications and their dependencies. Docker is a tool that manages containers running on the same operating system kernel. Key Docker components include the client, server, images, and containers. Popular use cases of Docker include Google running over a billion containers per week and Finnish Railways saving 50% of cloud costs with Docker.
Containers and Docker
This document discusses containers and Docker. It begins by explaining that cloud infrastructures comprise virtual resources like compute and storage nodes that are administered through software. Docker is introduced as a standard way to package code and dependencies into portable containers that can run anywhere. Key benefits of Docker include increased efficiency, consistency, and security compared to traditional virtual machines. Some weaknesses are that Docker may not be suitable for all applications and large container management can be difficult. Interesting uses of Docker include malware analysis sandboxes, isolating Skype sessions, and managing Raspberry Pi clusters with Docker Swarm.
Introduction to Docker
Docker is a tool that allows applications to run in isolated containers to make them portable and consistent across environments. It provides benefits like easy developer onboarding, eliminating application conflicts, and consistent deployments. Docker tools include the Docker Engine, Docker Client, Docker Compose, and Docker Hub. Key concepts are images which are templates for containers, and containers which are where the code runs based on an image. The document outlines how to build custom images from Dockerfiles, communicate between containers using linking or networks, and deploy containers using Docker Compose or in the cloud.
Docker Basics
Docker is an open source containerization platform that allows applications to be easily deployed and run across various operating systems and cloud environments. It allows applications and their dependencies to be packaged into standardized executable units called containers that can be run anywhere. Containers are more portable and provide better isolation than virtual machines, making them useful for microservices architecture, continuous integration/deployment, and cloud-native applications.
Docker.pptx
This document provides an introduction and overview of Docker and containers. It discusses what containers are, how they differ from virtual machines, and how Docker works. Key points covered include common Docker commands, the Docker architecture, building images with Dockerfiles, and using Docker Compose to run multi-container applications. The benefits of containers for streamlining deployment and rapid scaling are also highlighted.
Docker Container Introduction
This document discusses Docker containers and provides an introduction. It begins with an overview of Docker and how it uses containerization technology like Linux containers and namespaces to provide isolation. It describes how Docker images are composed of layers and how containers run from these images. The document then explains benefits of Docker like portability and ease of scaling. It provides details on Docker architecture and components like images, registries and containers. Finally, it demonstrates how to simply run a Docker container with a command.
Docker internals
This document discusses Docker internals and components. It covers:
1. Docker provides build once, configure once capabilities to deploy applications everywhere reliably, consistently, efficiently and cheaply.
2. Docker components include the Docker daemon, libcontainer, cgroups, namespaces, AUFS/BTRFS/dm-thinp, and the kernel-userspace interface.
3. Docker uses filesystem isolation through layering, copy-on-write, caching and differencing using union filesystems like AUFS to provide efficient sharing of files between containers.
Recommended
Introduction to Docker Containers - Docker Captain
This document provides an introduction to Docker. It begins with an overview of the shift from monolithic to microservices architecture and how Docker addresses the complexity problems that arise. Docker is described as a tool that packages applications and dependencies into standardized units called containers that can run on any Linux server. Key differences between Docker containers and traditional virtual machines are outlined. The document then covers Docker concepts like images, containers, and the Docker Engine. It demonstrates the Docker build, ship, and run workflow and introduces common Docker commands and tools.
Docker 101 : Introduction to Docker and Containers
This document provides an introduction to Docker and containers. It discusses why containers are useful for software deployment given changes in the industry. Containers provide lightweight isolation of applications and their dependencies. Docker is a tool that manages containers running on the same operating system kernel. Key Docker components include the client, server, images, and containers. Popular use cases of Docker include Google running over a billion containers per week and Finnish Railways saving 50% of cloud costs with Docker.
Containers and Docker
This document discusses containers and Docker. It begins by explaining that cloud infrastructures comprise virtual resources like compute and storage nodes that are administered through software. Docker is introduced as a standard way to package code and dependencies into portable containers that can run anywhere. Key benefits of Docker include increased efficiency, consistency, and security compared to traditional virtual machines. Some weaknesses are that Docker may not be suitable for all applications and large container management can be difficult. Interesting uses of Docker include malware analysis sandboxes, isolating Skype sessions, and managing Raspberry Pi clusters with Docker Swarm.
Introduction to Docker
Docker is a tool that allows applications to run in isolated containers to make them portable and consistent across environments. It provides benefits like easy developer onboarding, eliminating application conflicts, and consistent deployments. Docker tools include the Docker Engine, Docker Client, Docker Compose, and Docker Hub. Key concepts are images which are templates for containers, and containers which are where the code runs based on an image. The document outlines how to build custom images from Dockerfiles, communicate between containers using linking or networks, and deploy containers using Docker Compose or in the cloud.
Docker Basics
Docker is an open source containerization platform that allows applications to be easily deployed and run across various operating systems and cloud environments. It allows applications and their dependencies to be packaged into standardized executable units called containers that can be run anywhere. Containers are more portable and provide better isolation than virtual machines, making them useful for microservices architecture, continuous integration/deployment, and cloud-native applications.
Docker.pptx
This document provides an introduction and overview of Docker and containers. It discusses what containers are, how they differ from virtual machines, and how Docker works. Key points covered include common Docker commands, the Docker architecture, building images with Dockerfiles, and using Docker Compose to run multi-container applications. The benefits of containers for streamlining deployment and rapid scaling are also highlighted.
Docker Container Introduction
This document discusses Docker containers and provides an introduction. It begins with an overview of Docker and how it uses containerization technology like Linux containers and namespaces to provide isolation. It describes how Docker images are composed of layers and how containers run from these images. The document then explains benefits of Docker like portability and ease of scaling. It provides details on Docker architecture and components like images, registries and containers. Finally, it demonstrates how to simply run a Docker container with a command.
Docker internals
This document discusses Docker internals and components. It covers:
1. Docker provides build once, configure once capabilities to deploy applications everywhere reliably, consistently, efficiently and cheaply.
2. Docker components include the Docker daemon, libcontainer, cgroups, namespaces, AUFS/BTRFS/dm-thinp, and the kernel-userspace interface.
3. Docker uses filesystem isolation through layering, copy-on-write, caching and differencing using union filesystems like AUFS to provide efficient sharing of files between containers.
kubernetes, pourquoi et comment
Docker nous permet de déployer nos applications dans des conteneurs. Du coup notre infrastructure se retrouve divisée dans différents conteneurs, un pour la base de données, un pour le front, un pour le backend. Voir même une division en services lorsque l’on est dans une approche micro-services.
Mais comment faire communiquer ces différents conteneurs? Comment orchestrer un cluster de conteneurs? Kubernetes est une réponse à ces questions.
presentation on Docker
Docker is an open platform for developing, shipping, and running distributed applications. It allows applications to be shipped and run in lightweight containers that can run on any Linux server. Docker uses operating-system-level virtualization and cgroups isolation to deliver lightweight containers quickly. Key features of Docker include portability, lightweight containers that share resources and isolate processes, and automated workflows.
Docker introduction (1)
Docker is an open platform for developing, shipping, and running applications. It allows packaging applications into standardized units for software called containers that can run on any infrastructure. The key components of Docker include images, containers, a client-server architecture using Docker Engine, and registries for storing images. Images act as templates for creating containers, which are run-time instances of images. Docker provides portability and isolation of applications using containers.
Docker Tutorial For Beginners | What Is Docker And How It Works? | Docker Tut...
This presentation about Docker tutorial will help you understand what is Docker, advantages of Docker, how does Docker work, components of Docker, virtual machine vs Docker, advanced concepts in Docker, basic Docker commands along with a demo. A Docker is an OS-level virtualization software that enables developers and IT administrators to create, deploy and run applications in a Docker container with all their dependencies. It is said to be a very light-weight software container and containerization platform. Docker engine or Docker is a client-server application that builds and executes using Docker components. Rapid deployment, portability, better efficiency, faster configuration, scalability, security are some of the advantages you get by using Docker.
Below topics are explained in this Docker presentation:
1. Virtual machine vs Docker
2. What is Docker?
3. Advantages of Docker
4. How does Docker work?
5. Components of Docker
6. Advanced concepts in Docker
7. Basic Docker commands
Why learn DevOps?
Simplilearn’s DevOps training course is designed to help you become a DevOps practitioner and apply the latest in DevOps methodology to automate your software development lifecycle right out of the class. You will master configuration management; continuous integration deployment, delivery and monitoring using DevOps tools such as Git, Docker, Jenkins, Puppet and Nagios in a practical, hands-on and interactive approach. The DevOps training course focuses heavily on the use of Docker containers, a technology that is revolutionizing the way apps are deployed in the cloud today and is a critical skillset to master in the cloud age.
After completing the DevOps training course you will achieve hands-on expertise in various aspects of the DevOps delivery model. The practical learning outcomes of this Devops training course are:
An understanding of DevOps and the modern DevOps toolsets
The ability to automate all aspects of a modern code delivery and deployment pipeline using:
1. Source code management tools
2. Build tools
3. Test automation tools
4. Containerization through Docker
5. Configuration management tools
6. Monitoring tools
Who should take this course?
DevOps career opportunities are thriving worldwide. DevOps was featured as one of the 11 best jobs in America for 2017, according to CBS News, and data from Payscale.com shows that DevOps Managers earn as much as $122,234 per year, with DevOps engineers making as much as $151,461. DevOps jobs are the third-highest tech role ranked by employer demand on Indeed.com but have the second-highest talent deficit.
This DevOps training course will be of benefit the following professional roles:
1. Software Developers
2. Technical Project Managers
3. Architects
4. Operations Support
5. Deployment engineers
6. IT managers
7. Development managers
You can learn more at https://www.simplilearn.com/cloud-computing/devops-practitioner-certification-training
Kubernetes Architecture
In this session, we will discuss the architecture of a Kubernetes cluster. we will go through all the master and worker components of a kubernetes cluster. We will also discuss the basic terminology of Kubernetes cluster such as Pods, Deployments, Service etc. We will also cover networking inside Kuberneets. In the end, we will discuss options available for the setup of a Kubernetes cluster.
Midi technique - présentation docker
J'ai retrouvé cette ancienne présentation que j'avais fait qui date de janvier 2016.
Docker
Docker allows building, shipping, and running applications in portable containers. It packages an application with all its dependencies into a standardized unit for software development. Major cloud providers and companies support and use Docker in production. Containers are more lightweight and efficient than virtual machines, providing faster launch times and allowing thousands to run simultaneously on the same server. Docker simplifies distributing applications and ensures a consistent environment.
Introduction to High-Performance Computing (HPC) Containers and Singularity*
Members of the Lawrence Berkley National Laboratory (LBNL) introduce Singularity*, review use cases, and discuss future development ideas.
What Is A Docker Container? | Docker Container Tutorial For Beginners| Docker...
This presentation on Docker Container will help you understand what is Docker, the architecture of Docker, what is a Docker Container, how to create a Docker Container, benefits of Docker Container, basic commands of Containers and you will also see a demo on creating Docker Container. Docker is a very lightweight software container and containerization platform. Docker containers provide a way to run software in isolation. It is an open source platform that helps to package an application and its dependencies into a Docker container for the development and deployment of software and a Docker COntainer is a portable executable package which includes applications and their dependencies. With Docker Containers, applications can work efficiently in different computer environments.
Below DevOps tools are explained in this Docker Container presentation:
1. What is Docker?
2. The architecture of Docker?
3. What is a Docker Container?
4. How to create a Docker Container?
5. Benefits of Docker Containers
6. Basic commands of Containers
Simplilearn's DevOps Certification Training Course will prepare you for a career in DevOps, the fast-growing field that bridges the gap between software developers and operations. You’ll become an expert in the principles of continuous development and deployment, automation of configuration management, inter-team collaboration and IT service agility, using modern DevOps tools such as Git, Docker, Jenkins, Puppet and Nagios. DevOps jobs are highly paid and in great demand, so start on your path today.
Why learn DevOps?
Simplilearn’s DevOps training course is designed to help you become a DevOps practitioner and apply the latest in DevOps methodology to automate your software development lifecycle right out of the class. You will master configuration management; continuous integration deployment, delivery and monitoring using DevOps tools such as Git, Docker, Jenkins, Puppet and Nagios in a practical, hands-on and interactive approach. The DevOps training course focuses heavily on the use of Docker containers, a technology that is revolutionizing the way apps are deployed in the cloud today and is a critical skillset to master in the cloud age.
After completing the DevOps training course you will achieve hands-on expertise in various aspects of the DevOps delivery model. The practical learning outcomes of this Devops training course are:
An understanding of DevOps and the modern DevOps toolsets
The ability to automate all aspects of a modern code delivery and deployment pipeline using:
1. Source code management tools
2. Build tools
3. Test automation tools
4. Containerization through Docker
5. Configuration management tools
6. Monitoring tools
DevOps jobs are the third-highest tech role ranked by employer demand on Indeed.com but have the second-highest talent deficit.
Learn more at https://www.simplilearn.com/cloud-computing/devops-practitioner-certification-training
Introduction to Docker
Docker allows applications to be packaged with all their dependencies and run consistently across computing environments. It provides isolation, security and portability for applications. This document discusses setting up an Eh Avatar application to run in Docker containers for Postgres, Redis and the application itself. It covers bringing up the dependency containers, building a custom Docker image for the application, and using Docker Compose to define and run the multi-container application. While this provides an introduction, there is still more to learn about optimizing Docker usage and avoiding common pitfalls.
Introduction to Kubernetes
Soft Introduction to Google's framework for taming containers in the cloud. For devs and architects that they just enter the world of cloud, microservices and containers
What is Docker
Docker allows you to package applications with their dependencies into standardized units called containers that can run on any Linux server. Containers are more portable and efficient than virtual machines, starting in milliseconds and using less disk space. Docker makes it easy to deploy and run applications without having to rebuild environments and guarantees that an application will run the same regardless of the infrastructure it is running on.
Virtual Machines and Docker
Presented Docker in 15 minutes with two of my classmates at school.
Presentation covering topics:
Virtualization
Virtual Machines
Container Technology (Docker)
Docker Compose
Docker Swarm
The demo can be found at:
https://github.com/DanishKhakwani/SimpleDockerDemo
Docker swarm introduction
Docker Swarm allows managing multiple Docker hosts as a single virtual Docker engine. The presenter demonstrates setting up a traditional Docker Swarm cluster with an external key-value store and load balancer. SwarmKit provides the core components of Docker Swarm as standalone binaries. Docker Swarm Mode is integrated directly into Docker Engine 1.12 and later, providing built-in orchestration without external components. The presenter then demonstrates a tutorial using Docker Swarm Mode to deploy a multi-container voting application across 3 Docker hosts and scale the service.
Docker and kubernetes
Docker allows building portable software that can run anywhere by packaging an application and its dependencies in a standardized unit called a container. Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. It groups containers that make up an application into logical units for easy management and discovery. Kubernetes can replicate containers, provide load balancing, coordinate updates between containers, and ensure availability. Defining applications as Kubernetes resources allows them to be deployed and updated easily across a cluster.
Rancher Rodeo 13 mai 2022
La présentation de notre rodéo du 13 mai.
Prochain rodéo le 1 juillet. Inscription sur https://more.suse.com/rancher-rodeo-fr-july.html
What is Docker | Docker Tutorial for Beginners | Docker Container | DevOps To...
This DevOps Docker Tutorial on what is docker ( Docker Tutorial Blog Series: https://goo.gl/32kupf ) will help you understand how to use Docker Hub, Docker Images, Docker Container & Docker Compose. This tutorial explains Docker's working Architecture and Docker Engine in detail. This Docker tutorial also includes a Hands-On session around Docker by the end of which you will learn to pull a centos Docker Image and spin your own Docker Container. You will also see how to launch multiple docker containers using Docker Compose. Finally, it will also tell you the role Docker plays in the DevOps life-cycle.
The Hands-On session is performed on an Ubuntu-64bit machine in which Docker is installed.
Linux 开源操作系统发展新趋势
A guest lecture at National University of Defense Technology (NUDT) in 2016 to postgraduate students in China about emerging technologies in the Linux operating system.
Survey of Container Build Tools
A survey of problems involved in building containers and build tools such as:
buildah
nixos-container
ansible-container
Smith
Distroless
Buildkit
Source to Image (s2i)
Habitat
More Related Content
What's hot
kubernetes, pourquoi et comment
Docker nous permet de déployer nos applications dans des conteneurs. Du coup notre infrastructure se retrouve divisée dans différents conteneurs, un pour la base de données, un pour le front, un pour le backend. Voir même une division en services lorsque l’on est dans une approche micro-services.
Mais comment faire communiquer ces différents conteneurs? Comment orchestrer un cluster de conteneurs? Kubernetes est une réponse à ces questions.
presentation on Docker
Docker is an open platform for developing, shipping, and running distributed applications. It allows applications to be shipped and run in lightweight containers that can run on any Linux server. Docker uses operating-system-level virtualization and cgroups isolation to deliver lightweight containers quickly. Key features of Docker include portability, lightweight containers that share resources and isolate processes, and automated workflows.
Docker introduction (1)
Docker is an open platform for developing, shipping, and running applications. It allows packaging applications into standardized units for software called containers that can run on any infrastructure. The key components of Docker include images, containers, a client-server architecture using Docker Engine, and registries for storing images. Images act as templates for creating containers, which are run-time instances of images. Docker provides portability and isolation of applications using containers.
Docker Tutorial For Beginners | What Is Docker And How It Works? | Docker Tut...
This presentation about Docker tutorial will help you understand what is Docker, advantages of Docker, how does Docker work, components of Docker, virtual machine vs Docker, advanced concepts in Docker, basic Docker commands along with a demo. A Docker is an OS-level virtualization software that enables developers and IT administrators to create, deploy and run applications in a Docker container with all their dependencies. It is said to be a very light-weight software container and containerization platform. Docker engine or Docker is a client-server application that builds and executes using Docker components. Rapid deployment, portability, better efficiency, faster configuration, scalability, security are some of the advantages you get by using Docker.
Below topics are explained in this Docker presentation:
1. Virtual machine vs Docker
2. What is Docker?
3. Advantages of Docker
4. How does Docker work?
5. Components of Docker
6. Advanced concepts in Docker
7. Basic Docker commands
Why learn DevOps?
Simplilearn’s DevOps training course is designed to help you become a DevOps practitioner and apply the latest in DevOps methodology to automate your software development lifecycle right out of the class. You will master configuration management; continuous integration deployment, delivery and monitoring using DevOps tools such as Git, Docker, Jenkins, Puppet and Nagios in a practical, hands-on and interactive approach. The DevOps training course focuses heavily on the use of Docker containers, a technology that is revolutionizing the way apps are deployed in the cloud today and is a critical skillset to master in the cloud age.
After completing the DevOps training course you will achieve hands-on expertise in various aspects of the DevOps delivery model. The practical learning outcomes of this Devops training course are:
An understanding of DevOps and the modern DevOps toolsets
The ability to automate all aspects of a modern code delivery and deployment pipeline using:
1. Source code management tools
2. Build tools
3. Test automation tools
4. Containerization through Docker
5. Configuration management tools
6. Monitoring tools
Who should take this course?
DevOps career opportunities are thriving worldwide. DevOps was featured as one of the 11 best jobs in America for 2017, according to CBS News, and data from Payscale.com shows that DevOps Managers earn as much as $122,234 per year, with DevOps engineers making as much as $151,461. DevOps jobs are the third-highest tech role ranked by employer demand on Indeed.com but have the second-highest talent deficit.
This DevOps training course will be of benefit the following professional roles:
1. Software Developers
2. Technical Project Managers
3. Architects
4. Operations Support
5. Deployment engineers
6. IT managers
7. Development managers
You can learn more at https://www.simplilearn.com/cloud-computing/devops-practitioner-certification-training
Kubernetes Architecture
In this session, we will discuss the architecture of a Kubernetes cluster. we will go through all the master and worker components of a kubernetes cluster. We will also discuss the basic terminology of Kubernetes cluster such as Pods, Deployments, Service etc. We will also cover networking inside Kuberneets. In the end, we will discuss options available for the setup of a Kubernetes cluster.
Midi technique - présentation docker
J'ai retrouvé cette ancienne présentation que j'avais fait qui date de janvier 2016.
Docker
Docker allows building, shipping, and running applications in portable containers. It packages an application with all its dependencies into a standardized unit for software development. Major cloud providers and companies support and use Docker in production. Containers are more lightweight and efficient than virtual machines, providing faster launch times and allowing thousands to run simultaneously on the same server. Docker simplifies distributing applications and ensures a consistent environment.
Introduction to High-Performance Computing (HPC) Containers and Singularity*
Members of the Lawrence Berkley National Laboratory (LBNL) introduce Singularity*, review use cases, and discuss future development ideas.
What Is A Docker Container? | Docker Container Tutorial For Beginners| Docker...
This presentation on Docker Container will help you understand what is Docker, the architecture of Docker, what is a Docker Container, how to create a Docker Container, benefits of Docker Container, basic commands of Containers and you will also see a demo on creating Docker Container. Docker is a very lightweight software container and containerization platform. Docker containers provide a way to run software in isolation. It is an open source platform that helps to package an application and its dependencies into a Docker container for the development and deployment of software and a Docker COntainer is a portable executable package which includes applications and their dependencies. With Docker Containers, applications can work efficiently in different computer environments.
Below DevOps tools are explained in this Docker Container presentation:
1. What is Docker?
2. The architecture of Docker?
3. What is a Docker Container?
4. How to create a Docker Container?
5. Benefits of Docker Containers
6. Basic commands of Containers
Simplilearn's DevOps Certification Training Course will prepare you for a career in DevOps, the fast-growing field that bridges the gap between software developers and operations. You’ll become an expert in the principles of continuous development and deployment, automation of configuration management, inter-team collaboration and IT service agility, using modern DevOps tools such as Git, Docker, Jenkins, Puppet and Nagios. DevOps jobs are highly paid and in great demand, so start on your path today.
Why learn DevOps?
Simplilearn’s DevOps training course is designed to help you become a DevOps practitioner and apply the latest in DevOps methodology to automate your software development lifecycle right out of the class. You will master configuration management; continuous integration deployment, delivery and monitoring using DevOps tools such as Git, Docker, Jenkins, Puppet and Nagios in a practical, hands-on and interactive approach. The DevOps training course focuses heavily on the use of Docker containers, a technology that is revolutionizing the way apps are deployed in the cloud today and is a critical skillset to master in the cloud age.
After completing the DevOps training course you will achieve hands-on expertise in various aspects of the DevOps delivery model. The practical learning outcomes of this Devops training course are:
An understanding of DevOps and the modern DevOps toolsets
The ability to automate all aspects of a modern code delivery and deployment pipeline using:
1. Source code management tools
2. Build tools
3. Test automation tools
4. Containerization through Docker
5. Configuration management tools
6. Monitoring tools
DevOps jobs are the third-highest tech role ranked by employer demand on Indeed.com but have the second-highest talent deficit.
Learn more at https://www.simplilearn.com/cloud-computing/devops-practitioner-certification-training
Introduction to Docker
Docker allows applications to be packaged with all their dependencies and run consistently across computing environments. It provides isolation, security and portability for applications. This document discusses setting up an Eh Avatar application to run in Docker containers for Postgres, Redis and the application itself. It covers bringing up the dependency containers, building a custom Docker image for the application, and using Docker Compose to define and run the multi-container application. While this provides an introduction, there is still more to learn about optimizing Docker usage and avoiding common pitfalls.
Introduction to Kubernetes
Soft Introduction to Google's framework for taming containers in the cloud. For devs and architects that they just enter the world of cloud, microservices and containers
What is Docker
Docker allows you to package applications with their dependencies into standardized units called containers that can run on any Linux server. Containers are more portable and efficient than virtual machines, starting in milliseconds and using less disk space. Docker makes it easy to deploy and run applications without having to rebuild environments and guarantees that an application will run the same regardless of the infrastructure it is running on.
Virtual Machines and Docker
Presented Docker in 15 minutes with two of my classmates at school.
Presentation covering topics:
Virtualization
Virtual Machines
Container Technology (Docker)
Docker Compose
Docker Swarm
The demo can be found at:
https://github.com/DanishKhakwani/SimpleDockerDemo
Docker swarm introduction
Docker Swarm allows managing multiple Docker hosts as a single virtual Docker engine. The presenter demonstrates setting up a traditional Docker Swarm cluster with an external key-value store and load balancer. SwarmKit provides the core components of Docker Swarm as standalone binaries. Docker Swarm Mode is integrated directly into Docker Engine 1.12 and later, providing built-in orchestration without external components. The presenter then demonstrates a tutorial using Docker Swarm Mode to deploy a multi-container voting application across 3 Docker hosts and scale the service.
Docker and kubernetes
Docker allows building portable software that can run anywhere by packaging an application and its dependencies in a standardized unit called a container. Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. It groups containers that make up an application into logical units for easy management and discovery. Kubernetes can replicate containers, provide load balancing, coordinate updates between containers, and ensure availability. Defining applications as Kubernetes resources allows them to be deployed and updated easily across a cluster.
Rancher Rodeo 13 mai 2022
La présentation de notre rodéo du 13 mai.
Prochain rodéo le 1 juillet. Inscription sur https://more.suse.com/rancher-rodeo-fr-july.html
What is Docker | Docker Tutorial for Beginners | Docker Container | DevOps To...
This DevOps Docker Tutorial on what is docker ( Docker Tutorial Blog Series: https://goo.gl/32kupf ) will help you understand how to use Docker Hub, Docker Images, Docker Container & Docker Compose. This tutorial explains Docker's working Architecture and Docker Engine in detail. This Docker tutorial also includes a Hands-On session around Docker by the end of which you will learn to pull a centos Docker Image and spin your own Docker Container. You will also see how to launch multiple docker containers using Docker Compose. Finally, it will also tell you the role Docker plays in the DevOps life-cycle.
The Hands-On session is performed on an Ubuntu-64bit machine in which Docker is installed.
What's hot (20)
Docker Tutorial For Beginners | What Is Docker And How It Works? | Docker Tut...
Docker Tutorial For Beginners | What Is Docker And How It Works? | Docker Tut...
Introduction to High-Performance Computing (HPC) Containers and Singularity*
Introduction to High-Performance Computing (HPC) Containers and Singularity*
What Is A Docker Container? | Docker Container Tutorial For Beginners| Docker...
What Is A Docker Container? | Docker Container Tutorial For Beginners| Docker...
What is Docker | Docker Tutorial for Beginners | Docker Container | DevOps To...
What is Docker | Docker Tutorial for Beginners | Docker Container | DevOps To...
Similar to Singularity: The Inner Workings of Securely Running User Containers on HPC Systems
Linux 开源操作系统发展新趋势
A guest lecture at National University of Defense Technology (NUDT) in 2016 to postgraduate students in China about emerging technologies in the Linux operating system.
Survey of Container Build Tools
A survey of problems involved in building containers and build tools such as:
buildah
nixos-container
ansible-container
Smith
Distroless
Buildkit
Source to Image (s2i)
Habitat
Android Internals at Linaro Connect Asia 2013
Karim Yaghmour's "Android Internals - Embedded Android Workshop Part 1" at Linaro Connect Asia 2013 in Hong Kong.
VASCAN - Docker and Security
Michael Irwin graduated from VT in 2011 and started using Docker for QA work in 2015. He attended his first DockerCon in 2016 and deployed Summit, his first production project using Docker, on AWS later that year. He started the Blacksburg Docker Meetup in 2016. In 2017, he was recognized as a Docker Captain. Docker provides containerization which isolates processes using kernel namespaces. Images are built from layers containing filesystem changes and metadata. Docker can be used to standardize environments for development, testing, and production.
Techtalks: taking docker to production
Docker networking uses Linux bridges which only connect containers on the same host. Containers cannot communicate across hosts. NAT and iptables are used to expose container ports to external networks. Docker's process model runs containers as child processes of the Docker daemon, so if the daemon fails all containers will fail. Alternative container runtimes like Rocket have a more robust process model.
JOSA TechTalk: Taking Docker to Production
Docker and Containers are proven solutions, but are they ready to replace your current deployment? And more importantly, are you aware of the changes you'll have to make to accommodate them? Are there any risks involved? This talk will answer these questions and talk about how to plan, automate, build, deploy, and orchestrate the whole process.
Leveraging Android's Linux Heritage at AnDevCon IV
BusyBox is a package that provides many common UNIX commands as a single executable. It contains implementations of utilities like cat, cp, grep, ls, mkdir, rm, and others. Using BusyBox allows embedded Linux systems to have full functionality while minimizing disk space usage since it packages multiple utilities into a single binary.
Android Security, From the Ground Up
This document provides an overview of Android security from the ground up, covering key aspects like the CPU, bootloader, kernel, native user-space, framework, and updates. It discusses goals of protecting user data and system resources while providing application isolation. Key security features include a robust Linux kernel, mandatory application sandboxing, secure IPC, and application signing.
Container images for OpenShift
These are the slides of a session I gave at EMEA Red Hat Tech Exchange 2017, a gathering of all Red Hat solution architects and consultants across EMEA. It is about considerations and good practices when creating images that will run on OpenShift. Available as blog at: https://developers.redhat.com/blog/2017/10/11/container-images-openshift-part-1-objectives/
JOSA TechTalks - Docker in Production
Docker is an open platform for developers and system administrators to build, ship and run distributed applications. Using Docker, companies in Jordan have been able to build powerful system architectures that allow speeding up delivery, easing deployment processes and at the same time cutting major hosting costs.
George Khoury shares his experience at Salalem in building flexible and cost effective architectures using Docker and other tools for infrastructure orchestration. The result allows them to easily and quickly move between different cloud providers.
Docker_AGH_v0.1.3
Docker is an open platform for building and running distributed applications across multiple isolated containers. It allows encapsulating applications into lightweight executable packages called containers that include all necessary dependencies. The document discusses Docker's architecture, components, image layers, repositories, building images using Dockerfiles, and challenges in implementing Docker in real-world systems due to limitations in separating runtime configuration and data. Competition in the container space from projects like Rocket, Project Atomic, and Vagga is also discussed.
Optimizing VM images for OpenStack with KVM/QEMU
This document discusses optimizing VM images for OpenStack with KVM/QEMU. It covers disk and container formats like RAW, QCOW2, and AMI. It also discusses tools for manipulating disk files, launching an instance, image OS preparation using cloud-init, authentication models, networking configuration, and hotplug support. The goal is to provide optimized images that support features like snapshots while allowing faster instance launching and increased storage efficiency.
Rootless Containers
In this talk we will discuss how to build and run containers without root privileges. As part of the discussion, we will introduce new programs like fuse-overlayfs and slirp4netns and explain how it is possible to do this using user namespaces. fuse-overlayfs allows to use the same storage model as "root" containers and use layered images. slirp4netns emulates a TCP/IP stack in userland and allows to use a network namespace from a container and let it access the outside world (with some limitations).
We will also introduce Usernetes, and how to run Kubernetes in an unprivileged user namespace
https://sched.co/Jcgg
Exploring Docker Security
These are the slides from a talk I gave for our Secure Systems course in the degree program Computer Science and Media at Stuttgart Media University.
Introduction to Docker and Containers
This document introduces Docker and containers. It discusses that Docker provides a way to easily deploy applications by using lightweight virtualization to isolate applications into containers. These containers can then be easily shared between systems. The document outlines what problem containers solve, provides an overview of how Docker works, and discusses how to author Docker images using Dockerfiles or by committing changes to containers. It also discusses the Docker community and ecosystem and Docker's long term roadmap to further simplify deployment of applications.
Scale Big With Docker — Moboom 2014
This document discusses Docker, an open-source containerization platform. It begins by outlining why Docker is useful for deploying applications reliably and at scale across various environments. It then explains the container metaphor and how Docker addresses challenges of shipping code similarly to how shipping containers standardized shipping goods. The document provides an overview of using Docker and building images with Dockerfiles. It concludes by discussing the Docker community, upcoming features, and the goals for Docker 1.0.
Rootless Kubernetes
Kubernetes supports several security mechanisms such as Seccomp, Apparmor, SELinux, and runAsUser for protecting the hosts from container-breakout attacks. However, these mechanisms are not sufficient for the security demand because Kubelet and CRI/OCI runtimes require the root privileges on the hosts, and these components are seriously bug-prone. The dependency on the root privileges has been also problematic for promoting Kubernetes to the HPC world, where users are often disallowed to install software as the root.
In this talk, Akihiro and Giuseppe will show the community’s ongoing work for making Kubernetes deployable and runnable as a non-root user, by using User Namespaces. The main topics of discussion will be UID/GID mapping, unprivileged Copy-on-Write filesystems, Usermode networking (Slirp), and Cgroups.
https://fosdem.org/2019/schedule/event/containers_k8s_rootless/
Containers: from development to production at DevNation 2015
In Docker, applications are shipped using a lightweight format, managed with a high-level API, and run within software containers which abstract the host environment. Operating details like distributions, versions, and network setup no longer matter to the application developer.
Thanks to this abstraction level, we can use the same container across all steps of the life cycle of an application, from development to production. This eliminates problems stemming from discrepancies between those environments.
Even so, these environments will always have different requirements. If our quality assurance (QA) and production systems use different logging systems, how can we still ship the same container to both? How can we satisfy the backup and security requirements of our production stack without bloating our development stack?
In this sess, you will learn about the unique features in containers that allow you to cleanly decouple system administrator tasks from the core of your application. We’ll show you how this decoupling results in smaller, simpler containers, and gives you more flexibility when building, managing, and evolving your application stacks.
Introduction to Docker at SF Peninsula Software Development Meetup @Guidewire
This document provides an introduction and overview of Docker and containers. It discusses that Docker is an open source tool that allows applications to be packaged with all their dependencies and run as isolated processes on any machine. Containers provide lightweight virtualization that improves efficiency by sharing resources but still isolating processes. The document outlines how Docker uses containers powered by Linux namespaces and cgroups to package and deploy applications easily and consistently across environments.
OpenNebula Conf 2014: CentOS, QA an OpenNebula - Christoph Galuschka
CentOS, the Community Enterprise OS, uses Opennebula as virtualization plattform for its automated QA-process. The opennebula setup consists of 3 nodes, all running CentOS-6, who handle the following tasks:
– sunstone as cloud controller
– local mirror/DNS-Server/http-Server for the VMs to pull in packages
– one VM to run a jenkins instance to launch the various tests (ci.de.centos.org)
– nginx on the cloud controller to forward http traffic to the jenkins VM
A public git repository (http://www.gitorious.org/testautomation) is used to allow whoever wants to contribute to pull the current test suite – t_functional, a series of bash scripts used to do funtional tests of various applications, binaries, configuration files and Trademark issues. As new tests are added to the repo via personal clones and merge requests, those tests first need to complete a test run via jenkins. Each test run currently consists of 4 VMs (one for each arch for C5 and C6 – C7 to come), which run the complete test suite. All VMs used for theses tests are instantiated and torn down on demand, whenever the call to testrun a personal clone is issued (via IRC).
Once completed successfully, the request is merged into the main repo. The jenkins node monitors this repository and which automatically triggers another complete test run.
Besides these triggered test runs, the test suite is automatically triggered daily to run. This is used to verify functionality of published updates – a handfull of failty updates have allready been discovered this way.
Besides t_functional, the Linux Test Project Suite of tests is also run on a daily basis, also to verify functionality of the OS and all updates.
The third setup is used to test the available and functional integrity of published docker images for CentOS.
All these tests are later – during the QA-phase of a point release – used to verify functionality of new packages inside the CentOS QA-Setup.
Similar to Singularity: The Inner Workings of Securely Running User Containers on HPC Systems (20)
Leveraging Android's Linux Heritage at AnDevCon IV
Leveraging Android's Linux Heritage at AnDevCon IV
Containers: from development to production at DevNation 2015
Containers: from development to production at DevNation 2015
Introduction to Docker at SF Peninsula Software Development Meetup @Guidewire
Introduction to Docker at SF Peninsula Software Development Meetup @Guidewire
OpenNebula Conf 2014: CentOS, QA an OpenNebula - Christoph Galuschka
OpenNebula Conf 2014: CentOS, QA an OpenNebula - Christoph Galuschka
More from inside-BigData.com
Major Market Shifts in IT
The document discusses the top 5 technologies that all organizations must understand: digital transformation, quantum computing, IoT, 5G, and AI/HPC. It provides an overview of each technology including opportunities and threats to organizations. The document emphasizes that understanding these emerging technologies is mandatory as the information revolution changes many aspects of life and business.
Preparing to program Aurora at Exascale - Early experiences and future direct...
In this deck from IWOCL / SYCLcon 2020, Hal Finkel from Argonne National Laboratory presents: Preparing to program Aurora at Exascale - Early experiences and future directions.
"Argonne National Laboratory’s Leadership Computing Facility will be home to Aurora, our first exascale supercomputer. Aurora promises to take scientific computing to a whole new level, and scientists and engineers from many different fields will take advantage of Aurora’s unprecedented computational capabilities to push the boundaries of human knowledge. In addition, Aurora’s support for advanced machine-learning and big-data computations will enable scientific workflows incorporating these techniques along with traditional HPC algorithms. Programming the state-of-the-art hardware in Aurora will be accomplished using state-of-the-art programming models. Some of these models, such as OpenMP, are long-established in the HPC ecosystem. Other models, such as Intel’s oneAPI, based on SYCL, are relatively-new models constructed with the benefit of significant experience. Many applications will not use these models directly, but rather, will use C++ abstraction libraries such as Kokkos or RAJA. Python will also be a common entry point to high-performance capabilities. As we look toward the future, features in the C++ standard itself will become increasingly relevant for accessing the extreme parallelism of exascale platforms.
This presentation will summarize the experiences of our team as we prepare for Aurora, exploring how to port applications to Aurora’s architecture and programming models, and distilling the challenges and best practices we’ve developed to date. oneAPI/SYCL and OpenMP are both critical models in these efforts, and while the ecosystem for Aurora has yet to mature, we’ve already had a great deal of success. Importantly, we are not passive recipients of programming models developed by others. Our team works not only with vendor-provided compilers and tools, but also develops improved open-source LLVM-based technologies that feed both open-source and vendor-provided capabilities. In addition, we actively participate in the standardization of OpenMP, SYCL, and C++. To conclude, I’ll share our thoughts on how these models can best develop in the future to support exascale-class systems."
Watch the video: https://wp.me/p3RLHQ-lPT
Learn more: https://www.iwocl.org/iwocl-2020/conference-program/
and
https://www.anl.gov/topic/aurora
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Transforming Private 5G Networks
In this deck, Greg Wahl from Advantech presents: Transforming Private 5G Networks.
Advantech Networks & Communications Group is driving innovation in next-generation network solutions with their High Performance Servers. We provide business critical hardware to the world's leading telecom and networking equipment manufacturers with both standard and customized products. Our High Performance Servers are highly configurable platforms designed to balance the best in x86 server-class processing performance with maximum I/O and offload density. The systems are cost effective, highly available and optimized to meet next generation networking and media processing needs.
“Advantech’s Networks and Communication Group has been both an innovator and trusted enabling partner in the telecommunications and network security markets for over a decade, designing and manufacturing products for OEMs that accelerate their network platform evolution and time to market.” Said Advantech Vice President of Networks & Communications Group, Ween Niu. “In the new IP Infrastructure era, we will be expanding our expertise in Software Defined Networking (SDN) and Network Function Virtualization (NFV), two of the essential conduits to 5G infrastructure agility making networks easier to install, secure, automate and manage in a cloud-based infrastructure.”
In addition to innovation in air interface technologies and architecture extensions, 5G will also need a new generation of network computing platforms to run the emerging software defined infrastructure, one that provides greater topology flexibility, essential to deliver on the promises of high availability, high coverage, low latency and high bandwidth connections. This will open up new parallel industry opportunities through dedicated 5G network slices reserved for specific industries dedicated to video traffic, augmented reality, IoT, connected cars etc. 5G unlocks many new doors and one of the keys to its enablement lies in the elasticity and flexibility of the underlying infrastructure.
Advantech’s corporate vision is to enable an intelligent planet. The company is a global leader in the fields of IoT intelligent systems and embedded platforms. To embrace the trends of IoT, big data, and artificial intelligence, Advantech promotes IoT hardware and software solutions with the Edge Intelligence WISE-PaaS core to assist business partners and clients in connecting their industrial chains. Advantech is also working with business partners to co-create business ecosystems that accelerate the goal of industrial intelligence."
Watch the video: https://wp.me/p3RLHQ-lPQ
* Company website: https://www.advantech.com/
* Solution page: https://www2.advantech.com/nc/newsletter/NCG/SKY/benefits.html
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
The Incorporation of Machine Learning into Scientific Simulations at Lawrence...
In this deck from the Stanford HPC Conference, Katie Lewis from Lawrence Livermore National Laboratory presents: The Incorporation of Machine Learning into Scientific Simulations at Lawrence Livermore National Laboratory.
"Scientific simulations have driven computing at Lawrence Livermore National Laboratory (LLNL) for decades. During that time, we have seen significant changes in hardware, tools, and algorithms. Today, data science, including machine learning, is one of the fastest growing areas of computing, and LLNL is investing in hardware, applications, and algorithms in this space. While the use of simulations to focus and understand experiments is well accepted in our community, machine learning brings new challenges that need to be addressed. I will explore applications for machine learning in scientific simulations that are showing promising results and further investigation that is needed to better understand its usefulness."
Watch the video: https://youtu.be/NVwmvCWpZ6Y
Learn more: https://computing.llnl.gov/research-area/machine-learning
and
http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
How to Achieve High-Performance, Scalable and Distributed DNN Training on Mod...
In this deck from the Stanford HPC Conference, DK Panda from Ohio State University presents: How to Achieve High-Performance, Scalable and Distributed DNN Training on Modern HPC Systems?
"This talk will start with an overview of challenges being faced by the AI community to achieve high-performance, scalable and distributed DNN training on Modern HPC systems with both scale-up and scale-out strategies. After that, the talk will focus on a range of solutions being carried out in my group to address these challenges. The solutions will include: 1) MPI-driven Deep Learning, 2) Co-designing Deep Learning Stacks with High-Performance MPI, 3) Out-of- core DNN training, and 4) Hybrid (Data and Model) parallelism. Case studies to accelerate DNN training with popular frameworks like TensorFlow, PyTorch, MXNet and Caffe on modern HPC systems will be presented."
Watch the video: https://youtu.be/LeUNoKZVuwQ
Learn more: http://web.cse.ohio-state.edu/~panda.2/
and
http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze ...
In this deck from the Stanford HPC Conference, Nick Nystrom and Paola Buitrago provide an update from the Pittsburgh Supercomputing Center.
Nick Nystrom is Chief Scientist at the Pittsburgh Supercomputing Center (PSC). Nick is architect and PI for Bridges, PSC's flagship system that successfully pioneered the convergence of HPC, AI, and Big Data. He is also PI for the NIH Human Biomolecular Atlas Program’s HIVE Infrastructure Component and co-PI for projects that bring emerging AI technologies to research (Open Compass), apply machine learning to biomedical data for breast and lung cancer (Big Data for Better Health), and identify causal relationships in biomedical big data (the Center for Causal Discovery, an NIH Big Data to Knowledge Center of Excellence). His current research interests include hardware and software architecture, applications of machine learning to multimodal data (particularly for the life sciences) and to enhance simulation, and graph analytics.
Watch the video: https://youtu.be/LWEU1L1o7yY
Learn more: https://www.psc.edu/
and
http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
HPC Impact: EDA Telemetry Neural Networks
The document discusses using systems intelligence and artificial intelligence/neural networks to enhance semiconductor electronic design automation (EDA) workflows by collecting telemetry data from EDA jobs and infrastructure and analyzing it using complex event processing, machine learning models, and messaging substrates to provide insights that could optimize EDA pipelines and infrastructure. The approach aims to allow both internal and external augmentation of EDA processes and environments through unsupervised and incremental learning.
Biohybrid Robotic Jellyfish for Future Applications in Ocean Monitoring
In this deck from the Stanford HPC Conference, Nicole Xu from Stanford University describes how she transformed a common jellyfish into a bionic creature that is part animal and part machine.
"Animal locomotion and bioinspiration have the potential to expand the performance capabilities of robots, but current implementations are limited. Mechanical soft robots leverage engineered materials and are highly controllable, but these biomimetic robots consume more power than corresponding animal counterparts. Biological soft robots from a bottom-up approach offer advantages such as speed and controllability but are limited to survival in cell media. Instead, biohybrid robots that comprise live animals and self- contained microelectronic systems leverage the animals’ own metabolism to reduce power constraints and body as an natural scaffold with damage tolerance. We demonstrate that by integrating onboard microelectronics into live jellyfish, we can enhance propulsion up to threefold, using only 10 mW of external power input to the microelectronics and at only a twofold increase in cost of transport to the animal. This robotic system uses 10 to 1000 times less external power per mass than existing swimming robots in literature and can be used in future applications for ocean monitoring to track environmental changes."
Watch the video: https://youtu.be/HrmJFyvInj8
Learn more: https://sanfrancisco.cbslocal.com/2020/02/05/stanford-research-project-common-jellyfish-bionic-sea-creatures/
and
http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Machine Learning for Weather Forecasts
In this deck from the Stanford HPC Conference, Peter Dueben from the European Centre for Medium-Range Weather Forecasts (ECMWF) presents: Machine Learning for Weather Forecasts.
"I will present recent studies that use deep learning to learn the equations of motion of the atmosphere, to emulate model components of weather forecast models and to enhance usability of weather forecasts. I will than talk about the main challenges for the application of deep learning in cutting-edge weather forecasts and suggest approaches to improve usability in the future."
Peter is contributing to the development and optimization of weather and climate models for modern supercomputers. He is focusing on a better understanding of model error and model uncertainty, on the use of reduced numerical precision that is optimised for a given level of model error, on global cloud- resolving simulations with ECMWF's forecast model, and the use of machine learning, and in particular deep learning, to improve the workflow and predictions. Peter has graduated in Physics and wrote his PhD thesis at the Max Planck Institute for Meteorology in Germany. He worked as Postdoc with Tim Palmer at the University of Oxford and has taken up a position as University Research Fellow of the Royal Society at the European Centre for Medium-Range Weather Forecasts (ECMWF) in 2017.
Watch the video: https://youtu.be/ks3fkRj8Iqc
Learn more: https://www.ecmwf.int/
and
http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
HPC AI Advisory Council Update
In this deck, Gilad Shainer from the HPC AI Advisory Council describes how this organization fosters innovation in the high performance computing community.
"The HPC-AI Advisory Council’s mission is to bridge the gap between high-performance computing (HPC) and Artificial Intelligence (AI) use and its potential, bring the beneficial capabilities of HPC and AI to new users for better research, education, innovation and product manufacturing, bring users the expertise needed to operate HPC and AI systems, provide application designers with the tools needed to enable parallel computing, and to strengthen the qualification and integration of HPC and AI system products."
Watch the video: https://wp.me/p3RLHQ-lNz
Learn more: http://hpcadvisorycouncil.com
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Fugaku Supercomputer joins fight against COVID-19
Today RIKEN in Japan announced that the Fugaku supercomputer will be made available for research projects aimed to combat COVID-19.
"Fugaku is currently being installed and is scheduled to be available to the public in 2021. However, faced with the devastating disaster unfolding before our eyes, RIKEN and MEXT decided to make a portion of the computational resources of Fugaku available for COVID-19-related projects ahead of schedule while continuing the installation process.
Fugaku is being developed not only for the progress in science, but also to help build the society dubbed as the “Society 5.0” by the Japanese government, where all people will live safe and comfortable lives. The current initiative to fight against the novel coronavirus is driven by the philosophy behind the development of Fugaku."
Initial Projects
Exploring new drug candidates for COVID-19 by "Fugaku"
Yasushi Okuno, RIKEN / Kyoto University
Prediction of conformational dynamics of proteins on the surface of SARS-Cov-2 using Fugaku
Yuji Sugita, RIKEN
Simulation analysis of pandemic phenomena
Nobuyasu Ito, RIKEN
Fragment molecular orbital calculations for COVID-19 proteins
Yuji Mochizuki, Rikkyo University
Energy Efficient Computing using Dynamic Tuning
In this deck from the Performance Optimisation and Productivity group, Lubomir Riha from IT4Innovations presents: Energy Efficient Computing using Dynamic Tuning.
"We now live in a world of power-constrained architectures and systems and power consumption represents a significant cost factor in the overall HPC system economy. For these reasons, in recent years researchers, supercomputing centers and major vendors have developed new tools and methodologies to measure and optimize the energy consumption of large-scale high performance system installations. Due to the link between energy consumption, power consumption and execution time of an application executed by the final user, it is important for these tools and the methodology used to consider all these aspects, empowering the final user and the system administrator with the capability of finding the best configuration given different high level objectives.
This webinar focused on tools designed to improve the energy-efficiency of HPC applications using a methodology of dynamic tuning of HPC applications, developed under the H2020 READEX project. The READEX methodology has been designed for exploiting the dynamic behaviour of software. At design time, different runtime situations (RTS) are detected and optimized system configurations are determined. RTSs with the same configuration are grouped into scenarios, forming the tuning model. At runtime, the tuning model is used to switch system configurations dynamically.
The MERIC tool, that implements the READEX methodology, is presented. It supports manual or binary instrumentation of the analysed applications to simplify the analysis. This instrumentation is used to identify and annotate the significant regions in the HPC application. Automatic binary instrumentation annotates regions with significant runtime. Manual instrumentation, which can be combined with automatic, allows code developer to annotate regions of particular interest."
Watch the video: https://wp.me/p3RLHQ-lJP
Learn more: https://pop-coe.eu/blog/14th-pop-webinar-energy-efficient-computing-using-dynamic-tuning
and
https://code.it4i.cz/vys0053/meric
Sign up for our insideHPC Newsletter: http://insidehpc.com/newslett
HPC at Scale Enabled by DDN A3i and NVIDIA SuperPOD
The document discusses how DDN A3I storage solutions and Nvidia's SuperPOD platform can enable HPC at scale. It provides details on DDN's A3I appliances that are optimized for AI and deep learning workloads and validated for Nvidia's DGX-2 SuperPOD reference architecture. The solutions are said to deliver the fastest performance, effortless scaling, reliability and flexibility for data-intensive workloads.
State of ARM-based HPC
In this deck, Paul Isaacs from Linaro presents: State of ARM-based HPC. This talk provides an overview of applications and infrastructure services successfully ported to Aarch64 and benefiting from scale.
"With its debut on the TOP500, the 125,000-core Astra supercomputer at New Mexico’s Sandia Labs uses Cavium ThunderX2 chips to mark Arm’s entry into the petascale world. In Japan, the Fujitsu A64FX Arm-based CPU in the pending Fugaku supercomputer has been optimized to achieve high-level, real-world application performance, anticipating up to one hundred times the application execution performance of the K computer. K was the first computer to top 10 petaflops in 2011."
Watch the video: https://wp.me/p3RLHQ-lIT
Learn more: https://www.linaro.org/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Versal Premium ACAP for Network and Cloud Acceleration
Today Xilinx announced Versal Premium, the third series in the Versal ACAP portfolio. The Versal Premium series features highly integrated, networked and power-optimized cores and the industry’s highest bandwidth and compute density on an adaptable platform. Versal Premium is designed for the highest bandwidth networks operating in thermally and spatially constrained environments, as well as for cloud providers who need scalable, adaptable application acceleration.
Versal is the industry’s first adaptive compute acceleration platform (ACAP), a revolutionary new category of heterogeneous compute devices with capabilities that far exceed those of conventional silicon architectures. Developed on TSMC’s 7-nanometer process technology, Versal Premium combines software programmability with dynamically configurable hardware acceleration and pre-engineered connectivity and security features to enable a faster time-to- market. The Versal Premium series delivers up to 3X higher throughput compared to current generation FPGAs, with built-in Ethernet, Interlaken, and cryptographic engines that enable fast and secure networks. The series doubles the compute density of currently deployed mainstream FPGAs and provides the adaptability to keep pace with increasingly diverse and evolving cloud and networking workloads.
Learn more: https://insidehpc.com/2020/03/xilinx-announces-versal-premium-acap-for-network-and-cloud-acceleration/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Zettar: Moving Massive Amounts of Data across Any Distance Efficiently
In this video from the Rice Oil & Gas Conference, Chin Fang from Zettar presents: Moving Massive Amounts of Data across Any Distance Efficiently.
The objective of this talk is to present two on-going projects aiming at improving and ensuring highly efficient bulk transferring or streaming of massive amounts of data over digital connections across any distance. It examines the current state of the art, a few very common misconceptions, the differences among the three major type of data movement solutions, a current initiative attempting to improve the data movement efficiency from the ground up, and another multi-stage project that shows how to conduct long distance large scale data movement at speed and scale internationally. Both projects have real world motivations, e.g. the ambitious data transfer requirements of Linac Coherent Light Source II (LCLS-II) [1], a premier preparation project of the U.S. DOE Exascale Computing Initiative (ECI) [2]. Their immediate goals are described and explained, together with the solution used for each. Findings and early results are reported. Possible future works are outlined.
Watch the video: https://wp.me/p3RLHQ-lBX
Learn more: https://www.zettar.com/
and
https://rice2020oghpc.rice.edu/program-2/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Scaling TCO in a Post Moore's Era
In this deck from the Rice Oil & Gas Conference, Bradley McCredie from AMD presents: Scaling TCO in a Post Moore's Law Era.
"While foundries bravely drive forward to overcome the technical and economic challenges posed by scaling to 5nm and beyond, Moore’s law alone can provide only a fraction of the performance / watt and performance / dollar gains needed to satisfy the demands of today’s high performance computing and artificial intelligence applications. To close the gap, multiple strategies are required. First, new levels of innovation and design efficiency will supplement technology gains to continue to deliver meaningful improvements in SoC performance. Second, heterogenous compute architectures will create x-factor increases of performance efficiency for the most critical applications. Finally, open software frameworks, APIs, and toolsets will enable broad ecosystems of application level innovation."
Watch the video:
Learn more: http://amd.com
and
https://rice2020oghpc.rice.edu/program-2/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
CUDA-Python and RAPIDS for blazing fast scientific computing
In this deck from the ECSS Symposium, Abe Stern from NVIDIA presents: CUDA-Python and RAPIDS for blazing fast scientific computing.
"We will introduce Numba and RAPIDS for GPU programming in Python. Numba allows us to write just-in-time compiled CUDA code in Python, giving us easy access to the power of GPUs from a powerful high-level language. RAPIDS is a suite of tools with a Python interface for machine learning and dataframe operations. Together, Numba and RAPIDS represent a potent set of tools for rapid prototyping, development, and analysis for scientific computing. We will cover the basics of each library and go over simple examples to get users started. Finally, we will briefly highlight several other relevant libraries for GPU programming."
Watch the video: https://wp.me/p3RLHQ-lvu
Learn more: https://developer.nvidia.com/rapids
and
https://www.xsede.org/for-users/ecss/ecss-symposium
Sign up for our insideHPC Newsletter: http://insidehp.com/newsletter
Introducing HPC with a Raspberry Pi Cluster
In this deck from FOSDEM 2020, Colin Sauze from Aberystwyth University describes the development of a RaspberryPi cluster for teaching an introduction to HPC.
"The motivation for this was to overcome four key problems faced by new HPC users:
* The availability of a real HPC system and the effect running training courses can have on the real system, conversely the availability of spare resources on the real system can cause problems for the training course.
* A fear of using a large and expensive HPC system for the first time and worries that doing something wrong might damage the system.
* That HPC systems are very abstract systems sitting in data centres that users never see, it is difficult for them to understand exactly what it is they are using.
* That new users fail to understand resource limitations, in part because of the vast resources in modern HPC systems a lot of mistakes can be made before running out of resources. A more resource constrained system makes it easier to understand this.
The talk will also discuss some of the technical challenges in deploying an HPC environment to a Raspberry Pi and attempts to keep that environment as close to a "real" HPC as possible. The issue to trying to automate the installation process will also be covered."
Learn more: https://github.com/colinsauze/pi_cluster
and
https://fosdem.org/2020/schedule/events/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Overview of HPC Interconnects
In this deck from ATPESC 2019, Ken Raffenetti from Argonne presents an overview of HPC interconnects.
"The Argonne Training Program on Extreme-Scale Computing (ATPESC) provides intensive, two-week training on the key skills, approaches, and tools to design, implement, and execute computational science and engineering applications on current high-end computing systems and the leadership-class computing systems of the future."
Watch the video: https://wp.me/p3RLHQ-luc
Learn more: https://extremecomputingtraining.anl.gov/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
More from inside-BigData.com (20)
Preparing to program Aurora at Exascale - Early experiences and future direct...
Preparing to program Aurora at Exascale - Early experiences and future direct...
The Incorporation of Machine Learning into Scientific Simulations at Lawrence...
The Incorporation of Machine Learning into Scientific Simulations at Lawrence...
How to Achieve High-Performance, Scalable and Distributed DNN Training on Mod...
How to Achieve High-Performance, Scalable and Distributed DNN Training on Mod...
Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze ...
Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze ...
Biohybrid Robotic Jellyfish for Future Applications in Ocean Monitoring
Biohybrid Robotic Jellyfish for Future Applications in Ocean Monitoring
HPC at Scale Enabled by DDN A3i and NVIDIA SuperPOD
HPC at Scale Enabled by DDN A3i and NVIDIA SuperPOD
Versal Premium ACAP for Network and Cloud Acceleration
Versal Premium ACAP for Network and Cloud Acceleration
Zettar: Moving Massive Amounts of Data across Any Distance Efficiently
Zettar: Moving Massive Amounts of Data across Any Distance Efficiently
CUDA-Python and RAPIDS for blazing fast scientific computing
CUDA-Python and RAPIDS for blazing fast scientific computing
Recently uploaded
QR Secure: A Hybrid Approach Using Machine Learning and Security Validation F...
QR Secure: A Hybrid Approach Using Machine Learning and Security Validation Functions to Prevent Interaction with Malicious QR Codes.
Aim of the Study: The goal of this research was to develop a robust hybrid approach for identifying malicious and insecure URLs derived from QR codes, ensuring safe interactions.
This is achieved through:
Machine Learning Model: Predicts the likelihood of a URL being malicious.
Security Validation Functions: Ensures the derived URL has a valid certificate and proper URL format.
This innovative blend of technology aims to enhance cybersecurity measures and protect users from potential threats hidden within QR codes 🖥 🔒
This study was my first introduction to using ML which has shown me the immense potential of ML in creating more secure digital environments!
Astute Business Solutions | Oracle Cloud Partner |
Your goto partner for Oracle Cloud, PeopleSoft, E-Business Suite, and Ellucian Banner. We are a firm specialized in managed services and consulting.
Essentials of Automations: Exploring Attributes & Automation Parameters
Building automations in FME Flow can save time, money, and help businesses scale by eliminating data silos and providing data to stakeholders in real-time. One essential component to orchestrating complex automations is the use of attributes & automation parameters (both formerly known as “keys”). In fact, it’s unlikely you’ll ever build an Automation without using these components, but what exactly are they?
Attributes & automation parameters enable the automation author to pass data values from one automation component to the next. During this webinar, our FME Flow Specialists will cover leveraging the three types of these output attributes & parameters in FME Flow: Event, Custom, and Automation. As a bonus, they’ll also be making use of the Split-Merge Block functionality.
You’ll leave this webinar with a better understanding of how to maximize the potential of automations by making use of attributes & automation parameters, with the ultimate goal of setting your enterprise integration workflows up on autopilot.
Session 1 - Intro to Robotic Process Automation.pdf
👉 Check out our full 'Africa Series - Automation Student Developers (EN)' page to register for the full program:
https://bit.ly/Automation_Student_Kickstart
In this session, we shall introduce you to the world of automation, the UiPath Platform, and guide you on how to install and setup UiPath Studio on your Windows PC.
📕 Detailed agenda:
What is RPA? Benefits of RPA?
RPA Applications
The UiPath End-to-End Automation Platform
UiPath Studio CE Installation and Setup
💻 Extra training through UiPath Academy:
Introduction to Automation
UiPath Business Automation Platform
Explore automation development with UiPath Studio
👉 Register here for our upcoming Session 2 on June 20: Introduction to UiPath Studio Fundamentals: https://community.uipath.com/events/details/uipath-lagos-presents-session-2-introduction-to-uipath-studio-fundamentals/
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Dmitrii Kamaev, PhD
Senior Product Owner - QIAGEN
"NATO Hackathon Winner: AI-Powered Drug Search", Taras Kloba
This is a session that details how PostgreSQL's features and Azure AI Services can be effectively used to significantly enhance the search functionality in any application.
In this session, we'll share insights on how we used PostgreSQL to facilitate precise searches across multiple fields in our mobile application. The techniques include using LIKE and ILIKE operators and integrating a trigram-based search to handle potential misspellings, thereby increasing the search accuracy.
We'll also discuss how the azure_ai extension on PostgreSQL databases in Azure and Azure AI Services were utilized to create vectors from user input, a feature beneficial when users wish to find specific items based on text prompts. While our application's case study involves a drug search, the techniques and principles shared in this session can be adapted to improve search functionality in a wide range of applications. Join us to learn how PostgreSQL and Azure AI can be harnessed to enhance your application's search capability.
Day 2 - Intro to UiPath Studio Fundamentals
In our second session, we shall learn all about the main features and fundamentals of UiPath Studio that enable us to use the building blocks for any automation project.
📕 Detailed agenda:
Variables and Datatypes
Workflow Layouts
Arguments
Control Flows and Loops
Conditional Statements
💻 Extra training through UiPath Academy:
Variables, Constants, and Arguments in Studio
Control Flow in Studio
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
How we managed to grow and scale a RAG application from zero to thousands of users in 7 months. Lessons from technical challenges around managing high load for LLMs, RAGs and Vector databases.
The Microsoft 365 Migration Tutorial For Beginner.pptx
This presentation will help you understand the power of Microsoft 365. However, we have mentioned every productivity app included in Office 365. Additionally, we have suggested the migration situation related to Office 365 and how we can help you.
You can also read: https://www.systoolsgroup.com/updates/office-365-tenant-to-tenant-migration-step-by-step-complete-guide/
What is an RPA CoE? Session 1 – CoE Vision
In the first session, we will review the organization's vision and how this has an impact on the COE Structure.
Topics covered:
• The role of a steering committee
• How do the organization’s priorities determine CoE Structure?
Speaker:
Chris Bolin, Senior Intelligent Automation Architect Anika Systems
Apps Break Data
How information systems are built or acquired puts information, which is what they should be about, in a secondary place. Our language adapted accordingly, and we no longer talk about information systems but applications. Applications evolved in a way to break data into diverse fragments, tightly coupled with applications and expensive to integrate. The result is technical debt, which is re-paid by taking even bigger "loans", resulting in an ever-increasing technical debt. Software engineering and procurement practices work in sync with market forces to maintain this trend. This talk demonstrates how natural this situation is. The question is: can something be done to reverse the trend?
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
At this talk we will discuss DDoS protection tools and best practices, discuss network architectures and what AWS has to offer. Also, we will look into one of the largest DDoS attacks on Ukrainian infrastructure that happened in February 2022. We'll see, what techniques helped to keep the web resources available for Ukrainians and how AWS improved DDoS protection for all customers based on Ukraine experience
QA or the Highway - Component Testing: Bridging the gap between frontend appl...
These are the slides for the presentation, "Component Testing: Bridging the gap between frontend applications" that was presented at QA or the Highway 2024 in Columbus, OH by Zachary Hamm.
Introducing BoxLang : A new JVM language for productivity and modularity!
Just like life, our code must adapt to the ever changing world we live in. From one day coding for the web, to the next for our tablets or APIs or for running serverless applications. Multi-runtime development is the future of coding, the future is to be dynamic. Let us introduce you to BoxLang.
Dynamic. Modular. Productive.
BoxLang redefines development with its dynamic nature, empowering developers to craft expressive and functional code effortlessly. Its modular architecture prioritizes flexibility, allowing for seamless integration into existing ecosystems.
Interoperability at its Core
With 100% interoperability with Java, BoxLang seamlessly bridges the gap between traditional and modern development paradigms, unlocking new possibilities for innovation and collaboration.
Multi-Runtime
From the tiny 2m operating system binary to running on our pure Java web server, CommandBox, Jakarta EE, AWS Lambda, Microsoft Functions, Web Assembly, Android and more. BoxLang has been designed to enhance and adapt according to it's runnable runtime.
The Fusion of Modernity and Tradition
Experience the fusion of modern features inspired by CFML, Node, Ruby, Kotlin, Java, and Clojure, combined with the familiarity of Java bytecode compilation, making BoxLang a language of choice for forward-thinking developers.
Empowering Transition with Transpiler Support
Transitioning from CFML to BoxLang is seamless with our JIT transpiler, facilitating smooth migration and preserving existing code investments.
Unlocking Creativity with IDE Tools
Unleash your creativity with powerful IDE tools tailored for BoxLang, providing an intuitive development experience and streamlining your workflow. Join us as we embark on a journey to redefine JVM development. Welcome to the era of BoxLang.
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
The typical problem in product engineering is not bad strategy, so much as “no strategy”. This leads to confusion, lack of motivation, and incoherent action. The next time you look for a strategy and find an empty space, instead of waiting for it to be filled, I will show you how to fill it in yourself. If you’re wrong, it forces a correction. If you’re right, it helps create focus. I’ll share how I’ve approached this in the past, both what works and lessons for what didn’t work so well.
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
This LF Energy webinar took place June 20, 2024. It featured:
-Alex Thornton, LF Energy
-Hallie Cramer, Google
-Daniel Roesler, UtilityAPI
-Henry Richardson, WattTime
In response to the urgency and scale required to effectively address climate change, open source solutions offer significant potential for driving innovation and progress. Currently, there is a growing demand for standardization and interoperability in energy data and modeling. Open source standards and specifications within the energy sector can also alleviate challenges associated with data fragmentation, transparency, and accessibility. At the same time, it is crucial to consider privacy and security concerns throughout the development of open source platforms.
This webinar will delve into the motivations behind establishing LF Energy’s Carbon Data Specification Consortium. It will provide an overview of the draft specifications and the ongoing progress made by the respective working groups.
Three primary specifications will be discussed:
-Discovery and client registration, emphasizing transparent processes and secure and private access
-Customer data, centering around customer tariffs, bills, energy usage, and full consumption disclosure
-Power systems data, focusing on grid data, inclusive of transmission and distribution networks, generation, intergrid power flows, and market settlement data
Containers & AI - Beauty and the Beast!?!
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
Keywords: AI, Containeres, Kubernetes, Cloud Native
Event Link: https://meine.doag.org/events/cloudland/2024/agenda/#agendaId.4211
Recently uploaded (20)
QR Secure: A Hybrid Approach Using Machine Learning and Security Validation F...
QR Secure: A Hybrid Approach Using Machine Learning and Security Validation F...
Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |
Essentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation Parameters
Session 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdf
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
"NATO Hackathon Winner: AI-Powered Drug Search", Taras Kloba
"NATO Hackathon Winner: AI-Powered Drug Search", Taras Kloba
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
The Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptx
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
QA or the Highway - Component Testing: Bridging the gap between frontend appl...
QA or the Highway - Component Testing: Bridging the gap between frontend appl...
Introducing BoxLang : A new JVM language for productivity and modularity!
Introducing BoxLang : A new JVM language for productivity and modularity!
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
Singularity: The Inner Workings of Securely Running User Containers on HPC Systems
- 1. The Inner Workings of Securely Running User Containers on HPC Systems Singularity Michael Bauer
- 2. About Me Michael Bauer @bauerm97 on GitHub bauerm@umich.edu
- 7. What is a Virtual Machine? “In computing, a virtual machine (VM) is an emulation of a computer system. Virtual machines are based on computer architectures and provide functionality of a physical computer.” Examples:
- 8. Pros Cons ● Run different OS on one set of hardware ● Save money (e.g. buy one laptop, have Windows, OSX, and Linux) ● Easy maintenance ● Slower performance ● Memory/storage reqs.
- 9. What are containers? ● Similar goal as VMs ● No kernel emulation ● Not architecture level virtualization, but rather software level
- 10. What does that mean? ● Don’t waste extra ~5% performance doing emulation ● Smaller footprint (~500 MB vs ~20 GB VM) ● Very small startup time interval (~1 s vs ~1 min VM) ● Multiple instances can share one “container image”
- 13. Containers for Scientific Computing
- 14. Why do we want containers in HPC? ● Escape “dependency hell” ● Local and remote code works identically every time ● One file contains everything and can be moved anywhere
- 16. Needs for HPC containers ● Any user can run containers without special privileges (root) ● Integrate seamlessly into existing infrastructure ● Portability between many systems ● Users created and provided containers (no administrative oversight)
- 17. HPC container software can never touch root
- 18. What about Docker? ● Root level process spawns containers as child processes ● Integration into infrastructure difficult (e.g. must consider other scheduler systems)
- 19. Singularity
- 20. Needs for HPC containers ● Any user can run containers without special privileges (root) ● Integrate seamlessly into existing infrastructure ● Portability between many systems ● Users created and provided containers (no administrative oversight)
- 21. ● Any container can be run by any user - same user inside container and on host ● No workflow changes necessary to use ● Single .img file contains everything necessary ● Safe to run any container without screening its contents Singularity
- 25. Basic Usage of Singularity
- 26. Singularity Workflow 1. Create image file $ sudo singularity create [image] 2. Bootstrap image $ sudo singularity bootstrap [image] [definition.def] 3. Run image $ singularity shell [image] $ singularity exec [image] [/path/to/executable] $ singularity run [image] $ ./image
- 30. SLURM Integration #!/bin/bash -l #SBATCH --image=~/centos7/latest #SBATCH -p debug #SBATCH -N 64 #SBATCH -t 00:20:00 #SBATCH -J my_job #SBATCH -L SCRATCH #SBATCH -C haswell srun -n 4096 ./mycode.exe # an extra -c 1 flag is optional for fully packed pure MPI with hyperthreading
- 31. ALICE Tier 2 Use Case
- 32. GSI Green Cube Darmstadt Germany ALICE Detector LHC Geneva Switzerland
- 33. ALICE Tier 2: Problem ● Run ALICE jobs on ~2k jobs at any time ● Host machines run Debian 7.x kernel 3.16 ● ALICE expects Scientific Linux 6 (SL6) ● Library incompatibilities cause frequent errors (much higher than expected)
- 34. ALICE Tier 2: Pre-Singularity Solution ● Correct library versions mounted in Lustre ● SLURM job submission script alters $LD_LIBRARY_PATH to point to Lustre ● And maybe more? Big Ugly Hack
- 35. ALICE Tier 2: Singularity Solution ● Package Scientific Linux 6 into container ● Modify SLURM submission script to run container ● No need to mount Lustre for access to library files ● Can test container locally before deploying to HPC SLC6 Dockerfile ALICE GitHub Repository Singularity Build File Import from slc6-builder Container
- 36. How does it work
- 37. How ● Installed as SUID binary owned by root ● All necessary files are mounted ● Hide inside namespaces when possible and requested
- 38. SUID Binary Security ● Make code possible for anybody to audit ● Function call to change eUID to 0 ● Function call to change eUID to calling user UID ● Wrap actions that require root in these two calls
- 39. User Namespaces? ● Map UID on host : UID inside user namespace ● Looks and feels like root (mostly) ● Potentially breaks portability
- 40. If any user code gets executed as root user, the system should be considered compromised.
- 42. Core Principles 1. Never run user code as real user root 2. Only use eUID 0 when necessary (escalate and drop permissions accordingly) 3. Drop permissions and capabilities when forking into new thread http://jdfinley.com/important-papers/safe/
- 43. Isolation ● Bind mount image file into host’s filesystem ● Use chroot to move into the mounted image’s root filesystem ● Mount /dev/, /etc/hosts, etc… into container filesystem ● Use namespaces when possible and requested (e.g User, PID, etc…) ● No isolation of network means no extra network configuration ● Can use host’s physical devices inside container (e.g. IPoIB)
- 44. Security Cont. ● User invokes non-SUID Singularity binary, which in turn calls SUID binary. Only non-SUID binary can run SUID binary ● singularity_priv_escalate() called to escalate privileges to eUID 0 singularity_priv_drop() called to drop privileges to eUID of calling user ● Only escalate privileges when necessary ● MS_NOSUID and O_CLOEXEC flags set when necessary
- 46. We are always looking for more Collaborators!