Docker storage drivers by Jérôme PetazzoniDocker, Inc.
The first release of Docker only supported AUFS, and AUFS was available (out of the box) only on Debian and Ubuntu kernel. Then Red Hat wanted Docker to run on its distros, and contributed the Device Mapper driver, and later the BTRFS driver, and recently the overlayfs driver.
Jérôme presents how those drivers compare from a high-level perspective, explaining their pros and cons.
Then he showed each driver in action, and look at low-level implementation details. We won't dive into the golang implementation code itself, but we will explain the concepts of each driver. This will help to better understand how they work, and give some hints when it comes to troubleshoot their behaviour.
An introduction to Linux Container, Namespace & Cgroup.
Virtual Machine, Linux operating principles. Application constraint execution environment. Isolate application working environment.
Docker storage drivers by Jérôme PetazzoniDocker, Inc.
The first release of Docker only supported AUFS, and AUFS was available (out of the box) only on Debian and Ubuntu kernel. Then Red Hat wanted Docker to run on its distros, and contributed the Device Mapper driver, and later the BTRFS driver, and recently the overlayfs driver.
Jérôme presents how those drivers compare from a high-level perspective, explaining their pros and cons.
Then he showed each driver in action, and look at low-level implementation details. We won't dive into the golang implementation code itself, but we will explain the concepts of each driver. This will help to better understand how they work, and give some hints when it comes to troubleshoot their behaviour.
An introduction to Linux Container, Namespace & Cgroup.
Virtual Machine, Linux operating principles. Application constraint execution environment. Isolate application working environment.
The OpenVZ/Virtuozzo developers from Odin (ex Parallels) have been working on Linux container technologies since 1999. What was originally a separate patchset is now mostly merged into the upstream Linux kernel, enabling the way for projects like LXC and Docker. In the mean time, the OpenVZ/Virtuozzo Linux kernel is still one step ahead of the vanilla kernel when it comes to containers. The talk will provide details about recent efforts towards Docker and Virtuozzo interoperability. This development is twofold. The first goal is to run Docker inside an OpenVZ container and the second goal is to use the proven OpenVZ kernel as a backend for Docker (via libcontainer).
Linux Container Brief for IEEE WG P2302Boden Russell
A brief into to Linux Containers presented to IEEE working group P2302 (InterCloud standards and portability). This deck covers:
- Definitions and motivations for containers
- Container technology stack
- Containers vs Hypervisor VMs
- Cgroups
- Namespaces
- Pivot root vs chroot
- Linux Container image basics
- Linux Container security topics
- Overview of Linux Container tooling functionality
- Thoughts on container portability and runtime configuration
- Container tooling in the industry
- Container gaps
- Sample use cases for traditional VMs
Overall, a bulk of this deck is covered in other material I have posted here. However there are a few new slides in this deck, most notability some thoughts on container portability and runtime config.
Internal presentation of Docker, Lightweight Virtualization, and linux Containers; at Spotify NYC offices, featuring engineers from Yandex, LinkedIn, Criteo, and NASA!
Cgroups, namespaces, and beyond: what are containers made from? (DockerCon Eu...Jérôme Petazzoni
Linux containers are different from Solaris Zones or BSD Jails: they use discrete kernel features like cgroups, namespaces, SELinux, and more. We will describe those mechanisms in depth, as well as demo how to put them together to produce a container. We will also highlight how different container runtimes compare to each other.
This talk was delivered at DockerCon Europe 2015 in Barcelona.
Containerization Is More than the New VirtualizationC4Media
Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1E5GzZX.
Jérôme Petazzoni borrows from his experience at Docker Inc. to explain live applications running in Docker, including reading logs, remote access, and troubleshooting tips. Filmed at qconsf.com.
Jérôme Petazzoni is a senior engineer at dotCloud, where he rotates between Ops, Support and Evangelist duties and the nickname of “master Yoda”, has earned.
If you're not familiar with Docker yet, here is your chance to catch up: a quick overview of the Open Source Docker Engine, and its associated services delivered through the Docker Hub. It also includes Jérôme will also discuss the new features of Docker 1.0, and briefly explain how you can run and maintain Docker on Azure. In addition, an Azure team member will demonstrate how deploy docker to Azure. The presentation will be followed by a Q&A session!
Describes what is lightweight virtualization and containers, and the low-level mechanisms in the Linux kernel that it relies on: namespaces, cgroups. It also gives details on AUFS. Those component together are the key to understanding how modern systems like Docker (http://www.docker.io/) work.
Introduction to Docker at Glidewell Laboratories in Orange CountyJérôme Petazzoni
In this presentation we will introduce Docker, and how you can use it to build, ship, and run any application, anywhere. The presentation included short demos, links to further material, and of course Q&As. If you are already a seasoned Docker user, this presentation will probably be redundant; but if you started to use Docker and are still struggling with some of his facets, you'll learn some!
presentation held at SUSE Linux Expert Forum December 2014
Linux container history and Linux namespaces
examples include:
* Move a VPN connection to its own namespace(p 25)
* User namespaces demo(p 28)
see collection of useful articles and advanced container usecases pp 29
While probably the most prominent, Docker is not the only tool for building and managing containers. Originally meant to be a "chroot on steroids" to help debug systemd, systemd-nspawn provides a fairly uncomplicated approach to work with containers. Being part of systemd, it is available on most recent distributions out-of-the-box and requires no additional dependencies.
This deck will introduce a few concepts involved in containers and will guide you through the steps of building a container from scratch. The payload will be a simple service, which will be automatically activated by systemd when the first request arrives.
Anatomy of a Container: Namespaces, cgroups & Some Filesystem Magic - LinuxConJérôme Petazzoni
Containers are everywhere. But what exactly is a container? What are they made from? What's the difference between LXC, butts-nspawn, Docker, and the other container systems out there? And why should we bother about specific filesystems?
In this talk, Jérôme will show the individual roles and behaviors of the components making up a container: namespaces, control groups, and copy-on-write systems. Then, he will use them to assemble a container from scratch, and highlight the differences (and likelinesses) with existing container systems.
Union FileSystem - A Building Blocks Of a ContainerKnoldus Inc.
Namespace, CGroup, and Union file-system are the basic building blocks of a container. Let’s have our focus on file-system. Why yet another file-system for the container? Is Conventional Linux file-systems like ext2, ext3, ext4, XFS, etc. not good enough to meet the purpose? In this blog post, I will try to answer these questions. Here we will be delving deeply into the Union File System and a few of its essential properties.
The OpenVZ/Virtuozzo developers from Odin (ex Parallels) have been working on Linux container technologies since 1999. What was originally a separate patchset is now mostly merged into the upstream Linux kernel, enabling the way for projects like LXC and Docker. In the mean time, the OpenVZ/Virtuozzo Linux kernel is still one step ahead of the vanilla kernel when it comes to containers. The talk will provide details about recent efforts towards Docker and Virtuozzo interoperability. This development is twofold. The first goal is to run Docker inside an OpenVZ container and the second goal is to use the proven OpenVZ kernel as a backend for Docker (via libcontainer).
Linux Container Brief for IEEE WG P2302Boden Russell
A brief into to Linux Containers presented to IEEE working group P2302 (InterCloud standards and portability). This deck covers:
- Definitions and motivations for containers
- Container technology stack
- Containers vs Hypervisor VMs
- Cgroups
- Namespaces
- Pivot root vs chroot
- Linux Container image basics
- Linux Container security topics
- Overview of Linux Container tooling functionality
- Thoughts on container portability and runtime configuration
- Container tooling in the industry
- Container gaps
- Sample use cases for traditional VMs
Overall, a bulk of this deck is covered in other material I have posted here. However there are a few new slides in this deck, most notability some thoughts on container portability and runtime config.
Internal presentation of Docker, Lightweight Virtualization, and linux Containers; at Spotify NYC offices, featuring engineers from Yandex, LinkedIn, Criteo, and NASA!
Cgroups, namespaces, and beyond: what are containers made from? (DockerCon Eu...Jérôme Petazzoni
Linux containers are different from Solaris Zones or BSD Jails: they use discrete kernel features like cgroups, namespaces, SELinux, and more. We will describe those mechanisms in depth, as well as demo how to put them together to produce a container. We will also highlight how different container runtimes compare to each other.
This talk was delivered at DockerCon Europe 2015 in Barcelona.
Containerization Is More than the New VirtualizationC4Media
Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1E5GzZX.
Jérôme Petazzoni borrows from his experience at Docker Inc. to explain live applications running in Docker, including reading logs, remote access, and troubleshooting tips. Filmed at qconsf.com.
Jérôme Petazzoni is a senior engineer at dotCloud, where he rotates between Ops, Support and Evangelist duties and the nickname of “master Yoda”, has earned.
If you're not familiar with Docker yet, here is your chance to catch up: a quick overview of the Open Source Docker Engine, and its associated services delivered through the Docker Hub. It also includes Jérôme will also discuss the new features of Docker 1.0, and briefly explain how you can run and maintain Docker on Azure. In addition, an Azure team member will demonstrate how deploy docker to Azure. The presentation will be followed by a Q&A session!
Describes what is lightweight virtualization and containers, and the low-level mechanisms in the Linux kernel that it relies on: namespaces, cgroups. It also gives details on AUFS. Those component together are the key to understanding how modern systems like Docker (http://www.docker.io/) work.
Introduction to Docker at Glidewell Laboratories in Orange CountyJérôme Petazzoni
In this presentation we will introduce Docker, and how you can use it to build, ship, and run any application, anywhere. The presentation included short demos, links to further material, and of course Q&As. If you are already a seasoned Docker user, this presentation will probably be redundant; but if you started to use Docker and are still struggling with some of his facets, you'll learn some!
presentation held at SUSE Linux Expert Forum December 2014
Linux container history and Linux namespaces
examples include:
* Move a VPN connection to its own namespace(p 25)
* User namespaces demo(p 28)
see collection of useful articles and advanced container usecases pp 29
While probably the most prominent, Docker is not the only tool for building and managing containers. Originally meant to be a "chroot on steroids" to help debug systemd, systemd-nspawn provides a fairly uncomplicated approach to work with containers. Being part of systemd, it is available on most recent distributions out-of-the-box and requires no additional dependencies.
This deck will introduce a few concepts involved in containers and will guide you through the steps of building a container from scratch. The payload will be a simple service, which will be automatically activated by systemd when the first request arrives.
Anatomy of a Container: Namespaces, cgroups & Some Filesystem Magic - LinuxConJérôme Petazzoni
Containers are everywhere. But what exactly is a container? What are they made from? What's the difference between LXC, butts-nspawn, Docker, and the other container systems out there? And why should we bother about specific filesystems?
In this talk, Jérôme will show the individual roles and behaviors of the components making up a container: namespaces, control groups, and copy-on-write systems. Then, he will use them to assemble a container from scratch, and highlight the differences (and likelinesses) with existing container systems.
Union FileSystem - A Building Blocks Of a ContainerKnoldus Inc.
Namespace, CGroup, and Union file-system are the basic building blocks of a container. Let’s have our focus on file-system. Why yet another file-system for the container? Is Conventional Linux file-systems like ext2, ext3, ext4, XFS, etc. not good enough to meet the purpose? In this blog post, I will try to answer these questions. Here we will be delving deeply into the Union File System and a few of its essential properties.
Historically, sharing a Linux server entailed all kinds of untenable compromises. In addition to the security concerns, there was simply no good way to keep one application from hogging resources and messing with the others. The classic “noisy neighbor” problem made shared systems the bargain-basement slums of the Internet, suitable only for small or throwaway projects.
Serious use-cases traditionally demanded dedicated systems. Over the past decade virtualization (in conjunction with Moore’s law) has democratized the availability of what amount to dedicated systems, and the result is hundreds of thousands of websites and applications deployed into VPS or cloud instances. It’s a step in the right direction, but still has glaring flaws.
Most of these websites are just piles of code sitting on a server somewhere. How did that code got there? How can it can be scaled? Secured? Maintained? It’s anybody’s guess. There simply isn’t enough SysAdmin talent in the world to meet the demands of managing all these apps with anything close to best practices without a better model.
Containers are a whole new ballgame. Unlike VMs, you skip the overhead of running an entire OS for every application environment. There’s also no need to provision a whole new machine to have a place to deploy, meaning you can spin up or scale your application with orders of magnitude more speed and accuracy.
his workshop will shed light on a modern solution to solve application portability, building, delivery, packaging, and system dependency issues. Containers especially Docker have seen accelerated adoption in the web, cloud and recently the enterprise. HPC environments are seeing something similar to the introduction of HPC containers Singularity and Shifter. They provide a good use case for solving software portability, not to mention ensure repeatability of results. Not to mention their ECO system provides for the better development, delivery, testing workflows that were alien to most of HPC environments. This workshop will cover the Theory and hands-on of containers and Its ecosystem. Introducing Docker and singularity containers; Docker as a general-purpose container for almost any app, Singularity as the particular container technology for HPC. The workshop will go over the foundations of the containers platform, including an overview of the platform system components: images, containers, repositories, clustering, and orchestration. The strategy is to demonstrate through "live demo, and hands-on exercises." The reuse case of containers in building a portable distributed application cluster running a variety of workloads including HPC workload.
Extending OpenShift Origin: Build Your Own Cartridge with Bill DeCoste of Red...OpenShift Origin
Extending OpenShift Origin: Build Your Own Cartridge
Presenters: Bill DeCoste
Cartridges allow developers to provide services running on top of the Red Hat OpenShift Platform-as-a-Service (PaaS). OpenShift already provides cartridges for numerous web application frameworks and databases. Writing your own cartridges allows you to customize or enhance an existing service, or provide new services. In this session, the presenter will discuss best practices for cartridge development and the latest changes in the OpenShift cartridge support.
* Latest changes made in the platform to ease cartridge development
* OpenShift Cartridges vs. plugins
* Outline for development of a new cartridge
* Customization of existing cartridges
* Quickstarts: leveraging a cartridge or cartridges to provide a complete application
Advanced cgroups and namespaces
This talk picks up where we left off in the previous cgroups and namespaces talk and dive in even deeper!
Agenda:
* cgroups v2 design (cgroup v2 was started to be merged in the current kernel, 4.4)
* cgroups v2 examples (migrating tasks, enabling and disabling controllers, and more).
* comparison between cgroup v2 unified hierarchy and cgroup v1 legacy hierarchy.
* PIDs namespaces (from kernel 4.3)
* cgroup namespaces (not merged yet)
Docker is the Open Source container engine. It lets you author, run, and manage software containers. Escape from dependency hell, and make deployment a breeze! This presentation includes the standard Docker intro (actualized for Docker 0.11) as well as some insights about how to perform orchestration and multi-host container linking.
Introduction to OS LEVEL Virtualization & ContainersVaibhav Sharma
This Presentation contains information about os level virtualization and Containers internals. It has used other material on slide share which is referenced in Notes of PPT
History and Basics of containers, LXC, Docker and Kubernetes. This presentation is given to Engineering colleage students at VIT DevFest 2018. Beginner to Intermediate level.
Containerization is more than the new Virtualization: enabling separation of ...Jérôme Petazzoni
Docker offers a new, lightweight approach to application
portability. Applications are shipped using a common container format,
and managed with a high-level API. Their processes run within isolated
namespaces which abstract the operating environment, independently of
the distribution, versions, network setup, and other details of this
environment.
This "containerization" has often been nicknamed "the new
virtualization". But containers are more than lightweight virtual
machines. Beyond their smaller footprint, shorter boot times, and
higher consolidation factors, they also bring a lot of new features
and use cases which were not possible with classical virtual machines.
We will focus on one of those features: separation of operational
concerns. Specifically, we will demonstrate how some fundamental tasks
like logging, remote access, backups, and troubleshooting can be
entirely decoupled from the deployment of applications and
services. This decoupling results in independent, smaller, simpler
moving parts; just like microservice architectures break down large
monolithic apps in more manageable components.
Introduction to Docker, December 2014 "Tour de France" EditionJérôme Petazzoni
Docker, the Open Source container Engine, lets you build, ship and run, any app, anywhere.
This is the presentation which was shown in December 2014 for the "Tour de France" in Paris, Lille, Lyon, Nice...
A guest lecture at National University of Defense Technology (NUDT) in 2016 to postgraduate students in China about emerging technologies in the Linux operating system.
Multiple Your Crypto Portfolio with the Innovative Features of Advanced Crypt...Hivelance Technology
Cryptocurrency trading bots are computer programs designed to automate buying, selling, and managing cryptocurrency transactions. These bots utilize advanced algorithms and machine learning techniques to analyze market data, identify trading opportunities, and execute trades on behalf of their users. By automating the decision-making process, crypto trading bots can react to market changes faster than human traders
Hivelance, a leading provider of cryptocurrency trading bot development services, stands out as the premier choice for crypto traders and developers. Hivelance boasts a team of seasoned cryptocurrency experts and software engineers who deeply understand the crypto market and the latest trends in automated trading, Hivelance leverages the latest technologies and tools in the industry, including advanced AI and machine learning algorithms, to create highly efficient and adaptable crypto trading bots
Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...Globus
Large Language Models (LLMs) are currently the center of attention in the tech world, particularly for their potential to advance research. In this presentation, we'll explore a straightforward and effective method for quickly initiating inference runs on supercomputers using the vLLM tool with Globus Compute, specifically on the Polaris system at ALCF. We'll begin by briefly discussing the popularity and applications of LLMs in various fields. Following this, we will introduce the vLLM tool, and explain how it integrates with Globus Compute to efficiently manage LLM operations on Polaris. Attendees will learn the practical aspects of setting up and remotely triggering LLMs from local machines, focusing on ease of use and efficiency. This talk is ideal for researchers and practitioners looking to leverage the power of LLMs in their work, offering a clear guide to harnessing supercomputing resources for quick and effective LLM inference.
Quarkus Hidden and Forbidden ExtensionsMax Andersen
Quarkus has a vast extension ecosystem and is known for its subsonic and subatomic feature set. Some of these features are not as well known, and some extensions are less talked about, but that does not make them less interesting - quite the opposite.
Come join this talk to see some tips and tricks for using Quarkus and some of the lesser known features, extensions and development techniques.
Globus Connect Server Deep Dive - GlobusWorld 2024Globus
We explore the Globus Connect Server (GCS) architecture and experiment with advanced configuration options and use cases. This content is targeted at system administrators who are familiar with GCS and currently operate—or are planning to operate—broader deployments at their institution.
First Steps with Globus Compute Multi-User EndpointsGlobus
In this presentation we will share our experiences around getting started with the Globus Compute multi-user endpoint. Working with the Pharmacology group at the University of Auckland, we have previously written an application using Globus Compute that can offload computationally expensive steps in the researcher's workflows, which they wish to manage from their familiar Windows environments, onto the NeSI (New Zealand eScience Infrastructure) cluster. Some of the challenges we have encountered were that each researcher had to set up and manage their own single-user globus compute endpoint and that the workloads had varying resource requirements (CPUs, memory and wall time) between different runs. We hope that the multi-user endpoint will help to address these challenges and share an update on our progress here.
Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...Anthony Dahanne
Les Buildpacks existent depuis plus de 10 ans ! D’abord, ils étaient utilisés pour détecter et construire une application avant de la déployer sur certains PaaS. Ensuite, nous avons pu créer des images Docker (OCI) avec leur dernière génération, les Cloud Native Buildpacks (CNCF en incubation). Sont-ils une bonne alternative au Dockerfile ? Que sont les buildpacks Paketo ? Quelles communautés les soutiennent et comment ?
Venez le découvrir lors de cette session ignite
Code reviews are vital for ensuring good code quality. They serve as one of our last lines of defense against bugs and subpar code reaching production.
Yet, they often turn into annoying tasks riddled with frustration, hostility, unclear feedback and lack of standards. How can we improve this crucial process?
In this session we will cover:
- The Art of Effective Code Reviews
- Streamlining the Review Process
- Elevating Reviews with Automated Tools
By the end of this presentation, you'll have the knowledge on how to organize and improve your code review proces
Understanding Globus Data Transfers with NetSageGlobus
NetSage is an open privacy-aware network measurement, analysis, and visualization service designed to help end-users visualize and reason about large data transfers. NetSage traditionally has used a combination of passive measurements, including SNMP and flow data, as well as active measurements, mainly perfSONAR, to provide longitudinal network performance data visualization. It has been deployed by dozens of networks world wide, and is supported domestically by the Engagement and Performance Operations Center (EPOC), NSF #2328479. We have recently expanded the NetSage data sources to include logs for Globus data transfers, following the same privacy-preserving approach as for Flow data. Using the logs for the Texas Advanced Computing Center (TACC) as an example, this talk will walk through several different example use cases that NetSage can answer, including: Who is using Globus to share data with my institution, and what kind of performance are they able to achieve? How many transfers has Globus supported for us? Which sites are we sharing the most data with, and how is that changing over time? How is my site using Globus to move data internally, and what kind of performance do we see for those transfers? What percentage of data transfers at my institution used Globus, and how did the overall data transfer performance compare to the Globus users?
How Recreation Management Software Can Streamline Your Operations.pptxwottaspaceseo
Recreation management software streamlines operations by automating key tasks such as scheduling, registration, and payment processing, reducing manual workload and errors. It provides centralized management of facilities, classes, and events, ensuring efficient resource allocation and facility usage. The software offers user-friendly online portals for easy access to bookings and program information, enhancing customer experience. Real-time reporting and data analytics deliver insights into attendance and preferences, aiding in strategic decision-making. Additionally, effective communication tools keep participants and staff informed with timely updates. Overall, recreation management software enhances efficiency, improves service delivery, and boosts customer satisfaction.
Unleash Unlimited Potential with One-Time Purchase
BoxLang is more than just a language; it's a community. By choosing a Visionary License, you're not just investing in your success, you're actively contributing to the ongoing development and support of BoxLang.
How Does XfilesPro Ensure Security While Sharing Documents in Salesforce?XfilesPro
Worried about document security while sharing them in Salesforce? Fret no more! Here are the top-notch security standards XfilesPro upholds to ensure strong security for your Salesforce documents while sharing with internal or external people.
To learn more, read the blog: https://www.xfilespro.com/how-does-xfilespro-make-document-sharing-secure-and-seamless-in-salesforce/
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...Globus
The U.S. Geological Survey (USGS) has made substantial investments in meeting evolving scientific, technical, and policy driven demands on storing, managing, and delivering data. As these demands continue to grow in complexity and scale, the USGS must continue to explore innovative solutions to improve its management, curation, sharing, delivering, and preservation approaches for large-scale research data. Supporting these needs, the USGS has partnered with the University of Chicago-Globus to research and develop advanced repository components and workflows leveraging its current investment in Globus. The primary outcome of this partnership includes the development of a prototype enterprise repository, driven by USGS Data Release requirements, through exploration and implementation of the entire suite of the Globus platform offerings, including Globus Flow, Globus Auth, Globus Transfer, and Globus Search. This presentation will provide insights into this research partnership, introduce the unique requirements and challenges being addressed and provide relevant project progress.
Designing for Privacy in Amazon Web ServicesKrzysztofKkol1
Data privacy is one of the most critical issues that businesses face. This presentation shares insights on the principles and best practices for ensuring the resilience and security of your workload.
Drawing on a real-life project from the HR industry, the various challenges will be demonstrated: data protection, self-healing, business continuity, security, and transparency of data processing. This systematized approach allowed to create a secure AWS cloud infrastructure that not only met strict compliance rules but also exceeded the client's expectations.
Check out the webinar slides to learn more about how XfilesPro transforms Salesforce document management by leveraging its world-class applications. For more details, please connect with sales@xfilespro.com
If you want to watch the on-demand webinar, please click here: https://www.xfilespro.com/webinars/salesforce-document-management-2-0-smarter-faster-better/
SOCRadar Research Team: Latest Activities of IntelBrokerSOCRadar
The European Union Agency for Law Enforcement Cooperation (Europol) has suffered an alleged data breach after a notorious threat actor claimed to have exfiltrated data from its systems. Infamous data leaker IntelBroker posted on the even more infamous BreachForums hacking forum, saying that Europol suffered a data breach this month.
The alleged breach affected Europol agencies CCSE, EC3, Europol Platform for Experts, Law Enforcement Forum, and SIRIUS. Infiltration of these entities can disrupt ongoing investigations and compromise sensitive intelligence shared among international law enforcement agencies.
However, this is neither the first nor the last activity of IntekBroker. We have compiled for you what happened in the last few days. To track such hacker activities on dark web sources like hacker forums, private Telegram channels, and other hidden platforms where cyber threats often originate, you can check SOCRadar’s Dark Web News.
Stay Informed on Threat Actors’ Activity on the Dark Web with SOCRadar!
Developing Distributed High-performance Computing Capabilities of an Open Sci...Globus
COVID-19 had an unprecedented impact on scientific collaboration. The pandemic and its broad response from the scientific community has forged new relationships among public health practitioners, mathematical modelers, and scientific computing specialists, while revealing critical gaps in exploiting advanced computing systems to support urgent decision making. Informed by our team’s work in applying high-performance computing in support of public health decision makers during the COVID-19 pandemic, we present how Globus technologies are enabling the development of an open science platform for robust epidemic analysis, with the goal of collaborative, secure, distributed, on-demand, and fast time-to-solution analyses to support public health.
Globus Compute wth IRI Workflows - GlobusWorld 2024Globus
As part of the DOE Integrated Research Infrastructure (IRI) program, NERSC at Lawrence Berkeley National Lab and ALCF at Argonne National Lab are working closely with General Atomics on accelerating the computing requirements of the DIII-D experiment. As part of the work the team is investigating ways to speedup the time to solution for many different parts of the DIII-D workflow including how they run jobs on HPC systems. One of these routes is looking at Globus Compute as a way to replace the current method for managing tasks and we describe a brief proof of concept showing how Globus Compute could help to schedule jobs and be a tool to connect compute at different facilities.
5. Limiting process resources in Linux - chroot
● chroot: It is an operation that changes the apparent root directory for the current
running process and their children. A program that is run in such a modified
environment cannot access files and commands outside that environmental directory
tree. This modified environment is called a chroot jail.
• It is a way to isolate apps
• Change the process directory root
• chroot /path/to/newRootDir
• Deboostrap
• A simple tool to install a bases debian system in any subdirectory
• https://wiki.debian.org/Debootstrap
Martín Baez
Debooststrap example
# mkdir /stable-chroot
# debootstrap stable /stable-chroot
http://deb.debian.org/debian/
6. Limiting process resources in Linux - cgroups
● cgroups
• Started in 2006 with the name “process containers”(Paul Menage and Rohit Seth)
• Included in Linux Kernel when version 2.6.24 was released
• "Cgroups allow you to allocate resources — such as CPU time, system memory,
network bandwidth, or combinations of these resources — among user-defined groups
of tasks (processes) running on a system"
• All processes on a Linux system are child processes of a common parent:
the init process(or systemd), which is executed by the kernel at boot time and starts
other processes (which may in turn start child processes of their own). Because all
processes descend from a single parent, the Linux process model is a single hierarchy,
or tree.
• Additionally, every Linux process except init inherits the environment (such as the PATH
variable)
• Many different hierarchies of cgroups can exist simultaneously on a system. If the Linux
process model is a single tree of processes, then the cgroup model is one or more
separate, unconnected trees of tasks (i.e. processes).
•
Martín Baez
7. Limiting process resources in Linux – cgroups(cont.)
● cgroup model
• each hierarchy is attached to one or more subsystems.
• A subsystem represents a single resource
● cgroup subsystems
• cpu
• To provide cgroup tasks access to the CPU.
• memory
• Sets limits on memory use by tasks in a cgroup
• devices
• Allows or denies access to devices by tasks in a cgroup.
• ns
• Namespaces subsystem
• Others
• Freezer, net_cls,net_prio,perf_event
Martín Baez
8. Limiting process resources in Linux –
cgroups(Example)
• Let's create two groups:
• Assing 70% cpu time to one cgroup(red)
• Assing 70% cpu time to the other(blue)
We will create two cgroups in cpu subsystem: cpu_high and cpu_low
# mkdir /sys/fs/cgroup/cpu/cpu_high
# mkdir /sys/fs/cgroup/cpu/cpu_low
cgroup cpu_high:70% of CPU time and cpu_low will get 30% cput time
# echo 717 > /sys/fs/cgroup/cpu/cpu_high/cpu.shares
# echo 307 > /sys/fs/cgroup/cpu/cpu_low/cpu.shares
taskset command allow us to attach a process to a core.
# taskset -c 0 xterm -bg blue &
# taskset -c 0 xterm -bg red &
Martín Baez
9. Limiting process resources in Linux –
cgroups(Example)
Martín Baez
$ echo $$ > /sys/fs/cgroup/cpu/cpu_high/cgroup.procs
$ md5sum /dev/urandom &
$ echo $$ > /sys/fs/cgroup/cpu/cpu_low/cgroup.procs
$ md5sum /dev/urandom &
10. Limiting process resources in Linux – Namespaces
● They allow for isolation of global system resources between independent processes.
For example, the PID namespace isolates the PID number space. This
means that two processes running on the same host can have the same
PID!
● Without namespaces, a process running in container A could, for example, umount an
important filesystem in container B.
● The idea is that you can't interfere with something if it’s not visible to you.
Martín Baez
11. GNU Linux Tools
/proc virtual file system
● The /proc filesystem contains a illusionary filesystem. It does not exist on a disk.
Instead, the kernel creates it in memory. It is used to provide information about the
system.
● $ man proc
● /proc/1
● /proc/cpuinfo
● /proc/meminfo
● /proc/stat
● There are many commands that do little more than read the above files and
format them for easier understanding
● top
● ps
● free
Some System calls collect information from the environment in GNU Linux
Martín Baez
13. Containers – Motivation (from the dev part of
devops)
● You don’t need to install a bunch of language environments on your system. You can
simply run the ruby / python / java application inside docker.
● Consistent development environments for the entire team.
● Different versions of same programming language without having to hack arounds your
machine.
● Think of many jvm version and vendors in you laptop(JAVA_HOME, PATH....)
● If it runs in your container, it will run on your Linux server
● If you’re having a hard time building / compiling the application code, then build it
inside Docker
● https://cloud.google.com/containers/
Martín Baez
14. Containers – They are not a new idea
● Linux Containers(LXC)
● Solaris Zones
● BSD Jails
● Docker
● Based on LXC in the past (actually it has its ows libraries -> libcontainer)
● OpenVZ
● Heroku
● Awesome! :)
Martín Baez
15. Containers – What they are
● Instead of virtualizing the hardware stack as with the virtual machines approach,
containers virtualize at the operating.
● "Containers are a method of operating system virtualization that allow you to run an
application and its dependencies in resource-isolated processes" (Amazon)
● This means that containers are far more lightweight: they share the OS kernel, start
much faster, and use a fraction of the memory compared to booting an entire OS.
● Docker is the most popular, open-source container format.
● Benefits
Consistent Environment
Run Anywhere
Isolation
Martín Baez
16. Containers vs VMs
● Own network space
● Own network interface
● Can install packages
● Can run processes
● Can be packaged into images
They are not VMS at all
Martín Baez
17. Containers – Example
lxc:
# ls -l /usr/share/lxc/templates/
# lxc-create -t /usr/share/lxc/templates/lxc-alpine -n lxc-alpine
# lxc-start -n lxc-alpine
# lxc-attach -n lxc-alpine
# lxc-stop -n lxc-apline
It is possible to run docker on lxc.
In the past docker was based on lxcMartín Baez
18. Containers – How they work?
● In general, Containers running on Linux makes use of kernel namespaces to provide the
isolated workspace called the container.
● When you run a container, Docker creates a set of namespaces for that container.
These namespaces provide a layer of isolation.
PID namespace for process isolation.
NET namespace for managing network interfaces.
IPC namespace for managing access to IPC resources.
MNT namespace for managing filesystem mount points.
UTS namespace for isolating kernel and version identifiers.
● They also makes use of kernel control groups for resource allocation and isolation. A
cgroup limits an application to a specific set of resources.
Martín Baez
19. Limiting Containers resources(Docker)
● Docker
Memory
-m / --memory
--memory-swap
If --memory and --memory-swap are set to the same value, this prevents containers
from using any swap
CPU
--cpus
--cpus="1.5", the container is guaranteed at most one and a half of the CPUs
Realtime scheduler
https://docs.docker.com/config/containers/resource_constraints
/
Martín Baez
20. GNU Linux Tools in Docker
Important Issues:
● $ docker run -it -m 512m centos bash
● [root@aba9f6744c3f /]# top
● [root@aba9f6744c3f /]# free –m
● [root@aba9f6744c3f /]# lscpu
● /proc/meminfo, /proc/vmstat and friends are not not cgroup-aware
● They will always display memory numbers from the host system
● Processes inside a container can not rely on free, top and others to determine how
much memory they have to work with
● Auto-scaling is usually a function of how much memory is available INSIDE the
container(this information needs to be accessible from inside the container).
21. Java ergonomics
The JVM provides platform-dependent default selections for the garbage collector, heap
size, and runtime compiler.
● Java processes in Linux don’t behave as expected
Java ergonomics
“Ergonomics is the process by which the Java Virtual Machine (JVM) and garbage collection
tuning, such as behavior-based tuning, improve application performance.”
The JVM provides platform-dependent default selections for the garbage collector, heap size, and
runtime compiler.”.
● Garbage Collector, Heap, and Runtime Compiler Default Selections
A class of machine referred to as a server-class machine has been defined as a machine with the following:
2 or more physical processors
2 or more GB of physical memory
On server-class machines, the following are selected by default: Throughput garbage
collector,v Initial heap size of 1/64 of physical memory up to 1 GB, maximum heap size of 1/4
of physical memory up to 1 GB, Server runtime compiler
https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/ergonomics.htm
l
22. An Example
● Java + Spring Boot + Embebed Tomcat
● Reference: https://spring.io/guides/gs/spring-boot-Docker/
Dockerfile:
FROM java:8
ADD /target/example*.jar javaopts.jar# Entry in json format
ENTRYPOINT [“java”, “-jar”, “/javaopts.jar”]
How we run it:
We build the image
$ docker build -t spring-boot-javaopts .
We create and run a container
$ docker run spring-boot-javaopts
Martín Baez
23. Java 9 support for Docker CPU and memory limits
Memory Issues:
• To tell the JVM to be aware of Docker memory limits( in the absence of setting a
maximum Java heap via –Xmx)
• There are two JVM command line options required, -XX:
+UnlockExperimentalVMOptions -
• XX:+UseCGroupMemoryLimitForHeap.
• The -XX:+UnlockExperimentalVMOptions is required because in a future release
transparent identification of Docker memory limits is the goal.
• When these two JVM command line options are used, and -Xmx is not specified, the JVM
will look at the Linux cgroup configuration, Docker containers also use
cgroups configuration for CPU limits too.
24. Java 9 support for Docker CPU and memory limits
CPU Issues:
• As of Java SE 8u131, and in JDK 9, the JVM is Docker-aware with respect to Docker CPU
limits transparently.
• If -XX:ParalllelGCThreads, or -XX:CICompilerCount are not specified as command line
options, the JVM will apply the Docker CPU limit as the number of CPUs the JVM sees on
the system. The JVM will then adjust the number of GC threads and JIT compiler threads
just like it would as if it were running on a bare metal system with number of CPUs set as
the Docker CPU limit.
• If -XX:ParallelGCThreads or -XX:CICompilerCount are specified as JVM command line
options, and Docker CPU limit are specified, the JVM will use the -XX:ParallelGCThreads
and -XX:CICompilerCount values.
25. Java 10 support for Docker
Issues:
● https://bugs.openjdk.java.net/browse/JDK-8146115
"When running in a container, the operating system functions used provide information
about the host and do not include the container configuration and limits. The VM and core
libraries will be modified as part of this RFE to first determine if the current running process
is running in a container."
26. References
oDocker Internals: http://docker-saigon.github.io/post/Docker-Internals/
oUnderstanding the Docker Internals:
https://medium.com/@nagarwal/understanding-the-docker-internals-7ccb052
ce9fe
oLimit a container's resources
https://docs.docker.com/config/containers/resource_constraints/
oJava inside docker: What you must know to not FAIL:
https://developers.redhat.com/blog/2017/03/14/java-inside-docker/
oMemory inside Linux containers: https://fabiokung.com/2014/03/13/memory-
inside-linux-containers/
30. ROSA
RIOJUEVES 11 DE OCTUBRE -
18:15 hs
Metropolitano Eventos
Salón Contemporáneo (Junín 501)
Abrimos la INSCRIPCIÓN GRATUITA el lunes 17 de
septiembre!
Sigan nuestras redes para estar atentos
EndavaLat
am
@EndavaLat
am