Using functional programming within an industrial product group: perspectives...Anil Madhavapeddy
We present a case-study of using OCaml within a large product development project, focussing on both the technical and non- technical issues that arose as a result. We draw comparisons between the OCaml team and the other teams that worked on the project, providing comparative data on hiring patterns and cross-team code contribution.
Docker 1.11 Meetup: Containerd and runc, by Arnaud Porterie and Michael Crosby Michelle Antebi
In this talk, Michal Crosby will present on runC and Containerd, the internals and how they work together to start and manage containers in Docker. Afterwards, Arnaud Porterie will touch on about what was shipped in 1.11 and how it will enable some of the things we are working on for 1.12.
You Call that Micro, Mr. Docker? How OSv and Unikernels Help Micro-services S...rhatr
OSv is the new open source unikernel technology that combines the power of virtualization and micro-services architecture. This combination allows unmodified applications to be packaged just like Docker containers while at the same time outperform bare-metal deployments. Yes. You've heard it right: for the first time ever we can stop asking the question of how much performance would I lose if I virtualize. OSv lets you ask a different question: how much would my application gain in performance if I virtualize it. This talk will start by looking into the architecture of OSv and the kind of optimizations it makes possible for native, unmodified applications. We will then focus on JVM-specific optimizations and specifically on speedups available to micro-service oriented applications when they are being deployed on OSv.
runC: The little engine that could (run Docker containers) by Docker Captain ...Docker, Inc.
With the announcement of the OCI by Solomon Hykes at last summer's DockerCon, a Docker-contributed reference implementation of the OCI spec, called runC, was born. While some of you may have tried runC or have a history of poking at the OS layer integration library to Linux namespaces, cgroups and the like (known as libcontainer), many of you may not know what runC offers. In this talk Phil Estes, Docker engine maintainer who has also contributed to libcontainer and runC, will show what's possible using runC as a lightweight and fast runtime environment to experiment with lower-level features of the container runtime. Phil will introduce a conversion tool called "riddler", which can inspect and convert container configurations from Docker into the proper OCI configuration bundle for easy conversion between the two environments. He'll also demonstrate how to make custom configurations for trying out security features like user namespaces and seccomp profiles.
KVM and docker LXC Benchmarking with OpenStackBoden Russell
Passive benchmarking with docker LXC and KVM using OpenStack hosted in SoftLayer. These results provide initial incite as to why LXC as a technology choice offers benefits over traditional VMs and seek to provide answers as to the typical initial LXC question -- "why would I consider Linux Containers over VMs" from a performance perspective.
Results here provide insight as to:
- Cloudy ops times (start, stop, reboot) using OpenStack.
- Guest micro benchmark performance (I/O, network, memory, CPU).
- Guest micro benchmark performance of MySQL; OLTP read, read / write complex and indexed insertion.
- Compute node resource consumption; VM / Container density factors.
- Lessons learned during benchmarking.
The tests here were performed using OpenStack Rally to drive the OpenStack cloudy tests and various other linux tools to test the guest performance on a "micro level". The nova docker virt driver was used in the Cloud scenario to realize VMs as docker LXC containers and compared to the nova virt driver for libvirt KVM.
Please read the disclaimers in the presentation as this is only intended to be the "chip of the ice burg".
Using functional programming within an industrial product group: perspectives...Anil Madhavapeddy
We present a case-study of using OCaml within a large product development project, focussing on both the technical and non- technical issues that arose as a result. We draw comparisons between the OCaml team and the other teams that worked on the project, providing comparative data on hiring patterns and cross-team code contribution.
Docker 1.11 Meetup: Containerd and runc, by Arnaud Porterie and Michael Crosby Michelle Antebi
In this talk, Michal Crosby will present on runC and Containerd, the internals and how they work together to start and manage containers in Docker. Afterwards, Arnaud Porterie will touch on about what was shipped in 1.11 and how it will enable some of the things we are working on for 1.12.
You Call that Micro, Mr. Docker? How OSv and Unikernels Help Micro-services S...rhatr
OSv is the new open source unikernel technology that combines the power of virtualization and micro-services architecture. This combination allows unmodified applications to be packaged just like Docker containers while at the same time outperform bare-metal deployments. Yes. You've heard it right: for the first time ever we can stop asking the question of how much performance would I lose if I virtualize. OSv lets you ask a different question: how much would my application gain in performance if I virtualize it. This talk will start by looking into the architecture of OSv and the kind of optimizations it makes possible for native, unmodified applications. We will then focus on JVM-specific optimizations and specifically on speedups available to micro-service oriented applications when they are being deployed on OSv.
runC: The little engine that could (run Docker containers) by Docker Captain ...Docker, Inc.
With the announcement of the OCI by Solomon Hykes at last summer's DockerCon, a Docker-contributed reference implementation of the OCI spec, called runC, was born. While some of you may have tried runC or have a history of poking at the OS layer integration library to Linux namespaces, cgroups and the like (known as libcontainer), many of you may not know what runC offers. In this talk Phil Estes, Docker engine maintainer who has also contributed to libcontainer and runC, will show what's possible using runC as a lightweight and fast runtime environment to experiment with lower-level features of the container runtime. Phil will introduce a conversion tool called "riddler", which can inspect and convert container configurations from Docker into the proper OCI configuration bundle for easy conversion between the two environments. He'll also demonstrate how to make custom configurations for trying out security features like user namespaces and seccomp profiles.
KVM and docker LXC Benchmarking with OpenStackBoden Russell
Passive benchmarking with docker LXC and KVM using OpenStack hosted in SoftLayer. These results provide initial incite as to why LXC as a technology choice offers benefits over traditional VMs and seek to provide answers as to the typical initial LXC question -- "why would I consider Linux Containers over VMs" from a performance perspective.
Results here provide insight as to:
- Cloudy ops times (start, stop, reboot) using OpenStack.
- Guest micro benchmark performance (I/O, network, memory, CPU).
- Guest micro benchmark performance of MySQL; OLTP read, read / write complex and indexed insertion.
- Compute node resource consumption; VM / Container density factors.
- Lessons learned during benchmarking.
The tests here were performed using OpenStack Rally to drive the OpenStack cloudy tests and various other linux tools to test the guest performance on a "micro level". The nova docker virt driver was used in the Cloud scenario to realize VMs as docker LXC containers and compared to the nova virt driver for libvirt KVM.
Please read the disclaimers in the presentation as this is only intended to be the "chip of the ice burg".
Over the last six years, Galois has been developing the Haskell Lightweight Virtual Machine, or HaLVM, a lightweight virtual machine that runs directly on the Xen hypervisor. The HaLVM's design is based on a notion of minimalism: Authors of HaLVM domains include only those libraries and features they require, allowing the HaLVM to have a very small initial resource footprint that scales in a very obvious fashion. In doing so, we hope to combine the minimalism and flexibility of a small kernel with an extended set of libraries with the simplicity and reliability of the strongly typed, high-level language Haskell.
While initially designed for running operating system design experiments, the HaLVM has grown over time to be a suitable platform for writing simple network appliances with a very narrow resource footprint. This work has been enabled by other Galois projects: a TCP-compliant network stack written in Haskell, and a fairly-complete file system written in Haskell. In the end, because we use Haskell and provide these libraries at the Haskell level, programmers can create complex software structures quickly, easily, and with the added assurance that the Haskell type system provides.
In this talk, we will provide an overview of the HaLVM and its design principles - pointing out where the HaLVM shines and where it is weak - and continue with some of our experiences using it over the last six years.
OSv: probably the best OS for cloud workloads you've never hear ofrhatr
OSv is the revolutionary new open source technology that combines the power of virtualization and micro-services architecture. This combination allows unmodified applications deployed in a virtualized environment to outperform bare-metal deployments. Yes. You've heard it right: for the first time ever we can stop asking the question of how much performance would I lose if I virtualize. OSv lets you ask a different question: how much would my application gain in performance if I virtualize it. This talk will start by looking into the architecture of OSv and the kind of optimizations it makes possible for native, unmodified applications. We will then focus on JVM-specific optimizations and specifically on speedups available to big data management distributed applications. Finally, we will look into the relationship between OSv and Docker and how that layering can help make OSv a secret sauce for turbo-charging Cloud Foundry application deployments.
CIF16: Building the Superfluid Cloud with Unikernels (Simon Kuenzer, NEC Europe)The Linux Foundation
The confluence of a number of relatively recent trends including the development of virtualization technologies, the deployment of micro datacenters at PoPs, and the availability of microservers, opens up the possibility of evolving the cloud, and the network it is connected to, towards a superfluid cloud: a model where parties other than infrastructure owners can quickly deploy and migrate virtualized services throughout the network (in the core, at aggregation points and at the edge), enabling a number of novel use cases including virtualized CPEs and on-the-fly services, among others. Towards this goal, we identify a number of required mechanisms and present early evaluation results of their implementation.
On an inexpensive commodity server, we are able to concurrently run up to 10,000 specialized virtual machines (based on unikernels), instantiate a VM in as little as 10 milliseconds, and migrate it in under 100 milliseconds.
Cgroups, namespaces and beyond: what are containers made from?Docker, Inc.
Linux containers are different from Solaris Zones or BSD Jails: they use discrete kernel features like cgroups, namespaces, SELinux, and more. We will describe those mechanisms in depth, as well as demo how to put them together to produce a container. We will also highlight how different container runtimes compare to each other.
Sharding Containers: Make Go Apps Computer-Friendly Again by Andrey Sibiryov Docker, Inc.
Go is, without doubt, a great language for writing massively concurrent programs. Nevertheless, our experience running Go under extreme load shows that there comes a point where assumptions and decisions made in Go runtime bite back on its users and lead to inferior performance, especially in high-throughput & high-load applications. This talk covers main reasons for this to happen and explores an interesting way to work around this issue: automatic local sharding with Docker.
Using Docker, local load-balancer and creativity, we can automatically shard & pin our apps in such a way so that the external observer (client, another microservice) would never see any difference. The result is that apps run faster, resource utilization is better, engineers are not frustrated when their Go suddenly breaks down and runs slow because they have a solution!
Overview of Docker 1.11 features(Covers Docker release summary till 1.11, runc/containerd, dns load balancing ipv6 service discovery, labels, macvlan/ipvlan)
Deploying containers and managing them on multiple Docker hosts, Docker Meetu...dotCloud
Docker is an open-source project to easily create lightweight, portable, self-sufficient containers from any application. The same container that a developer builds and tests on a laptop can run at scale, in production, on VMs, bare metal, OpenStack clusters, public clouds and more.
OSv presentation from Linux Foundation Collaboration SummitDon Marti
OSv is a lightweight operating system designed to improve performance and administration for applications deployed in the cloud. Learn about the speed and manageability wins from a brand-new OS that works on your private or public cloud.
Konrad Wilk is a Software Development Manager at Oracle. His group’s mission is to make Linux and Xen Project virtualization better and faster. As part of this work, Konrad has been the maintainer of the Xen Project subsystem in Linux, Xen Project maintainer and now also Release Manager for the 4.5 release of the Xen Project Hypervisor. Konrad has been active in the Linux and Xen Project communities for more than 6 years and was instrumental in adding Xen Project support to the Linux Kernel.
PaaSTA, Yelp's platform as a service (PaaS) built on top of open source tools, provides tooling for developers to quickly turn their microservice into a monitored, highly available application spanning multiple data centers and cloud regions. Nathan Handler outlines the technologies that power PaaSTA and discusses how Yelp uses PaaSTA to empower developers and solve key problems.
Video: https://youtu.be/vISUXKeoqXM
If you're not familiar with Docker yet, here is your chance to catch up: a quick overview of the Open Source Docker Engine, and its associated services delivered through the Docker Hub. It also includes Jérôme will also discuss the new features of Docker 1.0, and briefly explain how you can run and maintain Docker on Azure. In addition, an Azure team member will demonstrate how deploy docker to Azure. The presentation will be followed by a Q&A session!
DockerCon EU 2015: Docker Networking Deep DiveDocker, Inc.
with Jana Radhakrishnan, Lead Software Engineer at Docker, and David Tucker, Product Manager, Docker
Docker Networking is a feature of Docker Engine that allows you to create virtual networks and attach containers to them so you can create the network topology that is right for your application. The networked containers can even span multiple hosts, so you don’t have to worry about what host your container lands on. They can seamlessly communicate with each other wherever they are - thus enabling true distributed applications.
And Networking is pluggable, so you can use any third-party networking driver to power your networks without having to make any changes to your application.
Read more: https://blog.docker.com/2015/11/docker-multi-host-networking-ga/
--
Docker is an open platform for developers and system administrators to build, ship and run distributed applications. With Docker, IT organizations shrink application delivery from months to minutes, frictionlessly move workloads between data centers and the cloud and can achieve up to 20X greater efficiency in their use of computing resources. Inspired by an active community and by transparent, open source innovation, Docker containers have been downloaded more than 700 million times and Docker is used by millions of developers across thousands of the world’s most innovative organizations, including eBay, Baidu, the BBC, Goldman Sachs, Groupon, ING, Yelp, and Spotify. Docker’s rapid adoption has catalyzed an active ecosystem, resulting in more than 180,000 “Dockerized” applications, over 40 Docker-related startups and integration partnerships with AWS, Cloud Foundry, Google, IBM, Microsoft, OpenStack, Rackspace, Red Hat and VMware.
Learn more at www.docker.com
Before 1998, it was impossible to virtualize Intel x86 systems. VMware introduced a technique called binary translation. In 2006, Intel introduced VT-x virtualization technology for x86. This was hardware support for virtualization by Intel. This presentation discusses what kind of support does Intel provides for virtualization in detail. It also discusses different implementation techniques in brief.
Over the last six years, Galois has been developing the Haskell Lightweight Virtual Machine, or HaLVM, a lightweight virtual machine that runs directly on the Xen hypervisor. The HaLVM's design is based on a notion of minimalism: Authors of HaLVM domains include only those libraries and features they require, allowing the HaLVM to have a very small initial resource footprint that scales in a very obvious fashion. In doing so, we hope to combine the minimalism and flexibility of a small kernel with an extended set of libraries with the simplicity and reliability of the strongly typed, high-level language Haskell.
While initially designed for running operating system design experiments, the HaLVM has grown over time to be a suitable platform for writing simple network appliances with a very narrow resource footprint. This work has been enabled by other Galois projects: a TCP-compliant network stack written in Haskell, and a fairly-complete file system written in Haskell. In the end, because we use Haskell and provide these libraries at the Haskell level, programmers can create complex software structures quickly, easily, and with the added assurance that the Haskell type system provides.
In this talk, we will provide an overview of the HaLVM and its design principles - pointing out where the HaLVM shines and where it is weak - and continue with some of our experiences using it over the last six years.
OSv: probably the best OS for cloud workloads you've never hear ofrhatr
OSv is the revolutionary new open source technology that combines the power of virtualization and micro-services architecture. This combination allows unmodified applications deployed in a virtualized environment to outperform bare-metal deployments. Yes. You've heard it right: for the first time ever we can stop asking the question of how much performance would I lose if I virtualize. OSv lets you ask a different question: how much would my application gain in performance if I virtualize it. This talk will start by looking into the architecture of OSv and the kind of optimizations it makes possible for native, unmodified applications. We will then focus on JVM-specific optimizations and specifically on speedups available to big data management distributed applications. Finally, we will look into the relationship between OSv and Docker and how that layering can help make OSv a secret sauce for turbo-charging Cloud Foundry application deployments.
CIF16: Building the Superfluid Cloud with Unikernels (Simon Kuenzer, NEC Europe)The Linux Foundation
The confluence of a number of relatively recent trends including the development of virtualization technologies, the deployment of micro datacenters at PoPs, and the availability of microservers, opens up the possibility of evolving the cloud, and the network it is connected to, towards a superfluid cloud: a model where parties other than infrastructure owners can quickly deploy and migrate virtualized services throughout the network (in the core, at aggregation points and at the edge), enabling a number of novel use cases including virtualized CPEs and on-the-fly services, among others. Towards this goal, we identify a number of required mechanisms and present early evaluation results of their implementation.
On an inexpensive commodity server, we are able to concurrently run up to 10,000 specialized virtual machines (based on unikernels), instantiate a VM in as little as 10 milliseconds, and migrate it in under 100 milliseconds.
Cgroups, namespaces and beyond: what are containers made from?Docker, Inc.
Linux containers are different from Solaris Zones or BSD Jails: they use discrete kernel features like cgroups, namespaces, SELinux, and more. We will describe those mechanisms in depth, as well as demo how to put them together to produce a container. We will also highlight how different container runtimes compare to each other.
Sharding Containers: Make Go Apps Computer-Friendly Again by Andrey Sibiryov Docker, Inc.
Go is, without doubt, a great language for writing massively concurrent programs. Nevertheless, our experience running Go under extreme load shows that there comes a point where assumptions and decisions made in Go runtime bite back on its users and lead to inferior performance, especially in high-throughput & high-load applications. This talk covers main reasons for this to happen and explores an interesting way to work around this issue: automatic local sharding with Docker.
Using Docker, local load-balancer and creativity, we can automatically shard & pin our apps in such a way so that the external observer (client, another microservice) would never see any difference. The result is that apps run faster, resource utilization is better, engineers are not frustrated when their Go suddenly breaks down and runs slow because they have a solution!
Overview of Docker 1.11 features(Covers Docker release summary till 1.11, runc/containerd, dns load balancing ipv6 service discovery, labels, macvlan/ipvlan)
Deploying containers and managing them on multiple Docker hosts, Docker Meetu...dotCloud
Docker is an open-source project to easily create lightweight, portable, self-sufficient containers from any application. The same container that a developer builds and tests on a laptop can run at scale, in production, on VMs, bare metal, OpenStack clusters, public clouds and more.
OSv presentation from Linux Foundation Collaboration SummitDon Marti
OSv is a lightweight operating system designed to improve performance and administration for applications deployed in the cloud. Learn about the speed and manageability wins from a brand-new OS that works on your private or public cloud.
Konrad Wilk is a Software Development Manager at Oracle. His group’s mission is to make Linux and Xen Project virtualization better and faster. As part of this work, Konrad has been the maintainer of the Xen Project subsystem in Linux, Xen Project maintainer and now also Release Manager for the 4.5 release of the Xen Project Hypervisor. Konrad has been active in the Linux and Xen Project communities for more than 6 years and was instrumental in adding Xen Project support to the Linux Kernel.
PaaSTA, Yelp's platform as a service (PaaS) built on top of open source tools, provides tooling for developers to quickly turn their microservice into a monitored, highly available application spanning multiple data centers and cloud regions. Nathan Handler outlines the technologies that power PaaSTA and discusses how Yelp uses PaaSTA to empower developers and solve key problems.
Video: https://youtu.be/vISUXKeoqXM
If you're not familiar with Docker yet, here is your chance to catch up: a quick overview of the Open Source Docker Engine, and its associated services delivered through the Docker Hub. It also includes Jérôme will also discuss the new features of Docker 1.0, and briefly explain how you can run and maintain Docker on Azure. In addition, an Azure team member will demonstrate how deploy docker to Azure. The presentation will be followed by a Q&A session!
DockerCon EU 2015: Docker Networking Deep DiveDocker, Inc.
with Jana Radhakrishnan, Lead Software Engineer at Docker, and David Tucker, Product Manager, Docker
Docker Networking is a feature of Docker Engine that allows you to create virtual networks and attach containers to them so you can create the network topology that is right for your application. The networked containers can even span multiple hosts, so you don’t have to worry about what host your container lands on. They can seamlessly communicate with each other wherever they are - thus enabling true distributed applications.
And Networking is pluggable, so you can use any third-party networking driver to power your networks without having to make any changes to your application.
Read more: https://blog.docker.com/2015/11/docker-multi-host-networking-ga/
--
Docker is an open platform for developers and system administrators to build, ship and run distributed applications. With Docker, IT organizations shrink application delivery from months to minutes, frictionlessly move workloads between data centers and the cloud and can achieve up to 20X greater efficiency in their use of computing resources. Inspired by an active community and by transparent, open source innovation, Docker containers have been downloaded more than 700 million times and Docker is used by millions of developers across thousands of the world’s most innovative organizations, including eBay, Baidu, the BBC, Goldman Sachs, Groupon, ING, Yelp, and Spotify. Docker’s rapid adoption has catalyzed an active ecosystem, resulting in more than 180,000 “Dockerized” applications, over 40 Docker-related startups and integration partnerships with AWS, Cloud Foundry, Google, IBM, Microsoft, OpenStack, Rackspace, Red Hat and VMware.
Learn more at www.docker.com
Before 1998, it was impossible to virtualize Intel x86 systems. VMware introduced a technique called binary translation. In 2006, Intel introduced VT-x virtualization technology for x86. This was hardware support for virtualization by Intel. This presentation discusses what kind of support does Intel provides for virtualization in detail. It also discusses different implementation techniques in brief.
GoLightly: Building VM-Based Language Runtimes with Google GoEleanor McHugh
Expanded and updated version of my Strange Loop 2010 presentation on how to program in Google's Go language with an emphasis on the kind of tricks you might want to perform when implementing a language runtime.
UPDATED OCTOBER 2015: Unikernels are small, fast, easily deployable, and very secure application stacks. Lacking a traditional operating system layer, they provide a new way of looking at the cloud which goes beyond the methodologies used by Docker and other container technologies.
This is an update of the deck as delivered by Russell Pavlicek. This includes some ground-breaking work done in the Rump Kernel project to bring web servers, database, and scripting language into the world of Unikernels.
Deck result of the Ohio Linuxfest 2015 in Columbus, OH.
Linux Performance Analysis: New Tools and Old SecretsBrendan Gregg
Talk for USENIX/LISA2014 by Brendan Gregg, Netflix. At Netflix performance is crucial, and we use many high to low level tools to analyze our stack in different ways. In this talk, I will introduce new system observability tools we are using at Netflix, which I've ported from my DTraceToolkit, and are intended for our Linux 3.2 cloud instances. These show that Linux can do more than you may think, by using creative hacks and workarounds with existing kernel features (ftrace, perf_events). While these are solving issues on current versions of Linux, I'll also briefly summarize the future in this space: eBPF, ktap, SystemTap, sysdig, etc.
Talk for PerconaLive 2016 by Brendan Gregg. Video: https://www.youtube.com/watch?v=CbmEDXq7es0 . "Systems performance provides a different perspective for analysis and tuning, and can help you find performance wins for your databases, applications, and the kernel. However, most of us are not performance or kernel engineers, and have limited time to study this topic. This talk summarizes six important areas of Linux systems performance in 50 minutes: observability tools, methodologies, benchmarking, profiling, tracing, and tuning. Included are recipes for Linux performance analysis and tuning (using vmstat, mpstat, iostat, etc), overviews of complex areas including profiling (perf_events), static tracing (tracepoints), and dynamic tracing (kprobes, uprobes), and much advice about what is and isn't important to learn. This talk is aimed at everyone: DBAs, developers, operations, etc, and in any environment running Linux, bare-metal or the cloud."
Broken benchmarks, misleading metrics, and terrible tools. This talk will help you navigate the treacherous waters of Linux performance tools, touring common problems with system tools, metrics, statistics, visualizations, measurement overhead, and benchmarks. You might discover that tools you have been using for years, are in fact, misleading, dangerous, or broken.
The speaker, Brendan Gregg, has given many talks on tools that work, including giving the Linux PerformanceTools talk originally at SCALE. This is an anti-version of that talk, to focus on broken tools and metrics instead of the working ones. Metrics can be misleading, and counters can be counter-intuitive! This talk will include advice for verifying new performance tools, understanding how they work, and using them successfully.
Video: https://www.youtube.com/watch?v=JRFNIKUROPE . Talk for linux.conf.au 2017 (LCA2017) by Brendan Gregg, about Linux enhanced BPF (eBPF). Abstract:
A world of new capabilities is emerging for the Linux 4.x series, thanks to enhancements that have been included in Linux for to Berkeley Packet Filter (BPF): an in-kernel virtual machine that can execute user space-defined programs. It is finding uses for security auditing and enforcement, enhancing networking (including eXpress Data Path), and performance observability and troubleshooting. Many new open source tools that have been written in the past 12 months for performance analysis that use BPF. Tracing superpowers have finally arrived for Linux!
For its use with tracing, BPF provides the programmable capabilities to the existing tracing frameworks: kprobes, uprobes, and tracepoints. In particular, BPF allows timestamps to be recorded and compared from custom events, allowing latency to be studied in many new places: kernel and application internals. It also allows data to be efficiently summarized in-kernel, including as histograms. This has allowed dozens of new observability tools to be developed so far, including measuring latency distributions for file system I/O and run queue latency, printing details of storage device I/O and TCP retransmits, investigating blocked stack traces and memory leaks, and a whole lot more.
This talk will summarize BPF capabilities and use cases so far, and then focus on its use to enhance Linux tracing, especially with the open source bcc collection. bcc includes BPF versions of old classics, and many new tools, including execsnoop, opensnoop, funcccount, ext4slower, and more (many of which I developed). Perhaps you'd like to develop new tools, or use the existing tools to find performance wins large and small, especially when instrumenting areas that previously had zero visibility. I'll also summarize how we intend to use these new capabilities to enhance systems analysis at Netflix.
Video: https://www.youtube.com/watch?v=FJW8nGV4jxY and https://www.youtube.com/watch?v=zrr2nUln9Kk . Tutorial slides for O'Reilly Velocity SC 2015, by Brendan Gregg.
There are many performance tools nowadays for Linux, but how do they all fit together, and when do we use them? This tutorial explains methodologies for using these tools, and provides a tour of four tool types: observability, benchmarking, tuning, and static tuning. Many tools will be discussed, including top, iostat, tcpdump, sar, perf_events, ftrace, SystemTap, sysdig, and others, as well observability frameworks in the Linux kernel: PMCs, tracepoints, kprobes, and uprobes.
This tutorial is updated and extended on an earlier talk that summarizes the Linux performance tool landscape. The value of this tutorial is not just learning that these tools exist and what they do, but hearing when and how they are used by a performance engineer to solve real world problems — important context that is typically not included in the standard documentation.
OSCON: Advanced Docker developer workflows on Mac OS and WindowsDocker, Inc.
with Anil Madhavapeddy and Richard Mortier
Most developers use a Mac or Windows host to develop Docker Linux containers. This normally requires the installation of a Linux virtual machine as well as a complicated setup that includes a local networked filesystem for sharing data between the host and the Linux container, including UID mapping and case sensitivity, with inotify often being unreliable into the container; replicating Linux networking configuration locally to reflect the structure of the deployed microservices on the local laptop; and maintaining a separate Linux virtual machine and hypervisor such as VirtualBox, leading to heavyweight resource usage on a developer laptop.
Anil Madhavapeddy and Richard Mortier outline various methods to make using Docker on a Mac OS or Windows laptop as transparent as possible to the developer, including build scenarios from the conventional (Compose-based web stacks) to the deep end of systems (unikernel compilation) to the multi-CPU architecture (cross-compiling ARM containers on a normal Mac or Windows host).
The Information Technology have led us into an era where the production, sharing and use of information are now part of everyday life and of which we are often unaware actors almost: it is now almost inevitable not leave a digital trail of many of the actions we do every day; for example, by digital content such as photos, videos, blog posts and everything that revolves around the social networks (Facebook and Twitter in particular). Added to this is that with the "internet of things", we see an increase in devices such as watches, bracelets, thermostats and many other items that are able to connect to the network and therefore generate large data streams. This explosion of data justifies the birth, in the world of the term Big Data: it indicates the data produced in large quantities, with remarkable speed and in different formats, which requires processing technologies and resources that go far beyond the conventional systems management and storage of data. It is immediately clear that, 1) models of data storage based on the relational model, and 2) processing systems based on stored procedures and computations on grids are not applicable in these contexts. As regards the point 1, the RDBMS, widely used for a great variety of applications, have some problems when the amount of data grows beyond certain limits. The scalability and cost of implementation are only a part of the disadvantages: very often, in fact, when there is opposite to the management of big data, also the variability, or the lack of a fixed structure, represents a significant problem. This has given a boost to the development of the NoSQL database. The website NoSQL Databases defines NoSQL databases such as "Next Generation Databases mostly addressing some of the points: being non-relational, distributed, open source and horizontally scalable." These databases are: distributed, open source, scalable horizontally, without a predetermined pattern (key-value, column-oriented, document-based and graph-based), easily replicable, devoid of the ACID and can handle large amounts of data. These databases are integrated or integrated with processing tools based on the MapReduce paradigm proposed by Google in 2009. MapReduce with the open source Hadoop framework represent the new model for distributed processing of large amounts of data that goes to supplant techniques based on stored procedures and computational grids (step 2). The relational model taught courses in basic database design, has many limitations compared to the demands posed by new applications based on Big Data and NoSQL databases that use to store data and MapReduce to process large amounts of data.
Course Website http://pbdmng.datatoknowledge.it/
Contact me to download the slides
Bare-metal, Docker Containers, and Virtualization: The Growing Choices for Cl...Odinot Stanislas
(FR)
Introduction très sympathique autour des environnements Cloud avec un focus particulier sur la virtualisation et les containers (Docker)
(ENG)
Friendly presentation about Cloud solutions with a focus on virtualization and containers (Docker).
Author: Nicholas Weaver – Principal Architect, Intel Corporation
The Lies We Tell Our Code (#seascale 2015 04-22)Casey Bisson
We tell our code lies from development to deploy. The most common of these lies start with the simple act of launching a virtual machine. These lies are critical to our applications. Some of them protect applications from themselves and each other, some even improve performance. Some, however, decrease performance, and others create barriers to simply getting things done.
We lie about the systems, networks, storage, RAM, CPU and other resources our applications use, but how we tell those lies is critical to how the applications that depend on them perform. Joyent's Casey Bisson will explore the lies we tell our code and demonstrate examples of how they sometimes help and hurt us.
Slides as presented at http://www.meetup.com/Seattle-Scalability-Meetup/events/219709036/. Video from that meetup is on YouTube, https://www.youtube.com/watch?v=LtPS2z_c2v4.
The lies we tell our code, LinuxCon/CloudOpen 2015-08-18Casey Bisson
As presented at LinuxCon/CloudOpen 2015: http://sched.co/3Y3v
We tell our code lies from development to deploy. The most common of these lies start with the simple act of launching a virtual machine. These lies are critical to our applications. Some of them protect applications from themselves and each other, some even improve performance. Some, however, decrease performance, and others create barriers to simply getting things done.
We lie about the systems, networks, storage, RAM, CPU and other resources our applications use, but how we tell those lies is critical to how the applications that depend on them perform. Joyent's Casey Bisson will explore the lies we tell our code and demonstrate examples of how they sometimes help and hurt us.
Docker - Demo on PHP Application deployment Arun prasath
Docker is an open-source project to easily create lightweight, portable, self-sufficient containers from any application. The same container that a developer builds and tests on a laptop can run at scale, in production, on VMs, bare metal, OpenStack clusters, public clouds and more.
In this demo, I will show how to build a Apache image from a Dockerfile and deploy a PHP application which is present in an external folder using custom configuration files.
Seminar about docker and its containerization capabilities during the "Aggiornamento Agile" event of Club degli Sviluppatori in January 2015, in Bari (Italy)
Software Engineering, Software Consulting, Tech Lead, Spring Boot, Spring Cloud, Spring Core, Spring JDBC, Spring Transaction, Spring MVC, OpenShift Cloud Platform, Kafka, REST, SOAP, LLD & HLD.
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI AppGoogle
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
👉👉 Click Here To Get More Info 👇👇
https://sumonreview.com/ai-fusion-buddy-review
AI Fusion Buddy Review: Key Features
✅Create Stunning AI App Suite Fully Powered By Google's Latest AI technology, Gemini
✅Use Gemini to Build high-converting Converting Sales Video Scripts, ad copies, Trending Articles, blogs, etc.100% unique!
✅Create Ultra-HD graphics with a single keyword or phrase that commands 10x eyeballs!
✅Fully automated AI articles bulk generation!
✅Auto-post or schedule stunning AI content across all your accounts at once—WordPress, Facebook, LinkedIn, Blogger, and more.
✅With one keyword or URL, generate complete websites, landing pages, and more…
✅Automatically create & sell AI content, graphics, websites, landing pages, & all that gets you paid non-stop 24*7.
✅Pre-built High-Converting 100+ website Templates and 2000+ graphic templates logos, banners, and thumbnail images in Trending Niches.
✅Say goodbye to wasting time logging into multiple Chat GPT & AI Apps once & for all!
✅Save over $5000 per year and kick out dependency on third parties completely!
✅Brand New App: Not available anywhere else!
✅ Beginner-friendly!
✅ZERO upfront cost or any extra expenses
✅Risk-Free: 30-Day Money-Back Guarantee!
✅Commercial License included!
See My Other Reviews Article:
(1) AI Genie Review: https://sumonreview.com/ai-genie-review
(2) SocioWave Review: https://sumonreview.com/sociowave-review
(3) AI Partner & Profit Review: https://sumonreview.com/ai-partner-profit-review
(4) AI Ebook Suite Review: https://sumonreview.com/ai-ebook-suite-review
#AIFusionBuddyReview,
#AIFusionBuddyFeatures,
#AIFusionBuddyPricing,
#AIFusionBuddyProsandCons,
#AIFusionBuddyTutorial,
#AIFusionBuddyUserExperience
#AIFusionBuddyforBeginners,
#AIFusionBuddyBenefits,
#AIFusionBuddyComparison,
#AIFusionBuddyInstallation,
#AIFusionBuddyRefundPolicy,
#AIFusionBuddyDemo,
#AIFusionBuddyMaintenanceFees,
#AIFusionBuddyNewbieFriendly,
#WhatIsAIFusionBuddy?,
#HowDoesAIFusionBuddyWorks
Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...Globus
Large Language Models (LLMs) are currently the center of attention in the tech world, particularly for their potential to advance research. In this presentation, we'll explore a straightforward and effective method for quickly initiating inference runs on supercomputers using the vLLM tool with Globus Compute, specifically on the Polaris system at ALCF. We'll begin by briefly discussing the popularity and applications of LLMs in various fields. Following this, we will introduce the vLLM tool, and explain how it integrates with Globus Compute to efficiently manage LLM operations on Polaris. Attendees will learn the practical aspects of setting up and remotely triggering LLMs from local machines, focusing on ease of use and efficiency. This talk is ideal for researchers and practitioners looking to leverage the power of LLMs in their work, offering a clear guide to harnessing supercomputing resources for quick and effective LLM inference.
Large Language Models and the End of ProgrammingMatt Welsh
Talk by Matt Welsh at Craft Conference 2024 on the impact that Large Language Models will have on the future of software development. In this talk, I discuss the ways in which LLMs will impact the software industry, from replacing human software developers with AI, to replacing conventional software with models that perform reasoning, computation, and problem-solving.
Providing Globus Services to Users of JASMIN for Environmental Data AnalysisGlobus
JASMIN is the UK’s high-performance data analysis platform for environmental science, operated by STFC on behalf of the UK Natural Environment Research Council (NERC). In addition to its role in hosting the CEDA Archive (NERC’s long-term repository for climate, atmospheric science & Earth observation data in the UK), JASMIN provides a collaborative platform to a community of around 2,000 scientists in the UK and beyond, providing nearly 400 environmental science projects with working space, compute resources and tools to facilitate their work. High-performance data transfer into and out of JASMIN has always been a key feature, with many scientists bringing model outputs from supercomputers elsewhere in the UK, to analyse against observational or other model data in the CEDA Archive. A growing number of JASMIN users are now realising the benefits of using the Globus service to provide reliable and efficient data movement and other tasks in this and other contexts. Further use cases involve long-distance (intercontinental) transfers to and from JASMIN, and collecting results from a mobile atmospheric radar system, pushing data to JASMIN via a lightweight Globus deployment. We provide details of how Globus fits into our current infrastructure, our experience of the recent migration to GCSv5.4, and of our interest in developing use of the wider ecosystem of Globus services for the benefit of our user community.
Globus Compute wth IRI Workflows - GlobusWorld 2024Globus
As part of the DOE Integrated Research Infrastructure (IRI) program, NERSC at Lawrence Berkeley National Lab and ALCF at Argonne National Lab are working closely with General Atomics on accelerating the computing requirements of the DIII-D experiment. As part of the work the team is investigating ways to speedup the time to solution for many different parts of the DIII-D workflow including how they run jobs on HPC systems. One of these routes is looking at Globus Compute as a way to replace the current method for managing tasks and we describe a brief proof of concept showing how Globus Compute could help to schedule jobs and be a tool to connect compute at different facilities.
Unleash Unlimited Potential with One-Time Purchase
BoxLang is more than just a language; it's a community. By choosing a Visionary License, you're not just investing in your success, you're actively contributing to the ongoing development and support of BoxLang.
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptxrickgrimesss22
Discover the essential features to incorporate in your Winzo clone app to boost business growth, enhance user engagement, and drive revenue. Learn how to create a compelling gaming experience that stands out in the competitive market.
We describe the deployment and use of Globus Compute for remote computation. This content is aimed at researchers who wish to compute on remote resources using a unified programming interface, as well as system administrators who will deploy and operate Globus Compute services on their research computing infrastructure.
Navigating the Metaverse: A Journey into Virtual Evolution"Donna Lenk
Join us for an exploration of the Metaverse's evolution, where innovation meets imagination. Discover new dimensions of virtual events, engage with thought-provoking discussions, and witness the transformative power of digital realms."
Graspan: A Big Data System for Big Code AnalysisAftab Hussain
We built a disk-based parallel graph system, Graspan, that uses a novel edge-pair centric computation model to compute dynamic transitive closures on very large program graphs.
We implement context-sensitive pointer/alias and dataflow analyses on Graspan. An evaluation of these analyses on large codebases such as Linux shows that their Graspan implementations scale to millions of lines of code and are much simpler than their original implementations.
These analyses were used to augment the existing checkers; these augmented checkers found 132 new NULL pointer bugs and 1308 unnecessary NULL tests in Linux 4.4.0-rc5, PostgreSQL 8.3.9, and Apache httpd 2.2.18.
- Accepted in ASPLOS ‘17, Xi’an, China.
- Featured in the tutorial, Systemized Program Analyses: A Big Data Perspective on Static Analysis Scalability, ASPLOS ‘17.
- Invited for presentation at SoCal PLS ‘16.
- Invited for poster presentation at PLDI SRC ‘16.
First Steps with Globus Compute Multi-User EndpointsGlobus
In this presentation we will share our experiences around getting started with the Globus Compute multi-user endpoint. Working with the Pharmacology group at the University of Auckland, we have previously written an application using Globus Compute that can offload computationally expensive steps in the researcher's workflows, which they wish to manage from their familiar Windows environments, onto the NeSI (New Zealand eScience Infrastructure) cluster. Some of the challenges we have encountered were that each researcher had to set up and manage their own single-user globus compute endpoint and that the workloads had varying resource requirements (CPUs, memory and wall time) between different runs. We hope that the multi-user endpoint will help to address these challenges and share an update on our progress here.
OpenMetadata Community Meeting - 5th June 2024OpenMetadata
The OpenMetadata Community Meeting was held on June 5th, 2024. In this meeting, we discussed about the data quality capabilities that are integrated with the Incident Manager, providing a complete solution to handle your data observability needs. Watch the end-to-end demo of the data quality features.
* How to run your own data quality framework
* What is the performance impact of running data quality frameworks
* How to run the test cases in your own ETL pipelines
* How the Incident Manager is integrated
* Get notified with alerts when test cases fail
Watch the meeting recording here - https://www.youtube.com/watch?v=UbNOje0kf6E
Enhancing Research Orchestration Capabilities at ORNL.pdfGlobus
Cross-facility research orchestration comes with ever-changing constraints regarding the availability and suitability of various compute and data resources. In short, a flexible data and processing fabric is needed to enable the dynamic redirection of data and compute tasks throughout the lifecycle of an experiment. In this talk, we illustrate how we easily leveraged Globus services to instrument the ACE research testbed at the Oak Ridge Leadership Computing Facility with flexible data and task orchestration capabilities.
Enhancing Research Orchestration Capabilities at ORNL.pdf
Unikernels: Rise of the Library Hypervisor
1. Unikernels: the Rise of
the Library Hypervisor
Anil Madhavapeddy, @avsm
Mindy Preston, @yomimono
Martin Lucina
+the MirageOS and Docker for Mac/Win teams
Docker Inc, @docker
with contributions from IBM
Docker Distributed Systems Summit
7th October 2016, Berlin, Germany
2. Conventional hypervisors
• Run full guest operating
systems with complex
emulation needs.
• Scaffolding for device
emulation, instruction
emulation, etc.
• Hard to compose into existing
infrastructure without wrapping
a full hypervisor layer.
Xen Hypervisor
qemu
xenstored
xenconsoled
Hardware
Dom0DomU
3. Conventional hypervisors
CVE-2016-3710: VGA emulation
missing bounds checks causes exploit.
CVE-2016-5403: unbounded virtio
memory usage causes DoS.
CVE-2016-3672: unrestricted qemu
logging causes DoS.
CVE-2015-8554: qemu-dm buffer
overrun in MSI-X causes exploit.
CVE-2015-7504: heap overflow in
pcnet emulator causes exploit.
• Run full guest operating
systems with complex
emulation needs.
• Scaffolding for device
emulation, instruction
emulation, etc.
• Hard to compose into existing
infrastructure without wrapping
a full hypervisor layer.
4. How can distributed systems
use hardware protection more
flexibly and composably?
5. Recap: Unikernels
• "library operating systems"
break kernels into libraries.
• Link libraries with a boot layer,
scheduler and application.
• Portable microservices that boot
directly on hypervisors or Unix. Xen
Hardware
App
Linux
Hardware
DockerApp
Configuration Business Logic
HTTP JSON SSL
TCP/IP
Xen
Devices
Unix
libev
Unix
musl libc
Application
Libraries
Libraries
6. Recap: Unikernels
• Many benefits are lost when
deploying on existing clouds.
• Tiny binaries (200k) still require
scaffolding of a full OS to boot.
• Difficult to manage hypervisor
from inside a container as full
host privilege is needed.
• "library operating systems"
break kernels into libraries.
• Link libraries with a boot layer,
scheduler and application.
• Portable microservices that boot
directly on hypervisors or Unix.
7. Library Hypervisors
• Extend the "kit" model and break down hypervisor
functionality into libraries.
• Expose core functionality (CPU and memory) as library,
and other pieces (device emulation) are optional.
• Benefit: huge reduction in TCB, and better fit to
container-native infrastructure with privilege dropping.
• Drawback: no existing support in operating systems.
8. Library Hypervisors
• Extend the "kit" model and break down hypervisor
functionality into libraries.
• Expose core functionality (CPU and memory) as library,
and other pieces (device emulation) are optional.
• Benefit: huge reduction in TCB, and better fit to
container-native infrastructure with privilege dropping.
• Drawback: no existing support in operating systems.
But let's a closer look!
12. • Easy drag and drop installation, and
autoupdates to get latest Docker.
• Secure, sandboxed virtualisation
architecture without elevated privileges.
• Native networking support, with VPN and
network sharing compatibility.
• File sharing between container and host:
uid mapping, inotify events, etc.
Docker for Mac
Aiming for a native OSX experience
that works with existing developer
workflows.
13. • Uses the new HyperKit framework, which is in turn
based on xHyve and FreeBSD's bHyve.
• Sandbox friendly: processes largely run as non-
root, with privileges of the local user.
Virtualisation
14. • Uses the new HyperKit framework, which is in turn
based on xHyve and FreeBSD's bHyve.
• Sandbox friendly: processes largely run as non-
root, with privileges of the local user.
Virtualisation
OSX Kernel
Hypervisor.
framework
Hardware
virt: VMX,
nested
paging
15. • Uses the new HyperKit framework, which is in turn
based on xHyve and FreeBSD's bHyve.
• Sandbox friendly: processes largely run as non-
root, with privileges of the local user.
Virtualisation
OSX Kernel Userspace
Hypervisor.
framework
User Process
Thread/vCPU
Traps on I/O pages
Manages ACPI, PCI
devices
Hardware
virt: VMX,
nested
paging
16. • Uses the new HyperKit framework, which is in turn
based on xHyve and FreeBSD's bHyve.
• Sandbox friendly: processes largely run as non-
root, with privileges of the local user.
Virtualisation
OSX Kernel Userspace
Hypervisor.
framework
User ProcessHardware
virt: VMX,
nested
paging
Process
Linux Kernel
VirtIO IPC
VirtIO Block
VirtIO Net
Alpine Linux
Userspace
Latest Docker
preconfigured
QCow2
VPNKit
Logs redirected to
OSX host
17. • Uses the new HyperKit framework, which is in turn
based on xHyve and FreeBSD's bHyve.
• Embeds Linux: includes an embedded
lightweight Alpine Linux distribution optimised for
fast boot and stateless operation for containers.
Virtualisation
$ docker info
Containers: 358
Running: 13
Paused: 0
Stopped: 345
Images: 485
Server Version: 1.11.1
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge null host
Kernel Version: 4.4.9-moby
Operating System: Alpine Linux v3.3
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.858 GiB
18. HyperKit library structure
• In HyperKit, most functionality is linked as a library.
• If app doesn't need a protocol, it is not linked and
not part of the trusted computing base.
19. • Want to hide the gory details of virtualisation from
the user. The Linux VM should be "invisible".
• Not solving this leads to many user complaints:
• VPN software and corporate installations do not
like bridged virtual machines or custom routing.
Result: container traffic cannot connect to Internet.
• Services cannot be exposed on localhost or
the external interface and are instead on the Linux
VM IP address.
Result: breaks common web oAuth workflows.
Networking
23. • Want to hide the gory details of virtualisation from
the user. The Linux VM should be "invisible".
• Not solving this leads to many user complaints:
• VPN software and corporate installations do not
like bridged virtual machines or custom routing.
Result: container traffic cannot connect to Internet.
• Services cannot be exposed on localhost or
the external interface and are instead on the Linux
VM IP address.
Result: breaks common web oAuth workflows.
Networking
24. • Challenge: Services publishing ports should be
exposed on localhost without needing VM info.
• Solution: VPNKit forwards container port requests
to a OSX service which binds them natively on its
external interface.
• Benefits:
• docker run -P on the Mac now works without
requiring any knowledge of the VM innards.
• External oAuth workflows operate with web apps.
Networking
27. • Challenge: Deal with custom VPN software on the
host that makes it difficult to bridge.
• Solution: VPNKit, efficiently reconstructs container
traffic into separate TCP/IP flows and translates
them into native OSX/Windows sockets.
• Benefits:
• All network traffic is generated from normal socket
calls (e.g. gethostbyaddr) on the Mac, so
interacts well with firewalls, VPNs, and any local
security policies.
Networking
28. • Native OSX application, uses HyperKit to virtualise
for domain-specific purpose ("docker run")
• Links MirageOS unikernel libraries for networking
and storage translation between OS boundaries.
• The library approach let us glue together these
components really easily.
• Docker for Mac is quite a complex distributed
system internally, but (hopefully) hidden from user.
Docker for Mac + unikernels
29. MirageOS 3 + Solo5
•Unikernels have been gathering pace; next
challenge is to make them easily deployable.
•Build handled via Docker, but docker run
shouldn't need privileges (e.g. to start a VM).
•MirageOS 3 has a new library hypervisor for
Linux, developed by IBM, Docker and
Cambridge University contributors.
mirage.io
30. MirageOS 3 + Solo5
• Source: https://github.com/Solo5/solo5
• Runs as a Unix process and opens /dev/kvm for
hardware isolation.
• ukvm is a small, modular monitor that links only what is
needed. Can be 10k in size!
• Can run privilege separated: one process opens /dev/
kvm and drops privileges and executes the unikernel.
• Boot times are the same as process fork times, since all
the device setup is handled in-process.
31. MirageOS 3 + Solo5
Source: Dan Williams and Ricardo Koller, IBM Research, HotCloud 16
32. MirageOS 3 + Solo5
• Due for stable release in the next month.
• Intended to be "unikernel template" for
other projects to share hypervisor code.
• Liberally licensed under BSD/Apache2/ISC
to encourage adoption and embedding.
• BoF and tutorials tomorrow to demonstrate
it. Developers are all here and hacking!
34. How can distributed systems
use hardware protection more
flexibly and composably?
35. Questions?
Download free at
docker.com
Twitter: @avsm
https://github.com/docker/hyperkit
https://github.com/docker/vpnkit
https://github.com/docker/datakit
https://github.com/mirage/
We will be
hacking
tomorrow!
37. • Challenge: Share arbitrary OSX directory tree into
Linux container without requiring extensive
modification of either side.
• Solution: Use a FUSE forwarding layer and
translate Linux filesystem calls to OSX equivalents.
OSX Host Linux Host Container
VOLUMEcom.docker.osxfs
Track extra
metadata
Translate to OSX
filesystem calls
FUSE
Filesystem Sharing
38. • Challenge: Need filesystem activation so events on
the Mac wake up container servers and vice-versa.
• Solution: osxfs uses FSEvents API and injects
inotify activation events into container.
OSX Host Linux Host Container
VOLUMEcom.docker.osxfs
FSEvents watches
open files
Events from Linux
causes OSX apps
to wake up
FUSE
Filesystem Sharing
39. • Challenge: Need filesystem activation so events on
the Mac wake up container servers and vice-versa.
• Solution: osxfs uses FSEvents API and injects
inotify activation events into container.
OSX Host Linux Host Container
VOLUMEcom.docker.osxfs
FSEvents watches
open files
Events from Linux
causes OSX apps
to wake up
FUSE
Filesystem Sharing
40. • Challenge: Deal with custom VPN software on the
host that makes it difficult to bridge.
• Solution: VPNKit, efficiently reconstructs container
traffic into separate TCP/IP flows and translates
them into native OSX/Windows sockets.
OSX Host Linux Host Container
RUN <...>com.docker.hyperkit-net
Reconstruct traffic
TCP flows
Translate to OSX
socket calls
Ethernet bridge
DHCPv4
NTP
Networking
41. OSX Host Linux Host
Privileged Port
Service
Container
EXPOSE
Port Service
VSock Binder
RUN <...>
VSock Listener
Userland Proxy
• Challenge: Services publishing ports should be
exposed on localhost without needing VM info.
• Solution: VPNKit forwards container port requests
to a OSX service which binds them natively on its
external interface.
Networking
42. $ docker run resin/armv7hf-debian uname -a
Linux 7ed2fca7a3f0 4.1.12 #1 SMP Tue Jan 12 10:51:00
UTC 2016 armv7l GNU/Linux
$ docker run justincormack/ppc64le-debian uname -a
Linux edd13885f316 4.1.12 #1 SMP Tue Jan 12 10:51:00
UTC 2016 ppc64le GNU/Linux
Multi-CPU architectures