Docker allows creating isolated environments called containers from images. Containers provide a standard way to develop, ship, and run applications. The document discusses how Docker can be used for scientific computing including running different versions of software, automating computations, sharing research environments and results, and providing isolated development environments for users through Docker IaaS tools. K-scope is a code analysis tool that previously required complex installation of its Omni XMP dependency, but could now be run as a containerized application to simplify deployment.

1. Docker Application
to Scientific Computing
HPC Usability Research Team
Maeda Toshiyuki
Bryzgalov Peter

2. DOCKER
Container-based Virtualisation Technology

3. What is Docker?
• Container-based virtualisation technology (not VM)
• Create and manage isolated environments
(containers)
• Develop and distribute applications in containers

4. Containers vs. VMs

5. Applications Development and Distribution

6. Images and Containers
• What runs in a container
docker ps ̶ list containers
• New containers are created from images
docker images ̶ list images
docker run image command ̶ basic command format for running
containers
• Compare to VM: container VM, image snapshot
• Two ways of creating images:
• commit a container: docker commit container image
• Dockerfile: instructions to build an image

7. Public Docker registry for sharing images
Docker Hub registry.hub.docker.com
over 70 000 Dockerized Apps2015 / 01 / 15
• OS
• DB
• Framework
• Development env.
• Applications
• etc.

8. Test new software
• Developers install applications in
containers and share its images
• User tests multiple applications
with his dataset on his machine
docker run imageA
docker run imageB
• User removes unnecessary
containers and images
docker rm containerA
docker rmi imageA
Developer B
Image B
App B
Developer A
App A
Image A
User
Dataset
Container A Container B

9. Use different versions
• Use multiple programs that is
hard to use on the same
machine at the same time
• Mount source code folders
into containers
source code
Compiler BCompiler A
binary A binary B binary C
Compiler C
docker run -v srcdir:/mnt/src gcc:4.8
docker run -v srcdir:/mnt/src gcc:4.9
…
srcdir

10. Reproduce computing environment
• Containerised applications run
the same way everywhere*
* On computers with same CPU
architecture, OS and with Docker
installed
Image A
Container A Container A
Computer I Computer II

11. • Share input data, program and
visualisation software together
with research results
• Other researchers can rerun
computations for different
input data or computing
parameters
• Use visualisation software to
look at results from different
points of view
Image
Container
Researcher A Researcher B
Share research results
Data
Binaries
Container
Data
Binaries
Data
Binaries

12. container
A1
container
A2
container
A
container
B
Automate computing
• Run same computations with
different data sets
• Run different computations on
the same dataset
N is a parameter
dataset I dataset II
data
･･･
･･･
docker run -v dirN:/mnt/src <image>
docker run -v dir:/mnt/src <imageN>
dir 1 dir 2

13. container
A
container
B
Pipeline
• Pipeline data processing
･･･
data data data
dir 1 dir 2 dir 3
docker run -v dir1:/mnt/src -v dir2:/mnt/dst imageA
docker run -v dir2:/mnt/src -v dir3:/mnt/dst imageB
…

14. Docker ecosystem
Service discovery
Tools for registering and searching information about services provided
by applications running in containers (including multi-host applications)
Orchestration
Tools with main purpose of managing multi-host multi-container
applications. Usually help managing multiple containers and network
connections between them.
Automation
Tools that help : a. to make containers easier to use, b. to give containers
new features, c. to build a service powered by containers
Monitoring
Tools for monitoring resources used by containers, containers heath-
check, monitoring in-container environment
OS
Light-weight OS for running containers
Networking
Tools for organising inter-container and host-container communications
Data Management
Tools for managing data in containers

15. ClusterHQ
OpenShift
Project Atomic /Redhat
SoundCloud
Google
CoreOS Inc
Apache
Spotify
uses
uses
Runs on
includes
Docker ecosystem
Service Discovery Orchestration
Monitoring OS Networking Data Management
Automation
Helios
Fig
hroot
Shutit
Docker IaaS tools
StackatoPanamax
MesosMarathon
FlynnSerf
etcd
CoreOS Rudder
Flocker
fleet
pipework Tenus
weave
Docker Swarm
OpenShift Origin
geard
Kubernetes
Consul
Shipyard
Citadel
Mackerel
Datadog
Prometheus
libswarmProject Atomic
ment toolkit Citadel, Shipyard gives you the ability to manage
ers, hosts and more.
ement applications in that it promotes composability.At the core,
ntainers, etc). However, using "Extention Images" you can add
uting and load balancing, centralized logging, deployment and
ts to use that fit your needs.
ols a cluster of Docker hosts and exposes it as a single "virtual"
. It allows you to connect to a single Docker endpoint and run
for shared configuration and service discovery.
eator with an open-source app marketplace hosted in GitHub.
ace for users of Docker, Fleet & CoreOS. With Panamax, you can
y containerized app no matter how complex it might be.
You can configure links, ports and environment variables, as well
es into categories.
a git repository and share it with your friends. You can source
s or submit a pull request and have your template be canonical
orld.
Overlay network
https://github.com/coreos/rudder
https://coreos.com/blog/introducing-rudder/
Hroot provides transports and straightforward configura
version your containers, then distribute them offline or
ShutIt is a tool for managing your build process that is b
Complex Docker Builds Made Simple.
https://github.com/ianmiell/shutit
Tool set for creating basic single-host multi-tenant IaaS
https://github.com/pyotr777/dockerIaaSTools
ring services in your infrastructure. It provides several key
Checking, Key/Value Store, Multi Datacenter
on-centric IT architecture by providing an end-to-end solution for
ons quickly and reliably, with atomic update and rollback for
oject Atomic Host. This is a lightweight operating system that has
RPM content. It is designed to run applications in Docker contain-
nk Docker containers into systemd and coordinate these contain-
erel-docker
r Mesos, Mesos can run and manage Docker containers in conjunc-
's Marathon frameworks.
OpenShift Origin is the open source upstream of OpenSh
hosting platform developed by Red Hat.Also known as P
takes care of infrastructure, middleware, and manageme
app.
Flocker lets you move your Docker containers and their d
means that you can run your databases, queues and key-
them around as easily as the rest of your app. Even statel
services and currently running these services in Docker co
impossible. Flocker aims to solve this problem by providi
that allows you to port both your stateful and stateless co
libswarm is a toolkit for composing network services.
It defines a standard interface for services in a distributed
other.This lets you:
Compose complex architectures from reusable building b
Avoid vendor lock-in by swapping any service out with an
2015/02/03 Bryzgalov Peter @ Japan, RIKEN, AICS
Fleet ties together systemd and etcd into a distributed i
extension of systemd that operates at the cluster level i
project is very low level and is designed as a foundation
Kubernetes enables users to ask a cluster to run a set of c
ly chooses hosts to run those containers on. Has schedule
Architecturally, we want Kubernetes to be built as a collec
and layers, with the ability to use alternative schedulers,
mechanisms.
Kubernetes is intended to run on multiple cloud provider
A single Kubernetes cluster is not intended to span multi
Kubernetes is not currently suitable for use by multiple u
Kubernetes provides higher-level organizational construc
cluster-level usage patterns, currently focused on service
be expanded to batch and test workloads in the future.
Stackato is the first enterprise-grade, cloud application pl
open-source project focused on packaging and deploying
addition to securing your application from others in your
provide a number of benefits to organizations such as uti
efficiently, simplifying the application lifecycle and enabl
fig.sh
Fast, isolated development environments.
geard is a command-line client and agent for integrating
systemd across multiple hosts.
tenus is a Golang package which allows you to configure
devices programmatically. It communicates with Linux Ke
creation and configuration of network devices on the Linu
more advanced network setups with Linux containers inc
Weave creates a virtual network that connects Docker con
hosts.
Applications use the network just as if the containers wer
network switch, with no need to configure port mappings
application containers on the weave network can be mad
regardless of where those containers are running. Simila
be exposed to application containers irrespective of their
of its distributed init system, fleet.
vides many powerful features for starting, stopping and manag-
world, you will almost exclusively use systemd to manage the
.
ip, failure detection, and orchestration that is decentralized,
. Serf runs on every major platform: Linux, Mac OS X, and
ght: it uses 5 to 10 MB of resident memory and primarily commu-
sages.
ailure detection, and user events.
ents that out-of-the-box gives companies a reasonable starting
running their applications and services.
of independent projects that together make up a toolkit or loose
systems.
rts, depending on what is most useful for you.The common goal is
and best practices in building distributed systems. It is the
nd developers that makes both their lives easier.
tform for deploying and managing containers across an entire fleet.
oup accounting metrics to gather CPU, memory, network and I/O
seconds before they are forwarded to Datadog.
e monitoring system. It collects metrics from configured targets at
essions, displays the results, and can trigger alerts if some condi-
uling containers on a docker cluster.

16. Docker IaaS tools
• Give all users their
personal environments
(containers) on one
server
• Users login into
containers the same way
they connect to the
server:
• Unused containers are
stopped
ssh user@server

17. K-scope
• Static and Dynamic performance analysis of Fortran source code
• Uses Omni XMP compiler for static analysis
• Omni XMP runs only on Linux and is hard to install
www.aics.riken.jp/ungi/soft/kscope

18. K-scope + Docker IaaS
• Install Omni XMP compiler in container
• Mount source code folder inside the container
• Seamlessly build source code from K-scope
• No need to edit source code or make files

19. K-scope and Docker IaaS tools
K-scope + Docker Iaas tools demonstration