The document outlines a presentation on the use of containers, particularly Docker, in high-performance computing (HPC) environments, scheduled for a meetup on March 15, 2017. It covers topics such as container technology, its application in HPC workloads, and the evolution of Docker as a platform to enhance software delivery. Additionally, it discusses the benefits of containers in terms of reproducibility, portability, and efficient resource management in data centers.

1. Containers: Portable, repeatable user-oriented application delivery
HPC Saudi 2017 - KAUST
15 th March 2017
#dockerbday
@walidshaari
walid.shaari@gmail.com
https://www.linkedin.com/in/walidshaari/

2. $whoami
● Passionate about openness, open source, devops, Infosec
● Member of the Saudi Aramco Expec Computer Center/HPC team
● Red Hat Certified Architect RHCA
● SANS GIAC Incident handler, Forensics and Web security certified
● Dhahran Docker & Ansible meetup organizer/mentor
@walidshaari
Walid.shaari@gmail.com
https://www.linkedin.com/in/walidshaari/

3. AGENDA : Good Morning Containers
8:30 - 8:35 Introduction, Networking, Socializing
8:37 - 9:38 Interactive theory session "Presentation with Q&A"
9:40 - 10:15 Play with Docker Birthday 4 Labs
10:15 - 10:30 Coffee break
10:30 - 11:55 Singularity, rkt, lxd

4. © 2013-2016 Docker, Inc. All rights reserved
Docker 4th Birthday
#dockerbday

5. Docker Bday #4 celebrations worldwide!
• 150+ Bday meetups!
• 6000+ RSVPs
• 700+ mentors
#dockerbday

6. Join the Docker Student
Community! Sign up here:
http://dockr.ly/students (with your school email) for
access to our free Docker Student Developer Kit and
more!
Become a Docker
Campus Ambassador!
For leaders on campus who want to help their
peers learn Docker! Learn more and apply here:
http://dockr.ly/campus-ambassador
Are you a student?

7. Surveys and expectations
Assuming everyone knows a bit of Linux/Unix/Mac OSX CLI ?
Development, Operations, Security, Business, Others?
Devops
Configuration management
Containers
Schedulers
Containers eco system
Clusters, Load balancers, Orchestration

8. © 2013-2016 Docker, Inc. All rights reserved
HPC

9. What is HPC?
▪ HPC workloads mostly
▪ Runs on Linux
▪ Runs on bare-metal for maximum performance, lower overhead
▪ HPC Application
▪ Broken into smaller parallel distributed problems across cluster
nodes.
▪ Utilizes inter-process communications heavily, shared memory, or
across network.
▪ Scientific computing

10. HPC
▪ HPC dominated by Academics research and discovery
▪ Industry in the last 5-10 years seen an increase in HPC interest (Car , O&E)
▪ Possible constraints:
▪ Snowflake deployments, each HPC cluster/supercomputer is build in mind with
specific use cases
▪ Long lived nodes.
▪ Bloated/drift/unclean maybe diskless reboots
▪ Reboot time, or launching app could be long due to system/memory checks, bootstrapping
▪ Traditional Data Center Linux distribution
▪ Fixed installation based on single enterprise distro (Scientific, RHEL, SLES)
▪ Old kernel features
https://arxiv.org/pdf/1702.05513.pdf #cHPC

11. © 2013-2016 Docker, Inc. All rights reserved
Containers

12. First Step, Definition?
• The Application matters
• The application can be a process or a set of processes
• The use case might be not a running app
• Set of tools to develop an app
• Set of scripts "apps" that are part of a pipeline
• Isolated contained environment "Encapsulation"
• Synonyms
• chroot
• jail
• partition
• namespace
• zone

13. chroot/jail
A chroot on Unix operating systems is an operation that
changes the apparent root directory for the current running
process and its children. A program that is run in such a
modified environment cannot name (and therefore normally
cannot access) files outside the designated directory tree.
The term "chroot" may refer to the chroot(2) system call or
the chroot(8) wrapper program. The modified environment
is called a chroot jail.
https://en.wikipedia.org/wiki/Chroot

14. Thank the giants

15. CONTAINERS?
WHAT ARE THEY REALLY?
Linux features?
Namespace
cgroupsLXC
Union file systems
Configuration management?
Virtualization technology?
npm
jar
Packaging ?
rpm
deb
tar.gz
Virtual/environment management ?
Sandboxing?
chroo
t
BSD jail Solaris zones
IBM VM/370 (1972)
seccomp

16. IT DEPENDS
Manual
Configuration
Traditional VMs
Less Portable
Minimal overhead
Most Portable
Lots of overhead
Configuration
Management tools
Containers
Docker
Intel Clear Containers
Singularity
LXC/LXD
Non-Repeatable Repeatable
rkt

17. DEVELOPERS LOVE DOCKER
17https://www.slideshare.net/dberkholz/cloud-native-in-the-enterprise-realworld-data-on-container-and-microservice-
adoption

18. KUBERNETES SEEING THE MOST DEVELOPER
TRACTION
18https://www.slideshare.net/dberkholz/cloud-native-in-the-enterprise-realworld-data-on-container-and-microservice-
adoption

19. Container
Containment, isolation or encapsulation of an environment.
Machine container:
Encapsulates a complete system image. e.g. Ubuntu, RHEL, Scientific Linux.
Process container:
Encapsulates a service/process(es) . e.g. Django, ROR, Gitlab, redis, Openfoam, kafka, spark.
What is the smallest application container?

20. Container Runtime
docker < 1.11.0
└── systemd
└── docker run OpenFoam
└── Docker Engine
└── OpenFoam
docker > 1.11.0
└── systemd
└── docker run OpenFoam
└── Docker Engine
└── containerd
└── runc
└── OpenFoam
rkt > 1.0
└── systemd
└── rkt run OpenFoam
└── OpenFoam
singularity (2.2.x)
└── systemd/(init)
└── bash
└── OpenFoam
https://medium.com/@adriaandejonge/moving-from-docker-to-rkt-310dc9aec938#.1glm3o1t3

21. Other runtime

22. Image formats
Layered
Overlay filesystems/Graph drivers
chrootDirectory
Archive
#OCI
#ACI

23. Use Cases: Packaging
Agnostic packaging
Captures
○ Dependencies
○ Environment
○ Configurations
○ Executables
○ How about data?
○ What Else?
■ hint: m*
Pack once, Run everywhere
http://hpcbios.readthedocs.io/en/latest/HPCBIOS_2012-92.html
#EasyBuild #lmod #GUIX #NYU-Environment-POSTER

24. Use Case: Portability
Portable/Scalable across
● platforms
● Distributions
● Environments
Separation of concerns, e.g. development pack and ship, operations scale and deploy.
development ensures app is resilient, operations enure infra is HA resilient and scalable

25. Use Case: Portability
Portable/Scalable across
● systems
● subsystems
● Anywhere
#BYOE

26. Use Case: Reproducible
Paolo Di Tommaso from the Center for Genomic Regulation presented : Manage Reproducibility of Computational Workflows with Docker Containers and
Nextflow.
https://www.slideshare.net/insideHPC/reproducible-computational-pipelines-with-docker-and-nextflow
https://youtu.be/Doo9H2-gBAk

27. 27
Data Center current silo inefficient state
SchedulerScheduler
Jobs
Jobs
Jobs
Jobs
Jobs
Jobs
Scheduler
Jobs
Jobs
Jobs
Cluster Management A
Cluster Management B
Cluster Management C
Node as a work unit,
traditiontial single level
(silo) schedulers. No
holistic awareness of other
workloads

28. 28
Data Center
Efficient Secure Allocation of Resources
VC3
BigData
VC1
Infra
VC2
HPC
Schedu
ler
Schedu
ler
Schedu
ler
DataCenter
Scheduler
jobs
Jobs
Jobs
Jobs
Jobs
Jobs
Jobs
Jobs
2nd Generation Cluster Management
Containers as a work unit,
container aware workload
schedulers integrated with
cluster management
software

29. 29
Mesos DC/OS:
Example of Data Center/Container aware scheduler
▪ Mature, Open Source Apache Project
▪ Cluster Resource Manager
▪ Scalable to 10,000s of nodes
▪ Fault tolerant, no single point of failure
▪ Multi-tenancy with strong resource isolation
▪ Improved resource utilization
▪ Can schedule batch and interactive workloads for HPC and Big data.
https://people.eecs.berkeley.edu/~alig/papers/mesos.pdf
https://katacoda.com/courses/mesos/playground

30. 30
HPC workload
runs on the cloud
25%

31. 31
Which workloads and frameworks are running on
OpenStack?
Source : https://www.openstack.org/assets/survey/Public-User-Survey-Report.pdf
> 38%
scientific/technical
computing already
happening on
Openstack

32. EXAMPLE HPC Data Center Use Case
https://fosdem.org/2017/schedule/event/magnumcern/

33. 33
NVIDIA Example use case
https://github.com/NVIDIA/nvidia-docker
http://www.nvidia.com/object/docker-container.html

34. Possible HPC Caveats/Constraints
1. Memory/storage deduplication
2. Code Optimization for specific architecture
3. Hardware environment Optimizations
4. Limited take on HPC specific orchestration and scheduling
5. Hardware topology assumptions (e.g. GPU brand, interconect)
6. Chroot based containers have limited tooling (e.g. introspection,
history, search)
7. chroot based containers might be hard to scan for security
vulnerabilities, hardening, and composition.

35. Container image security
Black listed artifacts
e.g. passwords, keys
3rd party software
e.g. libraries/packages
compiled from sourceSecurity Permissions
Configuration
Packages
License
Network
Metadata
Environment Variables
Context

36. 36
MPI batch jobs
● use ssh inside container
● dssh http://www.qnib.org/2016/03/31/dssh/
● Capitalize on openmpi
○ Openmpi/pbs/TORQUE ( mpiexed does’t use ssh)
● Singularity examples uses Openmpi/Slurm
● Mesos mpi frameworks
● Commercial Univa/LSF/ support
● Research, and contribute ideas, pull requests to swarm,
kubernetes, slurm, mesos, and the alike.
● https://github.com/ambu50/wrapper-sq

37. 37
Docker performance benchmarks
http://www.theregister.co.uk/2014/08/18/docker_kicks_kvms_butt_in_ibm_tests

38. DISCLAIMER
@kelseyhightower :
The problem with most blog posts attempting to compare two different systems is
the author not having the sufficient experience to do so.
https://twitter.com/kelseyhightower/status/826974374536187905

39. © 2013-2016 Docker, Inc. All rights reserved
1. Introduction to Docker
#dockerbday

40. #dockerbday
Interesting Numbers
17k+
pull requests
40k+
stars
800k+
repos
10B+
downloads
2000+
contributors
280+
meetups
220k+
members
80+
countries

41. What is Docker?
The leading open source platform to pack, ship and run apps
as lightweight containers.
Developers: use Docker to eliminate “works on my machine” problems when
collaborating on code with co-workers.
Operators: use Docker to run and manage apps side-by-side in isolated
containers to get better compute density.
Enterprises: use Docker to build agile software delivery pipelines to ship new
features faster, more securely and with confidence for both
Linux and Windows Server apps.
#dockerbday

42. • Standardized packaging for
software and dependencies
• Isolate apps from each other
• Share the same OS kernel
• Works for all major Linux
distributions
• Containers native to Windows
Server 2016
What are Docker containers?

43. Comparing Containers and VMs
Containers are an app
level construct
VMs are an infrastructure level
construct to turn one machine
into many servers

44. Containers and VMs together
Containers and VMs together provide a tremendous amount of
flexibility for IT to optimally deploy and manage apps.

45. Evolution of the Docker Platform
Beginning
• Single purpose
• Linux developer community
#dockerbday

46. Evolution of the Docker Platform
Many purposes, users and infrastructure
Today
Developer
Community
Need to experiment
and innovate with
leading edge tech
Ops
Community
Enterprise
Partner
Ecosystem
Run business
critical apps at
scale anywhere
Extend and add
value to a platform
with a shared path
to monetization
Need a predictable
system to deploy
and run apps
#dockerbday

47. The Docker Platform
Developers Ops Enterprise Ecosystem
ONE PLATFORM
For Developers and IT
For Linux and Windows
On Premises and in the Cloud
Traditional Homegrown, Commercial ISV, Microservices
Docker Community Edition
Docker Enterprise Edition
Docker Certified
Docker Store
#dockerbday

48. What is a Docker Edition?
Making things simple for a great user experience
#dockerbday
NEW! Certification program for
Infrastructure, Plugins and Containers
Community EditionEnterprise Edition

49. Docker Community Edition (CE) & Enterprise Edition (EE)
Enterprise Edition (EE)
• CaaS enabled platform subscription
(integrated container orchestration,
management and security)
• Enterprise class support
• Quarterly releases, supported for
one year each with backported
patches and hotfixes.
• Certified Technology: Infrastructure,
Plugins, Containers
• Free Docker platform for “do it
yourself” dev and ops
• Monthly Edge release with latest
features for developers
• Quarterly release with maintenance
for ops
Community Edition (CE)
#dockerbday

50. Docker old versioning scheme
0.0.3 March
2013
1.0 June
2014
1.1 July
2014
1.2 August
2014
1.3 October
2014
1.4
December
2014
1.5
February
2015
1.6 April
2015
1.7 June
2015
1.8 August
2015
1.9
November
2015
1.10
Feburary
2016
1.11 April
2016
1.12.0 July
2016
1.12.1
August 2016
1.12.2
October
2016
1.12.3
October
2016

51. Product Versioning & Support
DockerCE
Edge
Stable
● NEW! Product Versioning follows a Year.Month model
● `docker-engine` package no longer exists. There’s only `docker-ce` and `docker-ee`.
● The binary formerly known as the engine is versioned YY.MM
DockerEE
EE
Released quarterly
Each version
supported for 1 year
v17.03 v17.04 v17.07v17.06v17.05 v17.08
v17.03
v17.06
v17.03
v17.06
v17.09 v17.10
v17.09
v17.09
#dockerbday

52. Where do you
download Docker
Community Edition?
#dockerbday

53. Docker Store!
• A marketplace for you to get the
latest trusted containers, plugins,
and Docker editions!
• You can search, browse, purchase
and manage from one location.
• Community Edition for:
− Mac
− AWS
− Fedora
− CentOS
−Windows
−Azure
−Ubuntu
−Debian
#dockerbday

54. Want to build and publish a container in Docker
Store?
Visit store.docker.com and click apply to publish through the Store
Publisher Program!

55. © 2013-2016 Docker, Inc. All rights reserved
2. Learn Docker with Bday #4 Labs!
#dockerbday

56. Lab Instructions
STEP 1: Visit
http://birthday.play-with-docker.com/
Join the slack channel - #docker-bday-4
Join the Docker Community - dockr.ly/community
#dockerbday

57. STEP 2: Select the lab
you’d like to take.
http://birthday.play-with-
docker.com/
Lab Instructions
#dockerbday

58. As a special thank you for attending, use this code for a 30%
discount to attend DockerCon in Austin!
Register: http://2017.dockercon.com/
Code: BDAY4

59. Take a
#dockerselfie
#dockerbday

60. © 2013-2016 Docker, Inc. All rights reserved
http://birthday.play-with-docker.com/
Join the slack channel: #docker-bday-4
Join the Docker Community:
https://community.docker.com/registrations/
groups/4316
#ISC2017 Docker Workshop
#dockerbday
#dockerselfie

61. © 2013-2016 Docker, Inc. All rights reserved
Singularity

62. Scientific computing container

63. Singularity Container Selection Criteria
http://hpcugent.github.io/easybuild/files/EUM17/20170208-1_Singularity.pdf

64. http://hpcugent.github.io/easybuild/files/EUM17/20170208-1_Singularity.pdf

66. http://hpcugent.github.io/easybuild/files/EUM17/20170208-1_Singularity.pdf

67. Singularity speculations against Docker
http://hpcugent.github.io/easybuild/files/EUM17/20170208-1_Singularity.pdf

68. Docker use in scientific computing
http://geekyap.blogspot.ch/2016/11/docker-vs-singularity-vs-shifter-in-hpc.html

69. Counter arguments I
Docker Singularity
privilege model namespace since 1.10
Feb 2016
suid, namespace added sep
2016
support current Linux
distro
kernel 3.10+ 2.6 kernel
Image build Dockerfile based build, some
configuration management tools
are trying to automate it, or
abstract it even more.
most of the time
bootstrapping from Docker is
the only working method out
of 4.
No additional network
configuration
configurable, one can use none,
host, or whatever network
plugin
None, which is fine for a
minimal HPC binary
No additional hardware shares kernel, view limited by
pid,user,ipc,mnt,network
except of network
namespace, chrooted
process shares host kernel

70. Counter arguments II
Docker Singularity
development maturity 5 years internal, 4 years
Open Source, 2000+
contributors
4 core developers, 1 year
old, limited community
security audited, scrutinized, running
in internet facing production
sites
- no key signing
- no introspection
- no vulnerability
scanner
- history, layer tracing
capabilities
….
eco system Huge eco system, vendor
support, and ISVs
small few companies
production usage Ubercloud, CERN, several
use cases presented in ISC
workshop
None, which is fine for a
minimal HPC binary

71. Counter arguments III
Docker Singularity
rdma Mellnox have provided
RDMA name space for multi
tenant hosts
None
Image caching works, options to inspect,
clean/prune it when needed
did not work for me on 2.2.0
rich API yes minimal functions, no restful
API to integrate with others,
other than SHUB
inspection, accounting yes None

72. https://singularity-hub.org/tools/compare

73. Play With Singularity
Demos
• https://asciinema.org/~bauerm
• https://asciinema.org/~vs
Vagrant Environment
• https://github.com/singularityware/singularity-vagrant
Workshop for last month Intel HPC devcon:
• https://github.com/singularityware/intel-hpc-devcon

74. Regardless of Singularity claims against Docker
Singularity benefits from Docker ecosystem
Given the context of internal HPC clusters not facing public internet and using in-house images.
- Singularity is minimalistic, simpler architecture, user interface and integration with existing HPC infrastructure.
- Doesn't require operations to install root Daemons.
- Enables separation of duties between Dev and Ops, allowing end users to bring their own packaged apps #BYOE
- Needs the support and contribution of the HPC and scientific community
Features wish list:
- Follow current standards, such as the OCI.
- Provide introspection and traceability
- Metadata
- Private SHUB
Scientific computing loves Singularity

75. © 2013-2016 Docker, Inc. All rights reserved
rkt

76. What is rkt?
From the rkt GitHub page, "rkt (pronounced "rock-it") is a CLI for
running app containers on Linux. rkt is designed to be secure,
compassable and standards-based.
#ACI

77. Why rkt not Docker?
§ Don’t want to run Docker’s daemon.
§ Don’t require the Docker’s rich feature set/eco system. #KISS
§ Can’t trust Docker security yet, even though it is no longer an issue.
§ Have a modern Linux distro :
kernel > 4.3 and systemd version > 222
Similar reasons to why Singularity not Docker apart from the last

79. rkt playground
https://www.katacoda.com/courses/coreos/rkt-hello-world
https://www.katacoda.com/courses/coreos/playground

80. © 2013-2016 Docker, Inc. All rights reserved
lxd

81. The Canonical Solution
https://www.slideshare.net/dustinkirkland/container-world-2017-the-questions-youre-afraid-to-ask-about-containers

82. § Front end for LXC
§ Complete Linux environment
§ Enables simple restful management API to LXC
§ Secure by default
§ Simpler and less confusing tools
§ Checkpoint, restore, snapshot support
§ No drastic change in Infrastructure
§ Controls multi local and remote containers
§ OpenStack Nova plug-in for managing virtual LXD hosts in the cloud
LXD

83. § https://linuxcontainers.org/lxd/try-it
§ https://stgraber.org/2017/03/05/run-your-own-lxd-demo-server/
Play with LXD

84. © 2013-2016 Docker, Inc. All rights reserved
Container Distributions

85. Minimalist Container Distributions
q Atomic http://www.projectatomic.io/
q Container OS ( previously called CoreOS) https://coreos.com/os/docs/latest
q Rancher http://rancher.com/rancher-os/
q VMware Photon https://vmware.github.io/photon/
q SUSE MicroOs https://www.suse.com/communities/blog/rise-caas-platform/

86. What is Next in application management?
Not yet viable for HPC, however, have brilliant ideas. claims to be for modern
and legacy app. Still less than a year old
When you create a container image with Habitat, You know exactly what
went into the container and what is configurable about the application
Build immutable infrastructure but allow
last mile Application config changes
Build containers with a
Minimum Viable OS
Decouple the application build from the final
production ready container
Orchestrate the application
launch order and topology
required
https://www.habitat.sh/

87. References
q https://www.nextplatform.com/2017/03/02/solving-hpc-conflicts-containers/
q http://geekyap.blogspot.co.za/2016/11/docker-vs-singularity-vs-shifter-in-hpc.html
q https://www.enterprisetech.com/2017/03/02/docker-platform-fills-gaps-container-ecosystem/
q https://arxiv.org/pdf/1702.05513.pdf #cHPC, the HPC container prototype
q https://www.fosdem.org/2017/schedule/event/singularity/ https://www.nextflow.io/blog/2016/more-fun-
containers-hpc.html
q http://jvns.ca/blog/2016/10/02/i-just-want-to-run-a-container/
q Videos from 2nd EasyBuild User Meeting : Singualirty, Lmod, XALT and EasyBuild
https://www.youtube.com/playlist?list=PLVA9BuLC1j-yfxp2w-wraAGDCmhjb3o5Y
q http://www.vanessasaur.us/

88. 88
Thank you