Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Containers - Portable, repeatable user-oriented application delivery. Build, ship, run any app anywhere!


Published on

his workshop will shed light on a modern solution to solve application portability, building, delivery, packaging, and system dependency issues. Containers especially Docker have seen accelerated adoption in the web, cloud and recently the enterprise. HPC environments are seeing something similar to the introduction of HPC containers Singularity and Shifter. They provide a good use case for solving software portability, not to mention ensure repeatability of results. Not to mention their ECO system provides for the better development, delivery, testing workflows that were alien to most of HPC environments. This workshop will cover the Theory and hands-on of containers and Its ecosystem. Introducing Docker and singularity containers; Docker as a general-purpose container for almost any app, Singularity as the particular container technology for HPC. The workshop will go over the foundations of the containers platform, including an overview of the platform system components: images, containers, repositories, clustering, and orchestration. The strategy is to demonstrate through "live demo, and hands-on exercises." The reuse case of containers in building a portable distributed application cluster running a variety of workloads including HPC workload.

Published in: Software
  • Be the first to comment

Containers - Portable, repeatable user-oriented application delivery. Build, ship, run any app anywhere!

  1. 1. Containers: Portable, repeatable user-oriented application delivery HPC Saudi 2017 - KAUST 15 th March 2017 #dockerbday @walidshaari
  2. 2. $whoami ● Passionate about openness, open source, devops, Infosec ● Member of the Saudi Aramco Expec Computer Center/HPC team ● Red Hat Certified Architect RHCA ● SANS GIAC Incident handler, Forensics and Web security certified ● Dhahran Docker & Ansible meetup organizer/mentor @walidshaari
  3. 3. AGENDA : Good Morning Containers 8:30 - 8:35 Introduction, Networking, Socializing 8:37 - 9:38 Interactive theory session "Presentation with Q&A" 9:40 - 10:15 Play with Docker Birthday 4 Labs 10:15 - 10:30 Coffee break 10:30 - 11:55 Singularity, rkt, lxd
  4. 4. © 2013-2016 Docker, Inc. All rights reserved Docker 4th Birthday #dockerbday
  5. 5. Docker Bday #4 celebrations worldwide! • 150+ Bday meetups! • 6000+ RSVPs • 700+ mentors #dockerbday
  6. 6. Join the Docker Student Community! Sign up here: (with your school email) for access to our free Docker Student Developer Kit and more! Become a Docker Campus Ambassador! For leaders on campus who want to help their peers learn Docker! Learn more and apply here: Are you a student?
  7. 7. Surveys and expectations Assuming everyone knows a bit of Linux/Unix/Mac OSX CLI ? Development, Operations, Security, Business, Others? Devops Configuration management Containers Schedulers Containers eco system Clusters, Load balancers, Orchestration
  8. 8. © 2013-2016 Docker, Inc. All rights reserved HPC
  9. 9. What is HPC? ▪ HPC workloads mostly ▪ Runs on Linux ▪ Runs on bare-metal for maximum performance, lower overhead ▪ HPC Application ▪ Broken into smaller parallel distributed problems across cluster nodes. ▪ Utilizes inter-process communications heavily, shared memory, or across network. ▪ Scientific computing
  10. 10. HPC ▪ HPC dominated by Academics research and discovery ▪ Industry in the last 5-10 years seen an increase in HPC interest (Car , O&E) ▪ Possible constraints: ▪ Snowflake deployments, each HPC cluster/supercomputer is build in mind with specific use cases ▪ Long lived nodes. ▪ Bloated/drift/unclean maybe diskless reboots ▪ Reboot time, or launching app could be long due to system/memory checks, bootstrapping ▪ Traditional Data Center Linux distribution ▪ Fixed installation based on single enterprise distro (Scientific, RHEL, SLES) ▪ Old kernel features #cHPC
  11. 11. © 2013-2016 Docker, Inc. All rights reserved Containers
  12. 12. First Step, Definition? • The Application matters • The application can be a process or a set of processes • The use case might be not a running app • Set of tools to develop an app • Set of scripts "apps" that are part of a pipeline • Isolated contained environment "Encapsulation" • Synonyms • chroot • jail • partition • namespace • zone
  13. 13. chroot/jail A chroot on Unix operating systems is an operation that changes the apparent root directory for the current running process and its children. A program that is run in such a modified environment cannot name (and therefore normally cannot access) files outside the designated directory tree. The term "chroot" may refer to the chroot(2) system call or the chroot(8) wrapper program. The modified environment is called a chroot jail.
  14. 14. Thank the giants
  15. 15. CONTAINERS? WHAT ARE THEY REALLY? Linux features? Namespace cgroupsLXC Union file systems Configuration management? Virtualization technology? npm jar Packaging ? rpm deb tar.gz Virtual/environment management ? Sandboxing? chroo t BSD jail Solaris zones IBM VM/370 (1972) seccomp
  16. 16. IT DEPENDS Manual Configuration Traditional VMs Less Portable Minimal overhead Most Portable Lots of overhead Configuration Management tools Containers Docker Intel Clear Containers Singularity LXC/LXD Non-Repeatable Repeatable rkt
  17. 17. DEVELOPERS LOVE DOCKER 17 adoption
  19. 19. Container Containment, isolation or encapsulation of an environment. Machine container: Encapsulates a complete system image. e.g. Ubuntu, RHEL, Scientific Linux. Process container: Encapsulates a service/process(es) . e.g. Django, ROR, Gitlab, redis, Openfoam, kafka, spark. What is the smallest application container?
  20. 20. Container Runtime docker < 1.11.0 └── systemd └── docker run OpenFoam └── Docker Engine └── OpenFoam docker > 1.11.0 └── systemd └── docker run OpenFoam └── Docker Engine └── containerd └── runc └── OpenFoam rkt > 1.0 └── systemd └── rkt run OpenFoam └── OpenFoam singularity (2.2.x) └── systemd/(init) └── bash └── OpenFoam
  21. 21. Other runtime
  22. 22. Image formats Layered Overlay filesystems/Graph drivers chrootDirectory Archive #OCI #ACI
  23. 23. Use Cases: Packaging Agnostic packaging Captures ○ Dependencies ○ Environment ○ Configurations ○ Executables ○ How about data? ○ What Else? ■ hint: m* Pack once, Run everywhere #EasyBuild #lmod #GUIX #NYU-Environment-POSTER
  24. 24. Use Case: Portability Portable/Scalable across ● platforms ● Distributions ● Environments Separation of concerns, e.g. development pack and ship, operations scale and deploy. development ensures app is resilient, operations enure infra is HA resilient and scalable
  25. 25. Use Case: Portability Portable/Scalable across ● systems ● subsystems ● Anywhere #BYOE
  26. 26. Use Case: Reproducible Paolo Di Tommaso from the Center for Genomic Regulation presented : Manage Reproducibility of Computational Workflows with Docker Containers and Nextflow.
  27. 27. 27 Data Center current silo inefficient state SchedulerScheduler Jobs Jobs Jobs Jobs Jobs Jobs Scheduler Jobs Jobs Jobs Cluster Management A Cluster Management B Cluster Management C Node as a work unit, traditiontial single level (silo) schedulers. No holistic awareness of other workloads
  28. 28. 28 Data Center Efficient Secure Allocation of Resources VC3 BigData VC1 Infra VC2 HPC Schedu ler Schedu ler Schedu ler DataCenter Scheduler jobs Jobs Jobs Jobs Jobs Jobs Jobs Jobs 2nd Generation Cluster Management Containers as a work unit, container aware workload schedulers integrated with cluster management software
  29. 29. 29 Mesos DC/OS: Example of Data Center/Container aware scheduler ▪ Mature, Open Source Apache Project ▪ Cluster Resource Manager ▪ Scalable to 10,000s of nodes ▪ Fault tolerant, no single point of failure ▪ Multi-tenancy with strong resource isolation ▪ Improved resource utilization ▪ Can schedule batch and interactive workloads for HPC and Big data.
  30. 30. 30 HPC workload runs on the cloud 25%
  31. 31. 31 Which workloads and frameworks are running on OpenStack? Source : > 38% scientific/technical computing already happening on Openstack
  32. 32. EXAMPLE HPC Data Center Use Case
  33. 33. 33 NVIDIA Example use case
  34. 34. Possible HPC Caveats/Constraints 1. Memory/storage deduplication 2. Code Optimization for specific architecture 3. Hardware environment Optimizations 4. Limited take on HPC specific orchestration and scheduling 5. Hardware topology assumptions (e.g. GPU brand, interconect) 6. Chroot based containers have limited tooling (e.g. introspection, history, search) 7. chroot based containers might be hard to scan for security vulnerabilities, hardening, and composition.
  35. 35. Container image security Black listed artifacts e.g. passwords, keys 3rd party software e.g. libraries/packages compiled from sourceSecurity Permissions Configuration Packages License Network Metadata Environment Variables Context
  36. 36. 36 MPI batch jobs ● use ssh inside container ● dssh ● Capitalize on openmpi ○ Openmpi/pbs/TORQUE ( mpiexed does’t use ssh) ● Singularity examples uses Openmpi/Slurm ● Mesos mpi frameworks ● Commercial Univa/LSF/ support ● Research, and contribute ideas, pull requests to swarm, kubernetes, slurm, mesos, and the alike. ●
  37. 37. 37 Docker performance benchmarks
  38. 38. DISCLAIMER @kelseyhightower : The problem with most blog posts attempting to compare two different systems is the author not having the sufficient experience to do so.
  39. 39. © 2013-2016 Docker, Inc. All rights reserved 1. Introduction to Docker #dockerbday
  40. 40. #dockerbday Interesting Numbers 17k+ pull requests 40k+ stars 800k+ repos 10B+ downloads 2000+ contributors 280+ meetups 220k+ members 80+ countries
  41. 41. What is Docker? The leading open source platform to pack, ship and run apps as lightweight containers. Developers: use Docker to eliminate “works on my machine” problems when collaborating on code with co-workers. Operators: use Docker to run and manage apps side-by-side in isolated containers to get better compute density. Enterprises: use Docker to build agile software delivery pipelines to ship new features faster, more securely and with confidence for both Linux and Windows Server apps. #dockerbday
  42. 42. • Standardized packaging for software and dependencies • Isolate apps from each other • Share the same OS kernel • Works for all major Linux distributions • Containers native to Windows Server 2016 What are Docker containers?
  43. 43. Comparing Containers and VMs Containers are an app level construct VMs are an infrastructure level construct to turn one machine into many servers
  44. 44. Containers and VMs together Containers and VMs together provide a tremendous amount of flexibility for IT to optimally deploy and manage apps.
  45. 45. Evolution of the Docker Platform Beginning • Single purpose • Linux developer community #dockerbday
  46. 46. Evolution of the Docker Platform Many purposes, users and infrastructure Today Developer Community Need to experiment and innovate with leading edge tech Ops Community Enterprise Partner Ecosystem Run business critical apps at scale anywhere Extend and add value to a platform with a shared path to monetization Need a predictable system to deploy and run apps #dockerbday
  47. 47. The Docker Platform Developers Ops Enterprise Ecosystem ONE PLATFORM For Developers and IT For Linux and Windows On Premises and in the Cloud Traditional Homegrown, Commercial ISV, Microservices Docker Community Edition Docker Enterprise Edition Docker Certified Docker Store #dockerbday
  48. 48. What is a Docker Edition? Making things simple for a great user experience #dockerbday NEW! Certification program for Infrastructure, Plugins and Containers Community EditionEnterprise Edition
  49. 49. Docker Community Edition (CE) & Enterprise Edition (EE) Enterprise Edition (EE) • CaaS enabled platform subscription (integrated container orchestration, management and security) • Enterprise class support • Quarterly releases, supported for one year each with backported patches and hotfixes. • Certified Technology: Infrastructure, Plugins, Containers • Free Docker platform for “do it yourself” dev and ops • Monthly Edge release with latest features for developers • Quarterly release with maintenance for ops Community Edition (CE) #dockerbday
  50. 50. Docker old versioning scheme 0.0.3 March 2013 1.0 June 2014 1.1 July 2014 1.2 August 2014 1.3 October 2014 1.4 December 2014 1.5 February 2015 1.6 April 2015 1.7 June 2015 1.8 August 2015 1.9 November 2015 1.10 Feburary 2016 1.11 April 2016 1.12.0 July 2016 1.12.1 August 2016 1.12.2 October 2016 1.12.3 October 2016
  51. 51. Product Versioning & Support DockerCE Edge Stable ● NEW! Product Versioning follows a Year.Month model ● `docker-engine` package no longer exists. There’s only `docker-ce` and `docker-ee`. ● The binary formerly known as the engine is versioned YY.MM DockerEE EE Released quarterly Each version supported for 1 year v17.03 v17.04 v17.07v17.06v17.05 v17.08 v17.03 v17.06 v17.03 v17.06 v17.09 v17.10 v17.09 v17.09 #dockerbday
  52. 52. Where do you download Docker Community Edition? #dockerbday
  53. 53. Docker Store! • A marketplace for you to get the latest trusted containers, plugins, and Docker editions! • You can search, browse, purchase and manage from one location. • Community Edition for: − Mac − AWS − Fedora − CentOS −Windows −Azure −Ubuntu −Debian #dockerbday
  54. 54. Want to build and publish a container in Docker Store? Visit and click apply to publish through the Store Publisher Program!
  55. 55. © 2013-2016 Docker, Inc. All rights reserved 2. Learn Docker with Bday #4 Labs! #dockerbday
  56. 56. Lab Instructions STEP 1: Visit Join the slack channel - #docker-bday-4 Join the Docker Community - #dockerbday
  57. 57. STEP 2: Select the lab you’d like to take. Lab Instructions #dockerbday
  58. 58. As a special thank you for attending, use this code for a 30% discount to attend DockerCon in Austin! Register: Code: BDAY4
  59. 59. Take a #dockerselfie #dockerbday
  60. 60. © 2013-2016 Docker, Inc. All rights reserved Join the slack channel: #docker-bday-4 Join the Docker Community: groups/4316 #ISC2017 Docker Workshop #dockerbday #dockerselfie
  61. 61. © 2013-2016 Docker, Inc. All rights reserved Singularity
  62. 62. Scientific computing container
  63. 63. Singularity Container Selection Criteria
  64. 64.
  65. 65.
  66. 66. Singularity speculations against Docker
  67. 67. Docker use in scientific computing
  68. 68. Counter arguments I Docker Singularity privilege model namespace since 1.10 Feb 2016 suid, namespace added sep 2016 support current Linux distro kernel 3.10+ 2.6 kernel Image build Dockerfile based build, some configuration management tools are trying to automate it, or abstract it even more. most of the time bootstrapping from Docker is the only working method out of 4. No additional network configuration configurable, one can use none, host, or whatever network plugin None, which is fine for a minimal HPC binary No additional hardware shares kernel, view limited by pid,user,ipc,mnt,network except of network namespace, chrooted process shares host kernel
  69. 69. Counter arguments II Docker Singularity development maturity 5 years internal, 4 years Open Source, 2000+ contributors 4 core developers, 1 year old, limited community security audited, scrutinized, running in internet facing production sites - no key signing - no introspection - no vulnerability scanner - history, layer tracing capabilities …. eco system Huge eco system, vendor support, and ISVs small few companies production usage Ubercloud, CERN, several use cases presented in ISC workshop None, which is fine for a minimal HPC binary
  70. 70. Counter arguments III Docker Singularity rdma Mellnox have provided RDMA name space for multi tenant hosts None Image caching works, options to inspect, clean/prune it when needed did not work for me on 2.2.0 rich API yes minimal functions, no restful API to integrate with others, other than SHUB inspection, accounting yes None
  71. 71.
  72. 72. Play With Singularity Demos • • Vagrant Environment • Workshop for last month Intel HPC devcon: •
  73. 73. Regardless of Singularity claims against Docker Singularity benefits from Docker ecosystem Given the context of internal HPC clusters not facing public internet and using in-house images. - Singularity is minimalistic, simpler architecture, user interface and integration with existing HPC infrastructure. - Doesn't require operations to install root Daemons. - Enables separation of duties between Dev and Ops, allowing end users to bring their own packaged apps #BYOE - Needs the support and contribution of the HPC and scientific community Features wish list: - Follow current standards, such as the OCI. - Provide introspection and traceability - Metadata - Private SHUB Scientific computing loves Singularity
  74. 74. © 2013-2016 Docker, Inc. All rights reserved rkt
  75. 75. What is rkt? From the rkt GitHub page, "rkt (pronounced "rock-it") is a CLI for running app containers on Linux. rkt is designed to be secure, compassable and standards-based. #ACI
  76. 76. Why rkt not Docker? § Don’t want to run Docker’s daemon. § Don’t require the Docker’s rich feature set/eco system. #KISS § Can’t trust Docker security yet, even though it is no longer an issue. § Have a modern Linux distro : kernel > 4.3 and systemd version > 222 Similar reasons to why Singularity not Docker apart from the last
  77. 77. rkt playground
  78. 78. © 2013-2016 Docker, Inc. All rights reserved lxd
  79. 79. The Canonical Solution
  80. 80. § Front end for LXC § Complete Linux environment § Enables simple restful management API to LXC § Secure by default § Simpler and less confusing tools § Checkpoint, restore, snapshot support § No drastic change in Infrastructure § Controls multi local and remote containers § OpenStack Nova plug-in for managing virtual LXD hosts in the cloud LXD
  81. 81. § § Play with LXD
  82. 82. © 2013-2016 Docker, Inc. All rights reserved Container Distributions
  83. 83. Minimalist Container Distributions q Atomic q Container OS ( previously called CoreOS) q Rancher q VMware Photon q SUSE MicroOs
  84. 84. What is Next in application management? Not yet viable for HPC, however, have brilliant ideas. claims to be for modern and legacy app. Still less than a year old When you create a container image with Habitat, You know exactly what went into the container and what is configurable about the application Build immutable infrastructure but allow last mile Application config changes Build containers with a Minimum Viable OS Decouple the application build from the final production ready container Orchestrate the application launch order and topology required
  85. 85. References q q q q #cHPC, the HPC container prototype q containers-hpc.html q q Videos from 2nd EasyBuild User Meeting : Singualirty, Lmod, XALT and EasyBuild q
  86. 86. 88 Thank you