The container land has been heavily influenced by Docker. After all, the huge ecosystem building Docker images and tools is hard to ignore. Docker, defines images as lists of layers. This helps container engines cache reused layers and avoid re-downloading the whole root filesystem when minor changes get pushed. The, fairly new, OCI image spec is adopting the same approach in standardising the image format. However, layers can be chanky in size and hard to manage. Additionally, it would be better to increase the caching granularity to single files (say /bin/bash) instead of large layers. Physicists in the LHC computing GRID had a similar problem. They solced it with CernVM-FS. This talk explores the idea of using it for container image distribution. George Lestaris is a software engineer working at Pivotal in the Garden team, the container runtime of Cloud Foundry (CF). Before Pivotal, he spent time with high throughput computing on the LHC computing grid and cloud computing research in CERN. He has given talks before in CHEP 2013 regarding virtual clusters and their use in high-energy physics, in PyCon UK 2015 on interactive cloud experimentation and in the CF summit Santa Clara 2016 on containers in CF. -- https://www.youtube.com/watch?v=g7dJCMh20iE

1. Alternatives to
layer-based
image distribution
using a CERN filesystem for
distributing container images
George Lestaris
@glestaris

2. • Software engineer at Pivotal
• working for Cloud
Foundry GrootFS
• github.com/cloudfoundry/
grootfs
• ex-CERNois
About me

3. docker run elasticsearch:2.3.5
Container image

4. Container image format
• Container images are formalized in: Docker, AppC
(ACI) and OCI Image spec
• Generally: image is the combination of:
• a set of layers
• metadata

5. Building an image
FROM python:3.5
ADD . /myapp
RUN pip install
-r /myapp/requirements.txt
ENTRYPOINT python /myapp/manage.py
runserver 0.0.0.0:8000

6. Container
images are
composed of
layers
Layer is a set of
files and
directories

7. FROM python:3.5
Layers help us to
inherit images

8. FROM python:3.5
ADD . /myapp
RUN pip install …

9. • Different image formats - different distributions
mechanisms
• Docker: download layers through HTTP
connections from a registry
• Helps reusing layers of base images
• Efficient container image fetching by parallelizing
the downloads
Container image distribution

10. Registry
ClientClientClientClient
New image
From cached base
Update dependencies

11. Distributing software
in HEP

13. Data
Data
Data
Data
Data
Data
Data
Data
Frequent
releases
Simulation engine
Analysis framework
Experiment geometry
Experiment software
Dependencies
Simulation engine
Analysis framework
Experiment geometry
Experiment software
Dependencies
Simulation engine
Analysis framework
Experiment geometry
Experiment software
Dependencies
Simulation engine
Analysis framework
Experiment geometry
Experiment software
Dependencies
Simulation engine
Analysis framework
Experiment geometry
Experiment software
Dependencies
Simulation engine
Analysis framework
Experiment geometry
Experiment software
Dependencies
Simulation engine
Analysis framework
Experiment geometry
Experiment software
Dependencies
Simulation engine
Analysis framework
Experiment geometry
Experiment software
Dependencies

14. WLCG
170 computing
centres
in 42 countries

15. CernVM-FS

16. • Network file system
• no packages and layers —> files and directories
• FUSE
• Lazily downloads the used files
• Deduplication Downloaded files get cached using
a content addressable storage
using a network filesystem

17. User application
VFS
FUSE kernel
module
CernVM-FS
FUSE
CernVM-FS service
GET catalog
Cache
stat sha256:…
GET /blob/sha256:…
open /dir/file catalog
/dir/file
—> sha256:…

18. Similarities between
HEP software and
container images

19. • Most images are based on a Linux distribution
• redis 3.2.3
• Image size: 190 MB (Compressed 74 MB)
• Used to boot: 11 MB - 5.7 %
• node 6.5.0 5.4 %
• nginx 1.11 3.1 %
Applications use a small fragment of the image

20. • nginx 1.10 to 1.11:
• Real changes: 4.02 MB
• Layer changes: 58 MB (two of the three layers)
• 14.4 times the size of the diff
• nginx 1.9 to 1.10: 4.8 times the size of the diff
Small changes between versions

21. Demo
CernVM-FS
and runC

22. • Small tool to create containers
• Low-level interface - not supposed to be a
container runtime
• Used by container runtimes (Docker, Garden)
internally
runC

23. Performance
comparison

24. • http://github.com/glestaris/container-camp
• Used iCE - see PyCon UK 2015
• 20 AWS VMs in eu-west (m4.large)
• 1 CernVM-FS server on an AWS VM (m4.large) in
eu-central
• Dockerhub
Experiment setup

25. • All VMs create a redis:3.2.3 container in parallel
• Comparing runC, Docker and Docker with warm cache
• Run the server and ping (wait for the server to came
up)
Scenario
redis-server --daemonize yes
while ! redis-cli ping; do
echo 'retrying'
done

27. • IPFS: InterPlanetary file system
• Deduplication Content addressed storage for
object
• History Versioned objects
• Decentralized P2P transfers
• Objects are files, directories or changes
(commits)
Other approaches

28. • CI server
• Large clusters that parallelly fetch images
• Network contention
• Maintaining a private registry
• Serverless (?)
Use cases