Container Runtime Meetup #3の発表資料です。 高レベルコンテナランタイムcontainerdの概要を紹介しています。 https://runtime.connpass.com/event/198071/

1. Copyright(c)2021 NTT Corp. All Rights Reserved
containerd
Container Runtime Meetup #3
2021/1/28

2. Copyright(c)2021 NTT Corp. All Rights Reserved
GitHub:@ktock / Twitter:@TokunagaKohei
containerd Stargz Snapshotter

3. Copyright(c)2021 NTT Corp. All Rights Reserved
. 4
1

4. Copyright(c)2021 NTT Corp. All Rights Reserved
. 4
1

5. Copyright(c)2021 NTT Corp. All Rights Reserved
containerd
l CNCF graduated 5
l Docker Docker
l Kubernetes
https://github.com/containerd/containerd
l GKE AWS Fargate AKS(preview) IKS
l Docker/moby BuildKit k3c PouchContainer
l K8s k3s kind minikube kubespray microk8s
l FaaS faasd
https://sysdig.com/blog/sysdig-
2021-container-security-usage-report/

6. Copyright(c)2021 NTT Corp. All Rights Reserved
containerd
2017 2018 2019 2020 2021
12 Docker containerd [1]
5 containerd CNCF [2]
v1.0.0 v1.1.0 v1.2.0 v1.3.0 v1.4.0
(12 ) (4 ) (10 ) (9 ) (8 )
2 containerd CNCF graduation[3]
7 AKS containerd (preview) [7]
8 IKS containerd [4]
9 GKE containerd GA[5]
4 AWS Fargate containerd [6]
v1.5.0-beta.0
(1 )
2016
[1] https://www.docker.com/docker-news-and-press/docker-extracts-and-donates-containerd-its-core-container-runtime-accelerate ; [2] https://www.docker.com/blog/containerd-joins-cncf/ ; [3]
https://www.cncf.io/announcements/2019/02/28/cncf-announces-containerd-graduation/ ; [4] https://cloud.ibm.com/docs/containers?topic=containers-changelog_archive#1112_1513 ; [5] https://cloud.google.com/kubernetes-
engine/docs/release-notes-archive#september_5_2019 ; [6] https://aws.amazon.com/jp/blogs/containers/aws-fargate-launches-platform-version-1-4/ ; [7] https://azure.microsoft.com/ja-jp/updates/azure-kubernetes-service-aks-support-for-
containerd-runtime-is-in-preview/

7. Copyright(c)2021 NTT Corp. All Rights Reserved
3 containerd
OCI
kubelet
CRI
OCI
containerd API
dockerd
OCI
containerd API
CRI Docker

8. Copyright(c)2021 NTT Corp. All Rights Reserved
1: Kubernetes CRI
kubectl apply
Pod
CRI
OCI
runc, gVisor, Kata Containers
OCI
apiserver
kubelet
CRI
pull/push

9. Copyright(c)2021 NTT Corp. All Rights Reserved
2: Docker
docker run
containerd
runc, gVisor, Kata Containers
OCI
containerd API
dockerd
pull/push
Docker API

10. Copyright(c)2021 NTT Corp. All Rights Reserved
3:
runc, gVisor, Kata Containers
OCI
l Docker
l containerd
l containerd containerd
l containerd
containerd API
BuildKit faasd
Pouch
Container
nerdctl

11. Copyright(c)2021 NTT Corp. All Rights Reserved
. 4
1

12. Copyright(c)2021 NTT Corp. All Rights Reserved
OS
plugins
containerd
l
l
lDocker BuildKit
l unix socket
containerd API CRI
• /run/containred/containerd.sock
l OCI
• OCI Firecracker
l containerd
container image tasks
namespace
leases version
introspection
events diff
Server
runtimes
) () 2
CRI
Client
containerd
API
Kubelet
4 .1

13. Copyright(c)2021 NTT Corp. All Rights Reserved
OS
container image tasks
namespace
leases version
introspection
events diff
containerd API
l Smart Client
l containerd API
l
l pull/push
l
l OCI config
l
Go
containerd
container image …
namespace
leases content
snapshots
events tasks
OCI spec
Server
plugins runtimes
API

14. Copyright(c)2021 NTT Corp. All Rights Reserved
OS
Container
ctr: https://github.com/containerd/containerd
l containerd contianerd
CLI
l containerd API
nerdctl: https://github.com/AkihiroSuda/nerdctl
l Docker containerd CLI by Akihiro Suda, NTT
l Docker
l Lazy pulling containerd
containerd
containerd
l Docker BuildKit faasd Pouch Container
container image tasks
namespace
leases version
introspection
events diff
Server
plugins runtimes
ctr, nerdctl, Docker, etc
containerd
API
Client lib
crictl: https://github.com/kubernetes-sigs/cri-tools
l Kubernetes sig-node CRI
CLI containerd API
l Server CRI

15. Copyright(c)2021 NTT Corp. All Rights Reserved
containerd
l
•
l unix socket API
l /run/containerd/containerd.sock
l
• API Go plugin
•
Ø containerd
container image tasks
namespace
leases version
snapshots
CRI tasks
container image tasks
namespace
leases version
introspection
events diff
OCI spec
Client
OS
plugins
shim
OCI

16. Copyright(c)2021 NTT Corp. All Rights Reserved
containerd
container image …
namespace
leases content
snapshots
CRI tasks
Metadata store
OS
l
• API
l persistent
metadata store (bbolt;
https://github.com/etcd-io/bbolt)
l CRI
•
Ø
• Pod
CNI
shim
OCI
CRI

17. Copyright(c)2021 NTT Corp. All Rights Reserved
OS
containerd
Content store
l pull
l
Snapshotter
l
“snapshot”
l snapshot rootfs
l snapshotter
Overlayfs btrfs aufs FUSE…
Runtime shim OCI
l V2 shim
container image …
namespace
leases content
snapshots
CRI tasks
C
o
n
t
e
n
t
s
t
o
r
e
S
n
a
p
s
h
o
t
t
e
r
R
u
n
t
i
m
e
Content store snapshotter containerd
Docker Graph Driver

18. Copyright(c)2021 NTT Corp. All Rights Reserved
OS
containerd
containerd ”tightly scoped”
l unix socket
gRPC API containerd
l Go plugin
l containerd API
l
container image …
namespace
leases content
snapshots
CRI tasks
shim
OCI
l Proxy content store
IPFS
l Proxy snapshotter rootfs
lazy pulling
l Stream processor
l V2 shim OCI
Kata
s
h
i
m

19. Copyright(c)2021 NTT Corp. All Rights Reserved
containerd
firecracker-containerd https://github.com/firecracker-microvm/firecracker-containerd
l AWS Firecracker microVM containerd
l Snapshotter v2 runtime microVM API control API
Stargz Snapshotter https://github.com/containerd/stargz-snapshotter
l containerd non-core subproject
l eStargz lazy pulling snapshotter ”remote” snapshotter
imgcrypt https://github.com/containerd/imgcrypt
l containerd non-core subproject
l stream processor
OCI runtime V2 runtime
l Kata Containers

20. Copyright(c)2021 NTT Corp. All Rights Reserved
. 4
1

21. Copyright(c)2021 NTT Corp. All Rights Reserved
containerd 1.4.x
Lazy pulling: https://github.com/containerd/containerd/pull/3793
l pull
l Stargz Snapshotter https://github.com/containerd/stargz-snapshotter proxy snapshotter
OCI eStargz lazy pull
0 5 10 15 20 25 30 35 40 45
estargz
estargz-noopt
legacy
Start up time of python:3.7 (print “hello”)
pull create run
Host: EC2 Oregon (m5.2xlarge, Ubuntu 20.04)
Registry: GitHub Container Registry (ghcr.io)
Commit 7f45f74
(See detailed info in the later slides)
[sec]
Cgroups v2 : https://github.com/containerd/containerd/issues/3726
l Fedora (> 31) cgroup v2 containerd
l cgroup rootless docker
--pids-limit Docker 20.10
SELinux MCS (CRI): https://github.com/containerd/cri/pull/1487
l CRI SELinux MCS Multi Category Security
Pod
l Pod Pod

22. Copyright(c)2021 NTT Corp. All Rights Reserved
Stargz Snapshotter lazy pulling
l containerd non-core
l OCI eStargz lazy pulling proxy snapshotter
• pull
Kubernetes
l Prefetch content verification
l Kaniko, go-containerregistry, ko, nerdctl eStargz
Stargz
Snapshotter
rootfs FUSE
Lazy
pull
eStargz
pull
https://github.com/containerd/stargz-snapshotter
https://www.slideshare.net/KoheiTokunaga/stargz-snapshotter-pullcontainerd-238429575
2 ” Stargz Snapshotter: pull containerd ”. CNDT2020

23. Copyright(c)2021 NTT Corp. All Rights Reserved
containerd
l ctr containerd containerd
l nerdctl https://github.com/AkihiroSuda/nerdctl Docker
NRI(Node Resource Interface) (1.5 ): https://github.com/containerd/nri
l CNI
l CNI NW NRI cgroup namespace path
CPU pinning
Sandbox API: https://github.com/containerd/containerd/issues/4131
l containerd API Pod
Overlayfs volatile option: https://github.com/containerd/containerd/pull/4785
l Overlayfs volatile option (Linux 5.10) upper dir sync
Higher level API: CRI v2 embedded kubelet build
l containerd API

24. Copyright(c)2021 NTT Corp. All Rights Reserved
plugins
shim
OCI
C
o
n
t
e
n
t
s
t
o
r
e
S
n
a
p
s
h
o
t
t
e
r
R
u
n
t
i
m
e
container image tasks
namespac
e leases version
snapshots
CRI tasks
container image …
namespac
e leases content
snapshots
events tasks
OCI spec
API
Metadata store
Contained
l Kubernetes CRI
l Docker
l
containerd
l
l smart client
l
containerd
l Lazy pulling cgroup v2
l
(
)