The document discusses estargz lazy pulling, a method that optimizes container image retrieval by enabling on-demand file access, thus improving container start times. It highlights various tools and frameworks compatible with this technique, such as Kubernetes, containerd, and buildkit, and provides links to implementations and further resources. Additionally, it references relevant research indicating the potential efficiency gains from reducing image pull times in container orchestration environments.

1. Copyright(c)2021 NTT Corp. All Rights Reserved
eStargz lazy pulling
2021/11/5
CloudNative Days Tokyo 2021

2. Copyright(c)2021 NTT Corp. All Rights Reserved
GitHub:@ktock / Twitter:@TokunagaKohei
containerd BuildKit
Container Runtime Meetup

3. Copyright(c)2021 NTT Corp. All Rights Reserved
TL;DR
l Pull 3
l eStargz lazy pulling pull
• https://github.com/containerd/stargz-snapshotter
l eStargz Lazy pulling
• Kubernetes k3s KinD containerd nerdctl Podman CRI-O BuildKit Kaniko
go-containerregistry ko buildpacks.io
l eStargz OCI Image Specification

4. Copyright(c)2021 NTT Corp. All Rights Reserved
pull
pulling packages accounts for 76% of container start time,
but only 6.4% of that data is read [Harter et al. 2016]
[Harter et al. 2016] Tyler Harter, Brandon Salmon, Rose Liu, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau. "Slacker: Fast Distribution with
Lazy Docker Containers". 14th USENIX Conference on File and Storage Technologies (FAST ’16). February 22–25, 2016, Santa Clara, CA, USA
l pull
l cold start pull
l
• e.g. 4

5. Copyright(c)2021 NTT Corp. All Rights Reserved
OCI
l pull
l tar+{gzip,zstd} 5
l

6. Copyright(c)2021 NTT Corp. All Rights Reserved
eStargz lazy pulling
Standard Container Registry
eStargz-aware runtimes
BuildKit
Lazy pull
Lazy pull
legacy
ctr-remote
nerdctl
BuildKit
Kaniko
Image builders/convertes
eStargz
creates
eStargz
still runnable on
eStargz-agnostic tools (e.g. Docker)
etc…
etc…
legacy
l OCI lazy pulling
• Pull
• 6
• 6
l Lazy pulling OCI
l BuildKit Kaniko nerdctl eStargz
https://github.com/containerd/stargz-snapshotter

7. Copyright(c)2021 NTT Corp. All Rights Reserved
eStargz
eStargz
bin/ls
usr/bin/apt
entrypoint.sh
bin/bash
Range Request
TOC(Table Of Contents) footer
/
gzip
HTTP Range Request
/
stargz
bin/ls
usr/bin/apt
entrypoint.sh
bin/bash
For more details: https://github.com/containerd/stargz-snapshotter/blob/master/docs/stargz-estargz.md
l
l OCI
l CRFS stargz
l 7

8. Copyright(c)2021 NTT Corp. All Rights Reserved
README https://github.com/containerd/stargz-snapshotter
Lower is better

9. Copyright(c)2021 NTT Corp. All Rights Reserved
lazy pulling

10. Copyright(c)2021 NTT Corp. All Rights Reserved
stargz-snapshotter 1 0
https://github.com/containerd/stargz-snapshotter
Kubernetes
Containerd
Stargz
Snapshotter
external process
gRPC API
kubelet
Lazy pull
Container Registry
eStargz
CRI-O
Stargz Store
external process
exposed on dir
kubelet
l CRI 0 lazy pulling
• Containerd Stargz Snapshotter
• CRI-O Stargz Store

11. Copyright(c)2021 NTT Corp. All Rights Reserved
K3s
l 1 Kubernetes
l containerd CRI
l v1.22.2 Stargz Snapshotter lazy pulling
https://rancher.com/docs/k3s/latest/en/advanced/#enabling-lazy-pulling-of-estargz-experimental
k3s server --snapshotter=stargz
K3s server/agent
Stargz
Snapshotter (library)
k8s
Lazy pull
Container Registry
eStargz
k3d cluster create --image=rancher/k3s:v1.22.2-k3s2 --k3s-arg='--snapshotter=stargz@server:*;agent:*'
l K3d (k3s in Docker) lazy pulling
builtin
https://github.com/k3s-io/k3s

12. Copyright(c)2021 NTT Corp. All Rights Reserved
KinD
l Kubernetes in Docker Kubernetes
l 1CRI containerd
l Stargz Snapshotter lazy pulling
• ghcr.io/stargz-containers/estargz-kind-node:0.9.0
• KinD 21 :https://github.com/kubernetes-
sigs/kind/pull/2076
$ kind create cluster --name estargz-demo --image ghcr.io/stargz-containers/estargz-kind-node:0.9.0
Stargz
Snapshotter
kubelet
Lazy pull
Container Registry
eStargz
https://github.com/kubernetes-sigs/kind

13. Copyright(c)2021 NTT Corp. All Rights Reserved
containerd nerdctl
proc
Stargz
Snapshotter
Fetches files/chunks on demand
Mounts rootfs as FUSE
Node
Lazy pull
Container Registry
eStargz
container
l Containerd Stargz Snapshotter3 lazy pulling
• Containerd remote snapshotter 3
l nerdctl Docker containerd CLI Stargz Snapshotter lazy pulling 1
nerdctl --snapshotter=stargz run ghcr.io/stargz-containers/python:3.9-esgz
nerdctl --snapshotter=stargz compose -f docker-compose.stargz.yaml up
https://github.com/containerd/containerd
https://github.com/containerd/nerdctl

14. Copyright(c)2021 NTT Corp. All Rights Reserved
Podman CRI-O
proc
container
Stargz Store
Provides layers
(mounted as FUSE)
Podman, CRI-O
Uses layers for rootfs
Node
Lazy pull
Container Registry
eStargz
l Stargz Store lazy pulling
l systemd service
yum install -y stargz-snaphsotter
systemctl enable --now stargz-store
podman run ghcr.io/stargz-containers/python:3.9-esgz
l Additional Layer Store Podman >= v3.3.0, CRI-O >= v1.22.0
• Stargz Store Podman/CRI-O FUSE 4 1
https://github.com/cri-o/cri-o
https://github.com/containers/podman
Additional Layer Store

15. Copyright(c)2021 NTT Corp. All Rights Reserved
BuildKit
FROM ghcr.io/stargz-containers/golang:1.15.3-esgz as dev
COPY ./hello.go /hello.go
RUN go build -o hello /hello.go
COPY and RUN without waiting for
the pull completion
• /usr/local/go/bin/go
• /usr/local/go/src/fmt/…
etc...
Fetch files/chunks on demand
Build on node
Lazy pull
Container Registry
eStargz
golang:1.15.3-esgz
l 5 lazy pulling1 v0.8.0
l 5 1pull 5
• 5
$ docker buildx create --use --name lazy-builder ¥
--driver docker-container --driver-opt image=moby/buildkit:master ¥
--buildkitd-flags '--oci-worker-snapshotter=stargz'
$ docker buildx inspect --bootstrap lazy-builder
$ docker buildx build .
https://github.com/moby/buildkit

16. Copyright(c)2021 NTT Corp. All Rights Reserved
eStargz

17. Copyright(c)2021 NTT Corp. All Rights Reserved
eStargz
l Kaniko : https://github.com/GoogleContainerTools/kaniko
• Kubernetes
• GGCR_EXPERIMENT_ESTARGZ=17
• eStargz
$ docker run --rm -e GGCR_EXPERIMENT_ESTARGZ=1 ¥
-v /tmp/context:/workspace -v ~/.docker/config.json:/kaniko/.docker/config.json:ro ¥
gcr.io/kaniko-project/executor:v1.6.0 --destination "ghcr.io/ktock/sample:esgz"
$ docker buildx build ¥
-o type=registry,name=ktokunaga/hello:esgz,oci-mediatypes=true,compression=estargz .
l BuildKit : https://github.com/moby/buildkit
• eStargz7compression type1 master
• Docker Buildx
https://github.com/moby/buildkit/blob/master/docs/stargz-estargz.md#creating-
stargzestargz-images

18. Copyright(c)2021 NTT Corp. All Rights Reserved
eStargz
l nerdctl : https://github.com/containerd/nerdctl
• Docker 1containerd CLI
• 1 eStargz
• nerdctl build 1eStargz
l ctr-remote : https://github.com/containerd/stargz-snapshotter
• Stargz Snapshotter containerd CLI
• 1 eStargz
• 1
l go-containerregistry : https://github.com/google/go-containerregistry
• Kaniko, ko, buildpacks.io 8
• GGCR_EXPERIMENT_ESTARGZ=1 eStargz
• ggcr 1 1 eStargz
$ nerdctl image convert --estargz --oci ghcr.io/ktock/foo:1 ghcr.io/ktock/foo:1-esgz
$ go install github.com/google/go-containerregistry/cmd/crane@latest
$ GGCR_EXPERIMENT_ESTARGZ=1 crane optimize –-insecure ghcr.io/ktock/foo:1 ghcr.io/ktock/foo:1-esgz
ctr-remote image optimize --oci ghcr.io/ktock/foo:1 ghcr.io/ktock/foo:1-esgz

19. Copyright(c)2021 NTT Corp. All Rights Reserved
eStargz 2021/10
*1 eStargz OCI OCI eStargz
*2 GGCR_EXPERIMENT_ESTARGZ=1
*3 Stargz Snapshotter ghcr.io/stargz-containers/estargz-kind-node lazy pulling

20. Copyright(c)2021 NTT Corp. All Rights Reserved
https://github.com/opencontainers/image-spec/issues/815
l OCI Image Specification
• gzip 2 application/vnd.oci.image.layer.v1.tar+gz
• 0 Annotation org.opencontainers.image.toc.digest

21. Copyright(c)2021 NTT Corp. All Rights Reserved
Join our discussion!
l Subproject (non-core) of CNCF graduated containerd
l github https://github.com/containerd/stargz-snapshotter
l slack https://slack.containerd.io/ (#containerd-dev channel at CNCF slack)
l Pre-converted images are available on ghcr.io/stargz-containers
l Pull 2
l eStargz 2 lazy pulling2 pull 1
• https://github.com/containerd/stargz-snapshotter
l 1 eStargz Lazy pulling
• Kubernetes k3s KinD containerd nerdctl Podman CRI-O BuildKit Kaniko
go-containerregistry ko buildpacks.io
l eStargz OCI Image Specification 1