Nydus_Image_Service_for_Confidential_Containers

1. Nydus Image Service for
Confidential Containers
Gerry Liu
Senior Staff Engineer
Alibaba Cloud

2. Introduction
• Gerry Liu <gerry@linux.alibaba.com>
• Working at OS Team, Alibaba Cloud
• Interested in Linux kernel and cloud-native related technologies
• Member of
• Kata Containers: <https://github.com/kata-containers/kata-containers>
• Confidential Containers: <https://github.com/confidential-containers>
• Nydus Image Service: <https://github.com/dragonflyoss/image-service >
• rust-vmm: <https://github.com/rust-vmm>

3. What We Will Discuss Today
• What makes CoCo image management special?
• What is Nydus Image Service?
• Nydus Image Service for CoCo

4. Project CoCo (Confidential Containers)
Confidential Containers are designed to
protect confidentiality and integrity of
container workloads
by using hardware TEEs.
What types of resources to protect?
We are going to talk about image
management
CPU Memory Image
Storage
Net Device
Hardware
TEE
Encryption
• dm-crypt
• dm-integrity
• ZFS
• TLS/HTTPS
?
?
Application Container

5. Container
Rootfs
Container
Rootfs
Host
File System
CoCo Image Management Challenges
1. download image
blobs from registry
2. store blobs
to host FS
Container
Rootfs
3. convert blobs to
container rootfs
4. access files from
container image
• Image layers are shared and reused among containers
• All contents are visible on host side
1
Images should be protected
on registry and host
2
Rootfs should be
mounted inside guest
3
Images should be reused
with low memory overhead

6. CoCo Image Management Solutions
Image formats
with integrity &
confidentiality
• ocicrypt
• ocicrypt-rs
• cosign
1
Image management
inside trusted
execution environment
• Containerd with remote
image management
• Kata Containers with image
service
• image-rs: content store and
snapshotter for CC
2
Image Content
Lazy Loading & Caching
• Downloading images for every
container
• Slow startup
• High pressure on local storage
• High pressure on registry
• How to achieve similar startup
time as native containers?
• Image Caching
• Image Lazy Loading
• Nydus Image Service
3

7. What We Will Discuss Today
• What makes CoCo image management special?
• What is Nydus Image Service?
• Nydus Image Service for CoCo

8. What is Nydus Image Service?
An image format w/ advanced features:
• Lazy loading
• Data deduplication
• Native or OCIv1 compatible modes
• Encryption(in progress)
A readonly filesystem for containers
(runC/Kata/Kata CC), AI models and
software packages by:
• Linux/MacOS FUSE
• Virtio-fs
• EROFS with page sharing
• User space library (in progress)
An image service integrated with ecosystem:
• OCI distribution compatible
• Integrated with buildkit/containerd/cri-
o/nerdctl/Kata/harbor/dragonfly
A node level storage subsystem with
P2P, cache and data deduplication
1
2
3
4

9. Nydus Image Format
• Native Mode:
• File data is split into chunks
• Chunks may be compressed and/or
encrypted
• Chunk deduplication by comparing
chunk digests at build time
• OCIv1 Compatible Mode:
• Contains no file data, referring file
data in OCIv1 images instead
• RAFS metadata for lazy loading, ~5%
size of original OCIv1 image
• RAFS Data Blob: contain both file data
and FS metadata
• RAFS Metadata Blob: only contain
merged FS metadata from all data
blobs. Avoid runtime overhead by
overlay layers at build time. Chunked file data
with compression
and/or encryption
Info to locate and
decode chunk data
Info to provide
filesystem view
& locate chunk info
1
Merged FS View
No File Chunk Data

10. Flexible FS for Read-only Workload
Flexible Deployment Modes
• RunC/Application: Nydusd + FsCache + EROFS
• RunC/Application: Nydusd + FUSE
• Kata: Nydusd + Virtio-fs
• Kata: Nydusd + virtio-fs + EROFS
• Kata-CC: Nydusd + virtio-blk + EROFS
• Kata-CC: Nydusd + FsCache + EROFS
• Application: Nydus userspace library
2
Nydus is EROFS Compatible
• Nydus filesystem format is EROFS compatible
• Linux in-kernel EROFS can mount Nydus images
• FsCache enables data cache and lazy-loading for EROFS

11. Runtime Data Deduplication
3
• Nydus supports chunk-based data
deduplication at build time, by
using a chunk dictionary.
• Nydus optionally supports local CAS,
to manages all chunks downloaded.
• When running a new image, all
chunks already exists in local CAS
will be reused.
It may dramatically reduce resource consumptions
in case of software upgrading/CVE fixing.

12. Integration with the Eco-system
4

13. What We Will Discuss Today
• What makes CoCo image management special?
• What is Nydus Image Service?
• Nydus Image Service for CoCo

14. Nydus Enhancements for CoCo
EK: Encryption Key
BEK: Blob Encryption Key
Manifest: contains wrapped EKs
Metadata Blob: contains BEKs
Data Blob: contains chunked file data
Encryption Integrity
Protected by
ocicrypt Protected by Nydus
Protected by Nydus
Protected by cosign

15. Lazy-Loading and Image Cache for CoCo
Image Sources
• Registry through network
• Image sharing with P2P
• Image cache on host through virtio-fs/blk
Working Modes
• Nydusd + FUSE
• Nydusd + FsCache + EROFS
• EROFS with decryption (researching)

16. Image Caching Modes
• Blob-Based Image Cache
• First lazy-loading without caching
• Then lazy-loading with blob cache
• At last image fs cache with EROFS
• Block-Based Image Cache
• Simple but inflexible
1
1
2
2
3
3

17. Development Plan
Nydus: targeting v2.3, 2023Q2
• Runtime data deduplication
• Data encryption
• ocicrypt for Nydus metadata blob
Useful Links:
• Nydus Image Service Project
• image-rs Project
• ocicrypt-rs Project
CoCo: targeting v0.5
• Improve quality of image-rs
• Integrate Nydus w/o encryption
• Integrate Nydus with encryption