Many ML pipelines depend on shared filesystems for input, output and intermediate data storage. Standards such as CSI have made it possible for applications in Kubernetes to access a variety of data storage systems. Yet, data scientists still have to deal with low-level details of data access in order to execute their pipelines in Kubernetes. Datashim is a framework that manages the lifecycle of a Dataset object, a CustomResourceDefinition that represents a source of data. Datashim takes care of the details of data access while Kubernetes pods can declaratively access the data by referencing a Dataset in their specifications. This talk will describe Datashim and the Dataset object, discuss its use in ML pipelines, and demonstrate how its pluggable architecture is designed for the development of caching, scheduling and governance plugins. Datashim is an incubating project of the Linux Foundation Data and AI Foundation. This talk was given by Srikumar Venugopal for DoK Day Europe @ KubeCon 2022.

1. Srikumar Venugopal
DoK Day Europe 2022 @ KubeCon
Datashim - a framework for declarative
management of datasets on Kubernetes

2. DoK Day Europe 2022 @ KubeCon
Data Science on Kubernetes

3. Introducing Datashim
Cloud-native data access abstraction
Open-Source (LF Data and AI Foundation Incubation): https://datashim.io
DoK Day Europe 2022 @ KubeCon

4. Operational Flow
DoK Day Europe 2022 @ KubeCon

5. Kubeflow Pipeline Example
DoK Day Europe 2022 @ KubeCon
kind: Dataset
metadata:
name: “my-dataset”
spec:
local:
type: “COS”
accessKeyID: ...
secretAccessKey: ...
import kfp
import kfp.dsl as dsl
from kfp.dsl import PipelineVolume
...
def volume_op_dag():
dataset = PipelineVolume(”my-dataset")
step1 = dsl.ContainerOp(
name="step1",
image="library/bash:4.4.23",
command=["sh", "-c"],
arguments=["echo 1|tee /data/file1"],
pvolumes={"/data": dataset}
)
step2 = dsl.ContainerOp(
name="step2",
image="library/bash:4.4.23",
command=["sh", "-c"],
arguments=["cp /data/file1 /data/file2"],
pvolumes={"/data": step1.pvolume}
)
...
PVC: my-dataset
Example from: https://github.com/datashim-io/datashim/wiki/PVCs-for-Pipelines-SDK

6. DoK Day Europe 2022 @ KubeCon
human reference genome
g1k_queries
g1k_genomes
FTP
S3
PVC
PVC
PVC
PVC
PVC
PVC S3
results
Pipeline Simplification
human reference genome
g1k_genomes
DS
DS
results
Samtools
Sidecar
Samtools
DS
DS
DS
DS
DS
Y. Gkoufas, D.Y. Yuan, C.Pinto, P. Koutsovasilis, S. Venugopal,
"Datashim and Its Applications in Bioinformatics", Proceedings
of International Conference on High Performance Computing
PVC – Persistent Volume Claim
DS – Datashim Dataset

7. Declarative Caching
DoK Day Europe 2022 @ KubeCon
P. Koutsovasilis, S. Venugopal, Y. Gkoufas and C. Pinto, "A Holistic Approach to
Data Access for Cloud-Native Analytics and Machine Learning," in 2021 IEEE
14th International Conference on Cloud Computing (CLOUD)

8. Roadmap
Ephemeral volume support for S3
Integration with COSI (when finalised)
Auto-discovery of CSI implementation capabilities
Support for more frameworks (Tekton, Flyte)
Focus on observability (Design phase)
DoK Day Europe 2022 @ KubeCon

9. Acknowledgments
Yiannis Gkoufas
Christian Pinto
Panagiotis (Panos) Koutsovasilis
and many other contributors
DoK Day Europe 2022 @ KubeCon