Skip to main content


KubeDL Model tracks a model's version and lineage in Kubernetes CRD. It leverages container image to take versioning of a model. Each model version will generate a corresponding image that includes all the model artifacts.

In short, KubeDL training generates the KubeDL model and can then later by referenced by KubeDL Serving to serve the model directly.

KubeDL provides two CRDs Model and ModelVersion to accomplish this:


ModelVersion describes a model's version, its location, who creates this model, the image repo for pushing the model and so on. The backend controller watches this CRD and uses Kaniko to generate a image that incorporates all the artifacts. Check the full CRD spec.

An example YAML looks like below:

kind: ModelVersion
name: mv-3
namespace: default
# The model name for the model version
modelName: model1
# The entity (user or training job) that creates the model
createdBy: user1
# The image repo to push the generated model
imageRepo: modelhub/resnet
# The storage will be mounted at /kubedl-model inside the training container.
# Therefore, the training code should export the model at /kubedl-model path.
# The local storage to store the model
# The local host path to export the model
path: /foo
# The node where the chief worker run to export the model
nodeName: kind-control-plane

KubeDL mounts the storage such as local fs, nfs at /kubedl-model inside each training container by default. Therefore, the training code inside the container must export the model under /kubedl-model, so that it is also present on the external storage.

The ModelVersion controller will then trigger a Kaniko container mounted with the same storage and generate an image that includes the model artifacts at /kubedl-model path. KubeDL also automatically injects an ENV KUBEDL_MODEL_PATH=/kubedl-model into the generated model image.


Model is associated with multiple ModelVersions. It just aggregates some information about the ModelVersions in its status, such as the latest ModelVersion like below. Once the Model is deleted, all its ModelVersions are cascadingly deleted. Check the full CRD spec.

imageName: modelhub/resnet:v1c072
modelVersion: mv-3