Metadata Persistency

Kubernetes api-server typically stores job information for a limited lifespan. KubeDL has built-in support to persist the job metadata into external storage to outlive api-server state. The KubeDL controller will persist the job metadata during the lifecycle of job such as job and pod creation/deletion.

Currently, only Mysql is supported.

DB Schema

Job Table

ColumnTypeDescription
idint(64)primary id auto incremented by underlying database
namevarchar(128)name of job
namespacevarchar(128)namespace of job
job_idvarchar(64)job uid generated by kubernetes
versionvarchar(32)resource version generated by kubernetes(etcd)
statusvarchar(32)current observed job status(Created/Running/Failed/Succeed/Restarting)
kindvarchar(32)kind of job: TFJob,PyTorchJob…
resourcestextjob requested resources, including replicas and resources of each role.
deploy_regionvarchar(64)deploy_region indicates the physical region(IDC) this job located in, reserved for jobs running in across-region-clusters
tenantvarchar(255)fields reserved for multi-tenancy job management scenarios, indicating which tenant this job belongs to and who’s the owner(user)
ownervarchar(255)owner of job
is_in_etcdtinyint(4)is_in_etcd indicates that whether record of this job has been removed from etcd
gmt_createddatetimetimestamp when job created
gmt_modifieddatetimetimestamp when last job status transited
gmt_finisheddatetimetimestamp when job failed or succeed

Pod Table

ColumnTypeDescription
idint(64)primary id auto incremented by underlying database
namevarchar(128)name of pod
namespacevarchar(128)namespace of pod
pod_idvarchar(64)pod uid generated by kubernetes
versionvarchar(32)resource version generated by kubernetes(etcd)
statusvarchar(32)current observed pod phase(Pending/Running/Failed/Succeed/Unkown)
imagevarchar(255)image name of main conatiner
job_idvarchar(64)job id of this pod controlled by
replica_typevarchar(32)replica type of this pod belongs to
resourcesvarchar(1024)resources this pod requested, marshaled from a ResourceRequirements object
host_ipvarchar(64)ip of the host this pod scheduled
pod_ipvarchar(64)ip of this pod allocated
deploy_regionvarchar(64)deploy_region indicates the physical region(IDC) this job located in
is_in_etcdtinyint(4)is_in_etcd indicates that whether record of this job has been removed from etcd
remarktextextended messaged for pod
gmt_createddatetimetimestamp when pod created
gmt_modifieddatetimetimestamp when last pod status transited
gmt_starteddatetimetimestamp when main container stared
gmt_finisheddatetimetimestamp when pod failed or succeed

How To Use

Below is an example to setup KubeDL to use Mysql as the persistency DB.

  1. Set up credentials for KubeDL to connect to DB. Create a Secret object like below:
apiVersion: v1
kind: Secret
metadata:
  name: kubedl-mysql-config
  namespace: kubedl-system
type: Opaque
stringData:
  host: my.host.com
  dbName: kubedl
  user: kubedl-user
  password: this-is-me
  port: "3306"
  1. Update the Kubedl Deployment spec to include --meta-storage mysql in the startup flag and reference the DB credentials via environment variables. The KubeDL controller uses the env to set up connection with DB.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kubedl
  namespace: kubedl-system
  labels:
    app: kubedl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kubedl
  template:
    metadata:
      labels:
        app: kubedl
    spec:
      containers:
      - image: kubedl/kubedl:v0.3.0
        imagePullPolicy: Always
        name: kubedl-manager
        args:
        - "--meta-storage"
        - "mysql"
        env:
        - name: MYSQL_HOST
          value:
          valueFrom:
            secretKeyRef:
              name: kubedl-mysql-config
              key: host
        - name: MYSQL_DB_NAME
          value:
          valueFrom:
            secretKeyRef:
              name: kubedl-mysql-config
              key: dbName
        - name: MYSQL_USER
          value:
          valueFrom:
            secretKeyRef:
              name: kubedl-mysql-config
              key: user
        - name: MYSQL_PASSWORD
          value:
          valueFrom:
            secretKeyRef:
              name: kubedl-mysql-config
              key: password

MySql Config

The configs defined in the aforementioned secret:

Config NameDescription
hostMysql host name
dbNameDB name
userUser name
passwordUser password
portThe mysql DB port to connect to

Contributions

Currently, only mysql is supported. You are welcome to contribute your own storage plugin.