Job MetaData Persistency

Kubernetes api-server typically stores job information for a limited lifespan. KubeDL has built-in support to persist the job metadata into external storage to outlive api-server state. The KubeDL controller will persist the job metadata during the lifecycle of job such as job and pod creation/deletion.

Currently, only Mysql is supported.

DB Schema

Job Table

Column	Type	Description
id	int(64)	primary id auto incremented by underlying database
name	varchar(128)	name of job
namespace	varchar(128)	namespace of job
job_id	varchar(64)	job uid generated by kubernetes
version	varchar(32)	resource version generated by kubernetes(etcd)
status	varchar(32)	current observed job status(Created/Running/Failed/Succeed/Restarting)
kind	varchar(32)	kind of job: TFJob,PyTorchJob...
resources	text	job requested resources, including replicas and resources of each role.
deploy_region	varchar(64)	deploy_region indicates the physical region(IDC) this job located in, reserved for jobs running in across-region-clusters
tenant	varchar(255)	fields reserved for multi-tenancy job management scenarios, indicating which tenant this job belongs to and who's the owner(user)
owner	varchar(255)	owner of job
is_in_etcd	tinyint(4)	is_in_etcd indicates that whether record of this job has been removed from etcd
gmt_created	datetime	timestamp when job created
gmt_modified	datetime	timestamp when last job status transited
gmt_finished	datetime	timestamp when job failed or succeed

Pod Table

Column	Type	Description
id	int(64)	primary id auto incremented by underlying database
name	varchar(128)	name of pod
namespace	varchar(128)	namespace of pod
pod_id	varchar(64)	pod uid generated by kubernetes
version	varchar(32)	resource version generated by kubernetes(etcd)
status	varchar(32)	current observed pod phase(Pending/Running/Failed/Succeed/Unkown)
image	varchar(255)	image name of main conatiner
job_id	varchar(64)	job id of this pod controlled by
replica_type	varchar(32)	replica type of this pod belongs to
resources	varchar(1024)	resources this pod requested, marshaled from a ResourceRequirements object
host_ip	varchar(64)	ip of the host this pod scheduled
pod_ip	varchar(64)	ip of this pod allocated
deploy_region	varchar(64)	deploy_region indicates the physical region(IDC) this job located in
is_in_etcd	tinyint(4)	is_in_etcd indicates that whether record of this job has been removed from etcd
remark	text	extended messaged for pod
gmt_created	datetime	timestamp when pod created
gmt_modified	datetime	timestamp when last pod status transited
gmt_started	datetime	timestamp when main container stared
gmt_finished	datetime	timestamp when pod failed or succeed

How To Use

Below is an example to setup KubeDL to use Mysql as the persistency DB.

1. Set up credentials for KubeDL to connect to DB. Create a Secret object like below:

apiVersion: v1
kind: Secret
metadata:
  name: kubedl-mysql-config
  namespace: kubedl-system
type: Opaque
stringData:
  host: my.host.com
  dbName: kubedl
  user: kubedl-user
  password: this-is-me
  port: "3306"

2. Update the Kubedl Deployment spec to include --meta-storage mysql in the startup flag and reference the DB credentials via environment variables. The KubeDL controller uses the env to set up connection with DB.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kubedl
  namespace: kubedl-system
  labels:
    app: kubedl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kubedl
  template:
    metadata:
      labels:
        app: kubedl
    spec:
      containers:
      - image: kubedl/kubedl:v0.3.0
        imagePullPolicy: Always
        name: kubedl-manager
        args:
        - "--meta-storage"
        - "mysql"
        env:
        - name: MYSQL_HOST
          value:
          valueFrom:
            secretKeyRef:
              name: kubedl-mysql-config
              key: host
        - name: MYSQL_DB_NAME
          value:
          valueFrom:
            secretKeyRef:
              name: kubedl-mysql-config
              key: dbName
        - name: MYSQL_USER
          value:
          valueFrom:
            secretKeyRef:
              name: kubedl-mysql-config
              key: user
        - name: MYSQL_PASSWORD
          value:
          valueFrom:
            secretKeyRef:
              name: kubedl-mysql-config
              key: password

MySql Config

The configs defined in the aforementioned secret:

Config Name	Description
host	Mysql host name
dbName	DB name
user	User name
password	User password
port	The mysql DB port to connect to

Contributions

Currently, only mysql is supported. You are welcome to contribute your own storage plugin.