Skip to main content

Run in Host Network

Background​

Network bandwidth is a bottleneck resource for communication-intensive jobs. Host mode networking can be useful to optimize performance. In addition, other scenarios (e.g: nvlink communications between containerized gpu processes) may depend on host network as well.

How To Use​

KubeDL provides a feature-gate to enable hostnetwork mode for jobs. Users only need to add an annotation kubedl.io/network-mode: host to the job specifications, for example:

    apiVersion: "training.kubedl.io/v1alpha1"
    kind: "TFJob"
    metadata:
      name: "mnist"
      namespace: kubedl
      annotations:
 +      kubedl.io/network-mode: 'host'
    spec:
      cleanPodPolicy: None
      tfReplicaSpecs:
        Worker:
          replicas: 3
          ...

Design​

The essence of hostnetwork-mode is to randomize container ports to avoid port collision and enable service discovery across workers. KubeDL achieves by following steps:

  1. Enable hostnetwork in Pod spec and set DNS policy as ClusterFirstWithHostNet;
  2. Choose a random port as container port.
  3. Change TargetPort of corresponding worker's Service to the previous randomized port, and set CluterIP as empty string(instead of None), so that kube-proxy will be able to forward traffic from Port to TargetPort.
  4. Change the job cluster spec (e.g. the TF_CONFIG) .
  5. Handle worker fail-over and use latest available port as the TargetPort in the new worker.

Here is a Tensorflow job example:

hostnetwork