Kubernetes GPU pods /Kubeflow

Question

I have 2 servers with 4 tesla K40 GPUs.

I have successfully created a kubernetes cluster installed kubeflow 1.0 and evertything needed for it to work fine.

I can successfully create a Jupyter notebook server with 4 GPUs and use keras multigpu model on it and everything works fine.

Can I use 8 gpu (4 from one server and 4 from another) to create jupyter notebook server or run any gpupod or only 4 for one gpupod is avaliable for me?

When I try use 8 GPUs I get 0/2 nodes are available: 2 Insufficient nvidia.com/gpu

Answer 1

Can I use 8 gpu (4 from one server and 4 from another) to create jupyter notebook server or run any gpupod or only 4 for one gpupod is avaliable for me?

No, Kubernetes pods can only use resources from one machine(node) at a time.

You can learn more here.

---

As a workaround, you can run your jupyter server on one cluster, but train on GPUs from multiple clusters with some minimal code changes, depending on which framework you're working.

kubeflow even has some nice UI support for that. Here are some sources to learn more about:

• Clustering multiple GPUs from different machines and remotely run Jupyter Notebook #2720
• Using IPython for parallel computing
• How-To: Multi-GPU training with Keras, Python, and deep learning

Answer 2

A kubeflow jupyter notebook is a pod, and a simple training job can scale up to the number of free GPUs you have in one GPU node of your kubernetes cluster.

To scale out your training horizontally and utilize more GPUs in your training job, you need to use a distributed training framework. Kubeflow provides tfjob for that purpose.