We need to train large networks using TensorFlow that take several days to complete on a GPU. Amazon offers GPU instances like p2.16xlarge
with e.g. 16 GPUs (NVIDIA K80). Now I was wondering if TensorFlow utilizes multiple GPUs efficiently or would I be just as fast using a desktop with a single Titan X?
Those GPU instances are quite expensive so I'm looking to build a machine myself (Linux based). SLI doesn't seem to work with CUDA so am I stuck with 1 GPU at a time?