Google Vertex AI Prediction: Why is TorchServe showing 0 GPUs?
I have deployed a trained PyTorch model to a Google Vertex AI Prediction endpoint. The endpoint is working fine, giving me predictions, but when I examine its logs in Logs Explorer, I see:
INFO 2023-01-11T10:34:53.270885171Z Number of GPUs: 0
INFO 2023-01-11T10:34:53.270888834Z Number of CPUs: 4
This is despite the fact that I set the endpoint to use NVIDIA_TESLA_T4 as the accelerator type:
Why does the log show 0 GPUs and does this mean TorchServe is not taking advantage of the accelerator GPU?
Solution
This is a common problem with PyTorch and CUDA. GPU support is only enabled when the right version of PyTorch is installed, i.e. one which compiles for CUDA. So it’s recommended that you use images which have PyTorch's CUDA capabilities.
- Torch installation results in not supported wheel on this platform
- How to write a raster plot?
- Understanding Unusual YOLO Label Formats and Their Impact on Training
- Pytorch 0.4.0: There are three ways to create tensors on CUDA device. Is there some difference between them?
- libtorch_cpu.so: undefined symbol: iJIT_IsProfilingActive
- Pytorch tensor to numpy array
- UserWarning: Applied workaround for CuDNN issue, install nvrtc.so
- How to make a dataloader with a directory of subfolders relevant to each class in Pytorch
- How to convert a tensor of booleans to ints in PyTorch?
- Training different stage of model with different loss
- torch.nn.functional vs torch.nn - Pytorch
- Why do we do batch matrix-matrix product?
- How does one set the pad token correctly (not to eos) during fine-tuning to avoid model not predicting EOS?
- Unable to update a latent vector using custom loss function in pytorch
- How to setup TF and Torch on one virtual environment with same CUDA
- detectron2 installation - No module named 'torch'
- pytorch s3fd pga_attack, problem in loss.backward() to get grad.data
- How is log_softmax() implemented to compute its value (and gradient) with better speed and numerical stability?
- torch matmul between 2D and 3D tensor
- Pytorch LSTM vs LSTMCell
- How to implement tf.gather_nd in Pytorch with the argument batch_dims?
- How to manually specify checkpoint path in PyTorchLightning
- Get paths of saved checkpoints from Pytorch-Lightning ModelCheckpoint
- How to combine slice assignment, mask assignment and broadcasting in PyTorch?
- Creating a train, test split for data nested in multiple folders
- load pytorch dataloader into GPU
- How exactly the forward and backward hooks work in PyTorch
- AttributeError: '_MultiProcessingDataLoaderIter' object has no attribute 'next'
- AMD ROCm with Pytorch on Navi10 (RX 5700 / RX 5700 XT)
- AttributeError : 'tuple' has no attribute 'to'