Mounting a NFS volume by a OpenShift 3.11 PersistentVolume: mount.nfs: mounting failed, reason given by server: No such file or directory

In our OpenShift 3.11 cluster, we are trying to use NFS through a PersistentVolume and a NFS volume previously created on a external NFS storage (a Isilon Storage). We created and applied succesfully the PersistentVolume and the PersistentVolumeClaim on the Kubernetes/OpenShift Layer. The PVC binds the PV correctly, but when checking the Deployment events we face an error in the mounting NFS phase.

PersistentVolume:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: tool1pv
spec:
  capacity:
    storage: 100Gi 
  accessModes:
  - ReadWriteOnce 
  nfs: 
    path: /tool1shareenv1
    server: tommytheserver.companydomain.priv
  persistentVolumeReclaimPolicy: Retain
  claimRef:
    name: tool1claimenv1
    namespace: ocpnamespace1

PersistentVolumeClaim:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: tool1claimenv1
spec:
 accessModes:
   - ReadWriteOnce
 resources:
  requests:
    storage: 100Gi
 volumeName: tool1pvenv1 

When checking the Development Events, we see a "No such file":

MountVolume.SetUp failed for volume "tool1pvenv1" : mount failed:
 exit status 32 Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient mount for /var/lib/origin/openshift.local.volumes/pods/f1cb1291-fe12-01ea-bb92-0050123aa39be/volumes/kubernetes.io~nfs/tool1pvenv1 --scope -- mount -t nfs tommytheserver.companydomain.priv:/tool1shareenv1
/var/lib/origin/openshift.local.volumes/pods/f1cb9191-fe73-11ea-bb92-005056ba12be/volumes/kubernetes.io~nfs/tool1pvenv1d Output: Running scope as unit run-74039.scope. **mount.nfs: mounting tommytheserver.companydomain.priv:/tool1env1 failed, reason given by server: No such file or directory**

We investigated the server and the path fields and tried different varations such as:

PersistentVolumeVersion2:

 apiVersion: v1
kind: PersistentVolume
metadata:
  name: tool1pv
spec:
  capacity:
    storage: 100Gi 
  accessModes:
  - ReadWriteOnce 
  nfs: 
    path: /tool1shareenv1
    server: tommytheserver.companydomain.priv/tool1shareenv1
  persistentVolumeReclaimPolicy: Retain
  claimRef:
    name: tool1claimenv1
    namespace: ocpnamespace1

but we're still faceing the same No such file error.

How can we troubleshoot it ?

Normally to troubleshoot something like this I would...

1. Double check that your share path actually exists
2. Get the IP address of your node where your pod ran and ssh onto it. You can get the IP like this:

kubectl get pod <podname> -o wide -n namespace

Then I would make sure I can connect to the nfs server where the share exists:

telnet <nfs server> port

3. Following that I would run dmesg to see mounting related errors
4. I would try to mount the volume myself using the same arguments your error is showing. ie-

mount -t nfs tommytheserver.companydomain.priv:/tool1shareenv1

It is difficult to provide a specific answer without seeing the results of these troubleshooting steps. But, that is the approach I would take.