DeeplabV3+ without pretrained backbone, will it the reason for bad DSC?

I segment multiple targets in medical image (CT) with DeeplabV3+, but with 3D volumes, so I can't load pretrained backbone(resnet...etc.) in the net.

And the details is:

1. patch size: 16, 256, 256(cannot edit)
2. batch size: 2(cause' GPU cannot afford the bigger one)
3. optimizer: SGD
4. loss: Dice+CrossEntropy(refer to nnUNet setting)
5. dataset: just about 20 cases.
6. the original code is for 2D situation, and I exchange each layer from 2D to 3D(like nn.Conv2d TO nn.Conv3d and something)

But finally, My validation DSC just reached 0.6 around, I have no idea what's wrong in my code? Could anyone give me a hand(idea), please? Thanks a lot!

Increase the performance of the model, because now I don't have any idea why my network is so bad. Thanks a lot.

You can try to use a few 3x3 convolutional layers on 3D volumes of images keeping dimensions (h and w) of the features constant and then convert such tensor to 3 channel tensor using 1x1 convolutional layer. Now you will have a tensor of same height and width of the image with 3 channels and you can use the pretrained models.

For reference, check here: https://segmentation-models.readthedocs.io/en/latest/tutorial.html#training-with-non-rgb-data