This is a problem in our lab work recently
So, would it be possible to do fourier transform on an image with holes? or could we map or decompose the concave image into rectangle(s)?
Two answers depending on exactly what you want to do:
1- Can you find the spatial domain equivalent of your magnitude modification, and apply a 2D convolution on the image but the holes?
2- Can you follow an iterative approach, in which you replace the "holes" with an average value, do fft, apply your magnitude transformation, do ifft, replace the holes with the original content, and repeat until you get sufficient convergence?