Fit CVPixelBuffer into a square image with resizing and aspect ratio saving

I have an image that should be preprocessed before passing to CoreML to 640x640 square with resizing and saving aspect ratio (check the image). I found a lot of helpful links about resizing using vImageScale_*, but haven't found anything similar to adding coloured paddings to the resized image.

I know that Vision has the scaleFit option, but the final output is a bit different, so I'm trying to make image centered.

The vImageScale_* functions scale to fit the destination, so the aspect ratio will change if the source and destination are different ratios.

vImage provides affine transform operations (with support for a background colour!). Take a look at https://developer.apple.com/documentation/accelerate/applying_geometric_transforms_to_images for more information. The final example, Apply a Complex Affine Transform to a vImage Buffer, does exactly what you need (you just need to remove the rotate step).