I would like to improve the feature matching between two images by using a pre-trained semantic segmentation model. Of course I am familiar with the ORB or SIFT features and matching with OpenCV in python, but I am curious if it is possible to further improve that by using a precise semantic segmentation model. The purpose of this is I would like to calculate the epipolar geometry between 2 images. Does it make any sense to do that? Is there any algorithm that does that? I could not find anything on the internet.
You don't have to go all the way to hard-decisions on semantic boundaries to enrich your correspondences with semantic information. You can replace SIFT/ORB, which are low-level features, with semantic descriptors, such as DINO-ViT features. Check out this project page and see how these features can be used for establishing semantic correspondences between images.