I am working on an image editor project, where you can rotate and crop an image.
The problem I'm currently facing is once I've rotated the image, I'd like to be able to drag the crop box anywhere inside the boundaries of the rotated image. So far I've been looking at the Liang-Barsky and Cohen-Sutherland line-clipping algorithms, and Separating Axis Theorem, but I'm struggling to see how I can implement these for my use case.
Can anyone point me in the right direction? Am I barking up the wrong tree?
Use the comment by Alain. To check insideness of the corners, it suffices to counter-rotate the image to make its edges axis-aligned. Then you have an easy point-in-axis-aligned-box problem.
(I don't mean that you really have to rotate the image, just the geometry.)