I have two convex polygons in 3D. They are always flat and vertically aligned. There only in the shapes of squares, pentagons, and a 4-sided polygon where two sides are parallel and the other two sides are not parallel (yes this polygon is ALWAYS convex).
For instance two of my polygons may look like: (Z-axis is the vertical axis)
Polygon 1: { (0, 0, 0) (1, 1, 0) (1, 1, 1) (0, 0, 1) }
Polygon 2: { (0.5, 0.5, 0) (1.5, 1.5, 0) (1.5, 1.5, 1) (0.5, 0.5, 1) }
and my expected output:
{ (0.5, 0.5, 0), (1, 1, 0), ( 1, 1, 1), (0.5, 0.5, 1) }
Then I can calculate the area of the polygon which is what I am ultimately looking for.
I tried to find the vertices that lie inside each polygon and append that with the vertices of line intersections. Then I attempted to find the convex hull of the vertices that way my vertices are ordered and I can calculate the overlapping area. However, calculating the 3D convex hull is not trivial. I honestly don't even know if this is the right approach to the problem considering the data is in 3D.
First of all, the fact that it's 3D is making it look like a more difficult problem, so start by transforming it into the 2D problem it wants to be. Easiest approach here is to find the normal of the shared plane, find a coordinate of that normal which isn't zero. and throw out that coordinate from your input points. For instance, if all your polygons are on a plane with the normal (0.6, 0, 0.8), you can take each x,y,z and keep only the y,z, or keep only the x,y. Once you're done, you can recover the missing coordinate using the normal. So for the remainder, I'll treat this as a 2D problem in the XY plane.
Finding the intersection of two convex polygons is quite simple using a "rotating calipers" approach. Basically, you'll walk around each of the polygons in parallel, finding their (up to) two points of intersection.
Start by finding the X-minimal vertex of each polygon. This is the vertex that is extremal (farthest along) in the negative X query direction. Look at the polygon edge which is immediately counter-clockwise of each direction, and see if they intersect. If they do, write down that intersection for later. Next, see which one is clockwise of the other one (use the "perpendicular dot product" for that), and transition to the next vertex in that polygon. Then keep going, checking for an intersection between those two edges (one of them will be the edge from last time, and the other one will be a new edge) and again walking along one of the polygons to the next vertex.
Eventually you'll get back to the X-minimal vertex on both polygons. You will have found 0, 1, or 2 intersections. If there were 2, then your intersection polygon consists of those two points along with the parts of the polygons between them. If there were 0 or 1, either one of the polygons is inside the other or there's no intersection; check which situation it is by picking an arbitrary point on each polygon and testing if it's inside the other polygon.