I am trying to make a prototype for an algorithm to find the coordinates of a ball on a plate, and I want to make it as efficient as possible because I have to implement it in an FPGA. The pictures of the ball and plate are not always in the same orientation, so I need to shift the coordinates of the center of the ball, based on the coordinates of the corners of the plate.
To get an understanding what I mean, see the image below where the white sheet represents the plate.
Now I have already found a way to determine the coordinates of the ball on the picture, and the coordinates of the corners of the plate in the picture, but I want to find out where the ball is located on the plate.
I tried to do some things with getPerspectiveTransform() and wrapPerspective() and it worked, but this involves a lot of matrix computations and I think this is a bit overkill when I just want to shift the coordinates of one pixel (the center of the ball).
Do you know a more efficient way on how I can determine the coordinates of the center pixel of the ball on the plate?
I solved it by using the answer of morotspaj of the post below and Matlab.
Calculate a 2D homogeneous perspective transformation matrix from 4 points in MATLAB
ui and vi are already known (these equal the resolution of the picture), so I filled them into the 8x8 matrix and got the following:
/ x0 y0 1 0 0 0 0 0 \ /m00\ / 0 \
| x1 y1 1 0 0 0 -x1*RES_H -y1*RES_H | |m01| |RES_H|
| x2 y2 1 0 0 0 0 0 | |m02| | 0 |
| x3 y3 1 0 0 0 -x3*RES_H -y3*RES_H |.|m10|=|RES_H|
| 0 0 0 x0 y0 1 0 0 | |m11| | 0 |
| 0 0 0 x1 y1 1 0 0 | |m12| | 0 |
| 0 0 0 x2 y2 1 -x2*RES_V -y2*RES_V | |m20| |RES_V|
\ 0 0 0 x3 y3 1 -x3*RES_V -y3*RES_V / \m21/ \RES_V/
Where RES_H = (640 - 1) and RES_V = (480 - 1). This equation can be seen as Ax=b. I putted the A matrix (8x8) and the b vector (8x1) in Matlab and used linsolve(A,b) to solve the linear system:
syms x0 x1 x2 x3 y0 y1 y2 y3 RES_H RES_V;
A = [ x0 y0 1 0 0 0 0 0 ; x1 y1 1 0 0 0 -x1*RES_H -y1*RES_H ; x2 y2 1 0 0 0 0 0 ; x3 y3 1 0 0 0 -x3*RES_H -y3*RES_H ; 0 0 0 x0 y0 1 0 0 ; 0 0 0 x1 y1 1 0 0 ; 0 0 0 x2 y2 1 -x2*RES_V -y2*RES_V ; 0 0 0 x3 y3 1 -x3*RES_V -y3*RES_V ];
b = [ 0 ; RES_H ; 0 ; RES_H ; 0 ; 0 ; RES_V ; RES_V ];
simplify(linsolve(A,B))
The result was the following:
m00 = -(RES_H*(y0 - y2)*(x0*y1 - x1*y0 - x0*y3 + x3*y0 + x1*y3 - x3*y1)*(x1*y2 - x2*y1 - x1*y3 + x3*y1 + x2*y3 - x3*y2))/(- x0*x0*x1*y1*y2*y2 + 2*x0*x0*x1*y1*y2*y3 - x0*x0*x1*y1*y3*y3 + x0*x0*x2*y1*y1*y2 - 2*x0*x0*x2*y1*y2*y3 + x0*x0*x2*y2*y3*y3 - 2*x0*x0*x3*y1*y1*y2 + x0*x0*x3*y1*y1*y3 + 2*x0*x0*x3*y1*y2*y2 - x0*x0*x3*y2*y2*y3 + x0*x1*x1*y0*y2*y2 - 2*x0*x1*x1*y0*y2*y3 + x0*x1*x1*y0*y3*y3 - 2*x0*x1*x2*y0*y1*y3 + 2*x0*x1*x2*y0*y2*y3 + 2*x0*x1*x2*y1*y3*y3 - 2*x0*x1*x2*y2*y3*y3 + 2*x0*x1*x3*y0*y1*y2 - 2*x0*x1*x3*y0*y2*y2 - 2*x0*x1*x3*y1*y2*y3 + 2*x0*x1*x3*y2*y2*y3 - x0*x2*x2*y0*y1*y1 + 2*x0*x2*x2*y0*y1*y3 - x0*x2*x2*y0*y3*y3 + 2*x0*x2*x3*y0*y1*y1 - 2*x0*x2*x3*y0*y1*y2 - 2*x0*x2*x3*y1*y1*y3 + 2*x0*x2*x3*y1*y2*y3 - x0*x3*x3*y0*y1*y1 + x0*x3*x3*y0*y2*y2 + 2*x0*x3*x3*y1*y1*y2 - 2*x0*x3*x3*y1*y2*y2 - x1*x1*x2*y0*y0*y2 + 2*x1*x1*x2*y0*y0*y3 - 2*x1*x1*x2*y0*y3*y3 + x1*x1*x2*y2*y3*y3 - x1*x1*x3*y0*y0*y3 + 2*x1*x1*x3*y0*y2*y3 - x1*x1*x3*y2*y2*y3 + x1*x2*x2*y0*y0*y1 - 2*x1*x2*x2*y0*y0*y3 + 2*x1*x2*x2*y0*y3*y3 - x1*x2*x2*y1*y3*y3 - 2*x1*x2*x3*y0*y0*y1 + 2*x1*x2*x3*y0*y0*y2 + 2*x1*x2*x3*y0*y1*y3 - 2*x1*x2*x3*y0*y2*y3 + x1*x3*x3*y0*y0*y1 - 2*x1*x3*x3*y0*y1*y2 + x1*x3*x3*y1*y2*y2 + x2*x2*x3*y0*y0*y3 - 2*x2*x2*x3*y0*y1*y3 + x2*x2*x3*y1*y1*y3 - x2*x3*x3*y0*y0*y2 + 2*x2*x3*x3*y0*y1*y2 - x2*x3*x3*y1*y1*y2);
m01 = (RES_H*(x0 - x2)*(x0*y1 - x1*y0 - x0*y3 + x3*y0 + x1*y3 - x3*y1)*(x1*y2 - x2*y1 - x1*y3 + x3*y1 + x2*y3 - x3*y2))/(- x0*x0*x1*y1*y2*y2 + 2*x0*x0*x1*y1*y2*y3 - x0*x0*x1*y1*y3*y3 + x0*x0*x2*y1*y1*y2 - 2*x0*x0*x2*y1*y2*y3 + x0*x0*x2*y2*y3*y3 - 2*x0*x0*x3*y1*y1*y2 + x0*x0*x3*y1*y1*y3 + 2*x0*x0*x3*y1*y2*y2 - x0*x0*x3*y2*y2*y3 + x0*x1*x1*y0*y2*y2 - 2*x0*x1*x1*y0*y2*y3 + x0*x1*x1*y0*y3*y3 - 2*x0*x1*x2*y0*y1*y3 + 2*x0*x1*x2*y0*y2*y3 + 2*x0*x1*x2*y1*y3*y3 - 2*x0*x1*x2*y2*y3*y3 + 2*x0*x1*x3*y0*y1*y2 - 2*x0*x1*x3*y0*y2*y2 - 2*x0*x1*x3*y1*y2*y3 + 2*x0*x1*x3*y2*y2*y3 - x0*x2*x2*y0*y1*y1 + 2*x0*x2*x2*y0*y1*y3 - x0*x2*x2*y0*y3*y3 + 2*x0*x2*x3*y0*y1*y1 - 2*x0*x2*x3*y0*y1*y2 - 2*x0*x2*x3*y1*y1*y3 + 2*x0*x2*x3*y1*y2*y3 - x0*x3*x3*y0*y1*y1 + x0*x3*x3*y0*y2*y2 + 2*x0*x3*x3*y1*y1*y2 - 2*x0*x3*x3*y1*y2*y2 - x1*x1*x2*y0*y0*y2 + 2*x1*x1*x2*y0*y0*y3 - 2*x1*x1*x2*y0*y3*y3 + x1*x1*x2*y2*y3*y3 - x1*x1*x3*y0*y0*y3 + 2*x1*x1*x3*y0*y2*y3 - x1*x1*x3*y2*y2*y3 + x1*x2*x2*y0*y0*y1 - 2*x1*x2*x2*y0*y0*y3 + 2*x1*x2*x2*y0*y3*y3 - x1*x2*x2*y1*y3*y3 - 2*x1*x2*x3*y0*y0*y1 + 2*x1*x2*x3*y0*y0*y2 + 2*x1*x2*x3*y0*y1*y3 - 2*x1*x2*x3*y0*y2*y3 + x1*x3*x3*y0*y0*y1 - 2*x1*x3*x3*y0*y1*y2 + x1*x3*x3*y1*y2*y2 + x2*x2*x3*y0*y0*y3 - 2*x2*x2*x3*y0*y1*y3 + x2*x2*x3*y1*y1*y3 - x2*x3*x3*y0*y0*y2 + 2*x2*x3*x3*y0*y1*y2 - x2*x3*x3*y1*y1*y2);
m02 = -(RES_H*(x0*y2 - x2*y0)*(x0*y1 - x1*y0 - x0*y3 + x3*y0 + x1*y3 - x3*y1)*(x1*y2 - x2*y1 - x1*y3 + x3*y1 + x2*y3 - x3*y2))/(- x0*x0*x1*y1*y2*y2 + 2*x0*x0*x1*y1*y2*y3 - x0*x0*x1*y1*y3*y3 + x0*x0*x2*y1*y1*y2 - 2*x0*x0*x2*y1*y2*y3 + x0*x0*x2*y2*y3*y3 - 2*x0*x0*x3*y1*y1*y2 + x0*x0*x3*y1*y1*y3 + 2*x0*x0*x3*y1*y2*y2 - x0*x0*x3*y2*y2*y3 + x0*x1*x1*y0*y2*y2 - 2*x0*x1*x1*y0*y2*y3 + x0*x1*x1*y0*y3*y3 - 2*x0*x1*x2*y0*y1*y3 + 2*x0*x1*x2*y0*y2*y3 + 2*x0*x1*x2*y1*y3*y3 - 2*x0*x1*x2*y2*y3*y3 + 2*x0*x1*x3*y0*y1*y2 - 2*x0*x1*x3*y0*y2*y2 - 2*x0*x1*x3*y1*y2*y3 + 2*x0*x1*x3*y2*y2*y3 - x0*x2*x2*y0*y1*y1 + 2*x0*x2*x2*y0*y1*y3 - x0*x2*x2*y0*y3*y3 + 2*x0*x2*x3*y0*y1*y1 - 2*x0*x2*x3*y0*y1*y2 - 2*x0*x2*x3*y1*y1*y3 + 2*x0*x2*x3*y1*y2*y3 - x0*x3*x3*y0*y1*y1 + x0*x3*x3*y0*y2*y2 + 2*x0*x3*x3*y1*y1*y2 - 2*x0*x3*x3*y1*y2*y2 - x1*x1*x2*y0*y0*y2 + 2*x1*x1*x2*y0*y0*y3 - 2*x1*x1*x2*y0*y3*y3 + x1*x1*x2*y2*y3*y3 - x1*x1*x3*y0*y0*y3 + 2*x1*x1*x3*y0*y2*y3 - x1*x1*x3*y2*y2*y3 + x1*x2*x2*y0*y0*y1 - 2*x1*x2*x2*y0*y0*y3 + 2*x1*x2*x2*y0*y3*y3 - x1*x2*x2*y1*y3*y3 - 2*x1*x2*x3*y0*y0*y1 + 2*x1*x2*x3*y0*y0*y2 + 2*x1*x2*x3*y0*y1*y3 - 2*x1*x2*x3*y0*y2*y3 + x1*x3*x3*y0*y0*y1 - 2*x1*x3*x3*y0*y1*y2 + x1*x3*x3*y1*y2*y2 + x2*x2*x3*y0*y0*y3 - 2*x2*x2*x3*y0*y1*y3 + x2*x2*x3*y1*y1*y3 - x2*x3*x3*y0*y0*y2 + 2*x2*x3*x3*y0*y1*y2 - x2*x3*x3*y1*y1*y2);
m10 = -(RES_V*(y0 - y1)*(x0*y2 - x2*y0 - x0*y3 + x3*y0 + x2*y3 - x3*y2)*(x1*y2 - x2*y1 - x1*y3 + x3*y1 + x2*y3 - x3*y2))/(- x0*x0*x1*y1*y2*y2 + 2*x0*x0*x1*y1*y2*y3 - x0*x0*x1*y1*y3*y3 + x0*x0*x2*y1*y1*y2 - 2*x0*x0*x2*y1*y2*y3 + x0*x0*x2*y2*y3*y3 - 2*x0*x0*x3*y1*y1*y2 + x0*x0*x3*y1*y1*y3 + 2*x0*x0*x3*y1*y2*y2 - x0*x0*x3*y2*y2*y3 + x0*x1*x1*y0*y2*y2 - 2*x0*x1*x1*y0*y2*y3 + x0*x1*x1*y0*y3*y3 - 2*x0*x1*x2*y0*y1*y3 + 2*x0*x1*x2*y0*y2*y3 + 2*x0*x1*x2*y1*y3*y3 - 2*x0*x1*x2*y2*y3*y3 + 2*x0*x1*x3*y0*y1*y2 - 2*x0*x1*x3*y0*y2*y2 - 2*x0*x1*x3*y1*y2*y3 + 2*x0*x1*x3*y2*y2*y3 - x0*x2*x2*y0*y1*y1 + 2*x0*x2*x2*y0*y1*y3 - x0*x2*x2*y0*y3*y3 + 2*x0*x2*x3*y0*y1*y1 - 2*x0*x2*x3*y0*y1*y2 - 2*x0*x2*x3*y1*y1*y3 + 2*x0*x2*x3*y1*y2*y3 - x0*x3*x3*y0*y1*y1 + x0*x3*x3*y0*y2*y2 + 2*x0*x3*x3*y1*y1*y2 - 2*x0*x3*x3*y1*y2*y2 - x1*x1*x2*y0*y0*y2 + 2*x1*x1*x2*y0*y0*y3 - 2*x1*x1*x2*y0*y3*y3 + x1*x1*x2*y2*y3*y3 - x1*x1*x3*y0*y0*y3 + 2*x1*x1*x3*y0*y2*y3 - x1*x1*x3*y2*y2*y3 + x1*x2*x2*y0*y0*y1 - 2*x1*x2*x2*y0*y0*y3 + 2*x1*x2*x2*y0*y3*y3 - x1*x2*x2*y1*y3*y3 - 2*x1*x2*x3*y0*y0*y1 + 2*x1*x2*x3*y0*y0*y2 + 2*x1*x2*x3*y0*y1*y3 - 2*x1*x2*x3*y0*y2*y3 + x1*x3*x3*y0*y0*y1 - 2*x1*x3*x3*y0*y1*y2 + x1*x3*x3*y1*y2*y2 + x2*x2*x3*y0*y0*y3 - 2*x2*x2*x3*y0*y1*y3 + x2*x2*x3*y1*y1*y3 - x2*x3*x3*y0*y0*y2 + 2*x2*x3*x3*y0*y1*y2 - x2*x3*x3*y1*y1*y2);
m11 = (RES_V*(x0 - x1)*(x0*y2 - x2*y0 - x0*y3 + x3*y0 + x2*y3 - x3*y2)*(x1*y2 - x2*y1 - x1*y3 + x3*y1 + x2*y3 - x3*y2))/(- x0*x0*x1*y1*y2*y2 + 2*x0*x0*x1*y1*y2*y3 - x0*x0*x1*y1*y3*y3 + x0*x0*x2*y1*y1*y2 - 2*x0*x0*x2*y1*y2*y3 + x0*x0*x2*y2*y3*y3 - 2*x0*x0*x3*y1*y1*y2 + x0*x0*x3*y1*y1*y3 + 2*x0*x0*x3*y1*y2*y2 - x0*x0*x3*y2*y2*y3 + x0*x1*x1*y0*y2*y2 - 2*x0*x1*x1*y0*y2*y3 + x0*x1*x1*y0*y3*y3 - 2*x0*x1*x2*y0*y1*y3 + 2*x0*x1*x2*y0*y2*y3 + 2*x0*x1*x2*y1*y3*y3 - 2*x0*x1*x2*y2*y3*y3 + 2*x0*x1*x3*y0*y1*y2 - 2*x0*x1*x3*y0*y2*y2 - 2*x0*x1*x3*y1*y2*y3 + 2*x0*x1*x3*y2*y2*y3 - x0*x2*x2*y0*y1*y1 + 2*x0*x2*x2*y0*y1*y3 - x0*x2*x2*y0*y3*y3 + 2*x0*x2*x3*y0*y1*y1 - 2*x0*x2*x3*y0*y1*y2 - 2*x0*x2*x3*y1*y1*y3 + 2*x0*x2*x3*y1*y2*y3 - x0*x3*x3*y0*y1*y1 + x0*x3*x3*y0*y2*y2 + 2*x0*x3*x3*y1*y1*y2 - 2*x0*x3*x3*y1*y2*y2 - x1*x1*x2*y0*y0*y2 + 2*x1*x1*x2*y0*y0*y3 - 2*x1*x1*x2*y0*y3*y3 + x1*x1*x2*y2*y3*y3 - x1*x1*x3*y0*y0*y3 + 2*x1*x1*x3*y0*y2*y3 - x1*x1*x3*y2*y2*y3 + x1*x2*x2*y0*y0*y1 - 2*x1*x2*x2*y0*y0*y3 + 2*x1*x2*x2*y0*y3*y3 - x1*x2*x2*y1*y3*y3 - 2*x1*x2*x3*y0*y0*y1 + 2*x1*x2*x3*y0*y0*y2 + 2*x1*x2*x3*y0*y1*y3 - 2*x1*x2*x3*y0*y2*y3 + x1*x3*x3*y0*y0*y1 - 2*x1*x3*x3*y0*y1*y2 + x1*x3*x3*y1*y2*y2 + x2*x2*x3*y0*y0*y3 - 2*x2*x2*x3*y0*y1*y3 + x2*x2*x3*y1*y1*y3 - x2*x3*x3*y0*y0*y2 + 2*x2*x3*x3*y0*y1*y2 - x2*x3*x3*y1*y1*y2);
m12 = -(RES_V*(x0*y1 - x1*y0)*(x0*y2 - x2*y0 - x0*y3 + x3*y0 + x2*y3 - x3*y2)*(x1*y2 - x2*y1 - x1*y3 + x3*y1 + x2*y3 - x3*y2))/(- x0*x0*x1*y1*y2*y2 + 2*x0*x0*x1*y1*y2*y3 - x0*x0*x1*y1*y3*y3 + x0*x0*x2*y1*y1*y2 - 2*x0*x0*x2*y1*y2*y3 + x0*x0*x2*y2*y3*y3 - 2*x0*x0*x3*y1*y1*y2 + x0*x0*x3*y1*y1*y3 + 2*x0*x0*x3*y1*y2*y2 - x0*x0*x3*y2*y2*y3 + x0*x1*x1*y0*y2*y2 - 2*x0*x1*x1*y0*y2*y3 + x0*x1*x1*y0*y3*y3 - 2*x0*x1*x2*y0*y1*y3 + 2*x0*x1*x2*y0*y2*y3 + 2*x0*x1*x2*y1*y3*y3 - 2*x0*x1*x2*y2*y3*y3 + 2*x0*x1*x3*y0*y1*y2 - 2*x0*x1*x3*y0*y2*y2 - 2*x0*x1*x3*y1*y2*y3 + 2*x0*x1*x3*y2*y2*y3 - x0*x2*x2*y0*y1*y1 + 2*x0*x2*x2*y0*y1*y3 - x0*x2*x2*y0*y3*y3 + 2*x0*x2*x3*y0*y1*y1 - 2*x0*x2*x3*y0*y1*y2 - 2*x0*x2*x3*y1*y1*y3 + 2*x0*x2*x3*y1*y2*y3 - x0*x3*x3*y0*y1*y1 + x0*x3*x3*y0*y2*y2 + 2*x0*x3*x3*y1*y1*y2 - 2*x0*x3*x3*y1*y2*y2 - x1*x1*x2*y0*y0*y2 + 2*x1*x1*x2*y0*y0*y3 - 2*x1*x1*x2*y0*y3*y3 + x1*x1*x2*y2*y3*y3 - x1*x1*x3*y0*y0*y3 + 2*x1*x1*x3*y0*y2*y3 - x1*x1*x3*y2*y2*y3 + x1*x2*x2*y0*y0*y1 - 2*x1*x2*x2*y0*y0*y3 + 2*x1*x2*x2*y0*y3*y3 - x1*x2*x2*y1*y3*y3 - 2*x1*x2*x3*y0*y0*y1 + 2*x1*x2*x3*y0*y0*y2 + 2*x1*x2*x3*y0*y1*y3 - 2*x1*x2*x3*y0*y2*y3 + x1*x3*x3*y0*y0*y1 - 2*x1*x3*x3*y0*y1*y2 + x1*x3*x3*y1*y2*y2 + x2*x2*x3*y0*y0*y3 - 2*x2*x2*x3*y0*y1*y3 + x2*x2*x3*y1*y1*y3 - x2*x3*x3*y0*y0*y2 + 2*x2*x3*x3*y0*y1*y2 - x2*x3*x3*y1*y1*y2);
Using m00, m01, m02, m10, m11 and m12, I constructed an M matrix. I omitted m20, m21 and m22 because they have no influence on the result of x and y.
Finally, I used this M matrix to shift the coordinates of the ball:
x_shifted = m00*x + m01*y + m02;
y_shifted = m10*x + m11*y + m12;
Which is based on this:
As you can see, the solution produced by Matlab results in extremely long equations. By accident, I found out when you swap xi with ui and yi with vi in the A matrix, you will get the inverse of the transformation matrix M for m00, m01, m02, m10, m11, m12, but the coefficients are computed with way shorter equations.
/ 0 0 1 0 0 0 0 0 \ /m00\ /x0\
| RES_H 0 1 0 0 0 -RES_H*x1 0 | |m01| |x1|
| 0 RES_V 1 0 0 0 0 -RES_V*x2 | |m02| |x2|
| RES_H RES_V 1 0 0 0 -RES_H*x3 -RES_V*x3 |.|m10|=|x3|
| 0 0 0 0 0 1 0 0 | |m11| |y0|
| 0 0 0 RES_H 0 1 -RES_H*y1 0 | |m12| |y1|
| 0 0 0 0 RES_V 1 0 -RES_V*y2 | |m20| |y2|
\ 0 0 0 RES_H RES_V 1 -RES_H*y3 -RES_V*y3 / \m21/ \y3/
When you let this solve by Matlab in the same way, you will get:
m00 = (x0*x2*y1 - x1*x2*y0 - x0*x3*y1 + x1*x3*y0 - x0*x2*y3 + x0*x3*y2 + x1*x2*y3 - x1*x3*y2)/(RES_H*(x1*y2 - x2*y1 - x1*y3 + x3*y1 + x2*y3 - x3*y2));
m01 = -(x0*x1*y2 - x1*x2*y0 - x0*x1*y3 + x0*x3*y1 - x0*x3*y2 + x2*x3*y0 + x1*x2*y3 - x2*x3*y1)/(RES_V*(x1*y2 - x2*y1 - x1*y3 + x3*y1 + x2*y3 - x3*y2));
m02 = x0;
m10 = (x0*y1*y2 - x1*y0*y2 - x0*y1*y3 + x1*y0*y3 - x2*y0*y3 + x3*y0*y2 + x2*y1*y3 - x3*y1*y2)/(RES_H*(x1*y2 - x2*y1 - x1*y3 + x3*y1 + x2*y3 - x3*y2));
m11 = -(x0*y1*y2 - x2*y0*y1 - x1*y0*y3 + x3*y0*y1 - x0*y2*y3 + x2*y0*y3 + x1*y2*y3 - x3*y1*y2)/(RES_V*(x1*y2 - x2*y1 - x1*y3 + x3*y1 + x2*y3 - x3*y2));
m12 = y0;
m20 = (x0*y2 - x2*y0 - x0*y3 - x1*y2 + x2*y1 + x3*y0 + x1*y3 - x3*y1)/(RES_H*(x1*y2 - x2*y1 - x1*y3 + x3*y1 + x2*y3 - x3*y2));
m21 = -(x0*y1 - x1*y0 - x0*y3 + x1*y2 - x2*y1 + x3*y0 + x2*y3 - x3*y2)/(RES_V*(x1*y2 - x2*y1 - x1*y3 + x3*y1 + x2*y3 - x3*y2));
m22 = 1.0;
When you put these coefficients into a matrix and invert this matrix, you will get the same results for m00, m01, m02, m10, m11, m12 as for the first method.
I used the program which I found here to invert the matrix: https://codingtech2017.wordpress.com/2017/05/03/c-program-to-inverse-a-matrix3x3/