Matching 3d object with camera image for augmented reality

I am working on a simple AR engine and I am having a problem with matching 3d object with a camera image. For better understanding, I illustrated it with the picture. Points A and B are in 3d space. Points C and D are given on a texture plane. The distance to the plane from the camera is known. I know how to obtain the coordinates of Anear, Bnear, Afar, Bfar, Cnear, Dnear, Cfar and Dfar. The problem is how to find points A' and B' in 3d space such as vector d==d' and points Anear == Cnear and Bnear == Dnear (the projection of 3d points to the screen should result with the same coordinates) Could anyone please help me with the math here, or at least point me to where to look for the answer?

PS. Seems like my problem description is not clear enough so to put it in other words: I have a pair of points in 3d space and a pair of points on texture plane (image from webcam). I need to put the points in 3d space at the correct distance from camera - so after perspective transformation they overlay the points on texture plane. The spatial relation of the 3d points need to be preserved. In the drawing the visual solution are points A' and B'. The dashed line illustrates the perspective transformation (where they are casted on near plane at the same location as points C and D).

So if I understand correct

given points in world-space are

• A
• B
• C
• D

also known is the distance d and implicitely the Camera.position origin and Camera.transform.forward direction.

Searched are

• A'
• B'

As I understand you could find the first point A' by finding the intersection point of the line (origin = A, direction = camera forward) and the line (origin = camera.position, direction = Camera.position -> C).

Equally also the second point B' by finding the intersection point of the line (origin = B, direction = camera.forward) and the line (origin = camera.position, direction = Camera.position -> D).

Unity offers some special Math3d that come to help here e.g.:

//Calculate the intersection point of two lines. Returns true if lines intersect, otherwise false.
//Note that in 3d, two lines do not intersect most of the time. So if the two lines are not in the 
//same plane, use ClosestPointsOnTwoLines() instead.
public static bool LineLineIntersection(out Vector3 intersection, Vector3 linePoint1, Vector3 lineVec1, Vector3 linePoint2, Vector3 lineVec2)
{
    Vector3 lineVec3 = linePoint2 - linePoint1;
    Vector3 crossVec1and2 = Vector3.Cross(lineVec1, lineVec2);
    Vector3 crossVec3and2 = Vector3.Cross(lineVec3, lineVec2);

    float planarFactor = Vector3.Dot(lineVec3, crossVec1and2);

    //is coplanar, and not parrallel
    if(Mathf.Abs(planarFactor) < 0.0001f && crossVec1and2.sqrMagnitude > 0.0001f)
    {
        float s = Vector3.Dot(crossVec3and2, crossVec1and2) / crossVec1and2.sqrMagnitude;
        intersection = linePoint1 + (lineVec1 * s);
        return true;
    }
    else
    {
        intersection = Vector3.zero;
        return false;
    }
}

So you could probably do something like

public static bool TryFindPoints(Vector3 cameraOrigin, Vector3 cameraForward, Vector3 A, Vector3 B, Vector3 C, Vector3 D, out Vector3 AMapped, out Vector3 BMapped)
{
    AMapped = default;
    BMapped = default;

    if(LineLineIntersection(out AMapped, A, cameraForward, cameraOrigin, C - cameraOrigin))
    {
        if(LineLineIntersection(out BMapped, B, cameraForward, cameraOrigin, D - cameraOrigin))
        {
            return true;
        }
    }

    return false;
}

and then use it like

if(TryFindPoints(Camera.transform.position, Camera.transform.forward, A, B, C, D, out var aMapped, out var bMapped))
{
    // do something with aMapped and bMapped
}
else
{
    Debug.Log("It was mathematically impossible to find valid points");
}

---

Note: Typed on smartphone but I hope the idea gets clear