AKKit: How to select a group of 3D points from a 2D frame?

so the quest is this, I got an ARPointCloud with a bunch of 3d points and I'd like to select them based on a 2d frame from the perspective of the camera / screen.

I was thinking about converting the 2d frame to a 3d frustum and check if the points where the 3d frustum box, not sure if this is the ideal method, and not even sure how to do that.

Would anyone know how to do this or have a better method of achieving this?

Given the size of the ARKit frame W x H and the camera intrinsics we can create planes for the view frustum sides.

For example using C++ / Eigen we can construct our four planes (which pass through the origin) as

    std::vector<Eigen::Vector3d> _frustumPlanes;
    frustumPlanes.emplace_back(Eigen::Vector3d( fx, 0,   cx - W));
    frustumPlanes.emplace_back(Eigen::Vector3d(-fx, 0,  -cx));
    frustumPlanes.emplace_back(Eigen::Vector3d( 0,  fy,  cy - H));
    frustumPlanes.emplace_back(Eigen::Vector3d( 0, -fy, -cy));

We can then clip a 3D point by checking its position against the z < 0 half-space and the four sides of the frustum:

    auto pointIsVisible = [&](const Eigen::Vector3d& P) -> bool {
        if (P.z() >= 0) return false;  // behind camera
        for (auto&& N : frustumPlanes) {
            if (P.dot(N) < 0)
                return false;  // outside frustum plane
        }
        return true;
    };

Note that it is best to perform this clipping in 3D (before the projection) since points behind or near the camera or points far outside the frustum can have unstable projection values (u,v).