Stitching Images with Two Different Camera Positions

It may not be possible to do what I want, but I thought I would throw the question out there just in case.

I am attempting to stitch the following two images together that are taken from two different camera position.

They are taken at the exact same time, and as you can see, they have different perspectives. I recognize that I could accomplish this by taking two photos at different angles from the same camera position, but in this instance I am specifically looking for a solution using different camera positions.

When I stitch them with this code:

images = []
image1 = cv2.imread('./VideosToStitch/video2.png')
images.append(image1)
image2 = cv2.imread('./VideosToStitch/video1.png')
images.append(image2)

stitcher = cv2.Stitcher.create()
status, stitched = stitcher.stitch(images,cv2.Stitcher_PANORAMA)

cv2.imwrite('./VideosToStitch/stitched.png', stitched)

I get this:

The two blue lines look good, but as you can see the red line is broken. I thought perhaps I could apply a perspective transformation on the images first and then stitch them, but when I apply the warp perspective, I get this:

image1 = cv2.imread('./VideosToStitch/video1.png')

tl = (907,0)
bl = (0,1280)
tr = (1047,0)
br = (1920,1280)

pts1 = np.float32([tl,bl,tr,br])
pts2 = np.float32([[0,0],[0,1280],[1920,0],[1920,1280]])

matrix = cv2.getPerspectiveTransform(pts1,pts2)
transformedFrame = cv2.warpPerspective(image1, matrix,(1920,1280), flags=3)

Here you can see the red and blue lines are now straight, but I have lost all the detail on the ice surface. This could work if it is possible to apply the transformation in the horizontal axis only, and not the vertical axis. Or, alternatively, I could stitch them so the resulting image is curved, and then straighten it.

Thoughts?

To add illustration to Christoph Rackwitz comments below, here is the result if I adjust the perspective so the ice surface lines align both horizontally and vertically. As you see, the players which are perpendicular to the surface will never align.

You have two different optical origins.

You are not gonna get a panorama of the spherical or any other "pretty" kind. Those you could get with arbitrary views/perspectives, as long as the cameras have practically the same optical origin.

Best you can hope for is to get a stitched view of any plane (e.g. the field), from any perspective (top-down, audience view, ...). Anything not in that plane, but sticking out of the plane, such as players, will get warped. This is unavoidable.

This is a fact of math (geometry), not due to implementation.

You have seen this in your attempt to create a top-down view of the field. Players, audience, and especially the stands, are not in-plane, so they get distorted.

The only way around this is to have 3D information. That's feasible for anything fixed (the stands), but anything moving (living creatures) is a lot harder to get 3D data for.