pythonopencvopticalflow

Create frame from previous frame and Optical Flow

I have a video and I need to simulate frames using Optical Flow; i.e. having a frame and the Optical Flow that represents the pixel translation for the next frame simulate this following resulting frame.

I am using Python and OpenCV as follows:

  1. Generate flow between two consecutive grayscale frames
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
flow = cv2.calcOpticalFlowFarneback(prev_gray, gray, None, pyr_scale = 0.5, levels = 5, winsize = 11, iterations = 5, poly_n = 5, poly_sigma = 1.1, flags = 0)

This is the flow between first and second frames:

First frame optical flow

This is the following (target frame):

Original second frame

  1. Now, I want to use the first frame and the Optical Flow to generate the second frame estimation. Based on this question
height = flow.shape[0]
width = flow.shape[1]
R2 = np.dstack(np.meshgrid(np.arange(width), np.arange(height)))
pixel_map = R2 + flow

prev = original_frames[0].astype("float32")
pixel_map = pixel_map.astype("float32")
new_frame = cv2.remap(prev, pixel_map, None, cv2.INTER_LINEAR)

The resulting image is:

Resulting frame

Solution

  • The issue was solved slightly updating the code as follows:

    h = flow.shape[0]
w = flow.shape[1]
flow[:,:,0] += np.arange(w)
flow[:,:,1] += np.arange(h)[:,np.newaxis]

new_frame = cv2.remap(original_frame, flow, None, cv2.INTER_LINEAR)