c++image-processingcomputer-visiongradientopticalflow

Lucas-Kanade optical flow - gradient calculation

I have implemented Lucas-Kanade method (per pixel version, not the one for features that is in OpenCV). However, I have a question regarding calculating gradients (dx, dy, dt). In a few implementations, I have seen this:

for (int y = 0; y < src.GetHeight() - 1; y++) {
    for (int x = 0; x < src.GetWidth() - 1; x++) {
        float src00 = src.GetPixelStart(x, y)[0];
        float src10 = src.GetPixelStart(x + 1, y)[0];
        float src01 = src.GetPixelStart(x, y + 1)[0];
        float src11 = src.GetPixelStart(x + 1, y + 1)[0];

        float dst00 = dst.GetPixelStart(x, y)[0];
        float dst10 = dst.GetPixelStart(x + 1, y)[0];
        float dst01 = dst.GetPixelStart(x, y + 1)[0];
        float dst11 = dst.GetPixelStart(x + 1, y + 1)[0];

        float srcVal = src10 - src00 + src11 - src01;                           
        float dstVal = dst10 - dst00 + dst11 - dst01;

        dFx.SetValue(0.25 * (srcVal + dstVal), 0, x, y);

        //------

        srcVal = src01 - src00 + src11 - src10;                
        dstVal = dst01 - dst00 + dst11 - dst10;

        dFy.SetValue(0.25 * (srcVal + dstVal), 0, x, y);

        //------

        dFt.SetValue(0.25 * (dst00 + dst10 + dst01 + dst11 - src00 - src10 - src01 - src11), 0, x, y);
    }
}

It works quite OK.

However, I have tried to use Sobel operator in X (ConvolutionFilterSeparable({ -1, 0, 1 }, { 1, 2, 1 }) and Y (ConvolutionFilterSeparable({ 1, 2, 1 }, { -1, 0, 1 })) direction to calculate first derivatives and then

 Image2d<float> dFx = (derivatorA.dx + derivatorB.dx);
 Image2d<float> dFy = (derivatorA.dy + derivatorB.dy);
 Image2d<float> dFt = (dst - src);

The problem is, that if I use this, the final optical flow is very small and does not correspond with the image movements.

Why the simplified calculation works better?

Solution

  • Your Sobel kernel is not normalized, and hence does not produce the correct magnitude for the derivative. It also has the wrong sign (assuming the function you call computes an actual convolution, not a correlation). A proper derivative would be computed with ConvolutionFilterSeparable({ 1/2, 0, -1/2 }, { 1/4, 2/4, 1/4 }).

    The first component is the finite difference derivative with a step size of 2. The second component is a smoothing filter that sums to 1.