Differences between two images with slightly different point of view and lighting conditions with OpenCV

With the method explained in CV - Extract differences between two images we can identify the differences between two aligned images.

How to do this with OpenCV when the camera angle (point of view) and the lighting condition are slightly different?

The code from How to match and align two images using SURF features (Python OpenCV )? helps to rotate / align the two images but as the result of the perspective transform ("homography") is not perfect, the "difference" algorithm will not work well here.

As an example, how to get only the green sticker (= the difference) from these 2 photos?

For the alignment of two images, you might use the affine transformation. To do so, you need three points pairs from both images. In order to get these points, I will use the object corners. Here are the steps I am following to get the corners.

1. Background subtraction (or object extraction) by Gaussian mixture model
2. Noise removal on the 1st step output
3. Get the corners by using the contours

I will be using opencv library for all of these functions.

import cv2
from sklearn.mixture import GaussianMixture as GMM
import matplotlib.pyplot as plt
import numpy as np
import math


def extract_object(img):

    img2 = img.reshape((-1,3))

    n_components = 2

    #covariance choices: full, tied, diag, spherical
    gmm = GMM(n_components=n_components, covariance_type='tied')
    gmm.fit(img2)
    gmm_prediction = gmm.predict(img2)

    #Put numbers back to original shape so we can reconstruct segmented image
    original_shape = img.shape
    segmented_img = gmm_prediction.reshape(original_shape[0], original_shape[1])

    # set background always to 0
    if segmented_img[0,0] != 0:
        segmented_img = cv2.bitwise_not(segmented_img)
    return segmented_img


def remove_noise(img):
    img_no_noise = np.zeros_like(img)

    labels,stats= cv2.connectedComponentsWithStats(img.astype(np.uint8),connectivity=4)[1:3]

    largest_area_label = np.argmax(stats[1:, cv2.CC_STAT_AREA]) +1

    img_no_noise[labels==largest_area_label] = 1
    return img_no_noise

def get_box_points(img):
    contours, _ = cv2.findContours(img.astype(np.uint8), cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)

    cnt = contours[0]
    rect = cv2.minAreaRect(cnt)
    box_points = cv2.boxPoints(rect)
    box_points = np.int0(box_points)

    return box_points

img = cv2.imread('choco.jpg',1)
img_paper = cv2.imread('choco_with_paper.jpg',1)

# remove background
img_bg_removed = extract_object(img) 
img_paper_bg_removed = extract_object(img_paper)

img_no_noise = remove_noise(img_bg_removed)
img_paper_no_noise = remove_noise(img_paper_bg_removed)

img_box_points = get_box_points(img_no_noise)
img_paper_box_points = get_box_points(img_paper_no_noise)

The corners of the image are slightly off, but they are good enough for this task. I am sure there is a better way to detect the corners, but this was the fastest solution to me :)

Next, I will apply the affine transformation to register/align the original image to the image with the piece of paper.

# Affine transformation matrix
M = cv2.getAffineTransform(img_box_points[0:3].astype(np.float32), img_paper_box_points[0:3].astype(np.float32))

# apply M to the original binary image
img_registered = cv2.warpAffine(img_no_noise.astype(np.float32), M, dsize=(img_paper_no_noise.shape[1],img_paper_no_noise.shape[0]))

# get the difference
dif = img_registered-img_paper_no_noise

# remove minus values
dif[dif<1]=0

Here is the difference between the paper image and the registered original image.

All I have to do is to get the largest component (i.e. the piece of paper) among these areas, and apply a hull convex to cover the most of the piece of paper.

dif = remove_noise(dif) # get the largest component
contours, _ = cv2.findContours(dif.astype(np.uint8), cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
drawing = dif.copy().astype(np.uint8)

hull = [cv2.convexHull(contours[0])]

cv2.drawContours(drawing, hull, 0, 255,-1)

img_paper_extracted = cv2.bitwise_and(img_paper,img_paper,mask=drawing)

Here is my final result.