pythonmatlabopencvimage-processingfft

Segmenting License Plate Characters


I am facing a problem in segmenting characters from a license plate image. I have applied following method to extract license plate characters"

  1. Adaptive threshold the license plate image.
  2. Select contours which having particular aspect ratio.

If there is any shade in the license plate image as in attached file, I am not able to properly segment the characters due to improper binarization. The shade in the image merges adjacent characters in the image.

I have thresholded the images with different window sizes. The results are attached. How can I segment characters from image if there is shade in the image? I am using OpenCV.

License plate image with shade

threshoded image 1

thresholded image 2

thresholded image 3

I have used following function in OpenCV to threshold my license plate image:

cvAdaptiveThreshold(licensePlateImg, threshImg, 255, CV_ADAPTIVE_THRESH_GAUSSIAN_C, CV_THRESH_BINARY_INV, wind);

I have tried with different window sizes (wind) and different adaptiveMethod (ADAPTIVE_THRESH_MEAN_C and ADAPTIVE_THRESH_GAUSSIAN_C) to get the thresholded images.


Solution

  • Before I start, I know you are seeking an implementation of this algorithm in OpenCV C++, but my algorithm requires the FFT and the numpy / scipy packages are awesome for that. As such, I will give you an implementation of the algorithm in OpenCV using Python instead. The code is actually quite similar to the C++ API that you can easily transcribe that over instead. That way, it minimizes the amount of time it will take for me to learn (or rather relearn...) the API and I would rather give you the algorithm and the steps I did to perform this task to not waste any time at all.

    As such, I will give you a general overview of what I would do. I will then show you Python code that uses numpy, scipy and the OpenCV packages. As a bonus for those who use MATLAB, I will show you the MATLAB equivalent, with MATLAB code to boot!


    What you can do is try to use homomorphic filtering. In basic terms, we can represent an image in terms of a product of illumination and reflectance. Illumination is assumed to be slowly varying and the main contributor of dynamic range. This is essentially low frequency content. Reflectance represents details of objects and assumed to vary rapidly. This is also the primary contributor to local contrast and is essentially high frequency content.

    The image can be represented as a product of these two. Homomorphic filtering tries and splits up these components and we filter them individually. We then combine the results together when we are finished. As this is a multiplicative model, it's customary to use a log operation so that we can express the product as a sum of two terms. These two terms are filtered individually, scaled to emphasize or de-emphasize their contributions to the image, summed, then the anti-log is taken.

    The shading is due to the illumination, and so what we can do is decrease the contribution that this shading does over the image. We can also boost the reflectance so we can get some better edges as edges are associated with high frequency information.

    We usually filter the illumination using a low-pass filter, while the reflectance with a high-pass filter. In this case, I'm going to choose a Gaussian kernel with a sigma of 10 as the low-pass filter. A high-pass filter can be obtained by taking 1 and subtracting with the low-pass filter. I transform the image into the log domain, then filter the image in the frequency domain using the low-pass and high-pass filters. I then scale the low pass and high pass results, add these components back, then take the anti-log. This image is now better suited to be thresholded as the image has low variation.

    What I do as additional post-processing is that I threshold the image. The letters are darker than the overall background, so any pixels that are lower than a certain threshold would be classified as text. I chose the threshold to be intensity 65. After this, I also clear off any pixels that are touching the border, then remove any areas of the image that have less than 160 (MATLAB) or 120 (Python) pixels of total area. I also crop out some of the columns of the image as they are not needed for our analysis.


    Here are a couple of caveats for you:

    Caveat #1 - Removing borders

    Removing any pixels that touch the border is not built into OpenCV. However, MATLAB has an equivalent called imclearborder. I'll use this in my MATLAB code, but for OpenCV, this was the following algorithm:

    I created a method called imclearborder(imgBW, radius) in my code, where radius is how many pixels within the border you want to clear stuff up.

    Caveat #2 - Removing pixel areas below a certain area

    Removing any areas where they are less than a certain amount is also not implemented in OpenCV. In MATLAB, this is conveniently given using bwareaopen. The basic algorithm for this is:

    I created a method called bwareaopen(imgBW) that does this for us.

    Caveat #3 - Area parameter for removing pixel areas

    For the Python code, I had to play around with this parameter and I settled for 120. 160 was used for MATLAB. For python, 120 got rid of some of the characters, which is not desired. I'm guessing my implementation of bwareaopen in comparison to MATLAB's is different, which is probably why I'm getting different results.


    Without further ado, here's the code. Take note that I did not use spatial filtering. You could use filter2D in OpenCV and convolve this image with the Gaussian kernel, but I did not do that as Homomorphic Filtering when using low-pass and high-pass filters are traditionally done in the frequency domain. You could explore this using spatial filtering, but you would also have to know the size of your kernels before hand. With frequency domain filtering, you just need to know the standard deviation of the filter, and that's just one parameter in comparison to two.

    Also, for the Python code, I downloaded your image on to my computer and ran the script. For MATLAB, you can directly reference the hyperlink to the image when reading it in with the Image Processing toolbox.


    Python code

    import cv2 # For OpenCV modules (For Image I/O and Contour Finding)
    import numpy as np # For general purpose array manipulation
    import scipy.fftpack # For FFT2 
    
    #### imclearborder definition
    
    def imclearborder(imgBW, radius):
    
        # Given a black and white image, first find all of its contours
        imgBWcopy = imgBW.copy()
        contours,hierarchy = cv2.findContours(imgBWcopy.copy(), cv2.RETR_LIST, 
            cv2.CHAIN_APPROX_SIMPLE)
    
        # Get dimensions of image
        imgRows = imgBW.shape[0]
        imgCols = imgBW.shape[1]    
    
        contourList = [] # ID list of contours that touch the border
    
        # For each contour...
        for idx in np.arange(len(contours)):
            # Get the i'th contour
            cnt = contours[idx]
    
            # Look at each point in the contour
            for pt in cnt:
                rowCnt = pt[0][1]
                colCnt = pt[0][0]
    
                # If this is within the radius of the border
                # this contour goes bye bye!
                check1 = (rowCnt >= 0 and rowCnt < radius) or (rowCnt >= imgRows-1-radius and rowCnt < imgRows)
                check2 = (colCnt >= 0 and colCnt < radius) or (colCnt >= imgCols-1-radius and colCnt < imgCols)
    
                if check1 or check2:
                    contourList.append(idx)
                    break
    
        for idx in contourList:
            cv2.drawContours(imgBWcopy, contours, idx, (0,0,0), -1)
    
        return imgBWcopy
    
    #### bwareaopen definition
    def bwareaopen(imgBW, areaPixels):
        # Given a black and white image, first find all of its contours
        imgBWcopy = imgBW.copy()
        contours,hierarchy = cv2.findContours(imgBWcopy.copy(), cv2.RETR_LIST, 
            cv2.CHAIN_APPROX_SIMPLE)
    
        # For each contour, determine its total occupying area
        for idx in np.arange(len(contours)):
            area = cv2.contourArea(contours[idx])
            if (area >= 0 and area <= areaPixels):
                cv2.drawContours(imgBWcopy, contours, idx, (0,0,0), -1)
    
        return imgBWcopy
    
    #### Main program
    
    # Read in image
    img = cv2.imread('5DnwY.jpg', 0)
    
    # Number of rows and columns
    rows = img.shape[0]
    cols = img.shape[1]
    
    # Remove some columns from the beginning and end
    img = img[:, 59:cols-20]
    
    # Number of rows and columns
    rows = img.shape[0]
    cols = img.shape[1]
    
    # Convert image to 0 to 1, then do log(1 + I)
    imgLog = np.log1p(np.array(img, dtype="float") / 255)
    
    # Create Gaussian mask of sigma = 10
    M = 2*rows + 1
    N = 2*cols + 1
    sigma = 10
    (X,Y) = np.meshgrid(np.linspace(0,N-1,N), np.linspace(0,M-1,M))
    centerX = np.ceil(N/2)
    centerY = np.ceil(M/2)
    gaussianNumerator = (X - centerX)**2 + (Y - centerY)**2
    
    # Low pass and high pass filters
    Hlow = np.exp(-gaussianNumerator / (2*sigma*sigma))
    Hhigh = 1 - Hlow
    
    # Move origin of filters so that it's at the top left corner to
    # match with the input image
    HlowShift = scipy.fftpack.ifftshift(Hlow.copy())
    HhighShift = scipy.fftpack.ifftshift(Hhigh.copy())
    
    # Filter the image and crop
    If = scipy.fftpack.fft2(imgLog.copy(), (M,N))
    Ioutlow = scipy.real(scipy.fftpack.ifft2(If.copy() * HlowShift, (M,N)))
    Iouthigh = scipy.real(scipy.fftpack.ifft2(If.copy() * HhighShift, (M,N)))
    
    # Set scaling factors and add
    gamma1 = 0.3
    gamma2 = 1.5
    Iout = gamma1*Ioutlow[0:rows,0:cols] + gamma2*Iouthigh[0:rows,0:cols]
    
    # Anti-log then rescale to [0,1]
    Ihmf = np.expm1(Iout)
    Ihmf = (Ihmf - np.min(Ihmf)) / (np.max(Ihmf) - np.min(Ihmf))
    Ihmf2 = np.array(255*Ihmf, dtype="uint8")
    
    # Threshold the image - Anything below intensity 65 gets set to white
    Ithresh = Ihmf2 < 65
    Ithresh = 255*Ithresh.astype("uint8")
    
    # Clear off the border.  Choose a border radius of 5 pixels
    Iclear = imclearborder(Ithresh, 5)
    
    # Eliminate regions that have areas below 120 pixels
    Iopen = bwareaopen(Iclear, 120)
    
    # Show all images
    cv2.imshow('Original Image', img)
    cv2.imshow('Homomorphic Filtered Result', Ihmf2)
    cv2.imshow('Thresholded Result', Ithresh)
    cv2.imshow('Opened Result', Iopen)
    cv2.waitKey(0)
    cv2.destroyAllWindows()
    

    MATLAB code

    clear all;
    close all;
    
    % Read in image
    I = imread('https://i.sstatic.net/5DnwY.jpg');
    
    % Remove some columns from the beginning and end
    I = I(:,60:end-20);
    
    % Cast to double and do log.  We add with 1 to avoid log(0) error.
    I = im2double(I);
    I = log(1 + I);
    
    % Create Gaussian mask in frequency domain
    % We must specify our mask to be twice the size of the image to avoid
    % aliasing.
    M = 2*size(I,1) + 1;
    N = 2*size(I,2) + 1;
    sigma = 10;
    [X, Y] = meshgrid(1:N,1:M);
    centerX = ceil(N/2);
    centerY = ceil(M/2);
    gaussianNumerator = (X - centerX).^2 + (Y - centerY).^2;
    
    % Low pass and high pass filters
    Hlow = exp(-gaussianNumerator./(2*sigma.^2));
    Hhigh = 1 - Hlow;
    
    % Move origin of filters so that it's at the top left corner to match with
    % input image
    Hlow = ifftshift(Hlow);
    Hhigh = ifftshift(Hhigh);
    
    % Filter the image, and crop
    If = fft2(I, M, N);
    Ioutlow = real(ifft2(Hlow .* If));
    Iouthigh = real(ifft2(Hhigh .* If));
    
    % Set scaling factors then add
    gamma1 = 0.3;
    gamma2 = 1.5;
    Iout = gamma1*Ioutlow(1:size(I,1),1:size(I,2)) + ...
           gamma2*Iouthigh(1:size(I,1),1:size(I,2));
    
    % Anti-log then rescale to [0,1]
    Ihmf = exp(Iout) - 1;
    Ihmf = (Ihmf - min(Ihmf(:))) / (max(Ihmf(:)) - min(Ihmf(:)));
    
    % Threshold the image - Anything below intensity 65 gets set to white
    Ithresh = Ihmf < 65/255;
    
    % Remove border pixels
    Iclear = imclearborder(Ithresh, 8);
    
    % Eliminate regions that have areas below 160 pixels
    Iopen = bwareaopen(Iclear, 160);
    
    % Show all of the results
    figure;
    subplot(4,1,1);
    imshow(I);
    title('Original Image');
    subplot(4,1,2);
    imshow(Ihmf);
    title('Homomorphic Filtered Result');
    subplot(4,1,3);
    imshow(Ithresh);
    title('Thresholded Result');
    subplot(4,1,4);
    imshow(Iopen);
    title('Opened Result');
    

    This is the result I get:

    Python

    Take note that I re-arranged the windows so that they're aligned in a single column.

    enter image description here

    MATLAB

    enter image description here