pythonopencvcomputer-visionspriteimage-segmentation

Unable to get contour when bounding boxes are not overlapping


I have some sprite sheets. In some cases, the bounding boxes of the sprites are overlapping, even though the sprites themselves are not overlapping:

Overlapping bounding boxes

In other cases, the bounding boxes are not overlapping:

Nonoverlapping bounding boxes

To extract the individual sprites, I am doing the following:

im = cv2.imread("trees.png") # read image
imGray = cv2.cvtColor(im, cv2.COLOR_BGR2GRAY) # convert to gray
contours, _ = cv2.findContours(imGray, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE) # contouring
sortedContours = sorted(contours, key=cv2.contourArea, reverse=True) # sorting, not necessary...
for contourIdx in range(0,len(sortedContours)-1): # loop with index for easier saving
    contouredImage = im.copy() # copy image
    contouredImage = cv2.drawContours(contouredImage, sortedContours, contourIdx, (255,255,255), -1) # fill contour with white
    extractedImage = cv2.inRange(contouredImage, (254,254,254), (255,255,255)) # extract white from image
    resultImage = cv2.bitwise_and(im, im, mask=extractedImage) # AND operator to get only one filled contour at a time
    x, y, w, h = cv2.boundingRect(sortedContours[contourIdx]) # get bounding box
    croppedImage = resultImage[y:y + h, x:x + w] # crop
    cv2.imwrite("contour_"+str(contourIdx)+".png", croppedImage) # save

This works great for the former image where the bounding boxes are overlapping, but fails in the latter case where the bounding boxes are not overlapping. Why is that and how can I fix it ?

EDIT: In the former case, as expected, it detects the individual contours and outputs each separately. But in the latter case, it doesn't seem to detect any individual contours, or rather the whole image is output.


Solution

  • The problem is that your images have not been created in the same way. If you run ExifTool on the overlap image, it has this:

    Bit Depth                       : 8
    Color Type                      : RGB with Alpha
    Compression                     : Deflate/Inflate
    Filter                          : Adaptive
    Interlace                       : Adam7 Interlace
    SRGB Rendering                  : Perceptual
    Exif Byte Order                 : Big-endian (Motorola, MM)
    Orientation                     : Horizontal (normal)
    X Resolution                    : 72
    Y Resolution                    : 72
    Resolution Unit                 : inches
    Software                        : Pixelmator 3.9.11
    Modify Date                     : 2024:01:21 12:01:94
    Color Space                     : sRGB
    Exif Image Width                : 526
    Exif Image Height               : 244
    Pixels Per Unit X               : 2835
    Pixels Per Unit Y               : 2835
    Pixel Units                     : meters
    XMP Toolkit                     : XMP Core 6.0.0
    ...
    Creator Tool                    : Pixelmator 3.9.11
    

    And the no-overlap image:

    Bit Depth                       : 8
    Color Type                      : RGB with Alpha
    Compression                     : Deflate/Inflate
    Filter                          : Adaptive
    Interlace                       : Adam7 Interlace
    Software                        : Adobe ImageReady
    XMP Toolkit                     : Adobe XMP Core 5.0-c060 61.134777, 2010/02/12-17:32:00
    Creator Tool                    : Adobe Photoshop CS5 Windows
    

    Which of those aspects is causing the exact issue doesn't really matter, but if you examine the image after loading it with OpenCV, you'll find that the transparent pixels in the no-overlap image are green, and the grey-scale conversion turns all the pixels into the same colour.

    However, this code works for both images:

    # read the image as is, no interpretation
    im = cv2.imread("trees.png", cv2.IMREAD_UNCHANGED)
    # get the first three channels as RGB
    rgb = im[:, :, :3]
    # replace all the transparent pixels (alpha = 0) with black
    rgb[im[:, :, 3] == 0] = [0, 0, 0]
    # generate a greyscale image from the resulting RGB channels
    imGray = cv2.cvtColor(rgb, cv2.COLOR_RGB2GRAY)
    

    Instead of picking black, you might pick white, depending on what you expect to see in the image. The change above works, but it doesn't means it's the best way to approach this. I would imagine the alpha channel (im[:, :, 3]) actually has everything you need and perhaps you should work with that instead of the RGB channel, to find the contours.

    Here's a rewrite of your code that's a bit easier to understand and uses the alpha channel for the contouring:

    import cv2
    
    
    def load_image(im_filename):
        return cv2.imread(im_filename, cv2.IMREAD_UNCHANGED)
    
    
    def extract_alpha_channel(im):
        return im[:, :, 3]
    
    
    def find_contours(channel):
        return cv2.findContours(channel, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)[0]
    
    
    def crop_and_save_contoured(im, contours):
        for idx, contour in enumerate(contours):
            x, y, w, h = cv2.boundingRect(contour)
            cropped_image = im[y:y + h, x:x + w]
            cv2.imwrite(f"output/contour_{idx}.png", cropped_image)
    
    
    def main():
        original_image = load_image("no_overlap.png")
        alpha_channel = extract_alpha_channel(original_image)
        contours = find_contours(alpha_channel)
        crop_and_save_contoured(original_image, contours)
    
    
    if __name__ == "__main__":
        main()