python image opencv image-processing medical

Straigthening hand images by drawing a line through the middle finger in OpenCV

I want to straighten the hand in the image based on the middle finger. I have about 10000 of these hand X-ray images to preprocess. The preprocess I did so far:

Apply Gaussian Blur and Threshold (Binary + Otsu) on the image.
Apply dilation to get a single object (in this case a hand).
Used cv2.findContours() to draw outline along the edges around the hand.
Apply cv2.convexHull() and then cv2.convexityDefects() to mark the points that are farthest away from the convex hull. To filter out the ones of interest between the fingers, I only considered those that are more than a certain distance from the convex hull.

The code below describes the above-mentioned:

import cv2
import numpy as np
from matplotlib import pyplot as plt

img_path = "sample_image.png"

# Getting the threshold of the image:
image = cv2.imread("sample_image.png")
original = image.copy() 
blank = np.zeros(image.shape[:2], dtype = np.uint8)
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
blur = cv2.GaussianBlur(gray, (127,127), 0)
thresh = cv2.threshold(blur, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)[1]

# Merge into a single contour
kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (5,5))
dilate = cv2.dilate(thresh, kernel, iterations = 2)

# Drawing the contours along the edges of the hand in X-ray:
contours, hierarchy = cv2.findContours(dilate, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
contours = max(contours, key = lambda x: cv2.contourArea(x))

cv2.drawContours(image, [contours], -1, (255,255,0), 2)

# To mark the points that are farthest away from the convex hull, one for each "convexity defect"
hull = cv2.convexHull(contours, returnPoints = False)
defects = cv2.convexityDefects(contours, hull)
for i in range(defects.shape[0]):
    _, _, farthest_point_index, distance = defects[i, 0]
    farthest_point = contours[farthest_point_index][0]
    if distance > 50_000:
        circle = cv2.circle(image, farthest_point, 20, [0,0,255], -1)

Input image = Sample_image.png
After running Contour on the image = Contour on the hand
Final output of the code after convexity hull = Convexity Hull

The next step would be:

To find out which of four points are the most central, since those are next to the middle finger.
Draw a line along the identified middle finger and rotate the image straight.

How would one approach and code this? I am really struggling...

What I want:

UPDATE:

After trying suggestion by @Sheldon, this works for some image types of hand X-ray. I just added the code suggested by @Sheldon and went through some samples.

Let me explain:

Hands that are straight or leaned towards right

Here are six examples of hand X-ray images. The images are nearly straight or slight leaned towards the right. After running these six samples through, I got the following result respectively (same order as the input image link (I labelled the respective image order)).

In the fourth example (Bad example), the middle finger is shorter than the surrounding fingers for some reason, the convex hull only has one convexity defect between the index and the ring finger and the method fails I think..., hence why I think the line of axis is off and the ellipse is off (in this case I will just skip these types of example unless there is a code to counter it?).

Hands that are leaned towards left

Here is another six examples of hand X-ray images that are slanted towards the left side. The results respective to the input link always points downwards after running through straightening code. I cannot think of a reason

This is what I have so far. Is there a method or code I can input to counter those images that are slanted towards left such that it points upwards always?

UPDATE 2:

According to @NickODell, the reason that left-leaning hand turning upwards down after straightening is because quote: "there are two ellipses which fit the points in the contour: the ellipse and a version of that ellipse which is rotated by 180 degrees. You can fix this by confining the ellipse_angle to the range [-90, 90]". Adding this line of code fixes it: ellipse_angle = ((ellipse_angle + 90) % 180) - 90.

Fixed output through fixed code using the same six example

Solution

After struggling for a while with trying to reproduce your results (there is an error with the distance threshold in your code example), I came up with an alternate solution that should do the trick. Instead of attempting to find the line that goes through the two central points, I suggest fitting an ellipse to the contour of the hand, and then rotating the image by the ellipse_angle.

STEP1: Fitting the contours with an ellipse:

(x,y),(MA,ma),ellipse_angle = cv2.fitEllipse(contours)
  
cv2.ellipse(image, (int(x),int(y)), (np.int(MA),np.int(ma)), ellipse_angle, np.int(ellipse_angle), np.int(ellipse_angle+360), (255,0,255), 1)
x1=np.int((int(x) + np.int(MA)*np.sin(ellipse_angle * np.pi / 180.0)))
y1=np.int((int(y) - np.int(MA)*np.cos(ellipse_angle * np.pi / 180.0)))     
x2=np.int((int(x) - np.int(MA)*np.sin(ellipse_angle * np.pi / 180.0)))
y2=np.int((int(y) + np.int(MA)*np.cos(ellipse_angle * np.pi / 180.0))) 
cv2.line(image, (x1, y1), (x2, y2),(255,0,255),4)

This snippet outputs:

Note that you can barely see the ellipse here, so I plotted the ellipse's longest axis using cv2.line(image, (x1, y1), (x2, y2),(255,0,255),4).

STEP2: Rotating the image:

Following this pyimagesearch tutorial:

(h, w) = image.shape[:2]
(cX, cY) = (w // 2, h // 2)

M = cv2.getRotationMatrix2D((cX, cY), ellipse_angle, 1.0)
rotated = cv2.warpAffine(image, M, (w, h))
cv2.imshow("Rotated by {} Degrees".format(str(ellipse_angle)), rotated)

This returns:

I would be curious to know how this script performs on the remainder of your data.