Detect rectangular portraits of people on images with OpenCV

I have many images of yearbooks with people portraits and I'm trying to build an algorytm that will detect those portraits. At least, to detect correct rectangular portraits. Example 1 Example 2

I'm trying to investigate three directions:

1. Face detection
2. Dark rectangles detection (Since portraits are usually darker shapes on brighter background)
3. People name extraction from OCR'ed texts

By combining results of three algorithms above, I hope to get some methodology, that will be applicable for many different yearbooks pages.

I would be very appreciate for any help for the rectangles detection. I started with Java and OpenCV 3.

Here is my code applied for an image:

System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
Mat source = Imgcodecs.imread("Path/to/image", Imgcodecs.CV_LOAD_IMAGE_ANYCOLOR);
Mat destination = new Mat(source.rows(), source.cols(), source.type());

Imgproc.cvtColor(source, destination, Imgproc.COLOR_RGB2GRAY);
Imgproc.GaussianBlur(destination, destination, new Size(5, 5), 0, 0, Core.BORDER_DEFAULT);

int threshold = 100;
Imgproc.Canny(destination, destination, 50, 100);
Imgproc.Canny(destination, destination, threshold, threshold*3);

At this point, I have such result:

Trying to find contours from the edges above:

    List<MatOfPoint> contourDetections = new ArrayList<>();
    Mat hierarchy = new Mat();

    // Find contours
    Imgproc.findContours(destination, contourDetections, hierarchy, Imgproc.RETR_EXTERNAL, Imgproc.CHAIN_APPROX_SIMPLE);

    // Draw contours 
    Imgproc.drawContours(source, contours, -1, new Scalar(255,0,0), 2);

Getting this result:

But not sure how to extract rectangles from those contours since many of lines are incomplete.

Getting back to edges and trying to find vertical and horizontal lines using HoughLinesP:

    Mat lines = new Mat();
    int thre = 50;
    int minLineSize = 250;
    int lineGap = 80;

    int ignoreLinesShorter = 300;

    Imgproc.HoughLinesP(destination, lines, 1, Math.PI/180, thre, minLineSize, lineGap);

    for(int c = 0; c < lines.rows(); c++) {

        double[] vec = lines.get(c, 0);

        double  x1 = vec[0],
                y1 = vec[1],
                x2 = vec[2],
                y2 = vec[3];

        // Filtering only verticat and horizontal lines
        if(x1 == x2 || y1 == y2) {

            // Filtering out short lines
            if(Math.abs(x1 - x2) > ignoreLinesShorter || Math.abs(y1 - y2) > ignoreLinesShorter) {

              Point start = new Point(x1, y1);
              Point end = new Point(x2, y2);

              // Draw line
              Imgproc.line(source, start, end, new Scalar(0,0,255), 2);
            }
        }
    }

Result:

Like with contours, I'm still not seeing correct rectangles that I could detect. Could you help me with a correct direction? Maybe there is an easier way to perform this task?

For detecting rectangular portraits (headshots), I've had some success with the following methodology.

1. Rectangle Detection:
   a. Convert to grayscale
   b. Change color of image border to background color
   c. Binary Thresholding
   d. Closing Morphological Transformation
   e. Invert image if necessary
   f. Find contours
   g. Choose rectangular contours based on aspect ratio and area
2. Face Detection: Find the rectangular contours that contain headshots using Haar cascade. To be a headshot portrait, a rectangle should contain only one face with dimensions of face specified relative to size of the rectangle. For example, when using OpenCv CascadeClassifier.detectMultiScale, set minSize=(0.4 * width, 0.3 * height) for face size where height and width are the dimensions of rectangle.
3. Portrait Validation: Check for grid structure of portrait rectangles. For missing rectangles in grid, check if face exists. For existing rectangles, use grid structure to correct rectangle dimensions if necessary.

---

1. Python code for rectangle detection (It should be easy to convert to Java.)

img = cv2.imread('example.jpg')
gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)

# Remove black border by cropping
bw = 6 # border width
ht, wd = img.shape[:2] # height, width
gray = gray[bw:ht-bw, bw:wd-bw]

# HISTOGRAM -- Put histogram function here to determine the following:
bg_color = (235,235,235) # background color
thresh_value = 220

# Add back border with background color
gray = cv2.copyMakeBorder(gray, bw, bw, bw, bw, cv2.BORDER_CONSTANT, value=bg_color)

# Binary Threshold
thresh = cv2.threshold(gray, thresh_value, 255, cv2.THRESH_BINARY)[1] # orig: 235

# Closing Morphological Transformation
kernel = np.ones((5,5),np.uint8)
closing = cv2.morphologyEx(thresh, cv2.MORPH_CLOSE, kernel)

# Invert Image
closing = np.invert(closing)

# Find contours
cnts = cv2.findContours(closing, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
cnts = cnts[0] if len(cnts) == 2 else cnts[1]

# Find portraits by specifying range of sizes and aspect ratios
img_area = ht * wd
for cnt in cnts:
    x,y,w,h = cv2.boundingRect(cnt)
    if w*h < 0.005*img_area or w*h > 0.16*img_area or h/w < 0.95 or h/w > 1.55:
        continue
    cv2.rectangle(img, (x, y), (x+w, y+h), (0, 255, 0), 2)

cv2.imshow('Result', img)
cv2.waitKey(0)

---

Example 1 Result (First image is after inverting.)

Example 2 Result

---

2. Python code for face detection

def is_headshot(cnt_img):
    gray = cv2.cvtColor(cnt_img, cv2.COLOR_BGR2GRAY)
    height, width = cnt_img.shape[:2]
    min_size = int(max(0.4*width, 0.3*height))
    faces = face_cascade.detectMultiScale(gray, 
                                          scaleFactor=1.3, 
                                          minNeighbors=3, 
                                          minSize=(min_size, min_size))
    if len(faces) == 1:
        return True
    else:
        return False

face_cascade = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')
x,y,w,h = cv2.boundingRect(cnt) # bounding rectangle of contour found in code above
if is_headshot(img[y:y+h, x:x+w]):
    cv2.imwrite('headshot.jpg', img[y:y+h, x:x+w])

---

3. Python code for portrait validation
The grid structure can be found using code I posted in this stackoverflow question. Loop through the results of the completed grid. Each grid element is defined by (x,y,w,h) where w and h can be the average width and height of portraits found above. Use the function box1.intersection(box2) from shapely.geometry to determine if there are missing or missized portraits. If the intersection area is small or zero, there may be a missing portrait that should then be checked with face detection. I'm open to providing more details if there is any interest.