I have many images of yearbooks with people portraits and I'm trying to build an algorytm that will detect those portraits. At least, to detect correct rectangular portraits. Example 1 Example 2
I'm trying to investigate three directions:
By combining results of three algorithms above, I hope to get some methodology, that will be applicable for many different yearbooks pages.
I would be very appreciate for any help for the rectangles detection. I started with Java and OpenCV 3.
Here is my code applied for an image:
System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
Mat source = Imgcodecs.imread("Path/to/image", Imgcodecs.CV_LOAD_IMAGE_ANYCOLOR);
Mat destination = new Mat(source.rows(), source.cols(), source.type());
Imgproc.cvtColor(source, destination, Imgproc.COLOR_RGB2GRAY);
Imgproc.GaussianBlur(destination, destination, new Size(5, 5), 0, 0, Core.BORDER_DEFAULT);
int threshold = 100;
Imgproc.Canny(destination, destination, 50, 100);
Imgproc.Canny(destination, destination, threshold, threshold*3);
At this point, I have such result:
Trying to find contours from the edges above:
List<MatOfPoint> contourDetections = new ArrayList<>();
Mat hierarchy = new Mat();
// Find contours
Imgproc.findContours(destination, contourDetections, hierarchy, Imgproc.RETR_EXTERNAL, Imgproc.CHAIN_APPROX_SIMPLE);
// Draw contours
Imgproc.drawContours(source, contours, -1, new Scalar(255,0,0), 2);
But not sure how to extract rectangles from those contours since many of lines are incomplete.
Getting back to edges and trying to find vertical and horizontal lines using HoughLinesP:
Mat lines = new Mat();
int thre = 50;
int minLineSize = 250;
int lineGap = 80;
int ignoreLinesShorter = 300;
Imgproc.HoughLinesP(destination, lines, 1, Math.PI/180, thre, minLineSize, lineGap);
for(int c = 0; c < lines.rows(); c++) {
double[] vec = lines.get(c, 0);
double x1 = vec[0],
y1 = vec[1],
x2 = vec[2],
y2 = vec[3];
// Filtering only verticat and horizontal lines
if(x1 == x2 || y1 == y2) {
// Filtering out short lines
if(Math.abs(x1 - x2) > ignoreLinesShorter || Math.abs(y1 - y2) > ignoreLinesShorter) {
Point start = new Point(x1, y1);
Point end = new Point(x2, y2);
// Draw line
Imgproc.line(source, start, end, new Scalar(0,0,255), 2);
}
}
}
Result:
Like with contours, I'm still not seeing correct rectangles that I could detect. Could you help me with a correct direction? Maybe there is an easier way to perform this task?
For detecting rectangular portraits (headshots), I've had some success with the following methodology.
1. Python code for rectangle detection (It should be easy to convert to Java.)
img = cv2.imread('example.jpg')
gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
# Remove black border by cropping
bw = 6 # border width
ht, wd = img.shape[:2] # height, width
gray = gray[bw:ht-bw, bw:wd-bw]
# HISTOGRAM -- Put histogram function here to determine the following:
bg_color = (235,235,235) # background color
thresh_value = 220
# Add back border with background color
gray = cv2.copyMakeBorder(gray, bw, bw, bw, bw, cv2.BORDER_CONSTANT, value=bg_color)
# Binary Threshold
thresh = cv2.threshold(gray, thresh_value, 255, cv2.THRESH_BINARY)[1] # orig: 235
# Closing Morphological Transformation
kernel = np.ones((5,5),np.uint8)
closing = cv2.morphologyEx(thresh, cv2.MORPH_CLOSE, kernel)
# Invert Image
closing = np.invert(closing)
# Find contours
cnts = cv2.findContours(closing, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
cnts = cnts[0] if len(cnts) == 2 else cnts[1]
# Find portraits by specifying range of sizes and aspect ratios
img_area = ht * wd
for cnt in cnts:
x,y,w,h = cv2.boundingRect(cnt)
if w*h < 0.005*img_area or w*h > 0.16*img_area or h/w < 0.95 or h/w > 1.55:
continue
cv2.rectangle(img, (x, y), (x+w, y+h), (0, 255, 0), 2)
cv2.imshow('Result', img)
cv2.waitKey(0)
Example 1 Result (First image is after inverting.)
Example 2 Result
2. Python code for face detection
def is_headshot(cnt_img):
gray = cv2.cvtColor(cnt_img, cv2.COLOR_BGR2GRAY)
height, width = cnt_img.shape[:2]
min_size = int(max(0.4*width, 0.3*height))
faces = face_cascade.detectMultiScale(gray,
scaleFactor=1.3,
minNeighbors=3,
minSize=(min_size, min_size))
if len(faces) == 1:
return True
else:
return False
face_cascade = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')
x,y,w,h = cv2.boundingRect(cnt) # bounding rectangle of contour found in code above
if is_headshot(img[y:y+h, x:x+w]):
cv2.imwrite('headshot.jpg', img[y:y+h, x:x+w])
3. Python code for portrait validation
The grid structure can be found using code I posted in this stackoverflow question. Loop through the results of the completed grid. Each grid element is defined by (x,y,w,h) where w and h can be the average width and height of portraits found above. Use the function box1.intersection(box2) from shapely.geometry to determine if there are missing or missized portraits. If the intersection area is small or zero, there may be a missing portrait that should then be checked with face detection. I'm open to providing more details if there is any interest.