OpenCV.js contour-based cropping works for right/straight angles but fails when image is tilted left

I’m working on a browser-based image preprocessing task using OpenCV.js. My goal is to isolate a printed number on a black surface, which is typically white text over a dark/black background.

I’ve written a function that:

1. Converts the canvas image to grayscale.
2. Applies adaptive thresholding to highlight dark regions.
3. Finds contours.
4. Selects the largest one.
5. Uses cv.minAreaRect() and cv.getRotationMatrix2D() to deskew the image.
6. Crops the region of interest.
7. Applies Otsu’s binarization after the crop

It works well when the number is upright or tilted slightly to the right, but fails when tilted to the left — it can't straighten it.

export async function contourBasedCrop(inputCanvas: HTMLCanvasElement): Promise<void> {
  if (inputCanvas.width === 0 || inputCanvas.height === 0) {
    console.warn("Canvas is empty, skipping processing.");
    return;
  }
  const canvas = inputCanvas;
  const cv = await getOpenCv();
  const src = cv.imread(canvas);
  const gray = new cv.Mat();
  const binary = new cv.Mat();
  const contours = new cv.MatVector();
  const hierarchy = new cv.Mat();
  let largestContour = null;

  try {
    // Convert to grayscale
    cv.cvtColor(src, gray, cv.COLOR_RGBA2GRAY);

    // Create binary image to detect contours
    cv.adaptiveThreshold(
      gray,
      binary,
      255,
      cv.ADAPTIVE_THRESH_GAUSSIAN_C,
      cv.THRESH_BINARY_INV,
      15,
      2
    );

    const kernel = cv.Mat.ones(3, 3, cv.CV_8U);
    cv.erode(binary, binary, kernel);
    cv.dilate(binary, binary, kernel);
    kernel.delete();

    // Find contours
    cv.findContours(binary, contours, hierarchy, cv.RETR_EXTERNAL, cv.CHAIN_APPROX_SIMPLE);
    let maxArea = 0;
    for (let i = 0; i < contours.size(); i++) {
      const c = contours.get(i);
      const area = cv.contourArea(c);
      if (area > maxArea && area > 1000) {
        maxArea = area;
        if (largestContour) largestContour.delete();
        largestContour = c.clone();
      }
    }

    if (!largestContour) return;

    // Get rotation matrix
    const rotatedRect = cv.minAreaRect(largestContour);
    let { angle } = rotatedRect;
    if (rotatedRect.size.width < rotatedRect.size.height) angle += 90;
    const center = new cv.Point(rotatedRect.center.x, rotatedRect.center.y);
    const M = cv.getRotationMatrix2D(center, angle, 1);

    const rotated = new cv.Mat();
    cv.warpAffine(
      src,
      rotated,
      M,
      src.size(),
      cv.INTER_LINEAR,
      cv.BORDER_CONSTANT,
      new cv.Scalar()
    );

    // Crop the region of interest
    const x = Math.max(0, Math.floor(rotatedRect.center.x - rotatedRect.size.width / 2));
    const y = Math.max(0, Math.floor(rotatedRect.center.y - rotatedRect.size.height / 2));
    const width = Math.floor(rotatedRect.size.width);
    const height = Math.floor(rotatedRect.size.height);
    const safeWidth = Math.min(width, rotated.cols - x);
    const safeHeight = Math.min(height, rotated.rows - y);

    if (safeWidth <= 0 || safeHeight <= 0) return;
    const safeRect = new cv.Rect(x, y, safeWidth, safeHeight);
    const cropped = rotated.roi(safeRect);

    // Apply Otsu binarization after cropping
    const croppedGray = new cv.Mat();
    const croppedBinary = new cv.Mat();
    cv.cvtColor(cropped, croppedGray, cv.COLOR_RGBA2GRAY);
    cv.threshold(croppedGray, croppedBinary, 0, 255, cv.THRESH_BINARY + cv.THRESH_OTSU);
    cv.bitwise_not(croppedBinary, croppedBinary);

    canvas.width = croppedBinary.cols;
    canvas.height = croppedBinary.rows;
    cv.imshow(canvas, croppedBinary);

    cropped.delete();
    croppedGray.delete();
    croppedBinary.delete();
    M.delete();
    rotated.delete();
  } finally {
    gray.delete();
    binary.delete();
    contours.delete();
    hierarchy.delete();
    if (largestContour) largestContour.delete();
  }
}

You're on the right track with using cv.minAreaRect() to deskew the image, but it has a known limitation: it returns angles in the range [-90°, 0°) and can behave inconsistently depending on the aspect ratio of the bounding box. This often leads to incorrect rotations — especially for left-tilted text.

A more robust and consistent way to detect skew angle is to use PCA on the contour points. PCA determines the direction of maximum variance (the "long axis" of the shape), which is a good proxy for the orientation of printed numbers or text.

Here's how you can implement PCA-based angle detection in OpenCV.js:

function computeSkewAngleFromContour(contour: cv.Mat): number {
  const points = [];
  for (let i = 0; i < contour.rows; i++) {
    const pt = contour.intPtr(i);
    points.push([pt[0], pt[1]]);
  }

  const n = points.length;
  const meanX = points.reduce((sum, p) => sum + p[0], 0) / n;
  const meanY = points.reduce((sum, p) => sum + p[1], 0) / n;

  let covXX = 0, covXY = 0, covYY = 0;
  for (const [x, y] of points) {
    const dx = x - meanX;
    const dy = y - meanY;
    covXX += dx * dx;
    covXY += dx * dy;
    covYY += dy * dy;
  }

  covXX /= n;
  covXY /= n;
  covYY /= n;

  const trace = covXX + covYY;
  const det = covXX * covYY - covXY * covXY;
  const eigenvalue1 = trace / 2 + Math.sqrt(4 * covXY * covXY + (covXX - covYY) ** 2) / 2;

  const vx = covXY;
  const vy = eigenvalue1 - covXX;

  const angle = Math.atan2(vy, vx) * (180 / Math.PI);
  return angle;
}

Then in your main function, replace the minAreaRect and angle logic with:

const angle = getOrientationPCA(largestContour);
const center = new cv.Point(src.cols / 2, src.rows / 2);
const M = cv.getRotationMatrix2D(center, angle, 1);

This will properly straighten both left- and right-tilted numbers, since PCA gives a direct, unambiguous orientation based on the actual shape.