Text edge zigzag effect removal (OR finding the dominant color for a image region)

My goal is to draw the text bounding boxes for the following image. Since the two regions are colored differently, so this should be easy. I just need to select the pixels that match a certain color values to filter out the other text region and run a convex hull detection.

However, when I zoom in the image, I notice that the text regions has the zig-zag effect on the edges, so I'm not able to easily find the two color values (for the blue and green) from the above image.

I wonder is there a way to remove the zig-zag effect to make sure each phrase is colored consistently? Or is there a way to determine the dominant color for each text region?

The anti-aliasing causes the color to become lighter (or darker if against a black background) so you can think of the color as being affected by light. In that case, we can use light-invariant color spaces to extract the colors.

So first convert to hsv since it is a light invariant colorspace. Since the background can be either black or white, we will filter out them out (if the bg is always white and the text can be black you would need to change the filtering to allow for that).

I took the saturation as less than 80 as that will encompass white black and gray since they are the only colors with low saturation. (your image is not perfectly white, its 238 instead of 255 maybe due to jpg compression)

Since we found all the black, white and gray, the rest of the image are our main colors, so i took the inverse mask of the filter, then to make the colors uniform and unaffected by light, set the Saturation and Value of the colors to 255, that way the only difference between all the colors will be the hue. I also set bg pixels to 0 to make it easier for finding contours but thats not necissary

After this you can use whatever method you want to get the different groups of colors, I just did a quick histogram for the hue values and got 3 peaks but 2 were close together so they can be bundled together as 1. You can maybe use peak finding to try to find the peaks. There might be better methods of finding the color groups but this is what i just thought of quickly.

hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)
mask = hsv[:,:,1] < 80 # for white, gray & black
hsv[mask] = 0 # set bg pixels to 0
hsv[~mask,1:] = 255 # set fg pixels saturation and value to 255 for uniformity

colors = hsv[~mask]
z = np.bincount(colors[:,0])
print(z)

bgr = cv2.cvtColor(hsv, cv2.COLOR_HSV2BGR)
cv2.imshow('bgr', bgr)