Detect multiple circles in an image

I am trying to detect the count of water pipes in this picture. For this, I am trying to use OpenCV and Python-based detection. The results, I am getting is a little confusing to me because the spread of circles is way too large and inaccurate.

The code

import numpy as np
import argparse
import cv2

# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-i", "--image", required = True, help = "Path to the image")
args = vars(ap.parse_args())

# load the image, clone it for output, and then convert it to grayscale
image = cv2.imread(args["image"])
output = image.copy()
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

#detect circles in the image
#circles = cv2.HoughCircles(gray, cv2.HOUGH_GRADIENT, 1.2, param1=40,minRadius=10,maxRadius=35)
circles = cv2.HoughCircles(gray, cv2.HOUGH_GRADIENT, 8.5,70,minRadius=0,maxRadius=70)

#print(len(circles[0][0]))
# ensure at least some circles were found
if circles is not None:
    # convert the (x, y) coordinates and radius of the circles to integers
    circles = np.round(circles[0, :]).astype("int")
    # count = count+1   

    # print(count) 

    # loop over the (x, y) coordinates and radius of the circles
    for (x, y, r) in circles:
        # draw the circle in the output image, then draw a rectangle
        # corresponding to the center of the circle
        cv2.circle(output, (x, y), r, (0, 255, 0), 4)
        cv2.rectangle(output, (x - 5, y - 5), (x + 5, y + 5), (0, 128, 255), -1)

    # show the output image
   # cv2.imshow("output", np.hstack([output]))
    cv2.imwrite('output.jpg',np.hstack([output]),[cv2.IMWRITE_JPEG_QUALITY, 70])
    cv2.waitKey(0)

After I run this, I do see a lot of circles detected, however, the results are complete haywire. My question is, how do I improve this detection. Which parameters are specifically needed to optimize in the HoughCircles method to achieve greater accuracy? Or, should I take the approach of annotating hundreds of similar images via bounding boxes and then train them over a full-blown CNN like Yolo to perform detection?

Taking the approach mentioned in answer number 2 from here Measuring the diameter pictures of holes in metal parts, photographed with telecentric, monochrome camera with opencv . I got this output. This looks close to performing a count but misses on lot of actual pipes during the brightness transformation of the image.

You could do an adaptive threshold as preprocessing. This basically looks for areas that are relatively brighter than the neighboring pixels, your global threshold loses some of the pipes, this keeps them a little better.

import cv2
import matplotlib.pyplot as plt
import numpy as np

img = cv2.imread('a2MTm.jpg')
blur_hor = cv2.filter2D(img[:, :, 0], cv2.CV_32F, kernel=np.ones((11,1,1), np.float32)/11.0, borderType=cv2.BORDER_CONSTANT)
blur_vert = cv2.filter2D(img[:, :, 0], cv2.CV_32F, kernel=np.ones((1,11,1), np.float32)/11.0, borderType=cv2.BORDER_CONSTANT)
mask = ((img[:,:,0]>blur_hor*1.2) | (img[:,:,0]>blur_vert*1.2)).astype(np.uint8)*255

plt.imshow(mask)

You can then carry on with the same post processing steps.

Here are some example processing steps:

circles = cv2.HoughCircles(mask,
                           cv2.HOUGH_GRADIENT,
                           minDist=8,
                           dp=1,
                           param1=150,
                           param2=12,
                           minRadius=4,
                           maxRadius=10)
output = img.copy()
for (x, y, r) in circles[0, :, :]:
  cv2.circle(output, (x, y), r, (0, 255, 0), 4)

You can adjust the parameters to get what you would like, read about the parameters here.