Find number of valid squares from the provided number (>=4) of coordinates

I am trying to solve this question in O(N^3) time complexity but unable to get the right answer in C++. The question states: Design a function that takes in >=4 points and returns the number of squares that can be formed (i.e how many groups of four points (x, y) coordinates within the input points form a square). The complexity should be O(N^3). Below is my attempt at doing this problem in O(N^3) time but I got the wrong answer. I provided an input:

vector<Point> points = {
    {0, 0}, {1, 1}, {0, 1}, {1, 0}, // square 1
    {100, 5}, {50, 95},
    {2, 0}, {3, 0}, {2, 1}, {3, 1}  // square 2

};

I should be getting 2 as the right answer. But I am getting 17. I am storing the coordinates in an unordered_set:

int countSquares(const vector<Point>& points) {
unordered_set<string> pointSet;
for (const auto& p : points) {
    pointSet.insert(hashPoint(p));
}

int N = points.size();
set<string> uniqueSquares;  // Using set for automatic deduplication

This is my code that is iterating over all of the coordinates that I have stored in an unordered_set:

for (int i = 0; i < N; ++i) {
    for (int j = i + 1; j < N; ++j) {
        for (int k = j + 1; k < N; ++k) {
            Point A = points[i];
            Point B = points[j];
            Point C = points[k];

            auto trySquare = [&](Point a, Point c) {
                double cx = (a.first + c.first) / 2.0;
                double cy = (a.second + c.second) / 2.0;

                int vx = c.first - a.first;
                int vy = c.second - a.second;

                Point b = { static_cast<int>(cx + vy / 2.0), static_cast<int>(cy - vx / 2.0) };
                Point d = { static_cast<int>(cx - vy / 2.0), static_cast<int>(cy + vx / 2.0) };

                if (pointSet.count(hashPoint(b)) && pointSet.count(hashPoint(d))) {
                    vector<Point> square = {a, b, c, d};
                    sort(square.begin(), square.end()); // Sort points to prevent duplicate squares
                    string squareHash = hashPoint(square[0]) + hashPoint(square[1]) + 
                                        hashPoint(square[2]) + hashPoint(square[3]);
                    uniqueSquares.insert(squareHash); // Deduplicate squares
                }
            };

            trySquare(A, C);
            trySquare(B, C);
            trySquare(A, B);
        }
    }
}

return uniqueSquares.size();

}

I am using the formula

B = (cx - (-vy)/2, cy - vx/2) = (cx + vy/2, cy - vx/2)  
D = (cx + (-vy)/2, cy + vx/2) = (cx - vy/2, cy + vx/2)

to calculate the points b and d. vx, vy are the vectors we get when we rotate 90 degree counter clockwise from the midpoint of the diagonal of the points we are iterating with. Is there a better algorithm to solve this problem in O(N^3) time?

The main issue is that your cx and cy (the center point around which you want to turn) could be non-integers (with halves), so that your conversion to integer leads to wrong results.

It would be better to use the two chosen points as the vertices of one of the square's edges, and then use vx and vy to derive where the vertices of the parallel edge would be. Actually there would be two possible parallels for a given edge, so to get both of them, visit all pairs in both their orders.

Not the issue, but it is overkill to have three loops and then still check pairs of two separately. Just use two loops to get all pairs you want to check. This also avoids the need for an inline function:

    for (Point a : points) {
        for (Point b : points) { // Visit pairs in both directions
            if (a == b) continue; // Avoid counting squares with 0 surface

            int vx = b.first - a.first;
            int vy = b.second - a.second;

            // Get the vertices of a parallel edge of the square:
            Point c = { b.first - vy, b.second + vx };
            Point d = { a.first - vy, a.second + vx };

            if (pointSet.count(hashPoint(c)) && pointSet.count(hashPoint(d))) {
                std::vector<Point> square = {a, b, c, d};
                std::sort(square.begin(), square.end());
                std::string squareHash = hashPoint(square[0]) + hashPoint(square[1]) + 
                                         hashPoint(square[2]) + hashPoint(square[3]);
                uniqueSquares.insert(squareHash);
            }
        }
    }