Physics.OverlapBox being too big

I am doing a rubiks cube simulator. To make the sides turn, I have a collider on each of the sides, and I make the colliders parent everything inside them on click and then just turn the side.

To get every object inside the colliders, I use Physics.OverlapBox, and put every object except the other sides inside a list like this:

public List<GameObject> children = new List<GameObject>();

    private void Awake()
    {
        UpdateCubes();
    }

    void UpdateCubes()
    {
        Mesh mesh = GetComponent<MeshFilter>().mesh;

        children.Clear();

        foreach (Collider child in Physics.OverlapBox(transform.localPosition, 
            Vector3.Scale(mesh.bounds.size, transform.lossyScale) / 2, transform.rotation))
        {
            if (!child.transform.CompareTag("Side"))
            {
                children.Add(child.gameObject);
            }
        }
    }

Here is the problem: It seems like Physics.OverlapBox is way too big, because it gets every piece of the cube and some weird missing gameobjects, as seen here:

I have tried to change transform.localScale / 2 to transform.lossy Scale / 2 but it does'nt work. What should I do?

I got it working! The pieces from other sides were overlapping with other sides because they were so close together. I fixed it by putting more offset between them.