Border around a box not quite central

I'm trying to achieve this.

But as per this JSFiddle https://jsfiddle.net/52pv9srk/ the box isn't quite in the middle, no matter what I do, it always looks off a little bit! Plus I'm struggling to not have them move around on hover.

.attribute-block {
    margin-bottom: 30px;
}

.attribute-block a {
  text-decoration: none;
}
.attribute-colour {
    width: 40px;
    height: 40px;
    display: inline-block;
        text-decoration: none;
}
.attribute-colour:hover, .attribute-colour-selected {
    position: absolute;
    top: 3px;
    left: 3px;
    width: 32px;
    height: 33px;
}
.attribute-colour-hover-border {
    position: relative;
    display: inline-block;
    border: 0;
    width: 40px;
    height: 40px;
}
.attribute-colour-hover-border:hover, .attribute-colour-hover-border-selected {
    position: relative;
    border: 1px solid black;
    width: 40px;
    height: 40px;
    display: inline-block;
}

<p style="margin-bottom: 10px">Colour <strong>Black</strong></p>
<div class="attribute-block">
  <a href="">
    <div class="attribute-colour-hover-border-selected">
      <div class="attribute-colour-selected" style="background: #000000"></div>
    </div>
  </a>
  <a href="">
    <div class="attribute-colour-hover-border">
      <div class="attribute-colour" style="background: #ad2b50"></div>
    </div>
  </a>
  <a href="">
    <div class="attribute-colour-hover-border">
      <div class="attribute-colour" style="background: #406872"></div>
    </div>
  </a>
  <a href="">
    <div class="attribute-colour-hover-border">
      <div class="attribute-colour" style="background: #3a5127"></div>
    </div>
  </a>
</div>

Solution

Your computations are wrong : the parent box is 40x40, the child box is 32x32 and the absolute positionning is 3px top and left.

you could just put 4px for top and left, although I would rather recommand settings the parent box to

display: flex;
justify-content: center;
align-items: center;

Check snippet below:

.attribute-block {
    margin-bottom: 30px;
    display: flex;
    gap: 0.25rem;
}

.attribute-block a {
  text-decoration: none;
}

.attribute-colour-hover-border {
  position: relative;
  border: 0;
  width: 40px;
  height: 40px;
  display: flex;
  align-items: center;
  justify-content: center;
  box-sizing: border-box;
}
.attribute-colour-hover-border:hover {
  border: 2px solid blue;
}
.attribute-colour-hover-border-selected {
  border: 2px solid gold;
}

.attribute-colour {
  width: 40px;
  height: 40px;
  text-decoration: none;
}
.attribute-colour:hover, 
.attribute-colour-selected {
  width: 32px;
  height: 32px;
}

<p style="margin-bottom: 10px">Colour <strong>Black</strong></p>
<div class="attribute-block">
  <a href="">
    <div class="attribute-colour-hover-border attribute-colour-hover-border-selected">
      <div class="attribute-colour attribute-colour-selected" style="background: #000000"></div>
    </div>
  </a>
  <a href="">
    <div class="attribute-colour-hover-border">
      <div class="attribute-colour" style="background: #ad2b50"></div>
    </div>
  </a>
  <a href="">
    <div class="attribute-colour-hover-border">
      <div class="attribute-colour" style="background: #406872"></div>
    </div>
  </a>
  <a href="">
    <div class="attribute-colour-hover-border">
      <div class="attribute-colour" style="background: #3a5127"></div>
    </div>
  </a>
</div>