Consecutive same color cells should be in same column, gridbox

In my current project I have huge number of sections, each section has one or two rows ranging from 2 to 15 columns, of equal widths. The same colour cells are placed in same column. The positioning of a section is as shown below:

To achieve the above layout, I am using gridbox. I came close to the solution but the second row is kind of aligning to the left always.

.container {
  display: grid;
  width: 100%;
  grid-template-columns: repeat(auto-fit, minmax(0px, 1fr));
  gap: 5px 10px;
  background-color: steelblue;
}

.field {
  background-color: black;
  box-shadow: inset 0px 0px 2px #ffffff;
  height: 20px;
  color: white;
  text-align: center;
}

.second-row {
  grid-row-start: 2;
}


/* colours */

.colour-black {
  background: black;
}

.colour-blue {
  background: blue;
}

.colour-yellow {
  background: yellow;
}

.colour-red {
  background: red;
}

.colour-orange {
  background: orange;
}

.colour-purple {
  background: purple;
}

.colour-green {
  background: green;
}

.colour-brown {
  background: brown;
}

.colour-violet {
  background: violet;
}

<div class="container">
  <div class="field colour-black">1</div>
  <div class="field colour-blue">2</div>
  <div class="field colour-blue second-row">3</div>
  <div class="field colour-green">4</div>
  <div class="field colour-brown">5</div>
  <div class="field colour-orange">6</div>
  <div class="field colour-purple">7</div>
  <div class="field colour-red">8</div>
  <div class="field colour-red second-row">9</div>
  <div class="field colour-violet">10</div>
</div>

Somethings to note:

Same colour cells will be max to 2 only.
Same colour cells are always consecutive.
Only CSS solution. Can't do HTML changes other than adding classes.
Can't use JavaScript because that will be huge effort for my project and can't afford now.
Non-grid solution is also welcome.

Solution

You only need to add:

grid-auto-flow: column; to the container
grid-row: 1; to field

And it will work:

.container {
  display: grid;
  width: 100%;
  grid-template-columns: repeat(auto-fit, minmax(0px, 1fr));
  gap: 5px 10px;
  background-color: steelblue;
  grid-auto-flow: column; /* added */
}

.field {
  background-color: black;
  box-shadow: inset 0px 0px 2px #ffffff;
  height: 20px;
  color: white;
  text-align: center;
  grid-row: 1;  /* added */
}

.second-row {
  grid-row-start: 2;
}


/* colours */

.colour-black {
  background: black;
}

.colour-blue {
  background: blue;
}

.colour-yellow {
  background: yellow;
}

.colour-red {
  background: red;
}

.colour-orange {
  background: orange;
}

.colour-purple {
  background: purple;
}

.colour-green {
  background: green;
}

.colour-brown {
  background: brown;
}

.colour-violet {
  background: violet;
}

<div class="container">
  <div class="field colour-black">1</div>
  <div class="field colour-blue">2</div>
  <div class="field colour-blue second-row">3</div>
  <div class="field colour-green">4</div>
  <div class="field colour-brown">5</div>
  <div class="field colour-orange">6</div>
  <div class="field colour-purple">7</div>
  <div class="field colour-red">8</div>
  <div class="field colour-red second-row">9</div>
  <div class="field colour-violet">10</div>
</div>