How to Scale Up a div on hover and Scale Down all other divs

I've been trying to figure this out forever now but without success.

Below are 6 circular divs. When a user hovers over one of them, I want the hovered div to expand a bit (scale up) and all other divs to shrink a bit (scale down).

I tested the following:

#container div:hover {
  transform: scale(1.1);
}

#container div:not(:hover) {
  transform: scale(0.9);
}

However, this keeps all the divs scaled at 0.9 (I want them to only shrink when one of the other divs are hovered.

I also tested the following:

#container div:hover {
  transform: scale(1.1);
}

#container:hover div:not(:hover) {
  transform: scale(0.9);
}

However, since this shrinks all the divs when the entire container/parent element is hovered, not the individual divs, this solution doesn't work either.

Basically, when a circle (and only that circle) is hovered, I want it to expand and all others to shrink.

I'd very much prefer a pure CSS solution but I'm open to jQuery.

I also have attached a jsfiddle, this is why using hover on a container doesn't work: https://jsfiddle.net/e5vjL7k7/

Make class to scale-up and scale-down and on hover and add scale-up class to element that is hovered and scale-down class to its siblings.

$("li").hover(function() {
  $(this).toggleClass('scale-up').siblings().toggleClass('scale-down')
})

ul {
  list-style: none;
}
li {
  width: 50px;
  height: 50px;
  border: 1px solid black;
  display: inline-block;
  transition: all 0.3s ease-in;
  border-radius: 50%;
  margin: 10px;
}
.scale-up {
  transform: scale(1.2);
}
.scale-down {
  transform: scale(0.8);
}

<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
<ul>
  <li></li>
  <li></li>
  <li></li>
  <li></li>
  <li></li>
  <li></li>
</ul>