css :not(:hover) bugging keyframes

I'm having this issue that my not-on-hover animation returns to its normal state only once, after that it's bugged as shown in the gif below, and does not work correctly. I was thinking that there was an alternative to the inverse of on hover instead of what I'm using right now but no luck finding anything. Any advice?

.expanding-button {
  width: 7rem;
  height: 7rem;
  background-color: blue;
  color: white;
  border-radius: 50%;
  cursor: pointer;
  transition: all 0.5s ease-in-out;
}

.button__img {
  width: 5rem;
  height: auto;
  transition: all 1s ease-in-out;
}



.expanding-button:hover .button__img {
  animation-delay: 1s;
  animation: spin 1s forwards, move-left 1s forwards;
  transform-origin: center left;
}



.expanding-button:hover {
  background-color: darkblue;
  width: 700px;
  height: 100px;
  border-radius: 50px;
}

.expanding-button:not(:hover) .button__img {
  animation: move-right 0.5s forwards;
}

@keyframes spin {
  0% {
    transform: rotate(0deg);
  }

  100% {
    transform: rotate(360deg);
  }
}

@keyframes move-left {
  0% {
    transform: translateX(0);
  }


  100% {
    transform: translateX(-18rem);
  }
}

@keyframes move-right {
  0% {
    transform: translateX(-18rem);
  }

  100% {
    transform: translateX(0);
  }
}

Simply use transitions:

.expanding-button {
  width: 7rem;
  height: 7rem;
  background-color: blue;
  color: white;
  border-radius: 50%;
  cursor: pointer;
  position: relative;
  transition: all 0.5s ease-in-out;
}

.button__img {
  width: 5rem;
  height: auto;
  position: absolute;
  left: 5px;
  transition: all .5s ease-in-out;
}



.expanding-button:hover .button__img {
  transform: rotate(360deg);
  transform-origin: center left;
}

.expanding-button:hover {
  background-color: darkblue;
  width: 700px;
  height: 100px;
  border-radius: 50px;
}

Fix the left and top position your self as you desire. Also you can use flex instead of position and it will preserve your context as before but it requires changes to your html too.