Angular interceptor stopping all http requests

I have an interceptor that is supposed to refresh an access token if it's expired and continue the request with the new token, but its just looping a lot (not infinitely, something is stopping it after 100+ loops) and I can't seem to figure out why, this doesn't happen when token is not expired.

let auth = inject(AuthService);
if (!auth.AccessExpired()) return next(req);
return auth.refresh().pipe(
  catchError(() => next(req)),
  switchMap((res) => {
    var response = res as RefreshResponse;
    const newReq = req.clone({
      headers: req.headers.set("Authorization", `Bearer ${response.access_token}`)
    })
    return next(newReq);
  })
)

This is the refresh function:

refresh() {
  return this.http.post<RefreshResponse>(this.baseUrl + "refresh", {}, {
    headers: new HttpHeaders().set("refresh_token", this.refreshToken ?? "")
  }).pipe(
    tap(
      res => {
        console.log('responded' + res);
        this.accessToken = res.access_token
      }
    ),
    retry(2),
    catchError((err) => {
      console.log("errored", + err)
      this.logout();
      this.noti.showError("Your session expired, log in again.", 5000);
      return throwError(() => new Error(err));
    })
  )
}

As @Andrei says, the condition in the interceptor can be some like

let auth = inject(AuthService);
if (!auth.AccessExpired() || reg.Headers.has('refresh_token') return next(req);
....rest of your code..