Get snapshot of keys in Caffeine

I have a simple registry implemented as Caffeine cache:

class LocalFactorsRegistry(
    duration: Duration,
) : FactorsRegistry {
    private val factors =
        Caffeine
            .newBuilder()
            .expireAfterWrite(duration)
            .build<String, Boolean>()

    override fun getUsedFactors(): Set<String> = factors.asMap().keys

    override fun markFactorAsUsed(factorId: String) {
        factors.put(factorId, true)
    }
}

I try to test if item is evicted:

    @Test
    fun `should mark factor as used and wait for expiration`() {
        val factory = LocalFactorsRegistry(Duration.ofMillis(100))

        factory.getUsedFactors() shouldBe emptySet()

        factory.markFactorAsUsed("factor2")

        Thread.sleep(100)

        factory.getUsedFactors() shouldBe emptySet()
    }

However tests fails on last assertion:

java.util.NoSuchElementException
    at com.github.benmanes.caffeine.cache.BoundedLocalCache$EntryIterator.nextKey(BoundedLocalCache.java:3754)
    at com.github.benmanes.caffeine.cache.BoundedLocalCache$KeyIterator.next(BoundedLocalCache.java:3350)
    at kotlin.collections.CollectionsKt___CollectionsKt.toList(_Collections.kt:1323)
    at io.kotest.assertions.print.IterablePrint.print(IterablePrint.kt:6)
    at io.kotest.assertions.print.IterablePrint.print(IterablePrint.kt:3)
    at io.kotest.assertions.print.PrintKt.print(Print.kt:48)
    at io.kotest.assertions.eq.IterableEq.generateError(IterableEq.kt:221)
    at io.kotest.assertions.eq.IterableEq.checkSetEquality(IterableEq.kt:77)
    at io.kotest.assertions.eq.IterableEq.equals(IterableEq.kt:65)
    at io.kotest.assertions.eq.EqKt.eq(Eq.kt:51)
    at io.kotest.assertions.eq.EqKt.eq(Eq.kt:66)
    at io.kotest.matchers.ShouldKt$equalityMatcher$1.test(should.kt:73)
    at io.kotest.matchers.ShouldKt.invokeMatcher(should.kt:38)
    at io.kotest.matchers.ShouldKt.should(should.kt:33)
    at io.kotest.matchers.ShouldKt.shouldBe(should.kt:17)
    at com.example.LocalFactorsRegistryTest.should mark factor as used and wait for expiration(AbstractFactorsRegistryTest.kt:35)
    at java.base/java.lang.reflect.Method.invoke(Method.java:580)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)

I wonder how I can get an immutable snapshot of keys in the caffeine cache to assert it properly.

As mentioned by @ben-manes cleanUp() needs to be called prior factors.asMap().keys