$effect running into infinite loop

I'm trying to migrate from svelte4 to svelte5 and for that I would like to change some $: to $effect. A pattern that I used from time to time looks like that:

let {id} = $props();
let thing = $state();
let cache = new SvelteMap();

$effect = (async () => {
        if (!cache.has(id)) {
                cache.set(id, await get_thing(id));
        }
        thing = chache.get(id);
})

I like to update thing on id change and also whenever the cache entry is updated, because of that I like both to be reactive. However, I change cache (which is reactive) within an $effect, which I often read is a bad thing as it causes infinite loops. Is this a bad pattern? How else can I solve this?

The cache itself does not need to be reactive at all; whether entries exist or not should not affect anything in the UI. The items in the cache can still be reactive on their own.

thing ideally should just be a $derived depending on id, though top level async support is still experimental at this point. It then should just be:

const thing = $derived(await get_thing(id));

Where the function gets the thing from cache/updates the cache if necessary.

For now, using an $effect is probably fine, just use a regular Map or a plain object for the cache.