Azure API Management external Azure redis cache, cache miss when key set outside APIM

I have an Azure APIM that interacts with an external Azure Redis Cache. No matter how I set keys, via the console/cli or via the .NET SDK, APIM reports a cache miss in the trace. Example code below:

<cache-lookup-value key="@("ClientID:" + context.Variables["ClientId"])" variable-name="IsGreen" caching-type="external" />
<choose>
    <when condition="@(context.Variables.GetValueOrDefault<string>("IsGreen") == "true")">
        ...
    </when>
    <otherwise>
        ...
    </otherwise>
</choose>

cache-lookup-value (2.803 ms)
{
    "message": "Cache lookup resulted in a miss, variable will not be set.",
    "key": "2_ClientID:1",
    "variableName": "IsGreen"
}

However, in testing, I added to the cache via APIM and it then gets a cache hit. I have listed the keys via the console and debugged them in .NET and they look the same regardless of what interface sets the key:

<!-- This will result in cache hit next request -->
<cache-store-value key="ClientID:1" value="true" duration="1440" caching-type="external" />

I am almost certain this described issue here is the same thing I am experiencing.

A weird occurrence is the Redis Cache metrics show all cache hits which makes me think APIM is doing some additional logic after the fact to result in the false miss.

In .NET you can see where there are weird bytes prepended onto the value.

Redis itself allows for storing sequence of bytes only, but APIM allows you to store typed values like int, string, or certain objects even. That value prefix is used by APIM to identify type of value you're storing in cache. If APIM reads value from cache but can't parse it in expected way it will generate cache miss. Generally, value encoding syntax is not documented and if you were to reverse-engineer it and take dependency on the implementation your solution will very likely get broken when/if that syntax is updated, which may happen without any notice.

To manipulate values in APIM cache, even if external, it is advised to create an API in APIM, with operations to set and delete cache values, use APIM policies to implement those. And use that API to manage cache used by APIM.

Feel free to file feature request at https://aka.ms/apimwish