mqtt-vernemq

Is there a way to evict vernemq cached auth_on_register, auth_on_publish, auth_on_subscribe hook data from in memory

As per vernemq documents the hook data is stored in in memory cache and is not actively disposed.

We have around 360k clients distributed over cluster of 8 nodes. The client id, username and password do not change and are fixed for 320k clients, where as the rest 40k clients keep changing. These 40k clients also subscribe and publish to at most 3 topics. The clients tend to disconnect and connect back to any node from the cluster once in a day, due to which the hook data is being cached on all the nodes and increasing the memory. The memory keeps increasing on a daily basis, and the memory usage curve has not flattened.

Issue: I fear at some point of time we will get in OOM errors and the nodes can go down.

I tried clearing memory using echo commands (1 2 and 3) but only buff cache memory was cleared and the hook data was not. Is there a way to clear or evict the hook data from the in memory?

Solution

  • Since vernemq is written in erlang, the hook data is stored in Built-in term storage (ETS-Erlang term storage). These provide the ability to store very large quantities of data in an Erlang runtime system, and to have constant access time to the data. Find more details here https://erlang.org/doc/man/ets.html

    Note : Only the owner process can delete the table or all the objects from the table, here the owner process is vernemq broker itself.

    To answer my question, below are the following code changes made to the vernemq source code inorder to evict/ delete all the cache objects(it is mandatory to build the vernemq from source).

    1. There is a command ./vmq-admin webhooks cache show --reset which resets (deletes all objects) from the vmq_webhooks_stats ets table (resets the cache hits,entries,missess).
    2. This command is defined in vmq_webhooks_cli.erl, under cache_stats_cmd() function.
    3. Just replace vmq_webhooks_cache:reset_stats() with vmq_webhooks_cache:purge_all()

    Build the source and start the broker with the updated changes. On invoking ./vmq-admin webhooks cache show --reset command both the hook data and the stats will be deleted.

    This change helped me in solving the OOM issue, which was eventually faced after sometime.

    Update : Cache clear command added in 1.13.0 Command : ./vmq-admin webhooks cache clear