TL:DR - It was likely a problem with Jemalloc not being turned on. Now that Jemalloc is on, the problem appears to have gone away.
Whenever I run a job to write to a database in MariaDB, it takes RAM to perform the operation. However, even after the job has completed (and I can successfully query the database), the RAM is still being used. This is becoming a problem, because after about 80 jobs, all of the system memory is being held hostage by MariaDB, and I have to restart it. What would cause it to not release the RAM after finishing?
Update:
I recently moved from RHEL 7 to RHEL 8 to utilize column store, and thought that everything was copied exactly. I'm noticing that Jemalloc is no longer on my computer, so that might be part of the problem.
I have modified the my.cnf, but it's a copy of the RHEL 7 file. I have confirmed that the files are identical. I did not have this issue on RHEL 7.
Some other applications are running, but I'm confident in calling MariaDB the villain. While watching MariaDB in htop, I can see the MEM% increase when a database load starts. Back in RHEL 7, I would see it decrease after a job finished. However, I'm not seeing that now. The other applications go about their business as normal. Nothing else consumes an egregious amount of RAM, and the system runs fine until MariaDB builds up too much.
"80 jobs" is a bit of a misnomer. I am not running 80 things at the same time. 80 is just the approximate number after which MariaDB builds up too much, which can take a few days. The jobs definitely are completing. I use Open Grid Scheduler (OGS) as a resource manager for my jobs. I can see them clearing the running jobs state, and entering finished jobs. After that, I can query the databases with mysql, and see the updated results. In RHEL 7, this is also when MariaDB's MEM% would start to decrease. I do not think that they are still connected, although I am open to suggestions for ways of verifying this.
I have plenty of RAM for the amount of dataloads I am performing to not interfere with routine processes and application use. OGS helps ensure this by limiting the number of dataloads that can happen simultaneously. I used this same setup when I had RHEL 7. Based on past usage, I would not expect MariaDB to exceed 10% of my MEM%. I'm not using innodb, I actually use myisam. As such, my.cnf does not have innodb_buffer_pool_size.
For now, I can keep running things while working on debugging. At any time, I can kill and restart MariaDB to free up the memory it was holding. I have been doing this once it reaches about 80% of MEM%.
For what it is worth, variations of the FLUSH statement have done nothing. When I attempt them, I quickly get the message "Query OK, 0 rows affected", and the RAM usage remains the same.
Added snippets to pastebin: https://pastebin.com/SLaWxfjc
I'd like to thank the people who have been helping me. I think I have fixed the issue. When using RHEL 7, I was using jemalloc. When I moved to RHEL 8, it wasn't enabled and I thought it wasn't even installed on the machine. However, I found it lurking about somewhere unexpected and under an assumed name, trying to start a new life of leisure without having to do work. I've now put it back to work (which also involved updating the name in /etc/systemd/system/mariadb.service.d/override.cnf -> Environment=LD_PRELOAD=libjemalloc.so.2), and it appears that MariaDB is releasing RAM upon job completion. So, configuration management, consistency, and carefulness were once again my worst enemies. I'm going to be performing stress testing, but it appears to be working properly for now.
It also happens that what worked for me was suggested as an answer to an older question by user @GABRIEL.
Jemalloc had a slightly different name when I moved to RHEL 8, and my partial copying and partial reinstallation of files from RHEL 7 meant that I had a configuration mismatch. I had to find where Jemalloc now was, as well as updating the name in the following:
/etc/systemd/system/mariadb.service.d/override.cnf
Environment=LD_PRELOAD=libjemalloc.so.2