I am using ecCodes library in my project, and I have encountered an issue that memory is not freed between reading the files.
The minimal example representing the problem is this (and is basically a combination of those two library API usage examples [1](https://confluence.ecmwf.int/display/ECC/grib_get_keys) [2]:
#include <string>
#include <vector>
#include <iostream>
#include "eccodes.h"
int main() {
std::string filenames[] = {"../data/era5_model.grib", "../data/era5_model2.grib", "../data/era5_model3.grib",
"../data/era5_model4.grib"};
std::vector<long> vec = {};
for (auto & filename : filenames) {
FILE* f = fopen(filename.c_str(), "r");
int err = 0;
codes_handle* h;
while ((h = codes_handle_new_from_file(nullptr, f, PRODUCT_GRIB, &err)) != nullptr) {
long k1 = 0;
err = codes_get_long(h, "level", &k1);
vec.push_back(k1);
}
codes_handle_delete(h);
fclose(f);
}
std::cout << vec[52];
return 0;
}
In the example the program reads 4 identical ERA5 files, each of size 1.5GB. Before opening new file previous one is closed with codes_handle_delete() and fclose(). Therefore, the expected behaviour would be for the memory usage to stay at about 1.5GB. However, in reality the memory usage steadily increases to about 6.5GB and is freed when program closes (see screenshot below).
This particular example has been run on CLion with CMake (Release configuration), but the issue occurs with every other configuration and also in my other Rust project which calls ecCodes with FFI.
The library seems well tested and supported so it seems unlikely that it is a library bug. Therefore, is that an expected behaviour or is my code wrong? If the latter, how can I correct it?
I am using Ubuntu 21.04 and ecCodes 2.20.0 installed with apt
So I contacted the library authors and realized that I have not read this example carefully enough.
For the ecCodes to correctly free the memory codes_handle
should be deleted every time it is created (analogically to how you should free the memory every time you alloc it). Therefore in my example codes_handle_delete()
should be INSIDE the while
loop:
while ((h = codes_handle_new_from_file(nullptr, f, PRODUCT_GRIB, &err)) != nullptr) {
long k1 = 0;
err = codes_get_long(h, "level", &k1);
vec.push_back(k1);
codes_handle_delete(h);
}
After that change memory usage is almost unnoticeable.