c++cgribgrib-apieccodes

ecCodes (grib reading library) does not free the memory


I am using ecCodes library in my project, and I have encountered an issue that memory is not freed between reading the files.

The minimal example representing the problem is this (and is basically a combination of those two library API usage examples [1](https://confluence.ecmwf.int/display/ECC/grib_get_keys) [2]:

#include <string>
#include <vector>
#include <iostream>
#include "eccodes.h"

int main() {

    std::string filenames[] = {"../data/era5_model.grib", "../data/era5_model2.grib", "../data/era5_model3.grib",
                               "../data/era5_model4.grib"};

    std::vector<long> vec = {};

    for (auto & filename : filenames) {
        FILE* f = fopen(filename.c_str(), "r");
        int err = 0;
        codes_handle* h;

        while ((h = codes_handle_new_from_file(nullptr, f, PRODUCT_GRIB, &err)) != nullptr) {
            long k1 = 0;
            err = codes_get_long(h, "level", &k1);
            vec.push_back(k1);
        }

        codes_handle_delete(h);
        fclose(f);
    }

    std::cout << vec[52];

    return 0;
}

In the example the program reads 4 identical ERA5 files, each of size 1.5GB. Before opening new file previous one is closed with codes_handle_delete() and fclose(). Therefore, the expected behaviour would be for the memory usage to stay at about 1.5GB. However, in reality the memory usage steadily increases to about 6.5GB and is freed when program closes (see screenshot below).

screenshot showing memory usage

This particular example has been run on CLion with CMake (Release configuration), but the issue occurs with every other configuration and also in my other Rust project which calls ecCodes with FFI.

The library seems well tested and supported so it seems unlikely that it is a library bug. Therefore, is that an expected behaviour or is my code wrong? If the latter, how can I correct it?

I am using Ubuntu 21.04 and ecCodes 2.20.0 installed with apt


Solution

  • So I contacted the library authors and realized that I have not read this example carefully enough.

    For the ecCodes to correctly free the memory codes_handle should be deleted every time it is created (analogically to how you should free the memory every time you alloc it). Therefore in my example codes_handle_delete() should be INSIDE the while loop:

    while ((h = codes_handle_new_from_file(nullptr, f, PRODUCT_GRIB, &err)) != nullptr) {
        long k1 = 0;
        err = codes_get_long(h, "level", &k1);
        vec.push_back(k1);
        codes_handle_delete(h);
    }   
    

    After that change memory usage is almost unnoticeable.