hdfql

HDFql sequential writing to the structure dataset


I am trying to write a simple structure in hdf5 But I still can't add records to the already created dataset. The task itself is to store a set of bytes of arbitrary length and its size

void WriteStructToFile(vector<vector<char>> fl_stream)
{
    struct data
    {
        int size_array;
        char * value_array;
    };
    int status;
    string  query;
    string FileName = "Fly_stream_file.h5";
    string NameGroup = "Stream";
    string Namedataset = "Inclusions";

 //------------------------------- Created File and Dataset -------------
    query = "CREATE TRUNCATE FILE " + FileName;
    status = HDFql::execute(query.c_str());
    query = "USE FILE " + FileName;
    status = HDFql::execute(query.c_str());
    query = "CREATE GROUP " + NameGroup;
    status = HDFql::execute(query.c_str());
    query = "CREATE TRUNCATE CHUNKED DATASET "+ Namedataset +" AS COMPOUND(size_array AS INT, value_array AS VARCHAR)(UNLIMITED)";
    stringstream scriptst;
    scriptst <<"CREATE TRUNCATE CHUNKED DATASET "<< Namedataset <<" AS COMPOUND("<< "size_array AS INT OFFSET " << offsetof(struct data, size_array)<<" ,"<<" value_array AS VARCHAR OFFSET "<< offsetof(struct data, value_array) <<")(UNLIMITED)"<< " SIZE "<< sizeof(struct data);
    status = HDFql::execute(scriptst);
    //clear request
    scriptst.str(std::string());
    scriptst.clear();

//--------------- Fill Dataset ----------------
    //we simulate the arrival of data in a function 
    
    
    struct data realdata;
    int number = HDFql::variableRegister(&realdata);
    for (int i = 0;i < fl_stream.size();i++)
    {

        realdata.size_array = fl_stream[i].size();
        realdata.value_array = fl_stream[i].data();

        //write data to the dataset
        scriptst << "INSERT INTO " << Namedataset << "(-1) VALUES FROM MEMORY " << number<< " SIZE " << sizeof(struct data) << " OFFSET(" << offsetof(struct data, size_array) << ", "
<< offsetof(struct data,value_array) << ")";
        status = HDFql::execute(scriptst);

        HDFql::execute("ALTER DIMENSION Inclusions TO +1");
        scriptst.str(std::string());
        scriptst.clear();
    }
    HDFql::variableUnregister(&realdata);
status = HDFql::execute("CLOSE FILE");
}

the result of the corrected code


Solution

  • The script that creates the dataset has a syntax error and lacks the specification of members' offsets and the compound's size. It should be as follows:

    sprintf(script, "CREATE TRUNCATE CHUNKED DATASET Inclusions AS COMPOUND(sizear AS INT OFFSET %d, value AS VARCHAR OFFSET %d)(UNLIMITED) SIZE %d", offsetof(struct data, sizear), offsetof(struct data, value), sizeof(struct data));
    status = HDFql::execute(script);
    

    Also, the second writing that you do to dataset Inclusions should use an hyperslab/point selection (otherwise, previously written data will be overwritten). Therefore, do the following:

    sprintf(script, "INSERT INTO Inclusions[-1] VALUES FROM MEMORY %d SIZE %d OFFSET(%d, %d)", number, sizeof(struct data), offsetof(struct data, sizear), offsetof(struct data, value));
    hdfql_execute(script);
    

    Moreover, change the data type of member value of struct data to char *value; instead.

    Finally, to make the code simpler, remove the call to function c_str() from string query as you can pass it directly to function HDFql::execute().