c++vectortypesmpiderived-types

MPI: Sending a non-contiguous part of an array of objects with arbitrary sized vectors


I'm trying to learn MPI and am trying to develop a C++ program where I need to send a bunch of objects with arbitrary sized vectors.

Let the class be defined as

class MyClass
{
    int id;
    int a[3];
    vector<int> b;
}

I'm able to use MPI_Type_create_struct along with MPI_Type_create_resized in order to send one object from rank 0 to rank 1 as such:

int size = 5;
b.resize(size);
MPI_Datatype mytype;
inr lengths[3] = {1,3,size};
MPI_Aint disp[3];
MyClass obj;
MPI_Aint base, lb, extent;

MPI_Get_address(&obj, &base);
MPI_Get_address(&obj.id, &disp[0]);
MPI_Get_address(&obj.a, &disp[1]);
MPI_Get_address(&obj.b, &disp[2]);

for(int i=0; i<3; i++)
   disp[i] = MPI_Aint_diff(disp[i], base);

MPI_Datatype types[3] = {MPI_INT, MPI_INT, MPI_INT};
MPI_Type_create_struct(4, lengths, disp, types, &mytype);
MPI_Type_get_extent(mytype, &lb, &extent);
MPI_Type_create_resized(mytype, lb, sizeof(MYClass), &mytype);
MPI_Type_commit(&mytype);

Now, If I have an array of objects MyClass obj_arr[10] and I want to send a non-contiguous part of this array, I'm able to achieve this using MPI_Type_create_indexed_block. This of course works only if the size of the vector (size) is constant across all the objects. The problem is in my case, the size is not uniform across all the objects. I'm not sure what the right approach here is. Any help would be greatly appreciated.

My current solution is to create a custom MPI Datatype for each indivdiual object, and send it to the appropriate rank. This would remove the need for MPI_Type_create_indexed_block, but I'm guessing this approach would be very inefficient due to the potentially large number of datatype creations and sends/recvs. Since I'm new to MPI, I'm wondering if there is a better solution using a different approach I'm not aware of?


Solution

  • A C++ vector is a small block, containing the size, and the address of the actual data. So with

    MPI_Get_address(&obj.b, &disp[2]);
    

    you get the address of that small block. Instead you should get the address of obj.b.data().

    Next, you can not do this with an MPI_Type_Struct because the data of MyClass that you want to send is not contiguous. You could use MPI_Pack to pack the id,a and actual vector elements, plus an int describing how many elements needs to be unpacked at the other side.

    Finally, create_resized is the wrong tool here. Don't use that in this case.