[SOLVED] RDMA WRITE: ibv_wc.byte_len does not match what I try to write

RDMA WRITE: ibv_wc.byte_len does not match what I try to write

I'm an Infiniband/RDMA newbie playing with RDMA on Mellanox Infiniband hardware. I used the source code here. The program runs pretty normal at first glance: The server writes some bytes to the client memory using RDMA_WRITE operation. However, I found later that the number of bytes transferred, which is indicated by the write completion structure(ibv_wc) does not match the amount of data I put in the ibv_send_wr structure. Here is the code for ibv_send_wr initialization in rdma_write():

ctx->sge_list.addr = (uintptr_t)ctx->buf;
ctx->sge_list.length = ctx->size; //which is 65536
ctx->sge_list.lkey = ctx->mr->lkey;
ctx->wr.wr.rdma.remote_addr = data->remote_connection->vaddr;
ctx->wr.wr_id = RDMA_WRID;
ctx->wr.sg_list = &ctx->sge_list;
ctx->wr_num_sge = 1;
ctx->wr.opcode = IBV_WR_RDMA_WRITE;
ctx->wr.send_flags = IBV_SEND_SIGNALED;
ctx->wr.next = NULL;

Then I read ibv_wc.byte_len after I get a write completion entry in the write completion queue. It shows random numbers: 32537, 32743, 32533. I assume the transfer is successful because ibv_wc.status is equal to IBV_WC_SUCCESS. Did I do something wrong?

Solution

According to ibv_poll_cq() page at RDMAMojo blog, the byte_len field is only valid for a small subset of operations, so it is left uninitialized in your case of an RDMA write operation on the send queue.

byte_len: The number of bytes transferred. Relevant if the Receive Queue for incoming Send or RDMA Write with immediate operations. This value doesn't include the length of the immediate data, if such exists. Relevant in the Send Queue for RDMA Read and Atomic operations.