infinibandrdma

What is the logic behind setting valid region half the size of PSN range?

In order to make it possible for the responder to distinguish duplicate packets from out of order packets, a given send queue shall have a series of PSNs no greater than 8,388,608 outstanding at any given time. Therefore, a send queue shall have no more than 8,388,608 packets outstanding at any given time. This includes the sum of all SEND request packets plus all RDMA WRITE request packets plus all ATOMIC Operation request packets plus all expected RDMA READ response packets. Thus, the PSN space (consisting of a range of 16,777,216 PSNs) is divided into two regions, each occupying a range of 8,388,608 PSNs, called the valid region and the invalid region.

As I quoted from IBTA spec, why is it not possible to distinguish duplicate from the out-of-order packets if the valid region is bigger than half the size of the 2^24-sized PSN region?

enter image description here

Solution

  • Imagine that the total PSN range was smaller, to simplify the example, say 0..3. The valid region would be 2 packets if we follow the spirit of the spec, which would include the expected PSN and 1 previous duplicate PSN, but let's say we increase it to 3 packets.

    Take a look at the two scenarios below:

    Out of order scenario

    Sender sends  | Receiver sends
Send 0        | Ack 0
Send 1 (lost) |
Send 2 (lost) |
Send 3        | ?

    After the receiver receives Send 0, the expected PSN is 1. When the receiver gets the 4th packet it is an out-of-order packet, more advanced than the expected PSN by 2. The responder should treat this as a sequence error.

    Duplicate scenario

    Sender sends     | Receiver receives | Receiver sends
Send 3           | Send 3            | Ack 3 (lost)
Send 3 (delayed) |                   |
Send 0           | Send 0            | Ack 0
                 | Send 3 (delayed)  | ?

    Here the sender retransmits Send 3 after it times-out waiting on the lost ack. The retransmission is delayed in the network, and the receiver sees it only after it receives Send 0. The expected PSN on the receiver is 1, and it is receiving a packet within the valid region (2 packets behind), so it should treat it as a duplicate packet.

    Summary

    As you can see, in both scenarios the receiver state (expected PSN) is the same, and the received packet has the same PSN, so with a valid region of 3, it wouldn't be able to distinguish between the two scenarios. If we limit the valid region to 2, the first scenario wouldn't be possible, as the sender would have to wait for an acknowledgement for PSN 1 before sending PSN 3.