[SOLVED] SCTP over UDP: Stop retransmission message

SCTP over UDP: Stop retransmission message

I'm trying to send SCTP messages over UDP. The setup appears to work, but I'm getting a retransmission message every time. Here's the process:

Create a new UDP socket()
bind() the socket to a local port
connect() the socket to a remote port
usrsctp_init() using the local port and a send_callback.

(In the send_callback, call sendto() using the UDP socket, the data passed in, and remote port address)

usrsctp_register_address() using the remote port address
Create a new usrsctp_socket() using AF_CONN, SOCK_STREAM, IPPROTO_SCTP, and the remote port
set sock opt to non-blocking
usrsctp_bind() using the SCTP socket and port 5000 as required by mediasoup
usrsctp_connect() using the SCTP socket and the same required port
In a loop, call: recvfrom() using the UDP socket and local address, and then usrsctp_conninput() using the remote port address

Now, that seems to connect properly, as I'm getting the INIT -> INIT_ACK -> COOKIE_ECHO -> COOKIE_ACK handshake. But when I send a message using usrsctp_sendv() using the SCTP socket, I get the SACK response but usrsctp keeps retransmitting the message.

Summary: I'm using usrsctp as the client and mediasoup as the server. I'm getting a retransmission message by usrsctp each time even though I'm getting SACK messages back from the server. For some reason usrsctp keeps retransmitting the message. I've tried disabling the retransmission with the SCTP_PR_SCTP_RTX policy set to 0, but it's not working. I've tried so many different things, but the message keeps retransmitting.

#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <usrsctp.h>

#include <stdio.h>
#include <string.h>
#include <pthread.h>

int sock;
struct socket *sctp_socket;

void *keypress_listener(void *arg)
{
    getchar();

    struct sctp_sendv_spa spa;
    memset(&spa, 0, sizeof(spa));
    spa.sendv_flags = SCTP_SEND_SNDINFO_VALID;
    spa.sendv_sndinfo.snd_sid = 1;
    spa.sendv_sndinfo.snd_ppid = htonl(51);
    spa.sendv_sndinfo.snd_flags = SCTP_EOR; // end of message
    spa.sendv_prinfo.pr_policy = SCTP_PR_SCTP_RTX;
    spa.sendv_prinfo.pr_value = 0;

    const char *msg = "Hello World";
    int addr_len = 1;

    if (usrsctp_sendv(sctp_socket, (void *)msg, strlen(msg), NULL, 0, &spa, (socklen_t)sizeof(spa), SCTP_SENDV_SPA, 0) < 0)
    {
        printf("Failed to send message");
    }
}

int onSendSctpData(
    void *addr,
    void *data,
    size_t len,
    uint8_t tos,
    uint8_t setDf)
{
    // printf("on send data. len:%d\n", len);
    struct sockaddr_in *remote = (struct sockaddr_in *)addr;
    return sendto(sock, data, len, 0, (struct sockaddr *)remote, sizeof(*remote));
}

int onRecvSctpData(
    struct socket *socket /*sock*/,
    union sctp_sockstore saddr /*addr*/,
    void *data,
    size_t len,
    struct sctp_rcvinfo rcv,
    int flags,
    void *ulpInfo)
{
    // printf("received sctp data\n");
    return 0;
}

int main(int argc, char *argv[])
{
    // setup UDP transport
    sock = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);
    int local_port = 40001;
    int remote_port = 44444;

    struct sockaddr_in local_addr;
    memset(&local_addr, 0, sizeof(local_addr));
    local_addr.sin_family = AF_INET;
    local_addr.sin_port = htons(local_port);
    local_addr.sin_addr.s_addr = INADDR_ANY;

    bind(sock, (struct sockaddr *)&local_addr, sizeof(local_addr));

    struct sockaddr_in remote_addr;
    memset(&remote_addr, 0, sizeof(remote_addr));
    remote_addr.sin_family = AF_INET;
    remote_addr.sin_port = htons(remote_port);
    remote_addr.sin_addr.s_addr = INADDR_ANY; // inet_addr("<ip-address>");

    connect(sock, (struct sockaddr *)&remote_addr, sizeof(remote_addr));

    // setup SCTP over UDP
    usrsctp_init(local_port, onSendSctpData, NULL);
    usrsctp_register_address(&remote_addr); // required, else usrsctp_bind() won't work

    sctp_socket = usrsctp_socket(AF_CONN, SOCK_STREAM, IPPROTO_SCTP, onRecvSctpData, NULL, 256u, &remote_addr);

    usrsctp_set_non_blocking(sctp_socket, 1);

    struct sockaddr_conn sctp_addr;
    memset(&sctp_addr, 0, sizeof(sctp_addr));
    sctp_addr.sconn_family = AF_CONN;
    sctp_addr.sconn_port = htons(5000);
    sctp_addr.sconn_addr = &remote_addr;

    if (usrsctp_bind(sctp_socket, (struct sockaddr *)&sctp_addr, sizeof(sctp_addr)) < 0)
    {
        printf("bind failed");
    }

    printf("sctp connecting...");
    if (usrsctp_connect(sctp_socket, (struct sockaddr *)&sctp_addr, sizeof(sctp_addr)) < 0)
    {
        printf("connect failed"); // this appears to be normal for non-blocking
    }

    pthread_t thread;
    pthread_create(&thread, NULL, keypress_listener, NULL);
    pthread_join(thread, NULL);

    int addrlen = sizeof(local_addr);
    char buf[1024];
    while (1)
    {
        ssize_t len = recvfrom(sock, buf, sizeof(buf), 0, (struct sockaddr *)&local_addr, &addrlen);
        if (len > 0)
        {
            usrsctp_conninput(&remote_addr, buf, len, 0);
        }
    }

    return 0;
}

retransmission detail

Edit: added MRE. As a sidenote, I used tailscale for the remote address, since localhost was causing some issues for me.

Edit 2: added wireshark data and transmission detail

Edit 3: updated code to work with linux. steps to install usrsctp and run program:

pull usrsctp: git clone https://github.com/sctplab/usrsctp.git
build usrsctp: cd into usrsctp, ./bootstrap, ./configure, make, sudo make install, sudo ldconfig
create program.c (paste in my code)
compile with gcc program.c -o program -lusrsctp -L/usr/local/lib -pthread
run with ./program.exe

Solution

I cannot believe how easy the solution was... and I can't believe what I had to do to figure it out. I compiled the usrsctp library in visual studio and statically linked to it with debug symbols so I could step through the code from my program. Usrsctp is incredibly complex, and I stepped through thousands of lines of code until I found the line that was sending the retransmission. Turns out it wasn't any specific retransmission code, it was just the normal send call, but it was returning an error. I looked through the documentation but I couldn't find an error code that made any sense. Then I thought about it for awhile, and realized that the error code seemed to be the same as the amount of bytes returned from the socket sendto() function. Yea, I was returning the bytes which usrsctp believed was an error code and so it kept resending the data!

I simply had to return 0 in the onSendSctpData() function and it stopped retransmitting!!