Sockets using GetQueuedCompletionStatus and ERROR_MORE_DATA

I am trying to use GetQueuedCompletionStatus with winsocks, but I can't seem to get it right. The procedure is as follows:

void foo() {
    ...
    SOCKET sck = WSASocket(AF_INET, SOCK_DGRAM, 0, 
        NULL, 0, WSA_FLAG_OVERLAPPED);
    ....
    bind(sck,(struct sockaddr *)&addr,sizeof(struct sockaddr_in));
    HANDLE hPort = CreateIoCompletionPort((HANDLE)sck, NULL, 0, 0 );
    OVERLAPPED pOverlapped = {0,};
    WSARecvFrom(sck,NULL,0,NULL,NULL,(struct sockaddr *)&laddr,&lsize,&pOverlapped,0);
    BOOL bReturn = GetQueuedCompletionStatus(
            hPort,
            &rbytes,
            (LPDWORD)&lpContext,
            &pOutOverlapped,
            INFINITE);
    ...
}

I then send some network data to the bound port from an external tool. GetQueuedCompletionStatus returns FALSE, and GetLastError() returns ERROR_MORE_DATA, which sounds correct, since I hadn't provided a buffer in WSARecvFrom.

The question is how can I provide a buffer to actually get the data from the failed I/O operation?

I tried to issue a WSARecvFrom with the original overlapped structured, but it simply queues another read, and a subsequent call to GetQueuedCompletionStatus does not return until more network data is sent.

Calling WSARecvFrom without an overlapped structure blocks it, and it also doesn't return until more network data is sent.

So, how can I handle ERROR_MORE_DATA properly, without losing the data from the first operation?

You must provide a buffer to WSARecvFrom(), just like with any read operation regardless of whether you use IOCP or not. You must ensure the buffer stays valid in memory until the IOCP operation is complete. IOCP fills the buffer you provide and then notifies the completion port when finished.

UDP cannot transfer more than 65535 bytes in a single datagram, so you can use that as your max buffer size.

In your example, your code is written to run synchronously (defeating the purpose of using IOCP at all), so you can use a local buffer:

void foo() {
    ...
    SOCKET sck = WSASocket(AF_INET, SOCK_DGRAM, 0, NULL, 0, WSA_FLAG_OVERLAPPED);
    if (sck == INVALID_SOCKET)
    {
        // error, do something...
        return;
    }
    ....
    bind(sck,(struct sockaddr *)&addr, sizeof(struct sockaddr_in));
    HANDLE hPort = CreateIoCompletionPort((HANDLE)sck, NULL, 0, 0 );
    if (!hPort)
    {
        // error, do something...
        return;
    }

    WSAOVERLAPPED Overlapped = {0};
    Overlapped.hEvent = WSACreateEvent();

    BYTE buffer[0xFFFF];
    DWORD dwBytesRecvd = 0;
    DWORD dwFlags = 0;
    sockaddr_in fromaddr = {0};
    int fromaddrlen = sizeof(fromaddr);

    WSABUF buf;
    buf.len = sizeof(buffer);
    buf.buf = buffer;

    int iRet = WSARecvFrom(sck, &buf, 1, &dwBytesRecvd, &dwFlags, (sockaddr*)&fromaddr, &fromaddrlen, &Overlapped, NULL);
    if (iRet == SOCKET_ERROR)
    {
        if (WSAGetLastError() != WSA_IO_PENDING)
        {
           // error, do something...
           return;
        }

        DWORD rBytes;
        ULONG_PTR key;
        LPOVERLAPPED pOverlapped = NULL;

        if (!GetQueuedCompletionStatus(hPort, &rbytes, &key, &pOverlapped, INFINITE))
        {
            if (pOverlapped)
            {
                // WSARecvFrom() failed...
            }
            else
            {
                // GetQueuedCompletionStatus() failed...
            }

            // do something...
            return;
        }
    }

    // I/O complete, use buffer, dwBytesRecvd, dwFlags, and fromaddr as needed...
}

However, this defeats the purpose of IOCP. If you really want to be synchronous, you could just use recvfrom() instead and let it block the calling thread until data arrives. IOCP works best when you have a pool of threads servicing the completion port. Call WSARecvFrom() and let it work in the background, don't wait on it. Let a separate thread call GetQueuedCompletionPort() and process the data when it is received, eg:

struct MyOverlapped
{
    WSAOVERLAPPED overlapped;
    BYTE buffer[0xFFFF];
    DWORD buflen;
    DWORD flags;
    sockaddr_storage fromaddr;
    int fromaddrLen;
};

HANDLE hPort = NULL;

void foo() {
    ...
    SOCKET sck = WSASocket(AF_INET, SOCK_DGRAM, 0, NULL, 0, WSA_FLAG_OVERLAPPED);
    if (sck == INVALID_SOCKET)
    {
        // error, do something...
        return;
    }
    ....
    bind(sck,(struct sockaddr *)&addr, sizeof(struct sockaddr_in));
    hPort = CreateIoCompletionPort((HANDLE)sck, NULL, 0, 0 );
    if (!hPort)
    {
        // error, do something...
        return;
    }

    MyOverlapped *ov = new MyOverlapped;
    ZeroMemory(ov, sizeof(*ov));
    ov->overlapped.hEvent = WSACreateEvent();
    ov->fromaddrlen = sizeof(ov->fromaddr);

    WSABUF buf;
    buf.len = sizeof(ov->buffer);
    buf.buf = ov->buffer;

    int iRet = WSARecvFrom(sck, &buf, 1, &ov->buflen, &ov->flags, (sockaddr*)&ov->fromaddr, &ov->fromaddrlen, (WSAOVERLAPPED*)ov, NULL);
    if (iRet == SOCKET_ERROR)
    {
        if (WSAGetLastError() != WSA_IO_PENDING)
        {
           // error, do something...
           return;
        }

        // WSARecvFrom() is now operating in the background,
        // the IOCP port will be signaled when finished...
    }
    else
    {
        // data is already available,
        // the IOCP port will be signaled immediately...
    }

    ...
}

...

// in another thread...

{
    ...

    DWORD rbytes;
    ULONG_PTR key;
    MyOverlapped *ov = NULL;

    if (!GetQueuedCompletionStatus(hPort, &rbytes, &key, (LPOVERLAPPED*)&ov, INFINITE))
    {
        if (ov)
        {
            // WSARecvFrom() failed...
            // free ov, or reuse it for another operation...
        }
        else
        {
            // GetQueuedCompletionStatus() failed...
        }
    }
    else
    {
        // use ov as needed...
        // free ov, or reuse it for another operation...
    }

    ...
}