non-blocking `accept` return `EAGAIN` on connection burst

I observed weird behavior when performing non-blocking accept loop. When there are multiple connection requests in the queue, only the first accept attempt succeed. All subsequent attempt fail with EAGAIN. However, if I sleep for some time after the first successful accept, I can now retrieve subsequent connections successfully.

Here's a minimal example to reproduce:

#include <errno.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/socket.h>
#include <arpa/inet.h>

int main() {
  // 127.0.0.1:4200
  struct sockaddr_in addr;
  addr.sin_family = AF_INET;
  addr.sin_port = htons(4200);
  addr.sin_addr.s_addr = htonl((127 << 24) + 1);

  // create a non blocking socket, listening localhost 4200 port
  int listen_sock = socket(AF_INET, SOCK_STREAM | SOCK_NONBLOCK, 0);
  bind(listen_sock, (struct sockaddr*)&addr, sizeof(struct sockaddr_in));
  listen(listen_sock, 0);

  // make two non-blocking connection request to the above address
  int conn[2];
  for (int i = 0; i < 2; ++i) {
    conn[i] = socket(AF_INET, SOCK_STREAM | SOCK_NONBLOCK, 0);
    connect(conn[i], (struct sockaddr*)&addr, sizeof(struct sockaddr_in));
  }

  // after 10 seconds, the two connections above have succeeded
  sleep(10);

  // now perform three `accept` attempt on the listening socket.
  // since there are two connections above,
  // the first two `accept` attempt should succeed,
  // and the third one should return `EAGAIN`
  accept(listen_sock, 0, 0);
  accept(listen_sock, 0, 0);
  accept(listen_sock, 0, 0);

  // but oops, only the first `accept` attempt succeed.
  // the second and third attempt both return `EAGAIN`

  // now sleep for 2 more second and do two more `accept` attempt
  sleep(2);
  accept(listen_sock, 0, 0); // this one succeed
  accept(listen_sock, 0, 0); // this one fail with `EAGAIN` as expected

  close(conn[0]);
  close(conn[1]);
  close(listen_sock);
}

Running the above program with strace gives the following output:

socket(AF_INET, SOCK_STREAM|SOCK_NONBLOCK, IPPROTO_IP) = 3
bind(3, {sa_family=AF_INET, sin_port=htons(4200), sin_addr=inet_addr("127.0.0.1")}, 16) = 0
listen(3, 0)                            = 0
socket(AF_INET, SOCK_STREAM|SOCK_NONBLOCK, IPPROTO_IP) = 4
connect(4, {sa_family=AF_INET, sin_port=htons(4200), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in progress)
socket(AF_INET, SOCK_STREAM|SOCK_NONBLOCK, IPPROTO_IP) = 5
connect(5, {sa_family=AF_INET, sin_port=htons(4200), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in progress)
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=10, tv_nsec=0}, 0x7ffd05a80610) = 0
accept(3, NULL, NULL)                   = 6
accept(3, NULL, NULL)                   = -1 EAGAIN (Resource temporarily unavailable)
accept(3, NULL, NULL)                   = -1 EAGAIN (Resource temporarily unavailable)
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=2, tv_nsec=0}, 0x7ffd05a80610) = 0
accept(3, NULL, NULL)                   = 7
accept(3, NULL, NULL)                   = -1 EAGAIN (Resource temporarily unavailable)
close(4)                                = 0
close(5)                                = 0
close(3)                                = 0

====== edit ====== Did some more experiment, I am even more confused now.

if I make three connections, and do three rounds of accept with sleep in between, the first and second rounds accept both accept only one connection, and return EAGAIN afterwards. But the third round of accept always return EAGAIN, no matter how long I sleep
tried calling getsockopt on the three connection sockets, no error
tried writing to the connections to verify if the connections are established successfully, and:
- before the very first accept call, the first connection can be written, writing to the second and third connection returns EAGAIN
- immediately after the first accept call, all three connections are writable
- and they remain writable afterward

Solution

You should try to increase the backlog value when calling listen(listen_sock, 0).

setting backlog=0 can cause the connection queue to have length 0. So when you make your 1st accept, it goes ok, but 2nd accept gets EAGAIN because queue limit is up.

When I set listen(listen_sock, 1) in your example, I see this strace output:

socket(AF_INET, SOCK_STREAM|SOCK_NONBLOCK, IPPROTO_IP) = 3
bind(3, {sa_family=AF_INET, sin_port=htons(4200), sin_addr=inet_addr("127.0.0.1")}, 16) = 0
listen(3, 1)                            = 0
socket(AF_INET, SOCK_STREAM|SOCK_NONBLOCK, IPPROTO_IP) = 4
connect(4, {sa_family=AF_INET, sin_port=htons(4200), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in progress)
socket(AF_INET, SOCK_STREAM|SOCK_NONBLOCK, IPPROTO_IP) = 5
connect(5, {sa_family=AF_INET, sin_port=htons(4200), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in progress)
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=10, tv_nsec=0}, 0x7fff8ca4eda0) = 0
accept(3, NULL, NULL)                   = 6
accept(3, NULL, NULL)                   = 7
accept(3, NULL, NULL)                   = -1 EAGAIN (Resource temporarily unavailable)
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=2, tv_nsec=0}, 0x7fff8ca4eda0) = 0
accept(3, NULL, NULL)                   = -1 EAGAIN (Resource temporarily unavailable)
accept(3, NULL, NULL)                   = -1 EAGAIN (Resource temporarily unavailable)
close(4)                                = 0
close(5)                                = 0
close(3)                                = 0