csocketsforkepollpersistent-connection

Linux C socket how to prevent 2 forked process from accepting the same connection when using epoll?

for (int t = 0; t < physicalCoreCount; t++) {
        int pid = fork();
        if (pid==0) {
            // setting up epoll    
            epoll_fd = epoll_create1(0);                
            event.data.fd = listenSocketfd;
            event.events = EPOLLIN | EPOLLET;
            ret = epoll_ctl(epoll_fd, EPOLL_CTL_ADD, listenSocketfd, &event);                
            events = (epoll_event*)calloc(LISTENQ, sizeof(event));
           
            //*****/
            while (1) {
                int ndfs = epoll_wait(epoll_fd, events, curfdCount, -1);
                if (ndfs==-1) {                        
                    continue;
                }
                for (int i=0; i < ndfs; i++) {
                    if (events[i].data.fd == listenSocketfd) { // original listener
                        int new_connfd = accept(events[i].data.fd, (sockaddr*)&clientaddr, &clientlen);
                        if (new_connfd==-1) {
                            if (errno==EAGAIN || errno==EWOULDBLOCK) {
                                continue;
                            }
                            else exitPerrorLog("accept()");
                        }

                        set_non_block(new_connfd);
                        event.events = EPOLLIN | EPOLLET;
                        event.data.fd = new_connfd;
                        if (epoll_ctl(epoll_fd, EPOLL_CTL_ADD, new_connfd, &event) < 0)
                            exitPerrorLog("epoll_ctl");

                        clientaddrOfFd[new_connfd] = new sockaddr_in();
                        memcpy(clientaddrOfFd[new_connfd], &clientaddr, sizeof(clientaddr));
                        curfdCount++;
                    }   
                    else {
                        process(events[i].data.fd, clientaddrOfFd[events[i].data.fd]);
                        //epoll_ctl(epoll_fd, EPOLL_CTL_DEL, events[i].data.fd, &event);                           
                        if (curfdCount > 10000) curfdCount = 10000; //curfdCount--;
                    }
                }
            }
        }
    }

The problem appears because I'm trying to implement persistent connection (not closing after response). However, child 0 can accept() a socket file descriptor X, process it, but later child 1 can accept() the same file descriptor. Since I didn't close the connection on child 0 (to implement HTTP1.1 keep-alive), now 2 children are reading/writing to the same file descriptor.

What's a way to prevent this problem? Thank you.

Edit: Important update: so the main problem is that I thought 2 same FD means it's the same connection, I want to avoid that since it would lead to 2 children reading/writing the same thing. If that situation doesn't happen (read/write overlap) then I think the question is solved. Can someone confirm?

Solution

  • You are creating multiple epoll instances and registering for edge-triggered events on the listening socket in each one. Naturally, you will get an event from each one when a new connection becomes available to accept. However, only one process can successfully accept each connection. As observed in comments, two different children might accept connections that are assigned the same file descriptor number in the respective processes, but that doesn't mean they refer to the same socket.

    You have several options, but prominent among them are: