how to get unlinkat(dir_fd, ".", AT_REMOVEDIR) to work?

I need to unlink an empty directory with

unlinkat(dir_fd, ".", AT_REMOVEDIR)

and I am getting EINVAL errno, which, according to official GNU documentation, must mean "An invalid flag value was specified in flags", which is not the case, as AT_REMOVEDIR is (the only) allowed flag.

SSCCE is as follows. Note, that in the simplest code, I do have the directory name, so I could have used rmdir. In the real situation, I don't have the name, I only have the descriptor, to an empty directory, and I need to remove it - so I have to use unlinkat .

foobar.c:

#define _GNU_SOURCE
#include <errno.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>

int main()
{
    int dir_fd =
        open("dir", O_RDONLY | O_PATH | O_DIRECTORY);

    if (-1 == dir_fd) {
        perror("open");
        return(1);
    }
    
    if (unlinkat(dir_fd, ".", AT_REMOVEDIR)) {
        perror("unlinkat");
        fprintf(stderr, "errno %d\n", errno);
        return(1);
    }
}

and I get this behaviour:

$mkdir dir
$gcc -lc foobar.c
$./a.out
unlinkat: Invalid argument
errno 22
$errno 22
EINVAL 22 Invalid argument

and GNU documentation for unlinkat says

EINVAL An invalid flag value was specified in flags.

When called with AT_REMOVEDIR, unlinkat(2) behaves as rmdir(2), and ultimately fails to the same reasons.

For rmdir(2):

EINVAL pathname has . as last component.

You have seemingly flipped the interface. Looking at your example in isolation, it should be

/* foobar.c */
#define _GNU_SOURCE
#include <errno.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>

int main(void)
{
    int dir_fd = open(".", O_RDONLY | O_PATH | O_DIRECTORY);

    if (-1 == dir_fd) {
        perror("open");
        return(1);
    }

    if (unlinkat(dir_fd, "dir", AT_REMOVEDIR)) {
        perror("unlinkat");
        fprintf(stderr, "errno %d\n", errno);
        return(1);
    }

    close(dir_fd);
}

$ mkdir dir
$ ./a.out

where a relative pathname given to unlinkat is resolved relative to the directory referred to by dirfd.

(Or the special value of AT_FDCWD could be used for the first argument of unlinkat to remove a file relative to the current working directory.)

This particular example is obviously a rather complex way of doing the same thing rmdir("dir") does, and it certainly does not help you if you do not know the name of the directory to be removed.

For your actual problem: On Linux, you can try something along the lines of the following example, that uses readlink(2) to read /proc/self/fd/NNN, where NNN is the file descriptor (for which you do not have a name), in order to retrieve a pathname for rmdir.

(See: Retrieve filename from file descriptor in C)

#define _GNU_SOURCE
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

static int mystery_directory(void)
{
    char dir[] = "XXXXXX";

    if (!mkdtemp(dir))
        return -1;

    return open(dir, O_RDONLY | O_PATH | O_DIRECTORY);
}

int main(void)
{
    /* We have magically arrived at this file descriptor */
    int dir_fd = mystery_directory();

    if (-1 == dir_fd) {
        perror("mystery_directory");
        return EXIT_FAILURE;
    }

    char fdrl[128];
    int len = snprintf(fdrl, sizeof fdrl, "/proc/self/fd/%d", dir_fd);

    if (len < 0 || len >= sizeof fdrl) {
        fprintf(stderr, "Something has gone very wrong.\n");
        return EXIT_FAILURE;
    }

    printf("procfs entry: <%s>\n", fdrl);

    char path[4096];
    ssize_t r = readlink(fdrl, path, sizeof path - 1);

    if (r < 0) {
        perror("readlink");
        return EXIT_FAILURE;
    }

    close(dir_fd);
    path[r] = 0;

    printf("Attempting to remove directory: \n\t%s\n", path);

    if (-1 == rmdir(path)) {
        perror("rmdir");
        return EXIT_FAILURE;
    }

    puts("Success!");
}

$ pwd
/home/so
$ ./a.out
procfs entry: </proc/self/fd/3>
Attempting to remove directory: 
    /home/so/IOgkes
Success!