cmacosttyptylseek

What is lseek of a tty on macOS doing?

Consider the following C program, named weird.c (because the thing it is doing makes no sense):

#include <inttypes.h>
#include <stdint.h>
#include <stdio.h>
#include <unistd.h>

int main(void) {
    off_t result = lseek(STDOUT_FILENO, (off_t)INT64_MAX, SEEK_SET);
    if (result != (off_t)-1)
        printf("Seek successful to position %" PRId64 "\n", (int64_t)INT64_MAX);
    else
        perror("lseek failed");
    return 0;
}

Compile and run it from your terminal window, e.g. gcc -o weird weird.c && ./weird. It is important you are running it from a tty (terminal window, SSH session, etc), so stdout is a tty.

On Linux, it will fail with the error lseek failed: Illegal seek–which makes sense, seeking on a tty doesn't make much sense.

However, on macOS it behaves differently – the program exits without printing anything, and then – if you are using bash as your shell, the shell exits too. (By contrast, if you are using zsh as your shell, your shell prompt still works, but no commands display any output.) I can only assume this is because the tty has been put into some sort of illegal state. But what is actually going on here?

I was guessing macOS would behave similarly to FreeBSD. But FreeBSD 14.1-RELEASE (amd64) has a different behaviour again–it prints:

Seek successful to position 9223372036854775807

Which is more understandable than what macOS does (treating nonsensical operation as a no-op instead of erroring out on it like Linux does.)

Solution

  • User jepler on Hacker News pointed out the explanation.

    Notice this code in the vn_write function in bsd/vfs/vfs_vnops.c:

            if (write_offset == INT64_MAX) {
                /* writes are not possible */
                error = EFBIG;
                goto error_out;
        }

    Basically what happens is this:

    1. macOS, unlike Linux, lets you use lseek to set the offset of a terminal – even though (usually) the offset doesn't actually do anything
    2. However, once the offset reaches INT64_MAX, further attempts to write fail with EFBIG, unless you seek back to somewhere earlier.

    The reason why bash exits, is most shells exit when they can't write to stdout/stderr. As to why zsh doesn't, I'm not sure, but I speculate the zsh line editing code is opening the terminal device directly, and hence accesses it with an independent file position.

    If you modify my test program to seek to e.g. INT64_MAX-100, you will be able to print 100 bytes of output, before all output starts failing.

    This code appears to have been introduced in xnu-7195.50.7.100.1 (macOS 11 Big Sur) – it wasn't present in xnu-6153.11.26 (macOS 10.15 Catalina).

    FreeBSD also appears to allow you to lseek to arbitrary offsets in a terminal – but unlike macOS 11+, won't refuse further writes if you seek up to INT64_MAX.