How to use GNU readline with flex-lexer?

I think that using the GNU Readline library for a command-line prompt is good and I want that functionality for my shell that I'm working on. Now readline works for me (my environment is CLion, CMake, Ubuntu, BSD, C, flex-lexer and lemon-parser) but I also need flex and yacc to work at the same time to scan and parse the input but the codes seem "incompatible" - are they really?

    params[0] = NULL;
   printf("> ");

    i=1;
    do {
        lexCode = yylex(scanner);

        /*  snprintf(shell_prompt, sizeof(shell_prompt), "%s:%s $ ", getenv("USER"), getcwd(NULL, 1024));
         Display prompt and read input (NB: input must be freed after use)...*/



        text = strdup(yyget_text(scanner));
        /*
        input = readline(text);

        if (!input)
            break;

        add_history(input);

        free(input);*/
        printf("lexcode %i Text %s\n", lexCode, text);
        if (lexCode == 4) {
            params[i++] = mystring;
            if (strcmp(text, "\'\0")) {
                params[i++] = mystring;
            }
        } else
        if (lexCode != EOL) {
                params[i++] = text;
                printf("B%s\n", text);
        }
        Parse(shellParser, lexCode, text);
        if (lexCode == EOL) {
            dump_argv("Before exec_arguments", i, params);
            exec_arguments(i, params);
            corpse_collector();
            Parse(shellParser, 0, NULL);
            i=1;
        }
    } while (lexCode > 0);
    if (-1 == lexCode) {
        fprintf(stderr, "The scanner encountered an error.\n");
    }

The above code has the parsing and scanning working and commented out the readline functionality that won't work if I want both at the same time. Can I make it work?

Solution

Read documentation of GNU readline. Read the tty demystified page. You'll want to use readline only when your stdin is a tty, so use isatty(3) as isatty(STDIN_FILENO) to detect that.

The basic behavior of a shell using readline is simply to use the

char *readline (const char *prompt);

function. So you want to adapt your parser to read from a buffer, not from stdin. This is usual practice. Don't forget to test (against failure) your call to readline and to free the resulting buffer when you are done.

Then, you want to add some completion. That is the hard part. Look at what other shells (zsh, bash, fish...) are doing, at least for inspiration. Notice that the default file-name completion could be enough, at least when starting.

BTW, I won't use both lemon and bison to parse the same input. Actually, for a shell (since its syntax is quite simple), I'll just use some hand-written recursive-descent parser.