Is it the tty, the shell, or both that is responsible for echo-ing user input?

I seem to be observing conflicting evidence regarding how the tty and the shell share responsibility for displaying user input.

In the following interactive session, user input is no longer echoed to the terminal, suggesting that the tty performs the echo.

$ stty -echo
# I typed ls (not displayed) and we see the output of ls only:
Bash.log File1    TTY.log

On the other hand, if we use expect to log exactly what is being emitted by bash vs by the tty, we can see that bash seems to be sending the user's input ls to the tty.

tty-echo-investigate.exp

#!/usr/bin/env expect

spawn bash

log_user 0
set log_tty_fd [open "TTY.log" "w"]
set log_bash_fd [open "Bash.log" "w"]

expect {
  -i $spawn_id "?" {
    send_user -- [set expect_out(0,string)]
    puts -nonewline $log_bash_fd [set expect_out(0,string)]
    exp_continue
  }
  -i $tty_spawn_id "?" {
    send -- [set expect_out(0,string)]
    puts -nonewline $log_tty_fd [set expect_out(0,string)]
    exp_continue
  }
}

Bash.log

[?2004h~/Play4/X ls
[?2004l
Bash.log File1    TTY.log
[?2004h~/Play4/X exit
[?2004l
exit

TTY.log

ls
exit

When user input is echoed by the tty vs bash, and how do they coordinate so we (usually) see only one copy of user input if they both sometimes do it?

Solution

After further investigation, it appears that bash (via readline) rather than the tty is doing the echo. The reason I do not see the characters I type when I do stty -echo is because readline is reading and respecting the tty setting when deciding whether to echo user input.

The main readline function calls rl_prep_term_function before performing the actual read and rl_deprep_term_function afterwards.
These function pointers are mapped to rl_prep_terminal and rl_deprep_terminal in rltty.c
rl_prep_terminal calls prepare_terminal_settings which runs _rl_echoing_p = (oldtio.sgttyb.sg_flags & ECHO); and also turns off echo in the tty with tiop->c_lflag &= ~(ICANON | ECHO);
The documentation for this variable states the following:

/* Non-zero means echo characters as they are read.  Defaults to no echo;
   set to 1 if there is a controlling terminal, we can get its attributes,
   and the attributes include `echo'.  Look at rltty.c:prepare_terminal_settings
   for the code that sets it. */
int _rl_echoing_p = 0;

Furthermore, tty echo gets restored after each line is read, which is why it is very tricky to see what is really happening by running commands like stty -a from bash directly.

We can also observe the readline behavior using the following expect script, which will dump the tty state of bash's tty without invoking it via bash itself.

#!/usr/bin/env expect

set timeout -1
spawn bash


log_user 0
set log_tty_fd [open "TTY.log" "w"]
set log_bash_fd [open "Bash.log" "w"]

expect {
  -i $spawn_id "?" {
    send_user -- [set expect_out(0,string)]
    puts -nonewline $log_bash_fd [set expect_out(0,string)]
    exp_continue
  }
  -i $tty_spawn_id "dump" {
    puts [ stty -a < $spawn_out(slave,name) ]
    exp_continue
  }
  -i $tty_spawn_id "?" {
    send -- [set expect_out(0,string)]
    puts -nonewline $log_tty_fd [set expect_out(0,string)]
    exp_continue
  }
}

When invoking this, we can see the true state of the tty when readline is trying to read, which is different from what stty -a returns when invoked on the actual shell:

$ ./tty-echo-investigate.exp
spawn bash
$ stty -a
stty -a
speed 9600 baud; 48 rows; 196 columns;
lflags: icanon isig iexten echo echoe -echok echoke -echonl echoctl
    -echoprt -altwerase -noflsh -tostop -flusho pendin -nokerninfo
    -extproc
iflags: -istrip icrnl -inlcr -igncr ixon -ixoff ixany imaxbel -iutf8
    -ignbrk brkint -inpck -ignpar -parmrk
oflags: opost onlcr -oxtabs -onocr -onlret
cflags: cread cs8 -parenb -parodd hupcl -clocal -cstopb -crtscts -dsrflow
    -dtrflow -mdmbuf
cchars: discard = ^O; dsusp = ^Y; eof = ^D; eol = <undef>;
    eol2 = <undef>; erase = ^?; intr = ^C; kill = ^U; lnext = ^V;
    min = 1; quit = ^\; reprint = ^R; start = ^Q; status = ^T;
    stop = ^S; susp = ^Z; time = 0; werase = ^W;
$ dump
speed 9600 baud; 48 rows; 196 columns;
lflags: -icanon isig iexten -echo echoe -echok echoke -echonl echoctl
    -echoprt -altwerase -noflsh -tostop -flusho -pendin -nokerninfo
    -extproc
iflags: -istrip -icrnl -inlcr -igncr ixon -ixoff ixany imaxbel -iutf8
    -ignbrk brkint -inpck -ignpar -parmrk
oflags: opost onlcr -oxtabs -onocr -onlret
cflags: cread cs8 -parenb -parodd hupcl -clocal -cstopb -crtscts -dsrflow
    -dtrflow -mdmbuf
cchars: discard = <undef>; dsusp = <undef>; eof = ^D; eol = <undef>;
    eol2 = <undef>; erase = ^?; intr = ^C; kill = ^U;
    lnext = <undef>; min = 1; quit = ^\; reprint = ^R; start = ^Q;
    status = ^T; stop = ^S; susp = ^Z; time = 0; werase = ^W;