Why the set-uid operation can't be used in an excutable file?

When learning the system security in the ubuntu 20.04 on the VMware, I tried the set-uid operation and found the fllowing question:

With the excutable file catcall compiled by the source code caltcall.c：

#include<string.h>
#include<stdio.h>
#include<stdlib.h>

int main(int argc, char* argv[]){
    char *cat = "/bin/cat";

    if(argc < 2){
        printf("please type a file name.\n");
        return 1;
    }
    char *command = malloc(strlen(cat) + strlen(argv[1] + 2));
        sprintf(command, "%s %s",cat, argv[1]);
        system(command);    
    return 0;
}

I complete the set-uid operation through the following codes:

$ sudo chown root catcall
$ sudo chmod 4755 catcall

When excuting the excutable file catcall, I thought I can see the content of the file /etc/shadow, for the 'catcall' has been set to the Set-Uid programme. But the operation is denied when trying to access the etc/shadow:

/bin/cat: /etc/shadow: Permission denied

Why did the set-uid operation failed?

Each process actually has two user ids, a real uid and an effective uid, see Difference between Real User ID, Effective User ID and Saved User ID. Normally they are equal. Setting the setuid bit causes the effective uid to be set to the owner of the binary upon exec, but the real uid remains unchanged, and will normally still be the uid of the user who actually ran the program.

So when your program starts, the effective uid will be root and the real uid will be dim_shimmer. Now you have chosen to try to start your program via system(), which executes the specified command using the shell. However, many shells have a security feature where they compare the real and effective uid, and if they are not equal, they set the effective uid equal to the real uid. (For instance, this avoids a disaster in case the sysadmin accidentally does chmod +s /bin/sh). I suspect that's what's happening here. So by the time your cat command runs, your real and effective uid are both dim_shimmer again, and that user does not have permission to read /etc/shadow.

If you run your cat command using execl instead (or one of its relatives), so as to bypass the shell, then I suspect you'll find that it works.