I/O error when trying to change serial baud in C program after changing with unix STTY

I noticed something weird yet reproducible.

I first check my serial port settings:

    bash-3.1# stty -F /dev/ttyS0
    speed 0 baud; line = 0;
    intr = <undef>; quit = <undef>; erase = <undef>; kill = <undef>; eof = <undef>; start = <undef>;
    stop = <undef>; susp = <undef>; rprnt = <undef>; werase = <undef>; lnext = <undef>; flush = <undef>;
    min = 1; time = 0;
    -cread
    -brkint -icrnl -imaxbel
    -opost -onlcr
    -isig -icanon -iexten -echo -echoe -echok -echoctl -echoke

Then change speed to 1200bps:

bash-3.1# stty -F /dev/ttyS0 1200

I then execute this fragment of my program in a function to change the baud:

fd=open(dev,O_NOCTTY | O_NONBLOCK | O_RDWR);
struct termios ser[1];
tcflush(fd,TCIFLUSH);
tcflush(fd,TCOFLUSH);
cfmakeraw(ser);
 // I call tcsetattr after each terminal setting to make sure its applied.
if (tcsetattr(fd,TCSANOW,ser) < 0){
    return -1;
}
cfsetspeed(ser,B9600);
if (tcsetattr(fd,TCSANOW,ser) < 0){
  return -2; //returns this after manually setting port via STTY
}

The problem is the baud rate does NOT get changed properly. In fact, I get -2 returned from the function and strerror(errno) returns "input/output error".

After program execution, I check system port settings:

bash-3.1# stty -F /dev/ttyS0
speed 0 baud; line = 0;
intr = <undef>; quit = <undef>; erase = <undef>; kill = <undef>; eof = <undef>; start = <undef>;
stop = <undef>; susp = <undef>; rprnt = <undef>; werase = <undef>; lnext = <undef>; flush = <undef>;
min = 1; time = 0;
-cread
-brkint -icrnl -imaxbel
-opost -onlcr
-isig -icanon -iexten -echo -echoe -echok -echoctl -echoke

And it resets to zero bps even though I specifically asked for 9600bps.

Why does it do that? and how do I force the speed to go to 9600bps programmatically?

Solution

You have many mistakes in your code.

You open the tty device with O_NONBLOCK, so when you issue the ioctl calls (tc*attr(3) calls result in ioctl(2) syscalls, depending on the unix flavour you are using) you don't know if the device has been already open to be able to make the tc*attr(3) calls. The same applies to O_NOCTTY flag. You have put those flags without knowledge of what is their function on the open system call. O_NOCTTY is useless in a program that is run from inside a session, and O_NONBLOCK will make your tc*attr(3) calls to return with an error (EAGAIN) if the device is not open yet, when trying to make the parameter adjustments.
You don't check the result of the open(2) call. This can make an error if you try to use -1 as a file descriptor (ENODEV, ENOTTY or EBADF, EINVAL, ENXIO, etc)
You don't initialize the data of the struct termios structure, so probably that's the reason of the error you get. As you show (your sample code snippet is not complete, one reason to tell you to read How to create a Minimal, Complete, and Verifiable example) the struct termios you use is declared on an automatic variable (because its declaration is embedded into the code) so it's for sure uninitialized and with rubbish data. You need normally to do a tcgetattr() on it to intialize to proper values, and to be able to restore the settings after your program ends.
bash(1) makes ioctl(2)s to set and get termios parameters on the standard input descriptor when it is connected to a tty device. If you are working with stdin, you must consider the interference of bash(1). This makes different the values you get from the ones you set with stty.
In general unix operating systems (not true in linux, i'm afraid) the last close to a device normally resets the parameters of a tty to standard fixed values, so the flags you set when you change a non stdin device (not the standar input, which is not last closed when stty finishes) with stty, those parameters reset to default once stty terminates (on the last close of the tty). Do a sleep 999999999 </dev/ttyBlaBla & before issuing the stty(1) command, so the port remains open (by the redirection of sleep command) after setting with stty(1).
READ termios(3) page, so you can set parameters from your program itself. Only if your program doesn't normally deal with setting parameters, you will not have to do it programmatically. But then there's no sense in changing the terminal parameters, so it is best to learn how to program the device parameters.

A proper way to do should be something (copied from your snippet and edited) like this:

#include <string.h> /* for strerror */
#include <errno.h> /* for errno definition */

/* ... */

fd=open(dev,O_NOCTTY | O_NONBLOCK | O_RDWR);
if (fd < 0) {
    fprintf(stderr, "OPEN: %s (errno = %d)\n",
        strerror(errno), errno);
    return -1;
}
struct termios ser; /* why an array? */
tcflush(fd,TCIFLUSH); /* unneeded, you have not used the tty yet */
tcflush(fd,TCOFLUSH); /* idem. */
/******* THIS IS THE MOST IMPORTANT THING YOU FORGOT ***********/
int res = tcgetattr(fd, &ser); /* *****this initializes the struct termios ser***** */
if (res < 0) {
    fprintf(stderr, "TCGETATTR: %s (errno = %d)\n",
        strerror(errno), errno);
    return -2; /* cannot tcgetattr */
}
/***************************************************************/
cfmakeraw(&ser); /* now it is valid to set it */
cfsetspeed(&ser,B9600);  /* better do all in only one system call */
// I call tcsetattr after each terminal setting to make sure its applied.
/* nope, tcsetattr and tcgetattr are the only calls that make something 
 * on the tty, the rest only manipulate bits on the struct termios
 * structure, but don't do anything to the terminal, you begun with a
 * trashed struct termios, so it's normal you end with an error. */
if ((res = tcsetattr(fd, TCSANOW, &ser)) < 0){
    fprintf(stderr, "ERROR: %s (errno = %d)\n",
        strerror(errno), errno); /* better to know what happened. */
    return -3; /* couldn't tcsetattr */
}

finally, this code (as yours first) has not been tested, mainly because you didn't posted a complete, minimal and verifiable example. So you'll probably need to rework it a little before including it in your code. And please RTFM (the last meaning to read termios(3) completely, and most important: How to create a Minimal, Complete, and Verifiable example) :). Also, don't check tty settings on stdin if you are using bash(1), as it normally restores tty settings after command exit, before issuing the prompt.