Explain how the AF flag works in an x86 instructions?

I have a little 8086 emulator and I've had a long standing bug for like 2 years now that AF does not behave properly inside of sub and add instructions.

My current way of computing its value is this for 8 bit numbers and subtraction:

uint8_t base=... , subt=...
base=base&0xF;
subt=subt&0xF; //isolate bottom nibble
if((int16_t)base-subt>7 || (int16_t)base-subt<-7){
    flags.af=1;
}else{
   flags.af=0;
}

(assuming an instruction like sub base,subt )

And for adding it's like this:

uint8_t base=... , adder=...
base=base&0xF;
adder=adder&0xF; //isolate bottom nibble
if(base+adder>7 || base+adder<-7){
    flags.af=1;
}else{
   flags.af=0;
}

(for an instruction like add base,adder)

How do I properly calculate the AF flag in my emulator for such instructions?

flags.af = (((base-subt) & ~0xf) != 0);

Checks to see if the upper bits are anything except zero, which would indicate an overflow or underflow out of the bottom 4 bits.

Here's a version that's a little closer to your original. Note that the difference between two 4-bit quantities will never be greater than 15. Likewise the addition will never be less than 0.

flags.af = ((int16_t)base-subt < -15);

flags.af = ((int16_t)base+adder > 15);

Putting parentheses around a boolean expression is just a style preference of mine, I know they're redundant.