While creating instruction functions for my 6502/NES emulator, I got stuck at understanding the concept of signed bytes and two's complement in the 6502. So apparently, branch instructions such as BMI use a signed byte in memory for forwards/backwards branching, and some instructions allow arithmetic operations with negative numbers. Also the negative flag detects the 7th bit of the accumulator. (Two's complement)
Does this mean that all bytes in memory are signed and I could initialize the memory as int8_t CPUMEMORY[0x10000]; and not uint8_t CPUMEMORY[0x10000]?
The thing that is bothering me though, is that the carry flag is set when an operation result exceeds the unsigned 8 bit limit, which is 255. But shouldn't this be 127 if all bytes are signed? (The overflow flag does this, but then what is the point of having a carry flag?)
Bytes in the memory are just bytes: they are neither signed nor unsigned. The value of a byte can however be interpreted as signed number (int8_t) or unsigned number (uint8_t). The same for the value of two consecutive bytes (int16_t or uint16_t) and the value of four consecutive bytes (int32_t or uint32_t).
To handle these bytes, you have several 6502 instructions that allow you to perform several operations for both signed and unsigned interpretations. You just have to use the right instructions and use the right flags (carry, overflow, negative...) for what you need to do.