I'm trying to write self modifying asm code.
At some point, I try the following : (NASM compiler)
start_of_code:
; ... snip ...
cmp byte [rax], 0x66
jae above_label
add byte[rax], 0x20
; ... snip ...
above_label:
inc rax
loop start_of_code
gdb shows the following values at start x/8xb $rax :
0x12 0x12 0x11 0x20 0x18 0xfe 0x83 0x9b
The first two iterations are ok, the 0x12 are compared as I expect and no jump take place. With 0x11 however, cmp sets the OF flag instead of the CF and jae is executed
I have 2 guesses at the moment, either gdb doesn't report the right memory location first, 0x9b is in fact the address being compared, or someplace else. This seems unlikely because the add seems to work like I expect.
Or this might be an alignment problem. Adding NOP instructions before the cmp does seem to affect the outcome in some way. I really have no clue why
edit : When I try
mov bl, byte [rax]
bl gets 0xcc
This doesn't make sense to me. gdb doesn't report any value of 0xcc anywhere near [rax]
0xcc is the machine code for int3 which is the software breakpoint interrupt. gdb places this into your code at breakpoints so it gets control back. It doesn't play well with code that reads itself. Use the hbreak command instead, to set a hardware breakpoint that doesn't need to modify your code.