I'm trying to detect stack overflow in some function, and want to set a watchpoint to a memory pointed by the RSP register. I can't just set a watchpoint to a certain address as the function could be called from a different places. So I want to set breakpoints at the start and the end of the function, and force them to enable/disable the watchpoint.
But neither way of setting a watchpoint works for me. E.g. wa $rsp watches the register; the
set $myvar = $rsp
wa $myvar
makes watchpoint to trace... changes of the $myvar! Not the value it stores!
Weird, I'm sure, here is should be a way, but I don't know one...
UPD: looks I found a bug in gdb. Two:
(gdb) wa *$rsp
Attempt to dereference a generic pointer.
(gdb) set $myvar = $rsp
(gdb) wa *$myvar
Attempt to dereference a generic pointer.
UPD: don't know why, but the wa &*$myvar sets a breakpoint to the $myvar, though shouldn't. Bug?
GDB doesn't let you explicitly dereference void * pointers such as $rsp. It doesn't know the size or the format of the pointed-to value.
Do watch *(char *)$rsp or watch *(int *)$rsp, etc., instead to watch for a change in what the register points to. (Either by memory writes, or by the register changing to point somewhere else.)
Using watch -l *(int*)$rsp will watch for changes to that 4-byte memory location, even if $rsp later changes. (Snapshotting the address).
On x86 CPUs, watch -l can use a hardware watchpoint, unlike without -l where it's monitoring for a change in evaluating the whole expression.