I've been looking at a simulation of the 1950s 'Pegasus' computer, and come across the term "self-modified link". How does this work?
Let's first note that:
The Pegasus computer supported subroutine/function calls by passing two instructions (in one register) on how the callee should return — as data, i.e. as parameter. The callee would then take that parameter, store it into its own local memory, and transfer control to that — thus executing the two instructions that were passed as data by the caller.
The caller would do this:
load instructions from L1: into X1 register
... pass other parameters ...
load callee's first block into U0
transfer control to U0 at start
L1: load instructions from main block back into U0
transfer control to U0 at desired caller resume point (e.g. L2)
L2: ...caller resumes...
And the caller would do this:
... compute something ...
store X1 into L4 # writes the two instruction pair in X1 into L4
transfer control to L4 # may or may not be present depending on location of L4
L4: # starts empty but contains the two instructions that came from L1
# so control is transferred back to the caller
The sequence at L1 is never directly executed, but rather is a template for instructions to be passed as a parameter to the callee, who will store that template into its own local memory and then execute that copy.
They use the term "cue" for call (subroutine invocation, e.g. transfer of control to callee) and "link" for return (transfer of control from subroutine back to the proper caller) (and the term "orders" for machine code instructions).
So, when they speak of "self-modfying link", they mean that the mechanism for a subroutine to return to its caller uses self-modfying code, which is code that alters its own instruction(s) while executing, i.e. code that writes into instruction memory for imminent execution. Here it is the write of X1 (a copy of the code at L1) to L4, which gets executed just after being written.
Self-modifying code is generally frowned upon these days as it has several negative implications: the instruction cache needs to be kept in sync with the self-modification, which interferes with performance, and, the instruction memory must be writable, which interferes with security.
Self modification was often used in older processors, due to what we might today think of as missing instructions in the instruction set — for example the PDP-8 had I/O instructions but the I/O port was hard coded into the instruction — there was no indirect I/O port access. Thus, in order for a subroutine to access an I/O port taken as a parameter, self-modifying code was used to construct an I/O instruction to that port. (A large switch statement could have also been used, but would take quite a bit more code.) These older processors did not have instruction caches, nor did programmers & development tools enforce separation of instruction sections from data sections (i.e. they didn't attempt to protect instructions).
In the case of the Pegasus, there are no indirect branches, thwarting the ability to return to a caller-provided address. As it turns out, the caller's code may have also actually been overlaid — the ability to pass two instructions to have the callee execute allowed both to restore a block of registers for the caller's code and the jump within that block.
(And passing fully formed instructions as a parameter resulted in fewer instructions for return linkage than the other self-modifying alternative: to dynamically construct a jump instruction given an address parameter — that would have taken several more instructions for doing bitfield manipulations.)
https://en.wikipedia.org/wiki/Ferranti_Pegasus http://bitsavers.org/pdf/ferranti/pegasus/PegasusProgrammingMan_1962.pdf