[SOLVED] How do I change the code in a lambda while keeping the captured vars?

How do I change the code in a lambda while keeping the captured vars?

Lets say I have a REPL to a process running my Common Lisp code. It could be running SWANK/SLIME.

I want to update a function defined with defun in my live process. The function may have captured some variables in a let binding. Essentially, this function is a closure.

How can I update the code in that closure without losing the data that it has captured?

2019-11-03: I selected one answer below, but I recommend reading all of the answers. Each has an interesting insight.

Solution

The other answers explain that you can't do what you are after: here are some practical reasons why you can't.

Consider a fragment of code like this:

(let ((a 1) (b 3) (c 2))
  (lambda (x)
    (+ (* a x x) (* b x) c)))

Let's imagine this is being compiled: what is a reasonable compiler going to do? Well, pretty obviously it can turn it into this:

(lambda (x)
  (+ (* 1 x x) (* 3 x) 2)

(and then probably into

(lambda (x)
  (+ (* x x) (* 3 x) 2))

and perhaps further into

(lambda (x)
  (+ (* x (+ x 3)) 2))

) before finally compiling it.

None of these transformations of the function body reference any of the lexical bindings introduced by the closure. The whole environment has been compiled away.

So if I now wanted to replace that function, in that environent, bu some other one, well, there's no environment any more: I can't.

Well, you could argue that that's an unduly simple case since the function doesn't mutate any of its closed-over variables. Well, consider something like this:

(defun make-box (contents)
  (values
   (lambda ()
     contents)
   (lambda (new)
     (setf contents new))))

make-box returns two functions: a reader and a writer, which both close over the contents of the box. And this shared state can't be fully compiled away.

But there's absolutely no reason at all why the closed over state, for instance, still knows that the variable being closed-over was called contents after make-box has been compiled (or even before). For instance the two functions might both reference some vector of lexical state, and both know that the thing that is contents in the source is the first element of that vector. All the names are gone, so it would not be possible to replace one of the functions sharing this state with some other one.

Further, in implementations with distinct compilers and interpreters, there's no reason at all why the interpreter should share a common representation of closed-over lexical state with the compiler (and both compiler and interpreter may have several different representations). And in fact the CL spec addresses this problem – the entry for compile says:

The consequences are undefined if the lexical environment surrounding the function to be compiled contains any bindings other than those for macros, symbol macros, or declarations.

and this caveat is there to deal with cases like this: if you have a bunch of functions which share some lexical state then, if the compiler has a different representation of lexical state than the interpreter, compiling just one of them is problematic. (I discovered this while trying to do some terrible thing with shared lexical state on a D-machine in about 1989, and some kind member of the committee explained my confusion to me.)

The above examples should convince you that replacing functions which share lexical state with other functions is not possible in any simple way. But, well, 'not possible in any simple way' is not the same as 'not possible'. For instance the specification of the language could simply say that this should be possible and require implementations to somehow make it work:

for the first example, an implementation might either choose not to compile the environment away thus reducing performance, or might choose to compile it away but keep a record of what it was so it could be reinstated around some new function, possibly to be compiled away again;
for the later examples, implementations might either choose to use simple, low-performance representations of the closed-over environment in all cases, or to use high-performance representations but keep a record of what the environment looked like so that replacement functions could be inserted into it;
in any case, the language would require new functionality to support this kind of redefinition, which would need to be carefully specified and would make the language larger and more complicated.

Both of these cases are really saying that implementations of the language either need to accept rather low performance, or require heroic techniques, and in either case there would be more to implement than there already was. Well, one of the goals of the Common Lisp effort was that, while heroic techniques are allowed, they should not be required for high performance. Additionally the language was already felt to be quite large enough. Finally implementors would almost certainly simply have rejected such a suggestion: they already had quite enough work to do, and did not want any more, especially at this kind of 'you will need to completely reengineer the compiler to do this' level.

So that is why, pragmatically, what you're after is not possible.