In Common Lisp, is it possible to programmatically manipulate a lexical environment at runtime?
If I create a closure, typical let-over-lambda, as described in D. Hoytes LOL-book, how can I augment the closure with a new binding, or remove a binding? Consider something simple as this:
(use-package :sb-cltl2)
(let ((foo "foo"))
(defun print-foo ()
(format t "Foo: ~a~%" foo)))
How do I add a new variable '(bar "bar") to the lexical environment of this function, and how do I add a new function 'print-bar' to it. Is that even possible?
As a follow-up: are lexical closures only live on the stack during the execution, akin to activation records as when compiling C, or are they real objects on the heap, i.e. closures, live during the lifetime of the application? I understand it would be implementation specific, but any general pointer?
I have been looking into CLTL2 environments, but I am not really sure I understand how to use this stuff. Almost all functions and macros from that extension, require the environment to be passed in as an argument. But there does not seem to be a way to obtain an existing environment from the system? Perhaps I am wrong about it. I have been testing a bit 'define-declaration', which is the only one describe to return something likened to the environment, the alist passed to that handler, but I have hard time to understand what they mean and how to use that one. The wording is "standardeese" so I might be misinterpreting something there too. I am not overly familiar with Common Lisp either. I don't really understand how to connect the generated "handler" with that alist/environment they are supposed to pass in and the handler is supposed to return.
For the rest, if I do this (naively):
(defvar myenv (augment-environment nil :variable '(foo bar)))
I can see I have a lexical environment:
#S(SB-KERNEL:LEXENV
:FUNS NIL
:VARS ((FOO . #<SB-C::LAMBDA-VAR :%SOURCE-NAME FOO {100E721B83}>)
(BAR . #<SB-C::LAMBDA-VAR :%SOURCE-NAME BAR {100E721C43}>))
:BLOCKS NIL
:TAGS NIL
:TYPE-RESTRICTIONS NIL
:LAMBDA NIL
:CLEANUP NIL
:HANDLED-CONDITIONS NIL
:DISABLED-PACKAGE-LOCKS NIL
:%POLICY #<SB-C:POLICY ((SB-EXT:INHIBIT-WARNINGS 1) (SPEED 1) (SPACE 1)
(SAFETY 1) (DEBUG 1) (COMPILATION-SPEED 1))>
:USER-DATA NIL
:PARENT NIL
:VAR-CACHE NIL
:FLUSHABLE NIL)
However, if I try to modify it, I am stubmling against the wall seems like:
(defun print-bar ()
(funcall
(enclose '(lambda ()
(setf bar "bar")
(format t "Bar: ~a~%" bar)) myenv)))
print-bar will print "bar". However so will also just evaluating 'bar' in the global environment. In other words, enclose didn't really enclosed the lambda into the lexical environment (myenv), instead setf modified the global environment. How is this supposed to be used effectively?
I was also testing cl-environments by Gutov, however I didn't got it to work:
(defun print-foo ()
(in-lexical-environment (myenv)
(setf foo "Foo!")
(format t "Foo: ~a~%" foo)))
The effect was exactly the same, setf modified the global environment.
I have been looking at SBCL source code, in implementation of function-information and variable-information and I could steal their code to get this info, but that is of course non-portable and erroneous approach.
Obviously, I do misunderstand something quite badly, so ELI5-me please. I really have trouble to understand what they talk about there.
What you ask is not possible in standard CL, or in any implementation that I know of.
To see why it's not possible, consider what the word 'lexical' in 'lexical environment' means: it means that you (or a language processor) can read a section of code and know what bindings, declarations, and so on exist at any given point in that code. So, given the obvious macro to avoid the horrible antipattern of non-toplevel defun:
(defmacro define-function (name form)
`(progn
(declaim (ftype function ,name))
(setf (fdefinition ',name) ,form)
',name))
I can write
(define-function counter
(let ((v 0))
(lambda ()
(prog1 v
(incf v)))))
And both I and anything that processes this code, specifically including a compiler, can know what this means. Similarly if I change that definition to
(define-function counter
(let ((v 0) (u 3))
(lambda ()
(prog1 v
(incf v)))))
Both I and the compiler can tell that the binding of u is unused, and it can issue a warning about that, in at least a couple of implementations it does.
The ability to do that depends on the lexical environment being lexical: a thing you can know everything about by reading the code.
If it was possible to mutate the lexical environment at runtime, consider something like
(define-function counter
(lambda ()
(prog1 v
(incf v))))
(Assuming no global special declaration for v in the usual way).
What's the compiler to do? It's going to look at the code and say
what is
v? Is it a lexical variable? A dynamic variable? A symbol macro? I mean, really, why don't you come back to me when you've designed a language a compiler can actually do something with, OK? Until that time here's your stupid code back, you need someone else to do this job.
And then if you give it
(define-function counter
(let ((v 0))
(lambda ()
(prog1 v
(incf v)))))
it's going to say
OK, so now you're telling me that
vis a lexical binding, right? And you think that helps, right? But you reserve the right to change your mind at any future point? O...K... good, fine. You need to find somebody else to compile your code because I'm off to deal with a language with sane semantics, thanks: here's my notice.
And suddenly you've got a whole bunch of free memory and no compiler.
So a 'lexical environment' which can be modified, in the sense that bindings can be added or removed from it programmatically, after its creation isn't a lexical environment: it's something else.
Indeed in CL such an environment exists: it's the dynamic environment. And it is possible to programmatically extend this at runtime: that's what progv does.
The CLtL2 environment access features. [This is not part of the answer: it's an extended note to it.] The point of lexical environments is that they're lexical: you can read the code and know what is in the environment. And the time when you might want to read the code to know things about the lexical environment is before, or during, compile-time.
Well, CL's a Lisp: we spend a lot of time writing functions whose arguments are source code and whose values are other source code: macros. Being able to write macros is kind of the point of Lisp.
That has two problems in CL.
But the compiler (or interpreter) must necessarily have solutions to both these problems, because it needs to compile the language: the compiler needs to know what, say, v means. And because the environment is, well, lexical, because it can't be modified later to change the meaning of v, it can know this.
The CLtL2 environment access features address the second of these problems. They let the function corresponding to a macro both ask questions of the system and also to some extent to let the compiler (or other macros) know about things. So, for instance a macro can enquire about whether a lexical binding for a given variable will exist in its expansion. Note 'will exist' not 'does exist': the lexical environment information describes things that will be true in the code being processed, not things that are currently the case.
The CLtL2 environment access stuff wasn't considered sufficiently mature to be standardized, in particular in so far as it constrained compilers more than was desirable I think, and I think that's correct: it's not. Many implementations have it though.
So this addresses the second problem above. The first problem – that writing something which processes CL code and can tell you things about what the lexical environment will be – is something which can be solved by sufficiently-carefully-written user programs: the language is (or should be) well-enough specified that you can write a 'code walker' if you work hard enough. Some such exist.