After some feedback, I'm posting code that should reproduce the situation.
The purpose of my code is to generate and then execute a custom database query. The idea is to generate the query, define a macro with the query as the form, then execute the macro. Below is a very simplified idea of what's happening.
(defun test ()
(defvar quer '(query (:select '* :from 'table)))
(defmacro macquer () quer)
(macquer))
This macro throws the below error when executed by the function, but not when executed alone. Thoughts?
STYLE-WARNING:
MACQUER is being redefined as a macro when it was previously assumed to be a function.
debugger invoked on a UNDEFINED-FUNCTION in thread
#<THREAD "main thread" RUNNING {1000560083}>:
COMMON-LISP-USER::MACQUER is a macro, not a function.
NB: The query is from the postmodern library which hooks up to PostgreSQL. I wanted to show you the real query, but this condition holds for any form that I've tried.
This answer deals largely with the example given, and tries to explain why it can't work. At the end is some guesswork about what you might be trying to do and how to achieve that.
First of all it's useful to informally define various 'times' in the processing of a form by a CL system. These may be slightly different if the form is being compiled or merely evaluated, but they may not be. These definitions are informal and not completely correct: they're enough to answer the question but I've skated over some details.
Here is a rough enumeration of the various times:
defun
, defvar
and so on), then there may be many later 'use times' when the function is called, or the variable is referred to.The above times refer to a compiled implementation. In practice some interpreted implementations, which do not have times (3) and (4), may tangle up times (2) and (5). However you should always regard these times as happening strictly in order. That's because they do happen strictly in order for a compiler, and because an implementation need not have an interpreter: the evaluator can work by compilation. SBCL is an example of an implementation which (largely?) works this way.
This means that all macros in a form, apart from local macros, must be defined properly before evaluation or compilation starts, and anything they use in their expansion must therefore also be defined properly before evaluation or compilation starts.
The above description concerns the times when processing a single form. The file compiler has some additional behaviour which makes the language practical to use. This additional behaviour concerns top-level forms only (there is a subtle definition about what 'top-level' means which I will skate over here).
When processing a top-level form which is a defining form then the file compiler is required to take special note of at least two cases
There may be other cases: these are the two that concern us here.
Consider the following source file
(defmacro with-foo ((foo) &body forms)
`(call-with-foo (lambda (,foo)
,@forms)))
(defun call-with-foo (f)
...
(funcall f ...)
...)
(defun fash ()
(with-foo (my-foo)
(print my-foo)))
When compiling this the compiler is required to note that with-foo
is a macro, and it will therfore be able to expand the use of it in fash
. It does not need to make the definition of call-with-foo
available at compile time, and it will not in fact be available then. This file is fine.
Here is a file which is not fine
(defvar *print-hi* '(print "hi"))
(defmacro printing-hi ()
*print-hi*)
(defun cant-compile ()
(printing-hi))
So, OK, what happens when compiling this file?
*print-hi*
is a special variable and creates code to define it at load time;printing-hi
and also creates code to recreate it at load time;cant-compile
, expanding the macro printing-hi
... to a reference to *print-hi*
which does not yet existThe compilation thus fails.
Well, this is a common enough case that the designers of CL added a special hack to deal with it: you can tell the file compiler that certain toplevel forms should be evaluated immediately after compilation, and the results (side-effects, really) of this evaluation are thus available later in the compilation. There are many subtleties here which I am skipping over, but here is how you would do this:
(eval-when (:compile-toplevel :load-toplevel :execute)
(defvar *print-hi* '(print "hi")))
(defmacro printing-hi ()
*print-hi*)
(defun can-now-compile ()
(printing-hi))
The eval-when
special operator, when used like this, tells the compiler that it should evaluate the defvar
form, and thus define *print-hi*
at compile time. This means printing-hi
can now be expanded at compile time and everything will be well.
Note one thing however: the value of *print-hi*
that will be used is the value at compile time not any later value. The effective definition of can-now-compile
is
(defun can-now-compile ()
(print "hi"))
And this does not change if you later change *print-hi*
: the macro is already expanded.
So now, consider your form:
(defun test ()
(defvar quer '(query (:select '* :from 'table)))
(defmacro macquer () quer)
(macquer))
Let's approximate the expansion of the defun
, defvar
and defmacro
macros which appear in this form and which the compiler will expand to give an expansion like this:
(progn
;; expansion of DEFUN
...
;; Something like this
(setf (fdefinition 'test)
(lambda ()
(block test
(progn
;; expansion of DEFVAR
...
;; this or some equivalent will probably be there
;; somewhere
(eval-when (:compile-toplevel :load-toplevel :execute)
(proclaim '(special 'quer)))
...
;; this or some equivalent will be there
(unless (boundp 'quer)
(setf (symbol-value 'quer)
'(query (:select '* :from 'table))))
...)
(progn
;; expansion of DEFMACRO
...
;; Some equivalent of this must be there, the actual
;; macro function may be different
(setf (macro-function 'macquer)
(lambda (...) quer))
...)
(macquer))))
...
'test)
There are still macros remaining in this: setf
and lambda
, but I'm not going to expand them as they're not interesting here.
OK, so this form now gets compiled. The resulting code will, at evaluation time (so after compilation time!), install a definition for a function named test
. There is no definition of macquer
, so the compiler will assume that this is a function call, as it should.
Note that, even when this form is evaluated neither the variable quer
nor the macro macquer
become defined: what happens is that a function, test
, is defined and calling test
will cause quer
and macquer
to be defined.
When test
is later called, it will thus do three things:
quer
;macquer
, perhaps even with the correct definition depending on how the compiler handles the unknown variable reference in its body;macquer
(and fail).In addition the compilation process will almost certainly produce one, and perhaps two, warnings:
quer
which is not known;macquer
is made.OK, so what's the underlying problem here? It's that you are confusing the various times I've outlined above. The function test
is referring to something called macquer
, but that is (or will later be) a macro which is defined only when and if the function is called. This can't work. The macro function itself, when compiled, attempts to refer to a variable, quer
which is also not yet defined.
Additionally the function definition involves things like defvar
and defmacro
which are almost never correct: while this is technically legal, those forms are almost always a mistake when they occur other than at toplevel, where the compiler can notice them and treat them appropriately (see compilation and in particular file compilation). And in this case they are a mistake.
The important thing here is that macros are functions which transform source code to other source code, and as such they need to be fully known at the point that source code is processed, either by the compiler or the interpreter, if there is one. If the expansion of the macro refers to a variable then that variable must therefore also be known at that time. If the expansion of the macro involves a call to a function then that function must be known at that time.
To make this 'work' you need to make sure that things are defined when you need them to be. As an example, based on the above examples of the file compiler, this will work:
(eval-when (:compile-toplevel :load-toplevel :execute)
;; This variable must be defined at macroexpansion time
(defvar *query* '(query (:select '* :from 'table))))
(defmacro querying ()
*query*)
(defun test ()
(querying))
This works and shows that the expansion of a macro can be just a variable whose value is the source-code you want, but that variable needs to be known about at macroexpansion time, and the value that will be used will be its value at macroexpansion time.
But the chances are that this is not what you want. In particular this won't work:
(defun test ()
(compute-query)
(querying))
(defun compute-query ()
(setf *query* ...))
test
will use the value of *query*
at compile time, not any later value.
My guess is that you want to programmatically build up the query, and then evaluate it. Well, the answer to that is not to use macros at all: Postmodern allows you to dynamically compile its s-sql language, and its query
macro understands this.
So instead, do this, perhaps:
(defvar *default-query*
'(query (:select '* :from 'table)))
(defun test (&optional (query *default-query*))
(query (sql-compile query)))
Note that *default-query*
is now really an optional thing. There are things you can't do this way, I think. In particular query
allows you to refer to lexical variables in the expansion which sql-compile
, being a function, can't do.