How does one define a macro using variable to supply the form and have it function properly in SBCL?

After some feedback, I'm posting code that should reproduce the situation.

The purpose of my code is to generate and then execute a custom database query. The idea is to generate the query, define a macro with the query as the form, then execute the macro. Below is a very simplified idea of what's happening.

1. Define a function, which:
2. Defines our query string
3. And a macro whose form is the variable containing the query string
4. Then executes the macro

(defun test ()
  (defvar quer '(query (:select '* :from 'table)))
  (defmacro macquer () quer)
  (macquer))

This macro throws the below error when executed by the function, but not when executed alone. Thoughts?

STYLE-WARNING:
   MACQUER is being redefined as a macro when it was previously assumed to be a function.

debugger invoked on a UNDEFINED-FUNCTION in thread
#<THREAD "main thread" RUNNING {1000560083}>:
  COMMON-LISP-USER::MACQUER is a macro, not a function.

NB: The query is from the postmodern library which hooks up to PostgreSQL. I wanted to show you the real query, but this condition holds for any form that I've tried.

This answer deals largely with the example given, and tries to explain why it can't work. At the end is some guesswork about what you might be trying to do and how to achieve that.

Times

First of all it's useful to informally define various 'times' in the processing of a form by a CL system. These may be slightly different if the form is being compiled or merely evaluated, but they may not be. These definitions are informal and not completely correct: they're enough to answer the question but I've skated over some details.

Here is a rough enumeration of the various times:

1. read time, when the text of the form is read from a stream of characters and turned into an expression of some kind;
2. macroexpansion time, when macros in the form are expanded until none remain;
3. compile time when the form is compiled;
4. possibly another time here when a representation of the compiled form is written to a file
5. evaluation time when the form is evaluated, which may also be load time if the previous step happened;
6. if the form is some kind of defining form (defun, defvar and so on), then there may be many later 'use times' when the function is called, or the variable is referred to.

The above times refer to a compiled implementation. In practice some interpreted implementations, which do not have times (3) and (4), may tangle up times (2) and (5). However you should always regard these times as happening strictly in order. That's because they do happen strictly in order for a compiler, and because an implementation need not have an interpreter: the evaluator can work by compilation. SBCL is an example of an implementation which (largely?) works this way.

This means that all macros in a form, apart from local macros, must be defined properly before evaluation or compilation starts, and anything they use in their expansion must therefore also be defined properly before evaluation or compilation starts.

Some special cases in the file compiler

The above description concerns the times when processing a single form. The file compiler has some additional behaviour which makes the language practical to use. This additional behaviour concerns top-level forms only (there is a subtle definition about what 'top-level' means which I will skate over here).

When processing a top-level form which is a defining form then the file compiler is required to take special note of at least two cases

• if the form defines a dynamic variable then the compiler must take note that references to and bindings of this variable are special later in the file
• if the form defines a macro, then the compiler must make the macro available later in the file, so that uses of it can be expanded at compile time.

There may be other cases: these are the two that concern us here.

Examples of how the file compiler deals with things

Consider the following source file

(defmacro with-foo ((foo) &body forms)
  `(call-with-foo (lambda (,foo)
                    ,@forms)))

(defun call-with-foo (f)
  ...
  (funcall f ...)
  ...)

(defun fash ()
  (with-foo (my-foo)
    (print my-foo)))

When compiling this the compiler is required to note that with-foo is a macro, and it will therfore be able to expand the use of it in fash. It does not need to make the definition of call-with-foo available at compile time, and it will not in fact be available then. This file is fine.

Here is a file which is not fine

(defvar *print-hi* '(print "hi"))

(defmacro printing-hi ()
  *print-hi*)

(defun cant-compile ()
  (printing-hi))

So, OK, what happens when compiling this file?

• firstly the compiler notes that *print-hi* is a special variable and creates code to define it at load time;
• then it notes the definition of printing-hi and also creates code to recreate it at load time;
• then it tries to compile cant-compile, expanding the macro printing-hi ... to a reference to *print-hi* which does not yet exist

The compilation thus fails.

Well, this is a common enough case that the designers of CL added a special hack to deal with it: you can tell the file compiler that certain toplevel forms should be evaluated immediately after compilation, and the results (side-effects, really) of this evaluation are thus available later in the compilation. There are many subtleties here which I am skipping over, but here is how you would do this:

(eval-when (:compile-toplevel :load-toplevel :execute)
  (defvar *print-hi* '(print "hi")))

(defmacro printing-hi ()
  *print-hi*)

(defun can-now-compile ()
  (printing-hi))

The eval-when special operator, when used like this, tells the compiler that it should evaluate the defvar form, and thus define *print-hi* at compile time. This means printing-hi can now be expanded at compile time and everything will be well.

Note one thing however: the value of *print-hi* that will be used is the value at compile time not any later value. The effective definition of can-now-compile is

(defun can-now-compile ()
  (print "hi"))

And this does not change if you later change *print-hi*: the macro is already expanded.

Your form and its evils

So now, consider your form:

(defun test ()
  (defvar quer '(query (:select '* :from 'table)))
  (defmacro macquer () quer)
  (macquer))

Let's approximate the expansion of the defun, defvar and defmacro macros which appear in this form and which the compiler will expand to give an expansion like this:

(progn
  ;; expansion of DEFUN
  ...
  ;; Something like this
  (setf (fdefinition 'test)
        (lambda ()
          (block test
            (progn
              ;; expansion of DEFVAR
              ...
              ;; this or some equivalent will probably be there
              ;; somewhere
              (eval-when (:compile-toplevel :load-toplevel :execute)
                (proclaim '(special 'quer)))
              ...
              ;; this or some equivalent will be there
              (unless (boundp 'quer)
                (setf (symbol-value 'quer)
                      '(query (:select '* :from 'table))))
              ...)
            (progn
              ;; expansion of DEFMACRO
              ...
              ;; Some equivalent of this must be there, the actual
              ;; macro function may be different
              (setf (macro-function 'macquer)
                    (lambda (...) quer))
              ...)
            (macquer))))
  ...
  'test)

There are still macros remaining in this: setf and lambda, but I'm not going to expand them as they're not interesting here.

OK, so this form now gets compiled. The resulting code will, at evaluation time (so after compilation time!), install a definition for a function named test. There is no definition of macquer, so the compiler will assume that this is a function call, as it should.

Note that, even when this form is evaluated neither the variable quer nor the macro macquer become defined: what happens is that a function, test, is defined and calling test will cause quer and macquer to be defined.

When test is later called, it will thus do three things:

• define a global special variable named quer;
• define a macro named macquer, perhaps even with the correct definition depending on how the compiler handles the unknown variable reference in its body;
• then attempt to call a function named macquer (and fail).

In addition the compilation process will almost certainly produce one, and perhaps two, warnings:

• a reference is made to a variable quer which is not known;
• a call to an an unknown function, macquer is made.

Making this work part one

OK, so what's the underlying problem here? It's that you are confusing the various times I've outlined above. The function test is referring to something called macquer, but that is (or will later be) a macro which is defined only when and if the function is called. This can't work. The macro function itself, when compiled, attempts to refer to a variable, quer which is also not yet defined.

Additionally the function definition involves things like defvar and defmacro which are almost never correct: while this is technically legal, those forms are almost always a mistake when they occur other than at toplevel, where the compiler can notice them and treat them appropriately (see compilation and in particular file compilation). And in this case they are a mistake.

The important thing here is that macros are functions which transform source code to other source code, and as such they need to be fully known at the point that source code is processed, either by the compiler or the interpreter, if there is one. If the expansion of the macro refers to a variable then that variable must therefore also be known at that time. If the expansion of the macro involves a call to a function then that function must be known at that time.

To make this 'work' you need to make sure that things are defined when you need them to be. As an example, based on the above examples of the file compiler, this will work:

(eval-when (:compile-toplevel :load-toplevel :execute)
  ;; This variable must be defined at macroexpansion time
  (defvar *query* '(query (:select '* :from 'table))))

(defmacro querying ()
  *query*)

(defun test ()
  (querying))

This works and shows that the expansion of a macro can be just a variable whose value is the source-code you want, but that variable needs to be known about at macroexpansion time, and the value that will be used will be its value at macroexpansion time.

But the chances are that this is not what you want. In particular this won't work:

(defun test ()
  (compute-query)
  (querying))

(defun compute-query ()
  (setf *query* ...))

test will use the value of *query* at compile time, not any later value.

Making this work, part two

My guess is that you want to programmatically build up the query, and then evaluate it. Well, the answer to that is not to use macros at all: Postmodern allows you to dynamically compile its s-sql language, and its query macro understands this.

So instead, do this, perhaps:

(defvar *default-query* 
  '(query (:select '* :from 'table)))

(defun test (&optional (query *default-query*))
  (query (sql-compile query)))

Note that *default-query* is now really an optional thing. There are things you can't do this way, I think. In particular query allows you to refer to lexical variables in the expansion which sql-compile, being a function, can't do.