Multiple, perhaps most, language implementations that include a compiler at runtime neglect to garbage-collect discarded code (See, for example julia, where this leads to memory leaks in applications like genetic-programming)
My preliminary tests indicate that Chez Scheme does not leak memory here, but I would like to know with greater certainty, since I don't even know if f and g actually get compiled. (The old mantra: "Tests can only prove the presence of bugs, not their absence")
The test I tried: f and g call each other, and their definitions get replaced at runtime.
(define f)
(define g)
(define (make-f x)
(eval `(set! f (lambda (y)
(if (> y 100)
(+ (remainder ,x 3) (g y))
(+ y 1))))))
(define (make-g x)
(eval `(set! g (lambda (y)
(if (< y 10)
(+ (remainder ,x 5) (f y))
(div y 2))))))
(define (make-and-run-f n)
(begin
(make-f 1)
(make-g 1)
(let loop ((i 0) (acc 0))
(if (> i n)
acc
(begin
(make-f i)
(make-g i)
(loop (+ i 1) (+ acc (f 33))))))))
(time (make-and-run-f 1000000)) ; runs in 10 min and negligible memory
Given the importance of both procedures and garbage collection to Scheme, I would be surprised if Chez Scheme did not try to garbage collect any dynamically created objects. The R6RS Standard says [emphasis mine]:
All objects created in the course of a Scheme computation, including procedures and continuations, have unlimited extent. No Scheme object is ever destroyed. The reason that implementations of Scheme do not (usually!) run out of storage is that they are permitted to reclaim the storage occupied by an object if they can prove that the object cannot possibly matter to any future computation.
A procedure is an object, and any object may be garbage collected if the implementation can prove that the computation will not need it again. This is not a requirement, but that goes for any object, not just for procedures.
The Chez Scheme manual seems definitive, though (Chez Scheme Version 9 User's Guide, p. 82):
Since all Scheme objects, including code objects, can be relocated or even reclaimed by the garbage collector....
In the 1990s Kent Dybvig wrote a paper together with David Eby and Carl Bruggeman which may be of interest here, called Don’t Stop the BIBOP: Flexible and Efficient Storage Management for Dynamically Typed Languages, which describes the garbage collection strategy implemented in Chez Scheme. In the paper some time is spent discussing "code objects" and in particular how they are segregated and treated differently during the garbage collection process (since they may contain pointers to other objects).