In the process of developing a custom Jinja2 extension that creates namespaces with dynamically evaluated names, I need to use the result of the evaluation of a template expression as argument to a Name node.
The root of the issue is that the Name node's first argument must be of string type, while the result of parsing the template expression is an AST node. But the problem is, the AST results from static evaluation, while the expression requires runtime evaluation (because expressions can contain variables whose value is not known at parse time). Is there a way to bridge that gap?
Follows a sample extension (not the actual, more complex extension that I'm working on) that captures the essence of what I'm trying to accomplish:
{% set my_name = 'my_namespace' %}
{% namespace my_name %}
{# A namespace named 'my_namespace' should now exists #}
Of course, I can easily parse the name expression and have the result of its evaluation displayed with an Output node. I can also create namespaces with predefined literal names:
from jinja2 import nodes
from jinja2.ext import Extension
class NamespaceExtension(Extension):
tags = {"namespace"}
def __init__(self, environment):
super().__init__(environment)
def parse(self, parser):
lineno = next(parser.stream).lineno
eval_context = nodes.EvalContext(self.environment)
name = parser.parse_expression()
##
## This obviously works as expected and outputs the result
## of evaluating `name` as an expression.
##
return nodes.Output([name]).set_lineno(lineno)
##
## This below also works — as a proof of concept — but we
## need the namespace name to be evaluated dynamically.
##
# return nodes.Assign(
# nodes.Name('my_namespace', 'store'),
# nodes.Call(nodes.Name('namespace', 'load'), [], [], None, None)
# ).set_lineno(lineno)
namespace = NamespaceExtension
The following attempts, however, don't work (I actually only had hope in the first-to-last attempt, so not too surprised regarding most of the others, but here they are nonetheless, if only for the sake of completeness and demonstration).
return nodes.Assign(
name,
nodes.Call(nodes.Name('namespace', 'load'), [], [], None, None)
).set_lineno(lineno)
# SyntaxError: cannot assign to function call
Right, we need the expression's value.
return nodes.Assign(
name.as_const(),
nodes.Call(nodes.Name('namespace', 'load'), [], [], None, None)
).set_lineno(lineno)
# RuntimeError: if no eval context is passed, the node must have an attached environment.
Well, ok, easy enough.
return nodes.Assign(
name.as_const(eval_context),
nodes.Call(nodes.Name('namespace', 'load'), [], [], None, None)
).set_lineno(lineno)
# jinja2.nodes.Impossible
Might have better luck attaching the environment.
name.set_environment(self.environment)
return nodes.Assign(
name.as_const(),
nodes.Call(nodes.Name('namespace', 'load'), [], [], None, None)
).set_lineno(lineno)
# jinja2.nodes.Impossible
Of course, expressions are parsed by default with a 'load' context and we need a Name node with a 'store' context.
name.set_ctx('store')
return nodes.Assign(
name,
nodes.Call(nodes.Name('namespace', 'load'), [], [], None, None)
).set_lineno(lineno)
# SyntaxError: cannot assign to function call
How about wrapping the name expression in 'load' context inside of a Name node with a 'store' context?
return nodes.Assign(
nodes.Name(name, 'store'),
nodes.Call(nodes.Name('namespace', 'load'), [], [], None, None)
).set_lineno(lineno)
# TypeError: '<' not supported between instances of 'Getattr' and 'str'
The Name node's first argument must be a string, so then how about evaluating name as a constant?
return nodes.Assign(
nodes.Name(name.as_const(), 'store'),
nodes.Call(nodes.Name('namespace', 'load'), [], [], None, None)
).set_lineno(lineno)
# jinja2.nodes.Impossible
Or perhaps this? Out of sheer despair...
return nodes.Assign(
nodes.Name(nodes.Const(name.as_const()), 'store'),
nodes.Call(nodes.Name('namespace', 'load'), [], [], None, None)
).set_lineno(lineno)
#TypeError: '<' not supported between instances of 'str' and 'Getattr'
You're trying to dynamically create a variable whose name is only known at runtime, but as you already pointed out, the name of an assignment target is determined during compilation, so simply tinkering with the AST isn't going to help.
As a workaround you can make the symbol mapping table available at runtime by embedding it as a dict representation in the Python source code that Jinja2's compiler generates. With the mapping the runtime code can then translate the evaluated name to the corresponding compiled identifier to create the variable at runtime by adding the name to the current frame's f_locals attribute, which is now a write-through proxy dict since Python 3.13 with the implementation of PEP-667.
But no existing node type is going to generate the code that does all that for you. You'll need to create a custom node type with its own visitor method for the compiler's code generator.
Unfortunately, if you try creating a custom node type by subclassing an existing node type you'll get a TypeError: can't create custom node types exception because the constructor of the node type's metaclass, nodes.NodeType.__new__, is neutered with a dummy function after all built-in node types are created:
# make sure nobody creates custom nodes
def _failing_new(*args: t.Any, **kwargs: t.Any) -> "te.NoReturn":
raise TypeError("can't create custom node types")
NodeType.__new__ = staticmethod(_failing_new) # type: ignore
del _failing_new
So to work around it you can replace the dummy constructor with type.__new__, Python's default metaclass constructor, and make the new custom node type subclass an existing node type with an expression as a field, such as nodes.Assign, so the new node type can reuse the target field while having its own name and therefore its own vistor for the code generator.
Unlike an assignment, your namespace keyword does not need a RHS node so you can simply pass None as the node field:
from jinja2 import Environment, nodes, ext, compiler
nodes.NodeType.__new__ = type.__new__
class NamespaceStmt(nodes.Assign):
pass
class NamespaceCodeGenerator(compiler.CodeGenerator):
def visit_NamespaceStmt(self, node, frame):
self.writeline('import sys')
self.writeline(f'sys._getframe().f_locals[{frame.symbols.refs!r}[')
self.visit(node.target, frame)
self.write(']] = Namespace()')
class NamespaceExtension(ext.Extension):
tags = {"namespace"}
def parse(self, parser):
self.environment.code_generator_class = NamespaceCodeGenerator
lineno = next(parser.stream).lineno
return NamespaceStmt(parser.parse_expression(), None).set_lineno(lineno)
so that:
content = '''\
{% set my_name = 'my_namespace' -%}
{% namespace my_name -%}
{% set my_namespace.foo = 'bar' -%}
{{ my_namespace.foo }}
'''
print(Environment(extensions=[NamespaceExtension]).from_string(content).render())
outputs:
bar
Note that only names that are explicitly assigned to would appear in the symbol mapping table, so in the example above if you remove the line {% set my_namespace.foo = 'bar' -%}, the compiler would not know ahead of time to add an identifier for the name my_namespace into the symbol table, causing the reference to my_namespace in {{ my_namespace.foo }} to produce an error.
Since the identifier of a name is really implemented by adding a prefix to the given name where the prefix is fixed for the scope, you can "predict" the identifier of an undefined name by obtaining the prefix of a dummy variable, named _Namespace in the example below:
from jinja2 import Environment, nodes, ext, compiler
nodes.NodeType.__new__ = type.__new__
class NamespaceStmt(nodes.Assign):
pass
class NamespaceCodeGenerator(compiler.CodeGenerator):
def visit_NamespaceStmt(self, node, frame):
self.writeline('import sys')
self.writeline(f'sys._getframe().f_locals[{frame.symbols.refs!r}')
# get the identifier prefix by removing the trailing dummy name
self.write('["_Namespace"][:-10] + ')
self.visit(node.target, frame)
self.write('] = Namespace()')
class NamespaceExtension(ext.Extension):
tags = {"namespace"}
def parse(self, parser):
self.environment.code_generator_class = NamespaceCodeGenerator
lineno = next(parser.stream).lineno
return [
nodes.Assign(nodes.Name('_Namespace', 'store'), nodes.Const(None)),
NamespaceStmt(parser.parse_expression(), None).set_lineno(lineno)
]
so that the following snippet will render nothing as expected while the original test case still works:
content = '''\
{% set my_name = 'my_namespace' -%}
{% namespace my_name -%}
{{ my_namespace.foo }}
'''
print(Environment(extensions=[NamespaceExtension]).from_string(content).render())