I'm having an issue with my ANTLR4 grammar not parsing a string correctly. I'm more interested in learning how to solve my problem than solving my specific problem. How can I generate any type of debug information? I want to know what the parser is "thinking" as it parses the string.
The grammar can be found here: https://github.com/Metrink/metrink-fe/blob/master/metrink.g4
I'm using the simple test string: -1d metric('blah', 'blah', 'blah')
I get the following error: 1:2 missing TIME_INDICATOR at 'd'
The grammar defines TIME_INDICATOR
as [shmd]
so I'm not sure how it's missing a TIME_INDICATOR
at the character d
when that is one of the possible tokens. What am I missing here?
I'm using Python3 generated from ANTLR4.
What I usually do is first dump the tokens to see if the actual tokens the parser expects are created.
You can do that with a small test class like this (easily ported to Python):
public class Main {
static void test(String input) {
metrinkLexer lexer = new metrinkLexer(new ANTLRInputStream(input));
CommonTokenStream tokenStream = new CommonTokenStream(lexer);
tokenStream.fill();
System.out.printf("input: `%s`\n", input);
for (Token token : tokenStream.getTokens()) {
if (token.getType() != TLexer.EOF) {
System.out.printf(" %-20s %s\n", metrinkLexer.VOCABULARY.getSymbolicName(token.getType()), token.getText());
}
}
System.out.println();
}
public static void main(String[] args) throws Exception {
test("-1d metric('blah', 'blah', 'blah')");
}
}
If you run the code above, the following will get printed to your console:
input: `-1d metric('blah', 'blah', 'blah')`
MINUS -
INTEGER_LITERAL 1
IDENTIFIER d
METRIC metric
LPAREN (
STRING_LITERAL 'blah'
COMMA ,
STRING_LITERAL 'blah'
COMMA ,
STRING_LITERAL 'blah'
RPAREN )
As you can see, the d
is being tokenized as a IDENTIFIER
instead of an TIME_INDICATOR
. This is because the IDENTIFIER
rule is defined before your TIME_INDICATOR
rule. The lexer does not "listen" to what the parser might need, it simply matches the most characters as possible, and if two or more rules match the same amount of characters, the rule defined first "wins".
So, d
can either be tokenized as TIME_INDICATOR
or an IDENTIFIER
. If this is dependent on context, I suggest you tokenize it as a IDENTIFIER
(and remove TIME_INDICATOR
) and create a parser rule like this:
relative_time_literal:
MINUS? INTEGER_LITERAL time_indicator;
time_indicator:
{_input.LT(1).getText().matches("[shmd]")}? IDENTIFIER;
The { ... }?
is called a predicate: Semantic predicates in ANTLR4?
Also, FALSE
and TRUE
will need to be placed before the IDENTIFIER
rule.
Petr Pivonka wrote:
PAY ATTENTION! The note "easily ported to Python" needs to be explained! [...]
In Python that could look like this:
import antlr4
from metrinkLexer import metrinkLexer
def test(source):
lexer = metrinkLexer(antlr4.InputStream(source))
token_stream = antlr4.CommonTokenStream(lexer)
token_stream.fill()
print(f"input: {source}")
for token in [t for t in token_stream.tokens if t.type != -1]:
print(f" {lexer.symbolicNames[token.type].ljust(20)}{token.text}")
if __name__ == '__main__':
test("-1d metric('blah', 'blah', 'blah')")
which will print:
input: -1d metric('blah', 'blah', 'blah')
MINUS -
INTEGER_LITERAL 1
IDENTIFIER d
METRIC metric
LPAREN (
STRING_LITERAL 'blah'
COMMA ,
STRING_LITERAL 'blah'
COMMA ,
STRING_LITERAL 'blah'
RPAREN )
In other words:
metrinkLexer.VOCABULARY.getSymbolicName(type)
becomes metrinkLexer.symbolicNames[type]
token.getType()
becomes token.type
token.getText()
becomes token.text
Petr Pivonka wrote:
It took me one day of labouring before I found that Java and Python are completely different nad that Python one is just empty shell.
That is not true. Everything in the Java API is also in the Python API (and also in C#, JavaScript, TypeScript, etc).