yaccjison

With Jison, how do I scan right shift operator and nested generic type definitions


I'm working on a grammar for a language that supports the right shift operator and generic types. For example:

function rectangle(): Pair<Tuple<Float, Float>> {
    let x = 0 >> 2; 
}

My problem is, during scanning, the right shift operator is correctly tokenized, but the >> in Pair<Tuple<Float, Float>> becomes a single >> token instead of two separate > tokens (unless I add a space). This is because I have the >> before the > in my .jison file:

">>"    { return '>>' }
">"     { return '>' }

Is there a good way to resolve this in Jison? I feel like this is a common problem as my syntax is similar to every other C-style language, but I haven't found a solution to it yet (besides writing a pre-scan script that manually space-delimits the >s).


Solution

  • The easiest solution is to just not recognize >> as a single token in the lexer. Instead, in your parser, recognize two consecutive > tokens as a right shift, and then check to make sure there's nothing (no whitespace or comments) between them (and give a syntax error if there is).