rustmacros

Unexpected token when parsing rust code using syn


Trying to teach myself rust procedural macros using Tolnay's workshop. I'm on test 2 in seq: https://github.com/dtolnay/proc-macro-workshop/blob/master/seq/tests/02-parse-body.rs

I created a struct to hold the results of parsing and implemented Parse:

use proc_macro::TokenStream as TokenStream1;
use proc_macro2::*;
use syn::*;
use parse::{Parse,ParseStream};

#[derive(Debug)]
struct Seq {
  ident: Ident,
  in_token: Token![in],
  start: LitInt,
  dots_token: Token![..],
  end: LitInt,
  brace_token: token::Brace,
  body: TokenStream,
}

impl Parse for Seq {
  fn parse(input: ParseStream<'_>) -> Result<Self> {
    let content;
    let result = Seq {
      ident: input.parse()?,
      in_token: input.parse()?,
      start: input.parse()?,
      dots_token: input.parse()?,
      end: input.parse()?,
      brace_token: braced!(content in input),
      body: content.cursor().token_stream(),
    };
    Ok(dbg!(result))
  }
}

#[proc_macro]
pub fn seq(input: TokenStream1) -> TokenStream1 {
  let seq: Result<Seq> = syn::parse(input);
  dbg!(seq);

  TokenStream1::default()
}

Then I ran it using the following code:

use seq::seq;

macro_rules! expand_to_nothing {
    ($arg:literal) => {
        // nothing                                                              
    };
}

seq!(N in 0..4 {
    expand_to_nothing!(N);
});

fn main() {}

The two debug statements print as follows:

[seq/src/lib.rs:29:8] result = Seq {
    ident: Ident {
        ident: "N",
        span: #0 bytes(252..253),
    },
    in_token: In,
    start: LitInt {
        token: 0,
    },
    dots_token: DotDot,
    end: LitInt {
        token: 4,
    },
    brace_token: Brace,
    body: TokenStream [
        Ident {
            ident: "expand_to_nothing",
            span: #0 bytes(268..285),
        },
        Punct {
            ch: '!',
            spacing: Alone,
            span: #0 bytes(285..286),
        },
        Group {
            delimiter: Parenthesis,
            stream: TokenStream [
                Ident {
                    ident: "N",
                    span: #0 bytes(287..288),
                },
            ],
            span: #0 bytes(286..289),
        },
        Punct {
            ch: ';',
            spacing: Alone,
            span: #0 bytes(289..290),
        },
    ],
}
[seq/src/lib.rs:36:3] seq = Err(
    Error(
        "unexpected token, expected `}`",
    ),
)

The first of those statements is exactly what I want to see, it shows that the Parse implementation did exactly what I want it to and returned without error. But why did I then get an error from syn::parse?


Solution

  • The documentation of parse doesn't spell it out, but in parse2s (the proc_macro2 equivalent) you can find the following:

    This function will check that the input is fully parsed. If there are any unparsed tokens at the end of the stream, an error is returned.

    The problem is that you do not parse any tokens within the braces. cursor explicitly notes:

    Cursors are immutable so no operations you perform against the cursor will affect the state of this parse stream.

    So syn::parse expects there to be no tokens within the braces. Everything extra is considered extraneous and results in the error you see because you did not consume the whole input:ParseStream.

    Fortunately it's easy to fix, just replace

          body: content.cursor().token_stream(),
    

    which doesn't parse contents tokens, with

          body: content.parse()?,
    

    which does properly parse & consume the contents.