I'm creating a new programing language just for fun and to know understand languages at more basic level. I started with writing a lexer with moo js and everything is working fine except for NEWLINE. I tried many things but it just won't resolve. I even tried by copying exact piece of code from moo js's documentation but still no help.
Lexer Code:
const moo = require("moo");
const lexer = moo.compile({
whitespace: /[ \t]+/,
// comment: /\/\/.*?$/,
number: /0|[1-9][0-9]*/,
string: /"(?:\\["\\]|[^\n"\\])*"/,
leftParen: '(',
rightParen: ')',
// keyword: ['while', 'if', 'else', 'moo', 'cows'],
assignmentOp: "=",
identifier: /[a-zA-Z_][a-zA-Z0-9_]*/,
newline: { match: /\n/, lineBreaks: true },
});
module.exports = lexer;
text-lexer code:
const fs = require("fs").promises;
const lexer = require("./lexer");
async function main() {
const code = (await fs.readFile("example1.hin")).toString();
lexer.reset(code);
let token;
while (true) {
token = lexer.next();
if (token) {
console.log("Got token", token);
} else {
break;
}
}
}
main().catch(err => console.log(err.stack));
test-example:
n = 4
m = 6
I also faced the same issue and solution was to change the regex for new line as Windows and Linux handled newline differently ( To find out more. check this out ). The one you mentioned :
newline: { match: /\n/, lineBreaks: true },
works for Linux
To work on both, use this regex:
newline: { match: /\r?\n/, lineBreaks: true },