webpacknext.jsabstract-syntax-treetreesitter

How to add web-tree-sitter to a NextJS project?


I want to be able to generate ASTs by parsing code typed on my NextJS web app. I saw that I should be using web-tree-sitter to achieve such a thing.

I successfully used Docker to generate the tree-sitter-javascript.wasm file, moved it to be in the same directory as my Page.tsx file (where I want to call the parsing function), and loading the parser like this: (as stated on their npm package's site).

     locateFile(scriptName: string, scriptDirectory: string) {
       return scriptName;
     },
   });

And then defining the parser and the JavaScript grammar like this, just like it says

  const JavaScript = await Parser.Language.load('tree-sitter-javascript.wasm');
  parser.setLanguage(JavaScript);

I am getting this error:

./node_modules/web-tree-sitter/tree-sitter.js:1:1044
Module not found: Can't resolve 'fs'

I also read on the npm package's site that adding a webpack config that makes the bundler ignore Node's fs which doesn't exist in the browser. I'm trying to add it via next.config.js like this:

const nextConfig = {
    // Other next.js config
  
    // webpack: (config, options) => {
    //   config.resolve.fallback = {
    //     fs: false,
    //   }
  
    //   return config
    // }
  }
  
  module.exports = nextConfig

However, I'm still getting the Can't resolve fs error.

What should I do differently?


Solution

  • I managed to make web-tree-sitter work on my project. Here is how:

    Add the Webpack config to next.config.js file:

    module.exports = {
        webpack: (config, options) => {
        config.resolve.fallback = {
          fs: false,
        }
    
        return config
      },
    }
    

    Add the tree-sitter.wasm and tree-sitter-javascript.wasm files to the public/ folder.

    That should work with the latest NextJS. And here is an example use with the app router (page.tsx):

    "use client"
    
    import React from "react"
    import Parser from "web-tree-sitter"
    
    const exampleCode = `
    const MyComp = () => {
      const [count, setCount] = React.useState(0)
      return (
        <div>
          <h1 className="text-4xl">Hello World</h1>
          <button onClick={() => setCount(count + 1)}>Click Me</button>
          <p>Count: {count}</p>
        </div>
      )
    }
    `
    
    export default function TreeSitterTest() {
      const [isReady, setIsReady] = React.useState(false)
      const [code, setCode] = React.useState(exampleCode)
      const [AST, setAST] = React.useState<Parser.Tree | null>(null)
    
      const parserRef = React.useRef<Parser>()
    
      React.useEffect(() => {
        ;(async () => {
          await Parser.init({
            locateFile(scriptName: string, scriptDirectory: string) {
              return scriptName
            },
          })
    
          const parser = new Parser()
          const Lang = await Parser.Language.load("tree-sitter-javascript.wasm")
    
          setIsReady(true)
    
          parser.setLanguage(Lang)
    
          parserRef.current = parser
        })()
      }, [])
    
      React.useEffect(() => {
        if (!isReady) return
        if (!parserRef.current) return
    
        const tree = parserRef.current.parse(code)
        console.log("rootNode:", tree.rootNode)
    
        setAST(tree)
      }, [code, isReady])
    
      return (
        <div>
          <h1 className="p-2 text-4xl">Tree Sitter Test</h1>
    
          <div className="flex-cole flex h-96 gap-1 p-1">
            <textarea
              className="rounder-sm flex-1 overflow-auto whitespace-pre-wrap border border-gray-300 p-2"
              value={code}
              onChange={(e) => setCode(e.target.value)}
            />
            <div className="rounder-sm flex-1 overflow-auto whitespace-pre-wrap border border-gray-300 p-2">
              {AST?.rootNode.toString()}
            </div>
          </div>
        </div>
      )
    }
    

    The project I implemented this on is OSS, so you can check it out directly there: