How to use WASM Tiktoken in React Webpack JS

The build issues due to wasm instantiate or wasm not being imported.

wasm.__wbindgen_add_to_stack_pointer is not a function

To use tiktoken wasm in react for better performance, follow this for webpack 5.

In webpack config override add the following. You can directly use wasm from js cdn as well. Skip this step if you want to use cdn:

module.exports = (config, env) => {

  config.experiments = {
    asyncWebAssembly: true,
    layers: true,
  };

}

To use tiktoken with config override:

import { get_encoding, init } from 'tiktoken/init';
import wasm from 'tiktoken/tiktoken_bg.wasm';

  let tikInit = false;
  const countTokens= (str) =>{
    if (!tikInit) {
    tikInit = true;

      const wasmFile = await fetch(wasm);
      const buffer = await wasmFile.arrayBuffer();
      await init((imports) => WebAssembly.instantiate(buffer, imports));
    }

    const e = get_encoding('cl100k_base');
    const t = e.encode(str);
    e.free();
    return t.length;
  }

To use tiktoken with js cdn for wasm and without config override:

import { get_encoding, init } from 'tiktoken/init';
import wasm from 'tiktoken/tiktoken_bg.wasm';

  let tikInit = false;
  const countTokens= (str) =>{
    if (!tikInit) {
    tikInit = true;
}
   const wasmFile = await fetch(
      'https://cdn.jsdelivr.net/npm/tiktoken@1.0.15/tiktoken_bg.wasm'
    );
 
      const buffer = await wasmFile.arrayBuffer();
      await init((imports) => WebAssembly.instantiate(buffer, imports));
    }

    const e = get_encoding('cl100k_base');
    const t = e.encode(str);
    e.free();
    return t.length;
  }