How to import trained model without initializing weights

I converted a EfficientNet model that was pretrained on ImageNet to tensorflow-js using the tensorflowjs-converter. When I try to load the model into my script, it tries to initialize the weights with initializers, that are not implemented in tfjs. However, it is not necessary to initialize the weights, as the model was pretrained and the weights were also converted. The converted model is here: https://github.com/paulsp94/tfjs_efficientnet3_imagenet

Here is a CodePen example of the problem: https://codepen.io/paulsp94/pen/XLNdJq

const start = async () => {
  efficientNetURL = 'https://raw.githubusercontent.com/paulsp94/tfjs_efficientnet3_imagenet/master/model.json';

  console.log("Load Model");
  let model;
  try {
    model = await tf.loadLayersModel(efficientNetURL, {strict: true});
    console.log(model.summary());
   } catch (error) {
     console.error(error);
   }
};

start()

You have to open the browser's console, to see the proper error.

The error is:

Error: "Unknown initializer: EfficientConv2DKernelInitializer. This may be due to one of the following reasons: 1. The initializer is defined in Python, in which case it needs to be ported to TensorFlow.js or your JavaScript code. 2. The custom initializer is defined in JavaScript, but is not registered properly with tf.serialization.registerClass()."

Update: While I could get around the initializers, by replacing all unknown initializers with e.g. Zeros initializer. I ran into another problem with the custom layers (Swish layer) which currently can't be resolved.

According to the documentation:

TensorFlow.js Layers currently only supports Keras models using standard Keras constructs. Models using unsupported ops or layers—e.g. custom layers, Lambda layers, custom losses, or custom metrics—cannot be automatically imported, because they depend on Python code that cannot be reliably translated into JavaScript.

It is not currently possible to import model with custom layers