pythontensorflowtensorflow-hubelmo

Is it possible and how to reduce the dimensions of ELMO embeddings using tf-hub elmo model?

I am using ELMO model from tensorflow hub.

import tensorflow as tf
import tensorflow_hub as hub
from tensorflow.contrib import predictor

elmo = hub.Module("https://tfhub.dev/google/elmo/3", trainable=False)

The default output dimension is 1024. Is there any possible way to reduce the dimension of ELMO embeddings to 128 without using PCA/losing information?

Solution

  • Since ELMo is pre-trained models released by allennlp these will give the fixed output dimension based on the training which is of 1024.

    One way to reduce the output dimension is by trying dimensionality reduction techniques like PCA,tSNE, UMAP, but as you said it will be at the cost of losing information.

    Instead, you can train the ELMo from scratch, provided with a good amount of data and devices to train.
    You can follow this article which explains about training ELMo from scratch.
    To change the hyperparameter like output dimension and other things can be done in this file.