pythonmachine-learninghuggingface-transformerstext-classificationsafe-tensors

How to reset parameters from AutoModelForSequenceClassification?


Currently to reinitialize a model for AutoModelForSequenceClassification, we can do this:

from transformers import AutoModel, AutoConfig, AutoModelForSequenceClassification

m = "moussaKam/frugalscore_tiny_bert-base_bert-score"
config = AutoConfig.from_pretrained(m)
model_from_scratch = AutoModel(config)

model_from_scratch.save_pretrained("frugalscore_tiny_bert-from_scratch")

model = AutoModelForSequenceClassification(
  "frugalscore_tiny_bert-from_scratch", local_files_only=True
)

Is there some way to reinitialize the model weights without saving a new pretrained model initialized with AutoConfig?

model = AutoModelForSequenceClassification(
  "moussaKam/frugalscore_tiny_bert-base_bert-score", 
  local_files_only=True
  reinitialize_weights=True
)

or something like:

model = AutoModelForSequenceClassification(
  "moussaKam/frugalscore_tiny_bert-base_bert-score", 
  local_files_only=True
)

model.reinitialize_parameters()

Solution

  • That is the purpose of from_config (i.e. creating a model but not loading the respective weights):

    from transformers import AutoModel, AutoConfig, AutoModelForSequenceClassification
    
    m = "moussaKam/frugalscore_tiny_bert-base_bert-score"
    config = AutoConfig.from_pretrained(m)
    
    no_weights_model = AutoModelForSequenceClassification.from_config(config)
    weights_model = AutoModelForSequenceClassification.from_pretrained(m)
    
    import torch
    
    print(torch.allclose(no_weights_model.bert.embeddings.word_embeddings.weight,weights_model.bert.embeddings.word_embeddings.weight))
    

    Output:

    False