Currently to reinitialize a model for AutoModelForSequenceClassification
, we can do this:
from transformers import AutoModel, AutoConfig, AutoModelForSequenceClassification
m = "moussaKam/frugalscore_tiny_bert-base_bert-score"
config = AutoConfig.from_pretrained(m)
model_from_scratch = AutoModel(config)
model_from_scratch.save_pretrained("frugalscore_tiny_bert-from_scratch")
model = AutoModelForSequenceClassification(
"frugalscore_tiny_bert-from_scratch", local_files_only=True
)
AutoConfig
?model = AutoModelForSequenceClassification(
"moussaKam/frugalscore_tiny_bert-base_bert-score",
local_files_only=True
reinitialize_weights=True
)
or something like:
model = AutoModelForSequenceClassification(
"moussaKam/frugalscore_tiny_bert-base_bert-score",
local_files_only=True
)
model.reinitialize_parameters()
That is the purpose of from_config (i.e. creating a model but not loading the respective weights):
from transformers import AutoModel, AutoConfig, AutoModelForSequenceClassification
m = "moussaKam/frugalscore_tiny_bert-base_bert-score"
config = AutoConfig.from_pretrained(m)
no_weights_model = AutoModelForSequenceClassification.from_config(config)
weights_model = AutoModelForSequenceClassification.from_pretrained(m)
import torch
print(torch.allclose(no_weights_model.bert.embeddings.word_embeddings.weight,weights_model.bert.embeddings.word_embeddings.weight))
Output:
False