I am trying to follow this example in the huggingface documentation here https://huggingface.co/transformers/model_doc/longformer.html:
import torch
from transformers import LongformerModel, LongformerTokenizer
model = LongformerModel.from_pretrained('allenai/longformer-base-4096')
tokenizer = LongformerTokenizer.from_pretrained('allenai/longformer-base-4096')
SAMPLE_TEXT = ' '.join(['Hello world! '] * 1000) # long input document
input_ids = torch.tensor(tokenizer.encode(SAMPLE_TEXT)).unsqueeze(0) # batch of size 1
# Attention mask values -- 0: no attention, 1: local attention, 2: global attention
attention_mask = torch.ones(input_ids.shape, dtype=torch.long, device=input_ids.device) # initialize to local attention
global_attention_mask = torch.zeros(input_ids.shape, dtype=torch.long, device=input_ids.device) # initialize to global attention to be deactivated for all tokens
global_attention_mask[:, [1, 4, 21,]] = 1 # Set global attention to random tokens for the sake of this example
# Usually, set global attention based on the task. For example,
# classification: the <s> token
# QA: question tokens
# LM: potentially on the beginning of sentences and paragraphs
outputs = model(input_ids, attention_mask=attention_mask, global_attention_mask=global_attention_mask, output_hidden_states= True)
sequence_output = outputs[0].last_hidden_state
pooled_output = outputs.pooler_output
I suppose that this would return a document embedding for the sample text. However, I run into the following error:
AttributeError: 'Tensor' object has no attribute 'last_hidden_state'
Why isnt it possible to call last_hidden_state?
Do not select via index:
sequence_output = outputs.last_hidden_state
outputs
is a LongformerBaseModelOutputWithPooling object with the following properties:
print(outputs.keys())
Output:
odict_keys(['last_hidden_state', 'pooler_output', 'hidden_states'])
Calling outputs[0]
or outputs.last_hidden_state
will both give you the same tensor, but this tensor does not have a property called last_hidden_state
.