pytorchcross-entropy

Why am I getting the wrong answer using pyTorch's cross-entropy?


I have a simple logit and target tensors and computing the cross entropy using Pytorch's from torch.nn import functional as F module.

>>> F.cross_entropy(torch.tensor([[0.25, 0.5, 0.25]]), torch.tensor([0]))

tensor(1.1891)

I'm getting 1.1891 as the answer. But according to the equation of cross_entropy loss, shouldn't the answer be

-(1*ln(0.25) + 0*ln(0.5) + 0*ln(0.25) = -ln(0.25) = 1.386?

What is going on?


Solution

  • Because you are missing how it is calculated. It is mentioned in documentation (related docs)

    -torch.log(torch.softmax(a,dim=1))[0][0]
    

    It calculates softmax of the entries first and then calculates cross-entropy.

    In short

     F.nll_loss(F.log_softmax(a), torch.tensor([0]))
    

    Here is the linked documentation to which this refers to

    enter image description here