[SOLVED] Pytorch Conv2d outputs infinity

Pytorch Conv2d outputs infinity

My input x is a [1,256,60,120] shaped tensor. My Conv2d is defined as follows

import torch.nn as nn
conv2d = nn.Conv2d(
            256,
            256,
            kernel_size=2,
            stride=2,
            bias=False,
        ),

Some instances I see that conv2d(x).isinf().any() is True. Note that

x.max() = tensor(5140., device='cuda:0', dtype=torch.float16).

x.min() = tensor(0., device='cuda:0', dtype=torch.float16),

conv2d.weight.max() =tensor(1.5796, device='cuda:0')

conv2d.weight.min() = tensor(-0.8045, device='cuda:0')

Why do I get infinity?

Solution

This is a case of numerical overflow.

Consider:

import torch
import torch.nn as nn

# set seed
torch.manual_seed(42)

# random values between 0 and 5140, like your values
x = (torch.rand(1, 256, 60, 120) * 5140)

# create conv layer
conv2d = nn.Conv2d(
            256,
            256,
            kernel_size=2,
            stride=2,
            bias=False,
        )

# set conv weights to a similar range to your values
torch.nn.init.uniform_(conv2d.weight, a=-0.8, b=1.57)

# compute output
y = conv2d(x)

print(y.max())
> tensor(1334073.6250, grad_fn=<MaxBackward1>)

When I run the code above, I see a max output value of 1334073.6250. You are using fp16 values, which overflow at 32000. You're seeing inf values because your output values exceed the range of fp16.

You should preprocess your inputs to have a more reasonable numeric range. Neural networks are very sensitive to numerical scale. Typically you want your input values to be roughly mean 0, variance 1. A max input value of 5140 is huge and likely to cause numeric issues in any situation.