I calculate R^2 manually and compare the results to torcheval.metrics.regression.r2_score without multioutput but I do not tie out the total sum squared calculation, and hence manual R^2 is different than torcheval:
Manual approach:
#Manual approach code
ss_total = torch.sum((var1 - torch.mean(var1)) ** 2)
ss_residual = torch.sum((var1 - var2) ** 2)
r2 = 1 - (ss_residual / ss_total)
print("R^2 manual",r2, "my ss_total", ss_total, "ss_residual", ss_residual)
#R^2 manual tensor(-1.4128, device='cuda:0') my ss_total tensor(3.7081, device='cuda:0') ss_residual tensor(8.9471, device='cuda:0')
Torcheval.metrics approach tss formulae documentation without multioutput:
sum_squared_obs = torch.sum((actual - torch.mean(actual)) ** 2)
tss sum squared calculation = sum_squared_obs - torch.square(sum_obs) / num_obs
r_squared = 1 - (rss / tss)
#torcheval.metrics.regression.r2_score tested in script
metric = R2Score(device=device)
update = metric.update(var1, var2)
print("sum_squared_residual",update.sum_squared_residual)
print("sum_obs",update.sum_obs)
print("torch.square(sum_obs)",torch.square(update.sum_obs))
print("num_obs",len(var1))
print("sum_squared_obs",update.sum_squared_obs)
r2_py = metric.compute()
print("R^2 pytorch",r2_py)
#sum_squared_residual tensor(8.9471, device='cuda:0')
#sum_obs tensor(-29.9617, device='cuda:0')
#torch.square(sum_obs) tensor(897.7044, device='cuda:0')
#num_obs 64
#sum_squared_obs tensor(22.2245, device='cuda:0')
#R^2 pytorch tensor(-0.0914, device='cuda:0')
#R^2 var_weight pytorch tensor(-0.0914, device='cuda:0')
I cannot tie out tss.
Can someone explain what is the difference between the two approaches?
I have saved the actual and predicted values in excel and calculate R^2 with both approaches.
R2 is not symmetric. You get different results because you change which variable is the ground truth.
In your first section of code, you treat var1
as the ground truth values and var2
as the predicted values.
In your second section of code, you treat var2
as the ground truth and var1
as the predicted value.
import torch
from torcheval.metrics.regression import R2Score
var1 = torch.randn(64)
var2 = torch.randn(64)
def compute_r2(inputs, targets):
ss_total = torch.sum((targets - torch.mean(targets)) ** 2)
ss_residual = torch.sum((targets - inputs) ** 2)
r2 = 1 - (ss_residual / ss_total)
return r2
def compute_r2_metric(inputs, targets):
metric = R2Score()
metric.update(inputs, targets)
r2 = metric.compute()
return r2
# treat var1 as input, var2 as target
print(compute_r2(var1, var2))
> tensor(-0.8324)
print(compute_r2_metric(var1, var2))
> tensor(-0.8324)
# treat var2 as input, var1 as target
print(compute_r2(var2, var1))
> tensor(-1.9208)
print(compute_r2_metric(var2, var1))
> tensor(-1.9208)