I'm trying to write a code that calculates hinge loss and softmax loss for each picture of cifar10, and I'm getting this error: "index 1 is out of bounds for axis 0 with size 1" for the log line in softmax_loss function. How can I fix this?
import numpy as np
from keras.datasets import cifar10
# define the weight matrix
W = np.random.rand(10, 3072)
# load CIFAR-10 dataset
(x_train, y_train), (x_test, y_test) = cifar10.load_data()
# flatten the images
x_train_flat = x_train.reshape(x_train.shape[0], -1)
# compute the Hinge Loss for each image
def hinge_loss(y_true, y_pred):
margin = 1.
loss = np.maximum(0., margin - y_true * y_pred)
return loss
# compute the Softmax loss for each image
def softmax_loss(y_true, y_pred):
num_classes = y_pred.shape[1]
softmax = np.exp(y_pred) / np.sum(np.exp(y_pred), axis=1, keepdims=True)
loss = -np.log(softmax[range(num_classes), y_true])
return loss
hinge_losses = []
softmax_losses = []
# iterate over each image in the training set
for i in range(x_train_flat.shape[0]):
# calculate predictions
x_i = x_train_flat[i, :]
y_i = np.dot(W, x_i)
# calculate the loss using the Hinge Loss function
y_true_h = np.zeros_like(y_i)
y_true_h[y_train[i]] = 1
hinge_loss_value = np.sum(hinge_loss(y_true_h, y_i))
hinge_losses.append(hinge_loss_value)
# calculate the loss using the Softmax function
y_true_s = y_train[i]
softmax_loss_value = np.sum(softmax_loss(y_true_s, y_i.reshape(1, -1)))
softmax_losses.append(softmax_loss_value)
# print the average loss values
print("Average Hinge Loss:", np.mean(hinge_losses))
print("Average Softmax Loss:", np.mean(softmax_losses))
as i said I get the error "index 1 is out of bounds for axis 0 with size 1" for this line:
loss = -np.log(softmax[range(num_classes), y_true])
First of all, softmax is an activation function, the corresponding loss is called the cross entropy loss. It seems to me that you just want to select the softmax value of the element corresponding to y_true
in order to calculate the loss. I believe you can then simply update your softmax_loss
(cross-entropy loss!) to:
def softmax_loss(y_true, y_pred):
softmax = np.exp(y_pred) / np.sum(np.exp(y_pred))
loss = -np.log(softmax[0, y_true])
return loss
Although it seems like your y_pred
values are exploding because you have not normalized your dataset x_train_flat
. Consider normalizing first, i.e.:
x_train_flat = x_train.reshape(x_train.shape[0], -1) / 255
but it seems like you should also reduce the weights. For example:
W = np.random.rand(10, 3072) / 3072