pythonnumpymachine-learningsoftmax

index 1 is out of bounds for axis 0 with size 1 for softmax function


I'm trying to write a code that calculates hinge loss and softmax loss for each picture of cifar10, and I'm getting this error: "index 1 is out of bounds for axis 0 with size 1" for the log line in softmax_loss function. How can I fix this?

import numpy as np
from keras.datasets import cifar10

# define the weight matrix
W = np.random.rand(10, 3072)

# load CIFAR-10 dataset
(x_train, y_train), (x_test, y_test) = cifar10.load_data()

# flatten the images
x_train_flat = x_train.reshape(x_train.shape[0], -1)

# compute the Hinge Loss for each image
def hinge_loss(y_true, y_pred):
    margin = 1.
    loss = np.maximum(0., margin - y_true * y_pred)
    return loss
# compute the Softmax loss for each image
def softmax_loss(y_true, y_pred):
    num_classes = y_pred.shape[1]
    softmax = np.exp(y_pred) / np.sum(np.exp(y_pred), axis=1, keepdims=True)
    loss = -np.log(softmax[range(num_classes), y_true])
    return loss


hinge_losses = []
softmax_losses = []

# iterate over each image in the training set
for i in range(x_train_flat.shape[0]):
    # calculate predictions
    x_i = x_train_flat[i, :]
    y_i = np.dot(W, x_i)
    
    # calculate the loss using the Hinge Loss function
    y_true_h = np.zeros_like(y_i)
    y_true_h[y_train[i]] = 1
    hinge_loss_value = np.sum(hinge_loss(y_true_h, y_i))
    hinge_losses.append(hinge_loss_value)
    
    # calculate the loss using the Softmax function
    y_true_s = y_train[i]
    softmax_loss_value = np.sum(softmax_loss(y_true_s, y_i.reshape(1, -1)))
    softmax_losses.append(softmax_loss_value)

# print the average loss values
print("Average Hinge Loss:", np.mean(hinge_losses))
print("Average Softmax Loss:", np.mean(softmax_losses))

as i said I get the error "index 1 is out of bounds for axis 0 with size 1" for this line:

loss = -np.log(softmax[range(num_classes), y_true])

Solution

  • First of all, softmax is an activation function, the corresponding loss is called the cross entropy loss. It seems to me that you just want to select the softmax value of the element corresponding to y_true in order to calculate the loss. I believe you can then simply update your softmax_loss (cross-entropy loss!) to:

    def softmax_loss(y_true, y_pred):
        softmax = np.exp(y_pred) / np.sum(np.exp(y_pred))
        loss = -np.log(softmax[0, y_true])
        return loss
    

    Although it seems like your y_pred values are exploding because you have not normalized your dataset x_train_flat. Consider normalizing first, i.e.:

    x_train_flat = x_train.reshape(x_train.shape[0], -1) / 255
    

    but it seems like you should also reduce the weights. For example:

    W = np.random.rand(10, 3072) / 3072