[SOLVED] ROC curve *without* a model estimator?

ROC curve *without* a model estimator?

I have created an AI Tool which extracts content from an image and then reviews that content for completeness and accuracy. I am trying to evaluate the performance of this tool and am gathering metrics to do so.

I have a resulting table that looks similar to the below; the true values come from manual ground truth reviews of documents and the predicted values are the actual output of the tool.

ID | true | predicted |
-----------------------
1  |  0   |      1    |
2  |  0   |      0    |
3  |  1   |      1    |
4  |  1   |      0    |

I have been able to use the true and predicted columns to obtain various metrics using the code below:

def calculate_metrics(df, true, predicted):
    accuracy = accuracy_score(df[true], df[predicted])
    precision = precision_score(df[true], df[predicted])
    recall = recall_score(df[true], df[predicted])
    f1 = f1_score(df[true], df[predicted])
    roc_auc = roc_auc_score(df[true], df[predicted])
    return print(f"accuracy: {accuracy}\nprecision: {precision}\nrecall: {recall}\nf1: {f1}\nroc_auc: {roc_auc}")

Additionally, I would like to plot a ROC curve. I am able to obtain the roc_auc score and assumed I could plot from there, but I'm having a hard time wrapping my head around how exactly to do so. It looks like I need a model estimator to determine probabilities and then from there I can create the plot but I'm unclear how to do so with the data I've obtained.

Is it even possible to create a ROC curve using the results I have and, if so, how do I do so?

Solution

Is it even possible to create a ROC curve using the results I have and, if so, how do I do so?

No. You need to have the model produce the probability that an observation falls into a class, rather than the predicted class. The point of an ROC curve is that it allows you to visualize how different thresholds for probability of determining that the data falls into either class affects the true positive rate and false positive rate.

However, this data is after thresholding, which means that you can't visualize the effect of different thresholds. You need your model to output a probability rather than the predicted class in order to perform this visualization.