Gnuplot Confusion Matrix - Labels showing up as 0 despite coloring working

I am trying to write a gnuplot script for a confusion matrix (square grid of values), and I want to plot the confusion matrix (colored by cell value) and annotate the value in each box. I currently have this for my splot:

splot \
    "../data/reordered_confusion_matrix.dat" matrix using 1:2:3 with image notitle, \
    "../data/reordered_confusion_matrix.dat" matrix using 1:2:3 with labels notitle font 'Arial,10' textcolor rgb 'black'

The first line plots the confusion matrix as intended (each cell is colored by the value assigned). However, the second line annotates 0s in all boxes (not reading the true values correctly). Does anyone have any insight into this? Thank you in advance!

As an FYI, this is what the data file looks like:

49  0   0   0   0   0   0   0   0   0   0   0   1
0   49  0   0   0   0   0   0   0   0   0   1   0
0   0   50  0   0   0   0   0   0   0   0   0   0
0   0   0   50  0   0   0   0   0   0   0   0   0
0   0   0   0   50  0   0   0   0   0   0   0   0
0   0   0   0   0   50  0   0   0   0   0   0   0
0   0   0   0   0   0   50  0   0   0   0   0   0
0   0   0   0   0   0   0   31  19  0   0   0   0
0   0   0   0   0   0   0   16  34  0   0   0   0
0   0   0   0   0   0   0   0   0   50  0   0   0
0   0   0   0   0   0   0   0   0   0   50  0   0
0   0   0   0   0   0   0   0   0   0   0   50  0
0   0   0   0   0   0   0   0   0   0   0   0   0

Solution

Check the following example as starting point for further optimization. In gnuplot console, type and check help <YourKeyword> for all commands, functions and keywords.

for your case you don't need splot, plot is enough.
apparently, for with labels you need to format your matrix values as text via sprintf().

Script:

### plot matrix with colors and values
reset session

$Data <<EOD
49  0   0   0   0   0   0   0   0   0   0   0   1
0   49  0   0   0   0   0   0   0   0   0   1   0
0   0   50  0   0   0   0   0   0   0   0   0   0
0   0   0   50  0   0   0   0   0   0   0   0   0
0   0   0   0   50  0   0   0   0   0   0   0   0
0   0   0   0   0   50  0   0   0   0   0   0   0
0   0   0   0   0   0   50  0   0   0   0   0   0
0   0   0   0   0   0   0   31  19  0   0   0   0
0   0   0   0   0   0   0   16  34  0   0   0   0
0   0   0   0   0   0   0   0   0   50  0   0   0
0   0   0   0   0   0   0   0   0   0   50  0   0
0   0   0   0   0   0   0   0   0   0   0   50  0
0   0   0   0   0   0   0   0   0   0   0   0   0
EOD

set size ratio -1
set key noautotitle
set xrange noextend
set yrange reverse noextend
set tics out
set palette rgb 33,13,10

plot $Data matrix u 1:2:3 w image, \
        '' matrix u 1:2:(sprintf("%g",$3)) w labels font 'Arial,10' tc 'black'
### end of script

Result: