Assertion Failure in Eigen Matrix Broadcasting: Dimension Mismatch in Neural Network Forward Pass

I’m implementing a neural network in C++ using Eigen for linear algebra operations. In the DenseLayer::forward function, I’m encountering a dimension mismatch issue when trying to add a bias vector to the layer’s output matrix.

Here’s the structure of my code:

The input matrix has dimensions 64x74 (batch size 64).
weights is initialized with dimensions 128x784.
bias is a column vector of dimensions 128x1.

Expected Behavior:

After multiplying input * weights.transpose(), I get an output matrix of size 64x128. I need to add bias to each row of output so that the final result remains 64x128.

Current Approach:

Here's the forward function:

Eigen::MatrixXd DenseLayer::forward(const Eigen::MatrixXd& input) {
input_cache = input;
Eigen::MatrixXd output = input * weights.transpose();

// Attempt to replicate bias across rows to match output dimensions
Eigen::MatrixXd bias_replicated = bias.replicate(output.rows(), 1);

output += bias_replicated;
return output;
}

Error Message:

When I run the code, I encounter the following assertion failure:

Assertion failed: (dst.rows() == src.rows() && dst.cols() == src.cols()), function resize_if_allowed, file AssignEvaluator.h, line 754.

What I've Tried:

Using .colwise() + bias.col(0), .rowwise() + bias.transpose(), and directly adding bias with different reshaping/transposing.
I've checked dimensions through debug statements and confirmed that output is 64x128 and bias is 128x1.

Question: How can I correctly broadcast bias across each row of output to perform element-wise addition without Eigen raising a dimension mismatch error? Is there a better way to handle this broadcasting in Eigen, or am I missing something fundamental in matrix addition with Eigen?

Debug Output: To help, here are the dimensions printed before the error:

Output before bias addition size: 64x128
Bias size for addition: 128x1
Replicated bias size: 64x128

Any insights on achieving the correct broadcasting or a workaround for this would be greatly appreciated!

Solution

Here is how I fixed it:

Initial fix which is not efficient and based on Homer512's feedback, I revised my initial approach

Recap on the problem:

The assertion error arose because Eigen's .rowwise() approach requires bias to be treated as a vector for broadcasting along rows. However, Eigen's internal checks often flag mismatches if the vector doesn't conform exactly as expected, especially with transposed vectors in operations like .rowwise() addition.

Solution: Expanding bias into a Full Matrix Before Adding

To work around this limitation, I manually replicated the bias to match the output dimensions before adding it, This ensures that each row of the output matrix receives a copy of the bias vector, making Eigen's type system happy and avoiding the assertion errors.

I Updated forward Function Implementation:

Eigen::MatrixXd DenseLayer::forward(const Eigen::MatrixXd& input) {
     std::cout << "[DenseLayer Forward] Input size: " << input.rows() << "x" << input.cols() << std::endl;
     std::cout << "[DenseLayer Forward] Weights size: " << weights.rows() << "x" << weights.cols() << std::endl;
     std::cout << "[DenseLayer Forward] Bias size: " << bias.rows() << "x" << bias.cols() << std::endl;

     input_cache = input;
     Eigen::MatrixXd output = input * weights.transpose();

     // Create a matrix of the same size as output, where each row is a copy of the bias vector
     Eigen::MatrixXd bias_expanded = bias.transpose().replicate(output.rows(), 1);

     std::cout << "[DenseLayer Forward] Output before bias addition size: " << output.rows() << "x" << output.cols() << std::endl;
     std::cout << "[DenseLayer Forward] Expanded bias size: " << bias_expanded.rows() << "x" << bias_expanded.cols() << std::endl;

     output += bias_expanded;

     std::cout << "[DenseLayer Forward] Output after bias addition size: " << output.rows() << "x" << output.cols() << std::endl;
     return output;
}

Explanation:

Explicit Replication of bias:

bias.transpose().replicate(output.rows(), 1); creates a bias_expanded matrix where each row is a copy of bias. This ensures that bias_expanded has the exact dimensions as output, so if output is 64x128, bias_expanded will also be 64x128.

Avoiding .rowwise() + bias.transpose():

By using this explicit replication, I bypassed .rowwise() entirely. This approach circumvents Eigen's broadcasting limitations and type-checking issues, making the code more robust.

Better Approach: Most recent Fix

Thanks to Homer512's suggestion, I revised my approach for efficiency. Initially, I had bias set up as a matrix, which required full replication to match output dimensions. Now, bias is an Eigen::VectorXd, which broadcasts naturally:

Eigen::MatrixXd DenseLayer::forward(const Eigen::MatrixXd& input) {
   input_cache = input;
   Eigen::MatrixXd output = input * weights.transpose();
   output.rowwise() += bias.transpose(); // Efficient, no full replication needed
   return output;
}

This approach removes the overhead, thanks again!