Why does this raycaster slow down whenever the player is near a wall?

I am writing a 2.5d game and I have noticed that whenever I am up close to and facing a wall, the game slows down tremendously. This was exacerbated when I added in textures. I am using an inefficient rendering method and have researched how to improve it, so this question is not about how to make it faster in general. My original thought was that it was simply because there was more on the screen. This was partially correct, however I made sure that the renderer would clip off anything not on the screen. I also added a check to see how many pixels (rectangles) were being drawn per frame. When I did this, I noticed I would still get slower and slower as I got closer and closer, even with the rectangle count being the exact same. Why is this code slowing down even when drawing the same amount of rectangles?

int drawStart = y_offset > 0 ? y_offset : 0;
int drawEnd = (y_offset + bar_height) < 480 ? y_offset + bar_height : 480;
int texNum = map[cell_y][cell_x] - 49;

//calculate value of wallX
double wallX; //where exactly the wall was hit
if (side == 0) wallX = player->y + distance * ray_angle_y;
else           wallX = player->x + distance * ray_angle_x;
wallX -= floor(wallX);

//x coordinate on the texture
int texX = wallX * TEXTURE_WIDTH;
if (side == 0 && ray_angle_x > 0) texX = TEXTURE_WIDTH - texX - 1;
if (side == 1 && ray_angle_y < 0) texX = TEXTURE_WIDTH - texX - 1;

// How much to increase the texture coordinate per screen pixel
double step = 1.0 * TEXTURE_HEIGHT / bar_height;
double pixel_size = bar_height / TEXTURE_HEIGHT;
// Starting texture coordinate
double texPos = bar_height > 480 ? (-y_offset * step) : 0;
for(int y = drawStart; y < drawEnd; y++) {
    // Cast the texture coordinate to integer, and mask with (TEXTURE_HEIGHT - 1) in case of overflow
    int texY = (int)texPos & (TEXTURE_HEIGHT - 1);
    texPos += step;
    Uint32 color = textures[texNum].array[texY][texX];
    if(side == 1) color = (color >> 1) & 8355711;
    
    
    SDL_Rect rect = {i * rect_width, y, rect_width, pixel_size};
 /* even when this is called the same amount of times, it will be slower depending on how close to the wall the player is*/
    SDL_FillRect( screenSurface, &rect, color );
}

I followed @Ry- 's advice to try using a profiler on the program. In doing so, I was able to track down the offending code:

double pixel_size = bar_height / TEXTURE_HEIGHT;
// ...    
SDL_Rect rect = {i * rect_width, y, rect_width, pixel_size};

I was calculating the variable pixel_size based on the ratio between bar_height and TEXTURE_HEIGHT, however this was incorrect. I had written it this way intending for there to only be TEXTURE_HEIGHT amount of 'pixels' in each bar, but I have not yet implemented this, and so it was drawing each 'pixel' much longer than it needed to be.

In short: each rectangle was being draw longer than a pixel in length even though I was drawing a rectangle for each pixel

- shamelessly stolen from my response to Ry-