How to choose the correct tile from a tileset based on neighbors to avoid discontinuities in a maze (SDL2, tile-based rendering)

Body:

I'm generating a maze using a simple 2D array where each cell contains information about walls on 4 sides (encoded in 4 bits: top, right, bottom, left).

Currently, I display the maze using simple wall tiles based only on the wall information of each individual cell. But this causes visible discontinuities between neighboring cells, especially in the corners, because adjacent cells may have no wall between them, yet the graphics don't visually connect seamlessly.

I now have a tileset image ([tileset.png][1]) to preview the image https://ibb.co/99mmZJ2W that contains many pre-rendered tiles designed to fix these discontinuities, including:

simple walls (top, right, bottom, left),
combinations of walls,
special corner extensions that visually fill the gaps between adjacent tiles.

The tileset is organized as a grid of 10 rows and 9 columns (total image size 144x160 pixels), and each tile is 16x16 pixels.

walls.dat file : https://limewire.com/d/IkVO2#LcLwVi2quk

My problem:

I need to take into account the neighbors of each cell to correctly select the tile. For example, if a cell has no wall on its top side, but its top neighbor has no wall on its bottom side either, I need to pick a tile that includes a filled top-left or top-right corner to visually connect both cells smoothly. The same applies for all 4 corners.

My question:

How can I correctly select the appropriate tile for each cell based both on its own walls and on its neighboring cells? How would you structure the logic to decide which tile to use?

I'm using C with SDL2.

Here is my current code:

#define SDL_MAIN_HANDLED
#include <SDL2/SDL.h>
#include <SDL2/SDL_image.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>

#define CELL_SIZE 30
#define N 20
#define WINDOW_SIZE N * CELL_SIZE

void render_maze_with_tiles(SDL_Renderer *renderer, int walls[N][N], const char *tileset_path)
{
    SDL_Surface *tileset_surface = IMG_Load(tileset_path);
    if (!tileset_surface)
    {
        printf("Error loading tileset image: %s\n", IMG_GetError());
        return;
    }

    SDL_Texture *tileset_texture = SDL_CreateTextureFromSurface(renderer, tileset_surface);
    SDL_FreeSurface(tileset_surface);

    if (!tileset_texture)
    {
        printf("Error creating tileset texture: %s\n", SDL_GetError());
        return;
    }

    SDL_Rect dest;
    dest.w = CELL_SIZE;
    dest.h = CELL_SIZE;

    for (int y = 0; y < N; y++)
    {
        for (int x = 0; x < N; x++)
        {
            int cell_walls = walls[y][x];
            dest.x = x * CELL_SIZE;
            dest.y = y * CELL_SIZE;

            SDL_Rect src_tile = {0, 0, 16, 16};

            // Select the tile based on wall configuration
            switch (cell_walls)
            {
                case 0: src_tile.y = 16; src_tile.x = 16; break; // no walls
                case 1: src_tile.y = 0; src_tile.x = 16; break; // top wall
                case 2: src_tile.y = 16; src_tile.x = 32; break; // right wall
                case 3: src_tile.y = 0; src_tile.x = 32; break; // top + right
                case 4: src_tile.y = 32; src_tile.x = 16; break; // bottom wall
                case 5: src_tile.y = 64; src_tile.x = 112; break; // top + bottom
                case 6: src_tile.y = 32; src_tile.x = 32; break; // right + bottom
                case 7: src_tile.y = 64; src_tile.x = 128; break; // top + right + bottom
                case 8: src_tile.y = 16; src_tile.x = 0; break; // left wall
                case 9: src_tile.y = 0; src_tile.x = 0; break; // top + left
                case 10: src_tile.y = 32; src_tile.x = 128; break; // left + right
                case 11: src_tile.y = 16; src_tile.x = 128; break; // top + left + right
                case 12: src_tile.y = 32; src_tile.x = 0; break; // bottom + left
                case 13: src_tile.y = 64; src_tile.x = 96; break; // top + bottom + left
                case 14: src_tile.y = 48; src_tile.x = 128; break; // bottom + right + left
                case 15: src_tile.y = 48; src_tile.x = 112; break; // all walls
                default: src_tile.y = 0; src_tile.x = 0; break;
            }

            SDL_RenderCopy(renderer, tileset_texture, &src_tile, &dest);
        }
    }

    SDL_RenderPresent(renderer);
    SDL_DestroyTexture(tileset_texture);
}

int load_walls(const char *filename, int walls[N][N])
{
    FILE *f = fopen(filename, "rb");
    if (!f)
    {
        return 0;
    }
    fread(walls, sizeof(int), N * N, f);
    fclose(f);
    return 1;
}

void render_player(SDL_Renderer *renderer, SDL_Texture *player_texture, int player_x, int player_y)
{
    SDL_Rect dest;
    dest.x = player_x * CELL_SIZE + CELL_SIZE / 4; // centered
    dest.y = player_y * CELL_SIZE + CELL_SIZE / 4;
    dest.w = CELL_SIZE / 2;
    dest.h = CELL_SIZE / 2;
    SDL_RenderCopy(renderer, player_texture, NULL, &dest);
}

int main(int argc, char *argv[])
{
    int walls[N][N];
    load_walls("walls.dat", walls);

    SDL_Init(SDL_INIT_VIDEO);
    IMG_Init(IMG_INIT_PNG);
    SDL_Window *window = SDL_CreateWindow("Maze", SDL_WINDOWPOS_CENTERED,
        SDL_WINDOWPOS_CENTERED, WINDOW_SIZE, WINDOW_SIZE, SDL_WINDOW_SHOWN);
    SDL_Renderer *renderer = SDL_CreateRenderer(window, -1, SDL_RENDERER_ACCELERATED);

    // Load tileset
    SDL_Surface *tileset_surface = IMG_Load("./tileset.png");
    if (!tileset_surface)
    {
        printf("Error loading tileset: %s\n", IMG_GetError());
        exit(1);
    }
    SDL_Texture *tileset_texture = SDL_CreateTextureFromSurface(renderer, tileset_surface);
    SDL_FreeSurface(tileset_surface);

    // Load player texture
    SDL_Surface *player_surface = IMG_Load("man.png");
    if (!player_surface)
    {
        printf("Error loading player image: %s\n", IMG_GetError());
        exit(1);
    }
    SDL_Texture *player_texture = SDL_CreateTextureFromSurface(renderer, player_surface);
    SDL_FreeSurface(player_surface);

    int player_x = 0, player_y = 0;
    int quit = 0;
    SDL_Event event;

    while (!quit)
    {
        while (SDL_PollEvent(&event))
        {
            if (event.type == SDL_QUIT) quit = 1;
            else if (event.type == SDL_KEYDOWN)
            {
                int cell_walls = walls[player_y][player_x];
                if (event.key.keysym.sym == SDLK_UP && !(cell_walls & 1) && player_y > 0)
                    player_y--;
                if (event.key.keysym.sym == SDLK_DOWN && !(cell_walls & 4) && player_y < N - 1)
                    player_y++;
                if (event.key.keysym.sym == SDLK_LEFT && !(cell_walls & 8) && player_x > 0)
                    player_x--;
                if (event.key.keysym.sym == SDLK_RIGHT && !(cell_walls & 2) && player_x < N - 1)
                    player_x++;
            }
        }

        SDL_RenderClear(renderer);
        render_maze_with_tiles(renderer, walls, "./tileset.png");
        render_player(renderer, player_texture, player_x, player_y);
        SDL_RenderPresent(renderer);
    }

    // Cleanup
    SDL_DestroyTexture(tileset_texture);
    SDL_DestroyTexture(player_texture);
    SDL_DestroyRenderer(renderer);
    SDL_DestroyWindow(window);
    IMG_Quit();
    SDL_Quit();
    return 0;
}

Solution

Context

You're right: with the tile set you are using, the appropriate tile for a given cell depends not only on the positions of the walls in that cell, but also on the positions of some of the walls in horizontally and vertically adjacent cells. You're also right to be thinking in terms of the corners, but you haven't gone far enough in that direction.

How can I correctly select the appropriate tile for each cell based both on its own walls and on its neighboring cells? How would you structure the logic to decide which tile to use?

For a given overall style (e.g. earth vs stone), each tile can be characterized entirely in terms of the types of its corners. These appear to be the supported types:

Empty. This applies when there are no walls entering that corner, neither belonging to the current cell nor belonging to its neighbors. The corner is rendered as all floor.
Corner cap. This applies when the current cell has no walls entering the corner but at least one of the adjacent cells does (it doesn't matter which one(s)). The corner is rendered with a small nub of wall in the corner itself, but otherwise with floor running up to both edges.
Flush horizontal. This applies when the current cell has a horizontal wall entering the corner but not a vertical wall entering it. The corner is rendered with horizontal wall running all the way to the edge.
Flush vertical. This applies when the current cell has a vertical wall entering the corner, but not a horizontal one. The corner is rendered with vertical wall running all the way to the edge.
Internal corner. This applies where the current cell has two walls entering that corner.

With regard to those, note well that although there are five alternatives at each corner, they are not independent, because each adjacent pair of corners must be consistent with whether there is a wall along the edge connecting them. The corners at each end of a wall segment must each be either flush corresponding to the direction of the wall or internal. The corners at each end of an open edge must each be one of the three other choices.

Note also that the other cells factor in only for choosing between empty style and corner cap style.

Approach

I would

devise a compact encoding of those characteristics (see below)
build a table mapping the valid codes to tile numbers
write a function to compute the code for each cell.
assign tiles to cells by computing the appropriate code for the cell and looking up the corresponding tile in the table.

For example, you might encode each corner's characteristics in each cell with three bits:

bit 2	bit 1	bit 0
exterior wall enters	left wall enters	right wall enters

A combination of such a code for each of four corners fits easily an a 16-bit integer, and the total number of valid codes is quite small (there are fewer than 90 tiles in your whole tile set, covering two styles). With so few codes, you can probably get away with looking up these codes via a linear search, but you could also set up for binary searching or arrange the codes into a data structure that provides faster access, such as a tree or hash table.