Direct3D 11 and 2D: multiplication matrix-vector in HLSL does not give the correct result

I am still trying to improve my test program about Direct3D 11 in 2D, without any additional library (see direct3d 11 and 2D: pass coordinates of a vertex as int and not float).

Now, what I want to do is to rotate the content (client) area of the window. The GDI part is working, but not the D3D one. The function d3d_resize() takes an argument named rot and, according to its value (0, 1, 2 or 3), I want to rotate the client area.

For now, I focus on rot == 0, that is no rotation.

I use a constant buffer to store the rotation matrix, defined (mathematically) like this:

|r11 r12 t1|
|r21 r22 t2|

where the sub-matrix (r) is the rotation part, and the sub-vector (t) is the translation.

So for rot == 0, the matrix is basically the identity matrix + no translation. If (x,y) is my 2D vector, I expand it with 1 to take into account the translation, hence:

|1 0 0|   |x|   |x|
|0 1 0| * |y| = |y|
          |1|

that is, what I want if rot == 0. My C part for the rotation is:

typedef struct
{
    float rotation[2][3]; /* rotation + translation */
    float dummy[2]; /* for 16 bytes padding */
} Const_Buffer;

void d3d_resize(D3d *d3d, int rot, UINT width, UINT height)
{
    /* snip */

    switch (rot)
    {
        case 0:
            ((Const_Buffer *)mapped.pData)->rotation[0][0] = 1.0f;
            ((Const_Buffer *)mapped.pData)->rotation[0][1] = 0.0f;
            ((Const_Buffer *)mapped.pData)->rotation[0][2] = 0.0f;
            ((Const_Buffer *)mapped.pData)->rotation[1][0] = 0.0f;
            ((Const_Buffer *)mapped.pData)->rotation[1][1] = 1.0f;
            ((Const_Buffer *)mapped.pData)->rotation[1][2] = 0.0f;
            break;
        case 1:
            ((Const_Buffer *)mapped.pData)->rotation[0][0] = 0.0f;
            ((Const_Buffer *)mapped.pData)->rotation[0][1] = -1.0f;
            ((Const_Buffer *)mapped.pData)->rotation[0][2] = 2.0f;
            ((Const_Buffer *)mapped.pData)->rotation[1][0] = 1.0f;
            ((Const_Buffer *)mapped.pData)->rotation[1][1] = 0.0f;
            ((Const_Buffer *)mapped.pData)->rotation[1][2] = 0.0f;
            break;
        case 2:
            ((Const_Buffer *)mapped.pData)->rotation[0][0] = -1.0f;
            ((Const_Buffer *)mapped.pData)->rotation[0][1] = 0.0f;
            ((Const_Buffer *)mapped.pData)->rotation[0][2] = 0.0f;
            ((Const_Buffer *)mapped.pData)->rotation[1][0] = 0.0f;
            ((Const_Buffer *)mapped.pData)->rotation[1][1] = -1.0f;
            ((Const_Buffer *)mapped.pData)->rotation[1][2] = 0.0f;
            break;
        case 3:
            ((Const_Buffer *)mapped.pData)->rotation[0][0] = 0.0f;
            ((Const_Buffer *)mapped.pData)->rotation[0][1] = 1.0f;
            ((Const_Buffer *)mapped.pData)->rotation[0][2] = 0.0f;
            ((Const_Buffer *)mapped.pData)->rotation[1][0] = -1.0f;
            ((Const_Buffer *)mapped.pData)->rotation[1][1] = 0.0f;
            ((Const_Buffer *)mapped.pData)->rotation[1][2] = 2.0f;
            break;
    }

    /* snip */
}

The vertex shader is:

cbuffer cv_viewport : register(b0)
{
    float2x3 rotation_matrix;
}

struct vs_input
{
    float2 position : POSITION;
    float4 color : COLOR;
};

struct ps_input
{
    float4 position : SV_POSITION;
    float4 color : COLOR;
};

ps_input main_vs(vs_input input )
{
    ps_input output;
    float2 p = input.position;
    p = mul(rotation_matrix, float3(input.position, 1.0f));
    output.position = float4(p, 0.0f, 1.0f);
    output.color = input.color;
    return output;
}

I used Visual Studio graphic debugger to check for the const buffer values when rot == 0 and they seem correct.

but the yellow triangle and blue rectangle are not displayed when rot == 0. If I comment the line in the shader where the multiplication is done, they are displayed. I've done something wrong but I don't know what and where. Find below the link to the complete C code + the shader code in my github (it is too large to be posted inline)

C code: https://github.com/vtorri/d3d_rot/blob/master/d3d_rot.c

shader code: https://github.com/vtorri/d3d_rot/blob/master/shader_3.hlsl

screenshot without mul:

screenshot with mul:

so, actually, the problem is padding in the constant buffer using a matrix in C code, and specify row layout for the matrix in the shader (see https://learn.microsoft.com/en-us/windows/win32/direct3dhlsl/dx-graphics-hlsl-variable-syntax)

Solution:

constant buffer in C code:

float rotation[2][4];

constant buffer in shader:

row_major float2x3 rotation_matrix;

now, it's working :-)