Why are my billboarded quads being rendered this way?

So I am instancing a quad a bunch of times in an area in love2d, which seems to be going fine, but when I add some shader code to billboard them they render at an odd spot and look like they are all pivoting around (0, 0, 0) (the position of the mesh i am instancing many times). Do I maybe have to make a new matrix for each instance to get them all to face the camera from their position, or I am doing something else wrong?

Here is the vertex shader code for the regular, non-billboard version:

mat4 ModelView = viewMatrix*modelMatrix;
            
relativeWorldPosition = projectionMatrix * ModelView * (vertexPosition + vec4(instancePositions[love_InstanceID].xyz, 0));          
            
return relativeWorldPosition;

Here is the vertex shader code for the billboard version:

mat4 ModelView = viewMatrix*modelMatrix;
            
ModelView[0][0] = 1;
ModelView[0][1] = 0;
ModelView[0][2] = 0;

            // Column 1:
ModelView[1][0] = 0;
ModelView[1][1] = 1;
ModelView[1][2] = 0;

            // Column 2:
ModelView[2][0] = 0;
ModelView[2][1] = 0;
ModelView[2][2] = 1;

relativeWorldPosition = projectionMatrix * ModelView * (vertexPosition + vec4(instancePositions[love_InstanceID].xyz, 0));
            
            
return relativeWorldPosition;

I tried to add billboarding to each instance of a mesh (a quad). I expected them to face the camera like a billboard from their current position. My attempt resulted in the instances rendering in a weird position that changed based on the cameras rotation.

So basically I just added a made a new model matrix from the existing one and manually set the position of it like so:

        mat4 TransModelMatrix = modelMatrix; 
        TransModelMatrix[3][0] = InstData.x;
        TransModelMatrix[3][1] = InstData.y;
        TransModelMatrix[3][2] = InstData.z;
        
        
        mat4 ModelView = viewMatrix*-TransModelMatrix;
        
        ModelView[0][0] = 1;
        ModelView[0][1] = 0;
        ModelView[0][2] = 0;

        // Column 1:
        //ModelView[1][0] = 0;
        //ModelView[1][1] = 1;
        //ModelView[1][2] = 0;

        // Column 2:
        ModelView[2][0] = 0;
        ModelView[2][1] = 0;
        ModelView[2][2] = 1;

        relativeWorldPosition = projectionMatrix * -ModelView * (vertexPosition);