Normals rotates wrong way when object is rotated

I am rendering a simple torus in WebGL. Rotating the vertices works fine but I have a problem with the normals. When rotated around a single axis, they keep the correct direction but when the rotation around a second axis increases, the normals start rotating the wrong way up until one of the rotations are 180°, then the normals are rotating in the complete opposite of what they should. I assume the problem lies with the quaternion used for rotation, but I have not been able to determine what is wrong.

Here is a (slightly modified, but it still shows the problem) jsfiddle of my project: https://jsfiddle.net/dt509x8h/1/
In the html-part of the fiddle there is a div containing all the data from the obj-file I am reading to generate the torus (although a lower resolution one).

vertex shader:

attribute vec4 aVertexPosition;
attribute vec3 aNormalDirection;

uniform mat4 uMVPMatrix;
uniform mat3 uNMatrix;

varying vec3 nrm;

void main(void) {
    gl_Position = uMVPMatrix * aVertexPosition;
    nrm = aNormalDirection * uNMatrix;
}

fragment shader:

varying vec3 nrm;

void main(void) {
    gl_FragColor = vec4(nrm, 1.0);
}

Updating the matrices (run when there has been input):

mat4.perspective(pMatrix, Math.PI*0.25, width/height, clipNear, clipFar); //This is actually not run on input, it is just here to show the creation of the perspective matrix
mat4.fromRotationTranslation(mvMatrix, rotation, position);
mat3.normalFromMat4(nMatrix, mvMatrix);

mat4.multiply(mvpMatrix, pMatrix, mvMatrix);

var uMVPMatrix = gl.getUniformLocation(shaderProgram, "uMVPMatrix");
var uNMatrix = gl.getUniformLocation(shaderProgram, "uNMatrix");
gl.uniformMatrix4fv(uMVPMatrix, false, mvpMatrix);
gl.uniformMatrix3fv(uNMatrix, false, nMatrix);

Creating the rotation quaternion (called when mouse has moved):

var d = vec3.fromValues(lastmousex-mousex, mousey-lastmousey, 0.0);
var l = vec3.length(d);
vec3.normalize(d,d);
var axis = vec3.cross(vec3.create(), d, [0,0,1]);
vec3.normalize(axis, axis);

var q = quat.setAxisAngle(quat.create(), a, l*scale);
quat.multiply(rotation, q, rotation);

Rotating the torus only around the Y-axis, the normals point in the right directions:

Rotating the torus around two axes. The normals are pointing all over the place:

I am using glMatrix v2.3.2 for all matrix and quaternion operations.

Update: It seems that rotating only around the Z axis (by setting the input axis for quat.setAxisAngle explicitly to [0,0,1], or by using quat.rotateZ) also causes the normals to rotate in the opposite direction.
Zeroing the z-component of the axis does not help.

Update2: Rotating by quat.rotateX(q, q, l*scale); quat.rotateY(q, q, l*scale); quat.multiply(rotation, q, rotation); Seems correct, but as soon as rotation around Z is introduced the z normals starts to move around.
Using the difference in x or y mouse-values instead of l causes all normals to move, and so does using largely different scale-values for x and y.

Update3: changing the order of multiplication in the shader to uNMatrix * aNormalDirection causes the normals to always rotate the wrong way.

In my case, the problem was with how I loaded the data from an .obj-file. I had inverted the z-position of all vertices, but the normals were generated from the non-inverted vertices.
Using non-inverted z-positions and flipping the normal-matrix multiplication fixed the issues.