Three.js Physijs move mesh towards mouse?

So i've been sifting through posts on here and the docs to try and figure out how to get a BoxMesh to essentially follow the mouse. Eventually i'll do joystick touch controls, similar to Heroes of Loot, or other games that use a touch joystick. But for now I want to ensure I can get directional velocity working properly.

So here's what i have in my mouse down event:

if (scene.mouseControls.isDown) {
  var coords = scene.mouseControls.screenCoords;
  var vector = new THREE.Vector3();

  vector.set(
    ( scene.mouseControls.screenCoords.x / window.innerWidth ) * 2 - 1,
    0.5,
    - ( scene.mouseControls.screenCoords.y / window.innerHeight ) * 2 + 1
  );
  vector.unproject(scene.camera);

  var p1 = this.mesh.position;
  var p2 = vector;
  var angle = Math.atan2(p2.x - p1.x, p2.z - p1.z);
  var velX = Math.sin(angle) * 20;
  var velZ = Math.cos(angle) * 20;

  console.log(vector.x, vector.z);

  this.mesh.setLinearVelocity(new THREE.Vector3(velX, 0, velZ));
}
else {
  this.mesh.setLinearVelocity(notMovingVector);
}

What happens though is the determined velocity seems to flip back and forth, so the player is very very jittery.

Here's how I set mouse coords:

window.addEventListener(activeEventList[POINTER_MOVE], function (e) {
  if (e.touches) {
    var t = e.touches[0];
    _this.screenCoords.x = t.clientX;
    _this.screenCoords.y = t.clientY;
  }
  else {
    _this.screenCoords.x = e.clientX;
    _this.screenCoords.y = e.clientY;
  }
});

Not using scroll offset or anything, as the scene is full screen.

Edit

Removed the previous direction code. Using atan2 to calculate an angle, and apply that instead. Similar to how ive done this in 2d games.

Solution

Managed to figure this out. My change to use atan2 was a step in the right direction, but it needed the multiplyScalar back. The thing is I am using the cameras X & Y values to calculate distance. But visually, its X & Y covers more space than down at the plane.

So here's what i did:

if (scene.mouseControls.isDown) {
  var coords = scene.mouseControls.screenCoords;
  var vector = new THREE.Vector3();

  vector.set(
    ( scene.mouseControls.screenCoords.x / window.innerWidth ) * 2 - 1,
    - ( scene.mouseControls.screenCoords.y / window.innerHeight ) * 2 + 1,
    0.5
  );
  vector.unproject(scene.camera);

  var dir = vector.sub(scene.camera.position).normalize();
  var distance = - scene.camera.position.y / dir.y;

  var p1 = this.mesh.position;
  var p2 = scene.camera.position.clone().add(dir.multiplyScalar(distance));

  var angle = Math.atan2(p2.z - p1.z, p2.x - p1.x);
  var velX = Math.cos(angle) * 20;
  var velZ = Math.sin(angle) * 20;

  this.mesh.setLinearVelocity(new THREE.Vector3(velX, 0, velZ));
}