SharpDX GetPickRay issues

I am trying to get a picking ray from the current camera position/view so I can use it to select objects in the 3d world. The issue is that when I move the camera around, the ray becomes incorrect. Am I doing something wrong or is it a floating point precision issue?

ViewportF viewport = new ViewportF();
viewport.Height=307.0
viewport.MaxDepth=1.0
viewport.MinDepth=0.0
viewport.Width=689.0
viewport.X=0.0
viewport.Y=0.0

Matrix viewProjection = new Matrix(new float[] {
    1.075709f, 0, 0, 85067.07f, 
    0, 2.414213f, 0, -146011.6f, 
    0, 0, 1, 14.90265f, 
    0, 0, 1, 15.00265});

Ray ray = Ray.GetPickRay(263, 77, viewport, matrix);

The result of the ray is:

ray.Position {X:-79080,03 Y:60480,02 Z:-14,90265}
ray.Direction {X:-0,1926253 Y:0,1401343 Z:0,9712146}

And on screen it creates the following line under my mouse:
(top left)

As far as I understand, the ray should not be visible under my mouse, right?
Obviously it would become visible if I move the camera and don't change the ray that I am drawing.

edit: If I don't move the camera, the ray is not visible under my mouse, not even when rotating.
edit2: It seems that the ray.Position is correct, the Ray.Direction seems to be wrong, though don't quote me on it.

Eventually it wasn't much more than the following bit of code:

public Ray GetPickRay(float mouseX, float mouseY)
{
    return Ray.GetPickRay((int)mouseX, (int)mouseY, new ViewportF(0, 0, ViewportWidth, ViewportHeight), ViewProjectionMatrix);
}