Unable to mouse pick a quad rendered in a framebuffer

Struggling to mouse pick a point/quad. I believe I am either using coordinates in the wrong space or perhaps not accounting for the framebuffer's position/size (it's a sub window of the main window).

Tried converting to various different coordinate spaces and inverting the model matrix too. Currently projecting a ray in world space (hopefully correctly) and trying to compare it to the point's (quad) location. The point is specified in local space, but the entity is rendered at the origin (0f, 0f, 0f) therefore I don't think it should be any different in world space?

To get the mouse ray in world space:

private fun calculateRay(): Vector3f {
        val mousePosition = Mouse.getCursorPosition()
        val ndc = toDevice(mousePosition)
        val clip = Vector4f(ndc.x, ndc.y, -1f, 1f)
        val eye = toEye(clip)
        return toWorld(eye)
    }

private fun toDevice(mousePosition: Vector2f): Vector2f {
        mousePosition.x -= fbo.x // Correct thing to do?
        mousePosition.y -= fbo.y
        val x = (2f * mousePosition.x) / fboSize.x - 1
        val y = (2f * mousePosition.y) / fboSize.y - 1
        return Vector2f(x, y)
    }

private fun toEye(clip: Vector4f): Vector4f {
        val invertedProjection = Matrix4f(projectionMatrix).invert()
        val eye = invertedProjection.transform(clip)
        return Vector4f(eye.x, eye.y, -1f, 0f)
    }

    private fun toWorld(eye: Vector4f): Vector3f {
        val viewMatrix = Maths.createViewMatrix(camera)
        val invertedView = Matrix4f(viewMatrix).invert()
        val world = invertedView.transform(eye)
        return Vector3f(world.x, world.y, world.z).normalize()
    }

When hovering over a point (11.25, -0.75), the ray coords are (0.32847548, 0.05527423). I tried normalising the point's position and it is still not a match.

Feel like I am missing/overlooking something or just manipulating the coordinate systems incorrectly. Any insight would be greatly appreciated, thank you.

EDIT with more information:

The vertices of the quad are: (-0.5f, 0.5f, -0.5f, -0.5f, 0.5f, 0.5f, 0.5f, -0.5f)

Loading matrices to shader:

private fun loadMatrices(position: Vector3f, rotation: Float, scale: Float, viewMatrix: Matrix4f, currentRay: Vector3f) {
        val modelMatrix = Matrix4f()
        modelMatrix.translate(position)
        modelMatrix.m00(viewMatrix.m00())
        modelMatrix.m01(viewMatrix.m10())
        modelMatrix.m02(viewMatrix.m20())
        modelMatrix.m10(viewMatrix.m01())
        modelMatrix.m11(viewMatrix.m11())
        modelMatrix.m12(viewMatrix.m21())
        modelMatrix.m20(viewMatrix.m02())
        modelMatrix.m21(viewMatrix.m12())
        modelMatrix.m22(viewMatrix.m22())
        modelMatrix.rotate(Math.toRadians(rotation.toDouble()).toFloat(), Vector3f(0f, 0f, 1f))
        modelMatrix.scale(scale)
        shader.loadModelViewMatrix(viewMatrix.mul(modelMatrix))
        shader.loadProjectionMatrix(projectionMatrix)
    }

Calculating gl_Position in vertex shader:

gl_Position = projectionMatrix * modelViewMatrix * vec4(position, 0.0, 1.0);

EDIT 2: Altered my code after reading up some more material based on Rabbid's comments. Not sure if I require the division by 2 in the viewport size (I have a retina MacBook display).

mousePosition.sub(fboPosition)
        val w = (fboSize.x / 2).toInt()
        val h = (fboSize.y / 2).toInt()
        val y = h - mousePosition.y

        val viewMatrix = Maths.createViewMatrix(camera)
        val origin = Vector3f()
        val dir = Vector3f()
        Matrix4f(projectionMatrix).mul(viewMatrix)
                                  .unprojectRay(mousePosition.x, y, intArrayOf(0, 0, w, h), origin, dir)

The top left origin of window sapce is (0,0). So if you get (0, 0), if the mouse is in the the top left of the window you've to skip:

mousePosition.x -= fbo.x // Correct thing to do?
mousePosition.y -= fbo.y

Since the bottom left of the framebuffer is (0,0), the y coordinate has to be flipped:

val y = 1 - (2f * mousePosition.y) / fboSize.y

When a Cartesian coordinate is transformed by a (inverse) projection matrix, then the result is a Homogeneous coordinates. You've to do a Perspective divide, to get a Cartesian coordinate in view space:

val eye = invertedProjection.transform(clip)
return Vector3f(eye.x/eye.w, eye.y/eye.w, eye.z/eye.w)