OpenGL ray casting (picking): account for object's transform

For picking objects, I've implemented a ray casting algorithm similar to what's described here. After converting the mouse click to a ray (with origin and direction) the next task is to intersect this ray with all triangles in the scene to determine hit points for each mesh.

I have also implemented the triangle intersection test algorithm based on the one described here. My question is, how should we account for the objects' transforms when performing the intersection? Obviously, I don't want to apply the transformation matrix to all vertices and then do the intersection test (too slow).

EDIT:
Here is the UnProject implementation I'm using (I'm using OpenTK by the way). I compared the results, they match what GluUnProject gives me:

private Vector3d UnProject(Vector3d screen)
{
    int[] viewport = new int[4];
    OpenTK.Graphics.OpenGL.GL.GetInteger(OpenTK.Graphics.OpenGL.GetPName.Viewport, viewport);

    Vector4d pos = new Vector4d();

    // Map x and y from window coordinates, map to range -1 to 1 
    pos.X = (screen.X - (float)viewport[0]) / (float)viewport[2] * 2.0f - 1.0f;
    pos.Y = 1 - (screen.Y - (float)viewport[1]) / (float)viewport[3] * 2.0f;
    pos.Z = screen.Z * 2.0f - 1.0f;
    pos.W = 1.0f;

    Vector4d pos2 = Vector4d.Transform(pos, Matrix4d.Invert(GetModelViewMatrix() * GetProjectionMatrix()));
    Vector3d pos_out = new Vector3d(pos2.X, pos2.Y, pos2.Z);

    return pos_out / pos2.W;
}  

Then I'm using this function to create a ray (with origin and direction):

private Ray ScreenPointToRay(Point mouseLocation)
{
    Vector3d near = UnProject(new Vector3d(mouseLocation.X, mouseLocation.Y, 0));
    Vector3d far = UnProject(new Vector3d(mouseLocation.X, mouseLocation.Y, 1));

    Vector3d origin = near;
    Vector3d direction = (far - near).Normalized();
    return new Ray(origin, direction);
} 

You can apply the reverse transformation of each object to the ray instead.