How to get 2D coordinates on window for 3D object in javafx

In javafx if we have 2D HUD (made of Pane and then out of it we create SubScene object for 2D Hud) and 3D SubScene and inside 3D scene we have some object with coordinates (x,y,z) - how can we get 2D coordinates in our HUD of the object if it is in visual field of our perspective camera?

I tried to get first Scene coordinates of the object and then convert it (sceneToScreen) coordinates and the same for point (0,0) of Pane and then to subtract first point from second point but i don't get right result. Sorry because of my bad English.Can Someone help with this?

There is a way to convert the 3D coordinates of an object in a subScene to a 2D scene coordinates, but unfortunately it uses private API, so it is advised not to rely on it.

The idea is based on how the camera projection works, and it is based on the com.sun.javafx.scene.input.InputEventUtils.recomputeCoordinates() method that is used typically for input events from a PickResult.

Let's say you have a node in a sub scene. For a given point of that node, you can obtain its coordinates like:

Point3D coordinates = node.localToScene(Point3D.ZERO);

and you can find out about the sub scene of the node:

SubScene subScene = NodeHelper.getSubScene(node);

Now you can use the SceneUtils::subSceneToScene method that

Translates point from inner subScene coordinates to scene coordinates.

to get a new set of coordinates, referenced to the scene:

coordinates = SceneUtils.subSceneToScene(subScene, coordinates);

But these are still 3D coordinates.

The final step to convert those to 2D is with the use of CameraHelper::project:

final Camera effectiveCamera = SceneHelper.getEffectiveCamera(node.getScene());        
Point2D p2 = CameraHelper.project(effectiveCamera, coordinates);

The following sample places 2D labels in the scene, at the exact same position of the 8 vertices of a 3D box in a subScene.

private final Rotate rotateX = new Rotate(0, Rotate.X_AXIS);
private final Rotate rotateY = new Rotate(0, Rotate.Y_AXIS);

private double mousePosX;
private double mousePosY;
private double mouseOldX;
private double mouseOldY;

private Group root;

@Override
public void start(Stage primaryStage) {

    Box box = new Box(150, 100, 50);
    box.setDrawMode(DrawMode.LINE);
    box.setCullFace(CullFace.NONE);

    Group group = new Group(box);

    PerspectiveCamera camera = new PerspectiveCamera(true);
    camera.setNearClip(0.1);
    camera.setFarClip(10000.0);
    camera.setFieldOfView(20);
    camera.getTransforms().addAll (rotateX, rotateY, new Translate(0, 0, -500));
    SubScene subScene = new SubScene(group, 500, 400, true, SceneAntialiasing.BALANCED);
    subScene.setCamera(camera);
    root = new Group(subScene);

    Scene scene = new Scene(root, 500, 400);

    primaryStage.setTitle("HUD: 2D Labels over 3D SubScene");
    primaryStage.setScene(scene);
    primaryStage.show();

    updateLabels(box);

    scene.setOnMousePressed(event -> {
        mousePosX = event.getSceneX();
        mousePosY = event.getSceneY();
    });

    scene.setOnMouseDragged(event -> {
        mousePosX = event.getSceneX();
        mousePosY = event.getSceneY();
        rotateX.setAngle(rotateX.getAngle() - (mousePosY - mouseOldY));
        rotateY.setAngle(rotateY.getAngle() + (mousePosX - mouseOldX));
        mouseOldX = mousePosX;
        mouseOldY = mousePosY;
        updateLabels(box);
    });
}

private List<Point3D> generateDots(Node box) {
    List<Point3D> vertices = new ArrayList<>();
    Bounds bounds = box.getBoundsInLocal();
    vertices.add(box.localToScene(new Point3D(bounds.getMinX(), bounds.getMinY(), bounds.getMinZ())));
    vertices.add(box.localToScene(new Point3D(bounds.getMinX(), bounds.getMinY(), bounds.getMaxZ())));
    vertices.add(box.localToScene(new Point3D(bounds.getMinX(), bounds.getMaxY(), bounds.getMinZ())));
    vertices.add(box.localToScene(new Point3D(bounds.getMinX(), bounds.getMaxY(), bounds.getMaxZ())));
    vertices.add(box.localToScene(new Point3D(bounds.getMaxX(), bounds.getMinY(), bounds.getMinZ())));
    vertices.add(box.localToScene(new Point3D(bounds.getMaxX(), bounds.getMinY(), bounds.getMaxZ())));
    vertices.add(box.localToScene(new Point3D(bounds.getMaxX(), bounds.getMaxY(), bounds.getMinZ())));
    vertices.add(box.localToScene(new Point3D(bounds.getMaxX(), bounds.getMaxY(), bounds.getMaxZ())));
    return vertices;
}

private void updateLabels(Node box) {
    root.getChildren().removeIf(Label.class::isInstance);
    SubScene oldSubScene = NodeHelper.getSubScene(box);
    AtomicInteger counter = new AtomicInteger(1);
    generateDots(box).stream()
        .forEach(dot -> {
            Point3D coordinates = SceneUtils.subSceneToScene(oldSubScene, dot);
            Point2D p2 = CameraHelper.project(SceneHelper.getEffectiveCamera(box.getScene()), coordinates);
            Label label = new Label("" + counter.getAndIncrement() + String.format(" (%.1f,%.1f)", p2.getX(), p2.getY()));
            label.setStyle("-fx-font-size:1.3em; -fx-text-fill: blue;");
            label.getTransforms().setAll(new Translate(p2.getX(), p2.getY()));
            root.getChildren().add(label);
        });
}

The FXyz3D library has another similar sample.

EDIT

A late edit of this answer, but it is worthwhile mentioning that there is no need for private API. There is public API in the Node::localToScene methods that allows traversing the subScene.

So this just works (note the true argument):

Point3D p2 = box.localToScene(dot, true);

According to the JavaDoc for Node::localToScene:

Transforms a point from the local coordinate space of this Node into the coordinate space of its scene. If the Node does not have any SubScene or rootScene is set to true, the result point is in Scene coordinates of the Node returned by getScene(). Otherwise, the subscene coordinates are used, which is equivalent to calling localToScene(Point3D).

Without true the conversion is within the subScene, but with it, the conversion goes from the current subScene to the scene. In this case, this methods calls SceneUtils::subSceneToScene, so we don't need to do it anymore.

With this, updateLabels gets simplified to:

private void updateLabels(Node box) {
    root.getChildren().removeIf(Label.class::isInstance);
    AtomicInteger counter = new AtomicInteger(1);
    generateDots(box).stream()
            .forEach(dot -> {
                Point3D p2 = box.localToScene(dot, true);
                Label label = new Label("" + counter.getAndIncrement() + String.format(" (%.1f,%.1f)", p2.getX(), p2.getY()));
                label.setStyle("-fx-font-size:1.3em; -fx-text-fill: blue;");
                label.getTransforms().setAll(new Translate(p2.getX(), p2.getY()));
                root.getChildren().add(label);
            });
}