javawindowsawtdisplayjna

Java Multi-Display Handling under Windows - Bug with scaled displays?

tl;dr

Under Windows 10, if I put my secondary display to the right of the primary one, and apply a scaling (e.g. 150%) to the secondary, then the display coordinates (as returned by the Java API) overlap instead of letting the display bounds sit side by side. In other words, if I slowly move my mouse from the left edge of the primary to the right edge of the secondary, Java's API MouseInfo.getPointerInfo().getLocation() returns an increasing X-position from 0 to 1920, then once the cursor enters the second screen, the value jumps back down to 1280 and then increases again to 2560. So the 1280-1920 range is returned twice, for different areas.

At the end of the post, I have included an (updated) demo that makes the issue obvious. Don't hesitate to try it and report back.

The long version:

This text gives (too) much context but is also meant to share the things I learned while searching on the topic.

First, why bother ? Because I am building a screen capture application in Java that requires a correct handling of multi-display configurations, including displays where Windows' scaling feature is applied.

Using the Java API (GraphicsEnvironment.getLocalGraphicsEnvironment().getScreenDevices()), as long as the scaling is 100%, one can observe that the primary display has its top left corner at the origin (0,0), with the other displays having coordinates "next" to the main one.

The following pictures were made using the code at the end of the post.

E.g. if we have 2 full-hd displays, the main one has its top left corner is at (0,0), while...

enter image description here

enter image description here

enter image description here

enter image description here

enter image description here

enter image description here

However, if the secondary display is scaled, things go awry: it seems the scaling factor is applied not only to its dimensions, but also its origin, which gets closer to (0,0).

If the secondary is on the left, it makes sense. For example, when the secondary 1920x1080 is scaled at 150%, it makes a logical 1280x720 positioned at (-1280,0):

enter image description here

But if the secondary is on the right, the origin is also scaled to (1280,0), getting closer to the origin and causing it to "overlap" the primary one:

enter image description here

In other words, if the mouse is at (1800,0) - see red dot above - I see no way of knowing if it actually is positioned on the right of the first display (at 120px from the right edge) or on the left of the secondary one (at 520px of the left edge). When moving the mouse from the primary to the secondary display in this case, the X position of the mouse "jumps back" when it reaches the border of the primary display.

The same is true for positioning a window on the screens. If I set the X-position of a dialog to 1800, I have no way to know where it will open.

After much browsing, some answers like this one indicate that the only way to query Windows scaling is by using native calls. Indeed, using JNA, one can get the physical size of the displays (although the answer seems to indicate that call should return the logical size). I.e the JNA calls ignore the scaling factor, and behaves exactly like the Java API when scaling is at 100%:

enter image description here

So am I missing something ?

Not knowing the scaling factor is a small issue, but not being able to tell which display the mouse is over, or not being able to position a window on the display I want looks like a real problem to me. Is it a Java Bug ?

Note: Here is the code for the app used above, run on with OpenJDK14 on Windows 10 64b. It shows a scaled down version of your display setup and mouse position as perceived by Java. It can also place and move a small dialog across the real screens if you click and drag inside the small rectangles. Credit: The UI is inspired by the WheresMyMouse code posted here.

As is, the code uses only the Java API. If you want to compare with JNA, search for the 4 blocks marked "JNA_ONLY", uncomment them, and add the jna libs. The demo will then toggle between JNA and Java API for displaying screen bounds and mouse cursor at each right-click. The dialog positioning never uses JNA in this version.

// JNA_ONLY
//import com.sun.jna.platform.win32.User32;
//import com.sun.jna.platform.win32.WinDef;
//import com.sun.jna.platform.win32.WinUser;

import javax.swing.*;
import java.awt.*;
import java.awt.event.MouseAdapter;
import java.awt.event.MouseEvent;
import java.awt.font.FontRenderContext;
import java.awt.font.TextLayout;
import java.awt.geom.Rectangle2D;
import java.util.ArrayList;
import java.util.List;

/**
 * Java multi-display detection and analysis.
 * UI idea based on WheresMyMouse - https://stackoverflow.com/a/21592711/13551878
 */
public class ShowDisplays {

    private static boolean useJna = false;

    public static void main(String[] args) {
        EventQueue.invokeLater(() -> {
            JFrame frame = new JFrame("Display Configuration");
            frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
            frame.setLayout(new BorderLayout());
            frame.add(new TestPane());
            frame.pack();
            frame.setLocationRelativeTo(null);
            frame.setVisible(true);
        });
    }

    public static class TestPane extends JPanel {
        private List<Rectangle> screenBounds;
        JDialog dlg;

        public TestPane() {
            screenBounds = getScreenBounds();
            // refresh screen details every second to reflect changes in Windows Preferences in "real time"
            new Timer(1000, e -> screenBounds = getScreenBounds()).start();

            // Refresh mouse position at 25fps
            new Timer(40, e -> repaint()).start();

            MouseAdapter mouseAdapter = new MouseAdapter() {

                public void mouseClicked(MouseEvent e) {
                    if (e.getButton() != MouseEvent.BUTTON1) {
                        useJna = !useJna;
                        repaint();
                    }
                }

                @Override
                public void mousePressed(MouseEvent e) {
                    System.out.println(e.getButton());
                    if (e.getButton() == MouseEvent.BUTTON1) {
                        if (!dlg.isVisible()) {
                            dlg.setVisible(true);
                        }
                        moveDialogTo(e.getPoint());
                    }
                }


                @Override
                public void mouseDragged(MouseEvent e) {
                    moveDialogTo(e.getPoint());
                }


                private void moveDialogTo(Point mouseLocation) {
                    final Rectangle surroundingRectangle = getSurroundingRectangle(screenBounds);
                    double scaleFactor = Math.min((double) getWidth() / surroundingRectangle.width, (double) getHeight() / surroundingRectangle.height);

                    int xOffset = (getWidth() - (int) (surroundingRectangle.width * scaleFactor)) / 2;
                    int yOffset = (getHeight() - (int) (surroundingRectangle.height * scaleFactor)) / 2;

                    int screenX = surroundingRectangle.x + (int) ((mouseLocation.x - xOffset) / scaleFactor);
                    int screenY = surroundingRectangle.y + (int) ((mouseLocation.y - yOffset) / scaleFactor);

                    dlg.setLocation(screenX - dlg.getWidth() / 2, screenY - dlg.getHeight() / 2);
                }


            };

            addMouseListener(mouseAdapter);
            addMouseMotionListener(mouseAdapter);

            // Prepare the test dialog
            dlg = new JDialog();
            dlg.setTitle("Here");
            dlg.setSize(50, 50);
            dlg.setDefaultCloseOperation(JFrame.HIDE_ON_CLOSE);

        }

        @Override
        public Dimension getPreferredSize() {
            return new Dimension(400, 400);
        }

        @Override
        protected void paintComponent(Graphics g) {
            super.paintComponent(g);
            Graphics2D g2d = (Graphics2D) g.create();

            // Mouse position
            Point mousePoint = getMouseLocation();

            g2d.setColor(Color.BLACK);
            g2d.fillRect(0, 0, getWidth(), getHeight());

            final Rectangle surroundingRectangle = getSurroundingRectangle(screenBounds);
            double scaleFactor = Math.min((double) getWidth() / surroundingRectangle.width, (double) getHeight() / surroundingRectangle.height);

            int xOffset = (getWidth() - (int) (surroundingRectangle.width * scaleFactor)) / 2;
            int yOffset = (getHeight() - (int) (surroundingRectangle.height * scaleFactor)) / 2;

            g2d.setColor(Color.BLUE);
            g2d.fillRect(xOffset, yOffset, (int) (surroundingRectangle.width * scaleFactor), (int) (surroundingRectangle.height * scaleFactor));

            Font defaultFont = g2d.getFont();
            for (int screenIndex = 0; screenIndex < screenBounds.size(); screenIndex++) {
                Rectangle screen = screenBounds.get(screenIndex);
                Rectangle scaledRectangle = new Rectangle(
                        xOffset + (int) ((screen.x - surroundingRectangle.x) * scaleFactor),
                        yOffset + (int) ((screen.y - surroundingRectangle.y) * scaleFactor),
                        (int) (screen.width * scaleFactor),
                        (int) (screen.height * scaleFactor));

                // System.out.println(screen + " x " + scaleFactor + " -> " + scaledRectangle);
                g2d.setColor(Color.DARK_GRAY);
                g2d.fill(scaledRectangle);
                g2d.setColor(Color.GRAY);
                g2d.draw(scaledRectangle);

                // Screen text details
                g2d.setColor(Color.WHITE);

                // Display number
                final Font largeFont = new Font(defaultFont.getName(), defaultFont.getStyle(), (int) (screen.height * scaleFactor) / 2);
                g2d.setFont(largeFont);
                String label = String.valueOf(screenIndex + 1);
                FontRenderContext frc = g2d.getFontRenderContext();
                TextLayout layout = new TextLayout(label, largeFont, frc);
                Rectangle2D bounds = layout.getBounds();
                g2d.setColor(Color.WHITE);
                g2d.drawString(
                        label,
                        (int) (scaledRectangle.x + (scaledRectangle.width - bounds.getWidth()) / 2),
                        (int) (scaledRectangle.y + (scaledRectangle.height + bounds.getHeight()) / 2)
                );

                // Resolution + corner
                final Font smallFont = new Font(defaultFont.getName(), defaultFont.getStyle(), (int) (screen.height * scaleFactor) / 10);
                g2d.setFont(smallFont);

                // Resolution
                String resolution = screen.width + "x" + screen.height;
                layout = new TextLayout(resolution, smallFont, frc);
                bounds = layout.getBounds();
                g2d.drawString(
                        resolution,
                        (int) (scaledRectangle.x + (scaledRectangle.width - bounds.getWidth()) / 2),
                        (int) (scaledRectangle.y + scaledRectangle.height - bounds.getHeight())
                );

                // Corner
                String corner = "(" + screen.x + "," + screen.y + ")";
                g2d.drawString(
                        corner,
                        scaledRectangle.x,
                        (int) (scaledRectangle.y + bounds.getHeight() * 1.5)
                );

            }

            g2d.setFont(defaultFont);
            FontMetrics fm = g2d.getFontMetrics();

            if (mousePoint != null) {
                g2d.fillOval(xOffset + (int) ((mousePoint.x - surroundingRectangle.x) * scaleFactor) - 2,
                        yOffset + (int) ((mousePoint.y - surroundingRectangle.y) * scaleFactor) - 2,
                        4,
                        4
                );
                g2d.drawString("Mouse pointer is at (" + mousePoint.x + "," + mousePoint.y + ")", 4, fm.getHeight());
            }

            g2d.drawString("Click and drag in this area to move a dialog on the actual screens", 4, fm.getHeight() * 2);

            // JNA_ONLY
            // g2d.drawString("Now using " + (useJna ? "JNA" : "Java API") + ". Right-click to toggle", 4, fm.getHeight() * 3);

            g2d.dispose();
        }
    }

    public static Rectangle getSurroundingRectangle(List<Rectangle> screenRectangles) {
        Rectangle surroundingBounds = null;
        for (Rectangle screenBound : screenRectangles) {
            if (surroundingBounds == null) {
                surroundingBounds = new Rectangle(screenRectangles.get(0));
            }
            else {
                surroundingBounds.add(screenBound);
            }
        }
        return surroundingBounds;
    }

    private static Point getMouseLocation() {
        // JNA_ONLY
//        if (useJna) {
//            final WinDef.POINT point = new WinDef.POINT();
//            if (User32.INSTANCE.GetCursorPos(point)) {
//                return new Point(point.x, point.y);
//            }
//            else {
//                return null;
//            }
//        }
        return MouseInfo.getPointerInfo().getLocation();
    }

    public static List<Rectangle> getScreenBounds() {
        List<Rectangle> screenBounds;

        // JNA_ONLY
//        if (useJna) {
//            screenBounds = new ArrayList<>();
//            // Enumerate all monitors, and call a code block for each of them
//            // See https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-enumdisplaymonitors
//            // See http://www.pinvoke.net/default.aspx/user32/EnumDisplayMonitors.html
//            User32.INSTANCE.EnumDisplayMonitors(
//                    null, // => the virtual screen that encompasses all the displays on the desktop.
//                    null, // => don't clip the region
//                    (hmonitor, hdc, rect, lparam) -> {
//                        // For each found monitor, get more information
//                        // See https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-getmonitorinfoa
//                        // See http://www.pinvoke.net/default.aspx/user32/GetMonitorInfo.html
//                        WinUser.MONITORINFOEX monitorInfoEx = new WinUser.MONITORINFOEX();
//                        User32.INSTANCE.GetMonitorInfo(hmonitor, monitorInfoEx);
//                        // Retrieve its coordinates
//                        final WinDef.RECT rcMonitor = monitorInfoEx.rcMonitor;
//                        // And convert them to a Java rectangle, to be added to the list of monitors
//                        screenBounds.add(new Rectangle(rcMonitor.left, rcMonitor.top, rcMonitor.right - rcMonitor.left, rcMonitor.bottom - rcMonitor.top));
//                        // Then return "true" to continue enumeration
//                        return 1;
//                    },
//                    null // => No additional info to pass as lparam to the callback
//            );
//            return screenBounds;
//        }

        GraphicsEnvironment graphicsEnvironment = GraphicsEnvironment.getLocalGraphicsEnvironment();
        GraphicsDevice[] screenDevices = graphicsEnvironment.getScreenDevices();
        screenBounds = new ArrayList<>(screenDevices.length);
        for (GraphicsDevice screenDevice : screenDevices) {
            GraphicsConfiguration configuration = screenDevice.getDefaultConfiguration();
            screenBounds.add(configuration.getBounds());
        }
        return screenBounds;
    }

}
Solution

  • This looks like you've run into a manifestation of bug JDK-8211999:

    In a multi-monitor setting involving one HiDPI screen placed to the right of one regular monitor, on Windows 10, the bounds returned by GraphicsEnvironment.getLocalGraphicsEnvironment().getScreenDevices()[x].getDefaultConfiguration().getBounds() are overlapping. This causes various secondary bugs...

    Comments note that:

    The same bug exists on Linux as well, macOS is not affected.

    There does not seem to be a simple pure Java workaround.

    A fix has been proposed which works for Windows, by not even trying to do the coordinate math in Java, and delegating the solution to native code.

    Since it appears that using the JNA (native) implementation appears to work, this seems the best approach for JDK versions 9 to 15. The bug was allegedly fixed in JDK16 but comments indicates it still exists in JDK 17. There's a related bug JDK-8249164 which may or may not be the cause of these additional problems.

    According to the bug report, it affects JDK 9+, so it is possible that reverting to JDK 8 may fix the issue, although I saw conflicting accounts on that. Comments also indicate other workaround available in JavaFX, that I hope will find their way into another answer.