I want to set up automated testing of a GUI app (say Qt) on Ubuntu Server distribution (i.e. without phisical monitor and without window manager). How can I accomplish this?
GUI applications - particularly Qt-based ones- usually need some display to connect to. Otherwise they'll have problems rendering the GUI (e.g. fonts) and receive input (e.g. mouse).
The need for a physical display can be avoided through a virtual display however. See this Squish Knowledge Base artcile for hints on the usage of Xvnc, X11vnc, Xvfb and Xnest.
Running a window manager on the virtual display is still recommended. It does not have to be a full desktop environment. A simple WM does. GUI toolkits are not necessarily prepared