I have a 4k monitor and try to set my Ubuntu 14.04 LTS to use it. Since there is no 3840x2160 option available in system settings and NVIDIA X Server Settings - the only way to do that is using xrandr tool. So I type:
gtf 3840 2160 60
# 3840x2160 @ 60.00 Hz (GTF) hsync: 134.10 kHz; pclk: 712.34 MHz
Modeline "3840x2160_60.00" 712.34 3840 4152 4576 5312 2160 2161 2164 2235 -HSync +Vsync
xrandr --newmode test87 712.34 3840 4152 4576 5312 2160 2161 2164 2235 -HSync +Vsync
xrandr --addmode HDMI-0 test87
X Error of failed request: BadMatch (invalid parameter attributes)
Major opcode of failed request: 140 (RANDR)
Minor opcode of failed request: 18 (RRAddOutputMode)
Serial number of failed request: 35
Current serial number in output stream: 36
It seems like a common error, but other solution suggest modifying the /etc/X11/xorg.conf (that I don't have, I tried to create one and reboot but it does not help). Some others solutiond suggest using brackets around mode names - it does not help as well.
If I'm not entirely mistaken 4k@60Hz is simply beyond the bandwidth capabilities of HDMI-1.4; you either need to connect via DisplayPort or use a lower refresh rate; 30Hz would be a safe bet.
Given the EDID data dump everything looks fine so far. The xorg.conf
entries that correspond to your display would be these (generated by parse-edid, which would be equivalent to what the X.org server does).
# EDID version 1 revision 3
Section "Monitor"
# Block type: 2:0 3:fd
# Block type: 2:0 3:fc
Identifier "ASUS PB287Q"
VendorName "ACI"
ModelName "ASUS PB287Q"
# Block type: 2:0 3:fd
HorizSync 24-160
VertRefresh 30-80
# Max dot clock (video bandwidth) 300 MHz
# Block type: 2:0 3:fc
# DPMS capabilities: Active off:yes Suspend:no Standby:no
Mode "3840x2160" # vfreq 30.000Hz, hfreq 67.500kHz
DotClock 297.000000
HTimings 3840 4016 4104 4400
VTimings 2160 2168 2178 2250
Flags "-HSync" "+VSync"
EndMode
Mode "1920x1080" # vfreq 60.000Hz, hfreq 67.500kHz
DotClock 148.500000
HTimings 1920 2008 2052 2200
VTimings 1080 1084 1089 1125
Flags "+HSync" "+VSync"
EndMode
# Block type: 2:0 3:fd
# Block type: 2:0 3:fc
EndSection
Note the UHD resolution mode.
Hadn't you mentioned that the NVidia X Server Settings tool can be used I'd have suspected a broken driver installation. Anyway I strongly you double check, that the drivers are properly installed and nothing is interfering with them.