I am developing a document based application for Mac OS X. It's a kind of media player, but instead of playing audio or video files it is supposed to open text-files containing meta-data specifying OpenGL animations. I would like to mimic Apples QuickTime X window style. This means, i have to do all the window drawings myself, because Cocoa has no appropriate window style.
There is one thing which gives me headaches: The rounded corners usually to be found on Mac OS X windows. I tried using the borderless window mask and working some CGS magic - there are some private Apple headers which allow window shaping, but they are of course undocumented. I was able to cut rectangular holes in my windows edges, but i couldn't figure out how Apple achieves rounded corners.
Creating a transparent window and drawing the frame myself does not work, because an OpenGL viewport is always rectangular, and the only way to change it is to turn on NSOpenGLCPSurfaceOpacity for alpha transparency and using the stencil buffer or shaders to cut out the edges, which seems like a hell of a lot of overhead.
If i put an OpenGLView into a standard Cocoa window with titlebar, the bottom edges are rounded. It seems this is happening at the NSThemeFrame stage of the view hierarchy. Any ideas how this is done?
Since Robs suggestion didn't work and no one else contributed to the discussion i settled on using the stencil buffer to crop the windows corners. I did this by creating a texture from the windows background and rendering it into the stencil buffer, discarding all transparent pixels. Looks fine, but is slow when resizing the window :/