I'm trying to enable mutlisampling and alpha-to-coverage for an FBO. Using the default framebuffer, all I have to do is call glEnable(GL_MULTISAMPLE) and glEnable(GL_SAMPLE_ALPHA_TO_COVERAGE). However, I am unable to achieve the same effect using my own FBO.
My goal: Draw the scene to an FBO the same way it would be drawn to the default framebuffer with the above properties. From there I want to be able to use the image as a texture for future passes through a shader.
This works: Code for making an FBO without multisampling/alpha-to-coverage, 1 color attachment, 1 depth attachment:
// Generate the color attachment
glGenTextures(1,&defaultColorAttachment0);
glActiveTexture(GL_TEXTURE0);
glBindTexture(GL_TEXTURE_2D,defaultColorAttachment0);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glTexImage2D(GL_TEXTURE_2D,0,GL_RGBA,screenWidth,screenHeight,0,GL_RGBA,GL_UNSIGNED_BYTE,NULL);
// Bind the texture to the FBO
glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, defaultColorAttachment0,0);
// Generate the depth attachment
glGenRenderbuffers(1,&defaultDepthBuffer);
glBindRenderbuffer(GL_RENDERBUFFER, defaultDepthBuffer);
glRenderbufferStorage(GL_RENDERBUFFER, GL_DEPTH_COMPONENT, screenWidth, screenHeight);
glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_DEPTH_ATTACHMENT, GL_RENDERBUFFER, defaultDepthBuffer);
This doesn't work. Code trying to make a multisampled FBO:
// Generate the color attachment
glGenTextures(1,&defaultColorAttachment0);
glBindTexture(GL_TEXTURE_2D_MULTISAMPLE, defaultColorAttachment0);
glTexImage2DMultisample(GL_TEXTURE_2D_MULTISAMPLE, 4, GL_RGBA, screenWidth, screenHeight, GL_FALSE);
glFramebufferTexture(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, defaultColorAttachment0,0);
// Generate the depth attachment
glGenRenderbuffers(1,&defaultDepthBuffer);
glBindRenderbuffer(GL_RENDERBUFFER, defaultDepthBuffer);
glRenderbufferStorage(GL_RENDERBUFFER, GL_DEPTH_COMPONENT, screenWidth, screenHeight);
glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_DEPTH_ATTACHMENT, GL_RENDERBUFFER, defaultDepthBuffer);
I have tried looking through the OpenGL wiki on this, although the it's incomplete (various unfinished headings make it look unprofessional). glGetError never complains. I've tried messing around with this, but I either get a black screen or a screen full of garbage pixels.
Main Question: What things do I need to consider/change and where (FBO creation, textures, shaders) in order to get multisampling and alpha-to-coverage to work with an FBO?
You need to allocate a multisampled depth buffer for this to work correctly and give it the same number of samples as your color buffer. In other words, you should be calling glRenderbufferStorageMultisample (...) instead of glRenderbufferStorage (...).
Your FBO should be failing a completeness check the way it is allocated right now. A call to glCheckFramebufferStatus (...) ought to return GL_FRAMEBUFFER_INCOMPLETE_MULTISAMPLE because your depth buffer has exactly 1 sample and your color buffer attachment has 4.
Since you are also using a multisampled texture attachment in this FBO, you should be aware of differences between sampling a single-sampled texture vs. multisampled in GLSL shaders.
Multisampled textures have a special sampler uniform type (e.g. sampler2DMS) and you have to explicitly fetch each sample in the texture by its integer (non-normalized) texel coordinate and sample index using texelFetch (...). This also means that they cannot be filtered or mip-mapped.
You probably do not want a multisampled texture in this case, you probably want to use glBlitFramebuffer (...) to do the MSAA resolve into a single-sampled FBO. If you do this instead you can read the anti-aliased results in your shaders rather than having to fetch each sample and implement the anti-aliasing yourself.