Deferred MSAA, always has been a good problem. In the spatial anti-aliasing domain, MSAA is still the swiss-army knife, handle almost all the case.
Some other post-processing methods like nvidia’s FXAA, AMDs MLAA, or DLAA. FXAA is rather pleasing in many cases as well, especially if you are a video game developer, as long as your rasterization implementation does not screwed up. But for the case like grass rendering, fur rendering, when you have many layers of thin line, FXAA will fail you. Just like the pixels annotated in the image below.
Other option like Nvidia’s AGAA(aggregated G-buffer anti-aliasing) Or if you are
really a game developer, TAA maybe the way to go. But here we are talking about
good old MSAA technique(also, refer to this
article if you
can have non-even sample depth buffer). Here we are talking about MSAA, with
only OpenGL 4.0 capability, no special vendor requirement. The trick is simple
and effective: “MSAA edge detection”. If we can detect difference between simple
pixel and complex pixel, we can treat them differently. It is the similar idea
from this
method.
In the D3D API, you can use SV_Coverage, in GLSL, you have similar variable
GL_SampleMaskIn, these are the variable which are available in the pixel
shader, tell the GPU which sample to write after shading. Take 4xMSAA for
example, if all the samples come from the same fragment, then the sample mask is
1111 for that pixel(4 bits). If the fragment only writes to 2 samples, the
sample mask would be 1010, 1100 0011, 0101, etc. Which in turns mark
this pixel complex.
There is only one problem here. Through my experiement with opengl, if the pixel is not on the triangles edge, but on the intersection of two triangles, the sample mask does not reflect this problem. I guess that is why when you look the came at the ground in the video game, there is still aliasing effect at the intersection of grass and terrian. To solve this, you can apply apply a normal test on the samples, if the pixel locates on the intersection, the normal test fails and thus mark the pixel complex.
Voilà, here is your deffered MSAA, it runs really fast for the lighting pass, and it works all the time, unless(there is always an exception), you have co-planar z-fighting problem, then you would have probably bigger problem to worry about than anti-aliasing.