Realistic Deferred MSAA implementation

Deferred MSAA, always has been a good problem. In the spatial anti-aliasing domain, MSAA is still the swiss-army knife, handle almost all the case.

Some other post-processing methods like nvidia's FXAA, AMDs MLAA, or DLAA. FXAA is rather pleasing in many cases as well, especially if you are a video game developer, as long as your rasterization implementation does not screwed up. But for the case like grass rendering, fur rendering, when you have many layers of thin line, FXAA will fail you. Just like the pixels annotated in the image below.

FXAA-issue

FXAA issues, it the rasterization failed to produce fragments, FXAA cannot help

Other option like Nvidia's AGAA(aggregated G-buffer anti-aliasing) Or if you are really a game developer, TAA maybe the way to go. But here we are talking about good old MSAA technique(also, refer to this article if you can have non-even sample depth buffer). Here we are talking about MSAA, with only OpenGL 4.0 capability, no special vendor requirement. The trick is simple and effective: "MSAA edge detection". If we can detect difference between simple pixel and complex pixel, we can treat them differently. It is the similar idea from this method. In the D3D API, you can use SV_Coverage, in GLSL, you have similar variable GL_SampleMaskIn, these are the variable which are available in the pixel shader, tell the GPU which sample to write after shading. Take 4xMSAA for example, if all the samples come from the same fragment, then the sample mask is 1111 for that pixel(4 bits). If the fragment only writes to 2 samples, the sample mask would be 1010, 1100 0011, 0101, etc. Which in turns mark this pixel complex.

complex pixels

Complex pixels here are marked on the edges, whose sample mask is not full

There is only one problem here. Through my experiement with opengl, if the pixel is not on the triangles edge, but on the intersection of two triangles, the sample mask does not reflect this problem. I guess that is why when you look the came at the ground in the video game, there is still aliasing effect at the intersection of grass and terrian. To solve this, you apply apply a normal test on the samples, if the pixel locates on the intersection, the normal test fails and thus mark the pixel complex.

intersect

This method works even at the intersection of triangles.

Voilà, here is your deffered MSAA, it runs really fast for the lighting pass, and it works all the time, unless(there is always an exception), you have co-planar z-fighting problem, then you would have probably bigger problem to worry about than anti-aliasing.