Skip to content

[Feature] MSAA (Hardware Multisample Anti-Aliasing)#307

Open
keithwill wants to merge 3 commits into
ProwlEngine:mainfrom
keithwill:msaa
Open

[Feature] MSAA (Hardware Multisample Anti-Aliasing)#307
keithwill wants to merge 3 commits into
ProwlEngine:mainfrom
keithwill:msaa

Conversation

@keithwill

Copy link
Copy Markdown
Contributor

This PR adds MSAA as a per-camera rendering option, alongside the existing post-process AA effects (FXAA/SMAA/TAA). It gives Prowl its first anti-aliaser that works at rasterization time rather than reconstructing edges from an already-aliased image.

MSAA

FXAA, SMAA, and TAA all run at the end of the frame on a finished image: the geometry has already been rasterized with one sample per pixel, so the edge information is gone and has to be inferred. MSAA instead rasterizes coverage and depth at multiple samples per pixel and resolves them, so geometric edges are correct rather than reconstructed — no blurring of fine detail (FXAA), no ghosting or history buffers (TAA). The trade is memory and bandwidth instead of a fullscreen pass, and it only helps geometric aliasing, not shading or texture aliasing, so it composes with the post-process effects rather than replacing them.

Because it lives in the render targets, MSAA cannot be an ImageEffect — it has to be part of the pipeline itself.

How it was implemented

Camera.MSAA (None/X2/X4/X8, default None, clamped to Graphics.MaxSamples) makes the forward geometry passes rasterize into a multisampled color+depth target, which is resolved back to an ordinary single-sampled texture before anything samples it. Everything from PostProcess onward — effects, gizmos, the final blit — is untouched and behaves exactly as it does with MSAA off. Camera.Target stays single-sampled; MSAA is internal to the pipeline.

The resolve goes through glBlitFramebuffer and multisampled content is never sampled in a shader, so no GLSL changes and no sampler2DMS are needed. Being a public field, MSAA surfaces in the inspector through the reflective property grid with no editor code.

Notes on the less obvious parts

  • The prepass stays single-sampled. It feeds _CameraDepthTexture/Normals/Motion, which GTAO/SSR/TAA read as plain sampler2D.
  • The prepass → colorRT depth blit is skipped under MSAA. It only ever provided opaque early-Z: the skybox is ZTest Off/ZWrite Off and is overdrawn regardless, and opaques regenerate correct per-sample depth themselves. Its replacement depth clear is emitted outside the ClearFlags switch, because the Depth and Nothing branches never clear and colorRT is pooled — they would otherwise inherit stale depth.
  • The Depth/Nothing branches copy the target with a fullscreen quad rather than a framebuffer blit: RenderScale makes target and colorRT different sizes, and a scaling blit into a multisampled framebuffer is INVALID_OPERATION.
  • Blit.shader gains a Blend Off "Copy" pass. The existing pass is Blend Alpha, which would blend rather than replace when copying into a multisampled target, and HDR scene color carries no alpha = 1 guarantee.
  • GTAO/SSR read and write SceneColor, so under MSAA the target is resolved for them and copied back, leaving the multisampled depth buffer live so transparents still rasterize with real coverage. Only paid when those effects are present.
  • Multisampled textures reject all sampler state (INVALID_ENUM), so the Texture base constructor skips its default wrap/filter setup for them.
  • RenderTexture's pool key carries SampleCount in both the key struct and the rebuild in ReleaseTemporaryRT; missing either would silently alias multisampled and single-sampled targets.

Included fix: grab-pass RT format

Also included is a small independent fix that MSAA would have turned into a hard error. The grab-pass temporary RT was always allocated as Color4b even when the scene color it copies from is HDR (Short4). glBlitFramebuffer permits format conversion from a single-sampled read framebuffer, so this silently truncated HDR grab/refraction passes to 8 bits per channel instead of erroring. The grab RT now takes the source RT's own format.

Usage

camera.MSAA = MSAASamples.X4;

Testing

Added MSAATests — headless tests covering the enum's literal sample counts, the camera default, multisampled RenderTexture/Texture2D construction and its invalid-argument cases, pool behavior (no aliasing across sample counts, reuse within one), the Blend Off copy pass in Blit.shader, and the guards on blitting from a multisampled source and resolving mismatched sizes.

Verified on-device across a matrix of 16 configurations (sample counts, HDR, all four ClearFlags, RenderScale 0.5/2.0, GTAO/SSR, post-process stacks, and MSAA toggling for pool reuse) under a GL debug context: no GL errors. The check was falsified by temporarily removing the multisample sampler-state guard, which correctly reported GL_INVALID_ENUM on GL_TEXTURE_2D_MULTISAMPLE.

The grab-pass temporary RT was always allocated as Color4b, even when the
scene color it copies from is HDR (Short4). glBlitFramebuffer permits format
conversion when the read framebuffer is single-sampled, so this silently
truncated HDR grab/refraction passes to 8 bits per channel rather than
erroring.

Allocate the grab RT with the source RT's own format. Dimensions already
match on both sides, so the blit stays legal.
Prowl's only anti-aliasing was post-process (FXAA/SMAA/TAA), which reconstructs
edges from an already-aliased image. MSAA antialiases at rasterization time
instead, so it cannot be an ImageEffect -- it has to live in the render targets.

Adds Camera.MSAA (None/X2/X4/X8, default None) which makes the forward geometry
passes rasterize into a multisampled color+depth target, resolved back to an
ordinary single-sampled texture before anything samples it. Everything from
PostProcess onward -- effects, gizmos, the final blit -- is untouched and
behaves exactly as it does with MSAA off. Being a public field, it surfaces in
the inspector via the reflective property grid with no editor code.

Because the resolve goes through glBlitFramebuffer and multisampled content is
never sampled in a shader, no GLSL changes and no sampler2DMS are needed.

Notes on the less obvious parts:

- The prepass stays single-sampled. It feeds _CameraDepthTexture/Normals/Motion,
  which GTAO/SSR/TAA read as plain sampler2D.

- The prepass -> colorRT depth blit is skipped under MSAA. It only ever provided
  opaque early-Z: the skybox is ZTest Off/ZWrite Off and is overdrawn regardless,
  and opaques regenerate correct per-sample depth themselves (RasterizerState
  defaults to ZWrite On/LEqual). The replacement depth clear is emitted outside
  the ClearFlags switch because the Depth and Nothing branches never clear, and
  colorRT is pooled, so they would otherwise inherit stale depth.

- The Depth/Nothing branches copy the target with a fullscreen quad rather than
  a framebuffer blit: RenderScale makes target and colorRT different sizes, and
  a scaling blit into a multisampled framebuffer is INVALID_OPERATION.

- Blit.shader gains a Blend Off "Copy" pass. The existing pass is Blend Alpha,
  which would blend rather than replace when copying into a multisampled target,
  and HDR scene color carries no alpha=1 guarantee.

- GTAO/SSR read and write SceneColor, so under MSAA the target is resolved for
  them and copied back, leaving the multisampled depth buffer live so transparents
  still rasterize with real coverage. Only paid when those effects are present.

- Multisampled textures reject all sampler state (INVALID_ENUM), so the Texture
  base constructor skips its default wrap/filter setup for them.

- RenderTexture's pool key carries SampleCount in both the key struct and the
  rebuild in ReleaseTemporaryRT; missing either would silently alias multisampled
  and single-sampled targets.

Verified on-device across a matrix of 16 configurations (sample counts, HDR, all
four ClearFlags, RenderScale 0.5/2.0, GTAO/SSR, post-process stacks, and MSAA
toggling for pool reuse) with a GL debug context: no GL errors. The check was
falsified by temporarily removing the multisample sampler-state guard, which
correctly reported GL_INVALID_ENUM on GL_TEXTURE_2D_MULTISAMPLE.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant