How do you unit-test a picture?

Post #7 ended with a test that compiles every shader and builds every pipeline on a real GPU, headlessly, in CI. That catches a whole class of failure: a shader that does not compile, a bind group that does not match its layout, a pipeline that cannot be created.

It proves nothing whatsoever about the picture.

You can re-plumb the scene cache, rewire the frame graph, move buffer ownership around, watch every unit test pass, watch every shader compile, and still be rendering something wrong in a way only eyes catch. That gap is what makes renderer refactoring brave rather than safe, and I needed it to be safe, because the renderer had become 3,362 lines of one class holding every concern.

The test

npm run golden renders a fixed scene headlessly through Deno's WebGPU, reads the framebuffer back, and compares it against a stored baseline.

golden:save   →  render, store the baseline
golden        →  render, compare

The frame is 320x180. Small on purpose: large enough that geometry, shadows and fog are all visible in it, small enough that the readback and hash are instant, because a test you wait for is a test you skip.

The scene is chosen to exercise exactly the paths that renderer surgery touches:

• static geometry, for the merge and batching path
• a moved dynamic object, for transform syncing
• a transparent object, for the sorted pass
• a textured object, for the texture store's bind-group caches
• vertex colours
• a directional light with shadows, for the cascade fit and the static shadow cache
• the procedural sky, for the background pass
• fog

Deliberately no particles and no skinning. Both need the runtime package, and this harness stays a client of the render package alone. A test that drags in half the engine to check the other half stops being a test of anything in particular.

The part that makes it usable: per-adapter baselines

Rasterisation is only bit-stable on one device with one driver. Two GPUs will legitimately produce different pixels for the same scene, and neither is wrong.

A naive pixel test therefore fails on every machine except the author's, gets muted within a week, and rots.

So the baseline is keyed by adapter description, and a hash recorded on a different adapter is reported as "no baseline" rather than as a failure. On a new machine the first run records rather than fails.

That sounds like a weakening. It is the opposite: it is what makes the test true. The question the golden frame answers is not "does this frame match some canonical picture", it is "did my change alter the frame on this machine". That is precisely the question you have during a refactor, and it is answerable exactly.

flowchart TD
  A["golden:save
before surgery"] --> B["Render 320x180
headless WebGPU"]
  B --> C["Pixel hash + 4x4 region stats
keyed by adapter"]
  D["Refactor"] --> E["golden
after surgery"]
  E --> F["Re-render, compare"]
  F -->|"hash equal"| G["Identical frame"]
  F -->|"hash differs"| H["Region report:
WHERE it diverged"]

A hash cannot tell you where

An exact hash is a yes or a no. When it says no, you are looking at two 320x180 images wondering what moved.

So the baseline also stores mean colour over a 4x4 grid of regions. When the hash differs, the region comparison names which cells changed. "The top-left region got darker" is a different investigation from "the bottom strip changed", and the difference is worth the twelve extra numbers.

The regions do a second job: they are a tolerant fallback for driver-update-sized noise, where a per-region tolerance catches a genuine change without failing on a rounding difference in a blend.

And a third, which I like most. Two sanity checks run on the regions themselves, every time:

• The regions must differ from each other. A uniform frame means nothing drew.
• The top of the frame must be sky-lit above a threshold.

Without those, a change that broke rendering entirely would produce a consistent black frame, hash it, and pass forever. That is the classic way a golden test dies: it starts asserting that the failure is stable.

What it was built for

Behind this net, the Renderer came apart into six modules, each owning exactly the state it is the sole writer of:

Module	Owns
editorKit	grid, pick pipelines, selection outline. Injected by the editor only
textureStore	upload, format, mips, bind-group caches
particleBuffers	per-emitter GPU state
instanceBuffers	the five per-object buffers, and the rule that any reallocation rebinds both variants
shadowMaps	the depth arrays and the static-cache bookkeeping
meshStore	residency, the in-place skinned overwrite, the merge epoch

3,362 lines became 2,802 beside those six. What stayed is the scene cache, culling, the cascade fit and the frame graph, which is one algorithm and is kept together deliberately. Slicing it further would trade cohesion for file count.

Every one of those extractions ran golden:save before and golden after. Two of them changed the frame, and I found out in seconds rather than in a week.

It also caught me over-cutting

Twice during that work I extracted a span slightly too large: once a cut swallowed 15 KB of the constructor, once a doc-comment backtrack swallowed a whole method that belonged elsewhere. Both were caught by assertions in the extraction tooling rather than by the golden frame, and both taught the same lesson: cut at a section header, never at a line offset.

The golden frame's contribution was different and more valuable. It removed the fear. The reason a 3,362-line class stays a 3,362-line class is not that nobody can see the seams; it is that touching it is unverifiable, so every change is a gamble against a scene you have to eyeball. Once the frame is a hash, the gamble is a test.

What I would take to another project

The pattern is not specific to rendering. It is: find the cheapest possible fixed point of the output, and record it per environment.

The instinct with visual output is that it is too subjective to test, so you skip it and rely on review. The escape is to stop trying to assert that the picture is right, and assert only that it is unchanged. That is a far weaker claim, it is mechanically checkable, and during a refactor it is exactly the claim you want.

Then add the two sanity checks, because a test whose failure mode is a stable blank is not a test.