Give a response a fresh request ID and a timestamp, and a second run can produce this diff:
{
"title": "Welcome",
- "requestId": "request-123",
- "createdAt": "2024-09-12T10:00:00Z"
+ "requestId": "request-456",
+ "createdAt": "2024-09-12T10:00:01Z"
}
The title hasn't changed. But those two request fields are enough to fail a byte-for-byte comparison. I'd rather it told me whether I'd broken the page.
I've developed a huge fondness for golden-file tests. I can dump the whole result and compare it with the previous version rather than choosing a few fields to check. An object, a page, generated code. All the fields are there, including the ones I wasn't thinking about when I wrote the change. That matters to me when refactoring. I can move the internals around while checking the output against its saved version, and the snapshots live in Git so I get a history of behaviour I can actually read. Quite a lot of checking for quite little test code.
This appeals to my laziness.
Some parts of the output are supposed to change. Approving another timestamp tells me very little. Do it often enough and I'm going to stop paying attention to the diff, which defeats the point.
Hang on, the first saved result can be wrong too. A golden file records what the program did, including its mistakes. Once I've reviewed it, I want later comparisons to show changes worth investigating. So which differences should I keep?
Give the output an order
Map iteration is a familiar source of changing output in Go. Suppose a page renders a map of headers into these rows:
-<dt>X-Request-ID</dt><dd>abc-123</dd>
<dt>Content-Type</dt><dd>text/html</dd>
+<dt>X-Request-ID</dt><dd>abc-123</dd>
Both headers are still there. The map hasn't changed:
headers := map[string]string{
"Content-Type": "text/html",
"X-Request-ID": "abc-123",
}
A generated loop ranging over that map could write either header first. Go's range rules leave map iteration order unspecified, so Go isn't promising to repeat this order next time or assigning a probability to either ordering of these headers.
If I save one traversal as the expected page, I've approved an accident. Accepting the next diff swaps one accident for another.
Somebody reading the page should get a stable order too. That decision belongs in the generator. For string keys, it can produce this loop. Here it prints the keys and values so we can see the order:
keys := make([]string, 0, len(headers))
for key := range headers {
keys = append(keys, key)
}
sort.Strings(keys)
for _, key := range keys {
value := headers[key]
// Emit the key and value in this order.
fmt.Println(key, value)
}
Collect the keys, sort them, then read the values in that order. The map keeps doing the lookup job it suits. The generated rows now have alphabetical order, and the golden file can record that choice.
I could sort the rows only in the test. But then the page could keep changing order while the comparison passed. Here the actual output needs to settle down, so the sorting belongs in the code that produces it.
Keep the changing fields in the test
Request IDs are a different case. I want fresh IDs in production. Making every request use the same ID would be an impressive amount of damage for a green test, and I'd still have to work out how to tell one request from another.
Another way to quieten the diff is to strip the changing fields before comparing the saved and current results:
delete(result, "requestId")
delete(result, "createdAt")
The diff disappears.
It would also disappear if the application stopped returning a request ID altogether, because I've removed it from both sides.
So much for checking the whole object.
I'd keep the fields and control where their values come from. The code can accept a function that supplies the ID so the application can keep its usual generator while the golden test returns a known value without deleting the tested field. If the application drops it or puts it in the wrong field, the snapshot changes.
Pass in a clock function and let this test return a fixed instant, and I can compare the formatted timestamp rather than deleting it before comparing. Tests concerned with expiry or elapsed time can supply different instants to exercise those rules.
I'd prefer to spend the effort here. Give the test control over its inputs and I get to keep more of the output instead of adding another exception to the comparison every time a field appears.
HMACs can carry those changing inputs further. An HMAC authenticates a message using a secret key, and with the hash function fixed the same key and message produce the same code. Put a timestamp in the message and a new timestamp can change the HMAC too. The HMAC definition makes that dependency explicit.
A known test key and fixed message let me compare the HMAC as well. A changing message? If I let the message vary I need to verify its HMAC against the actual message it came with before removing either from the comparison, because the relationship between message and authentication code is what the placeholder would hide. Broken authentication could otherwise pass.
Absolute paths are another nuisance. A result might contain /home/me/site/templates/page.pk on my machine and /builds/site/templates/page.pk in CI. Which template? That matters to me. Where I happened to check out the project is less interesting.
Replacing the known project-root prefix with <PROJECT> in a copy of the output being compared makes both paths <PROJECT>/templates/page.pk, while the application still receives and returns its real paths.
The filename stays. So does the directory beneath the project root. And if a refactor starts selecting templates/error.pk, that's a change worth seeing. Replacing the entire path with <PATH> would throw away the part I cared about.
I'd keep the replacement tied to that known root and its directory boundary. A path outside it should remain visible. Otherwise, a test could make selecting a completely different directory look like a change of machine.
The same care applies if I can't supply a clock or ID and need a placeholder instead. Check that the field exists and has the expected format before replacing its value. Keep checks for relationships that matter, such as the same request ID reaching two parts of a response. A missing field mustn't turn into the expected placeholder.
With those choices made, I can pull the internals apart and compare the result again. The rows have a defined order, the test supplies its changing inputs, and the path comparison keeps the filename. If a field disappears now, I have something worth reading. I'll review that change before updating the golden rather than accepting it along with another second passing on the clock.