I'm writing more unit tests than I used to. AI makes the repetitive setup easier and helps me change more of the code, so regression tests check those changes against the behaviour I've agreed to preserve. I used to put most of my testing effort into integration and regression tests. I still love those. Give me a known input, run it through the system, then show me what changed. That's useful when I'm about to pull much of the implementation apart.
And I do like pulling it apart, show me an awkward package boundary, I immediately want to move it, even if it isn't stopping the program doing its job. I've never been particularly shy about a major refactor if it makes the code easier to maintain and gives the project room to survive. Pipit is coming out of Piko, partly so I can understand and change its parts without carrying the whole framework around them every time I want to work on the engine.
“While I'm in here....”
A very tempting argument for taking on more work.
The testing needs to help me do that work. It can be a headache.
I want the behaviour to survive while I change the code. If I change code Y, does code X still work? Can the page still render? Does the result still contain the same fields?
Unit tests tied closely to internal helpers can get in the way of that. I move the work into another package, change the helper's arguments or replace the representation, and suddenly there's a pile of tests to rewrite. The behaviour I care about hasn't changed, but the tests have plenty to say about the furniture moving. A small test of a clear rule can be very useful. But when the tests follow my implementation too closely, I can be rewriting them alongside the code I wanted to change. A second refactor.
Does the test share my assumption? I can make the same assumption while writing the code and writing its test. Checking a small function against the answer I already had in mind can leave that assumption untouched, while a larger interaction sometimes exposes something I wasn't thinking about at all.
My fondness for golden-file tests comes from that freedom to refactor. I still have to decide whether the behaviour I'm preserving is right.
So the laziness has a fairly elaborate explanation. I did say I liked refactoring.
AI helped me get past the repetitive setup in Testing my patience. I'm now enjoying having more unit tests and higher coverage. What haven't I exercised?
An untouched branch gives me somewhere else to look. What input reaches it and what would I expect to happen? Now the coverage report can send me back into the code with a question. That's a different feeling from having a percentage sitting over the project as a verdict.
Integration and regression tests stay. Small cases can help explain a local rule with the larger tests checking the system while I change its structure. More unit tests are useful to me when they make that work easier to take on.
If I ask the AI to write both the code and its tests, their agreement can come from the same mistaken assumption. With a saved regression result, I can look at the output for a known input, understand it and decide whether I agree it's right.
After I approve an outcome, regression tests help the AI preserve it while editing. I'm putting more work into making those comparisons repeatable, so a refactor can be checked against the same inputs and expected results. A mismatch gives the AI something concrete to investigate and correct, instead of leaving me to discover where it has gone haywire.
A finished diff can arrive before understanding does
I'm using AI for broader changes during the work towards standalone Pipit, and the amount of code stops being a particularly useful measure of progress.
When I write a change myself, some of the understanding arrives while I form the idea and follow it through the code, whereas receiving the finished change means recovering that context afterwards from the supplied diff. I can read every changed file and still struggle to explain the whole patch.
Take a register-lifetime fix, changing when the interpreter can stop keeping a value in one of its storage slots. It might change when a value is copied, which handler reads it and whether a native callback can still reach it. When is the slot reused? What still reaches the old value? Following that one value across the files gives me a way through the change, because reading the diffs separately lets me keep nodding without quite joining them up.
Producing more code doesn't increase the amount I can hold in my head.
For a long time shrinking a failing program was where I wanted AI's help most. Give me less code to get through before I can debug it. The tools couldn't inspect a stopped program the way I could. Where is the value now? Another speculative fix got irritating when what I wanted was to look at the state.
The smaller program, please. Another speculative fix can wait. Once it becomes a regression test I can rerun it after the patch, another optimisation or forgetting why we were discussing it, and the rule beside the reproducer helps me recover the decision. The test outlives the conversation.
What am I accepting?
For Pipit's golden corpus I check the generated .pkasm files into the repository. These are readable listings of the bytecode, so I can see when the compiler starts producing different instructions. A test program can still return the expected answer while its generated code changes underneath. Was that intended? I'm checking the compiler's output here rather than how its own helper functions are arranged. An optimisation may intentionally change those instructions, and Pipit's contribution guide asks for a prediction before updating goldens, so a snapshot moving when I expected unchanged instructions leaves me a disagreement to resolve. Regenerate it? Not until I've worked out whether the prediction or the compiler was wrong.
Making the comparison count
These comparisons also need to fail when the outputs disagree. Pipit's compiler is now running under Pipit, compiling the same programs as the native compiler. This is the 24 September report:
Programs compared: 1574
Identical: 1574
Mismatched: 0
I'm pleased with that. I started by wanting to run the Go in a page, and now the engine can run its own compiler.
How much is interpreted? The top-level compiler package runs under the interpreter while supporting packages and Go's parser and type checker are still native, and this compares their bytecode without executing every resulting program.
The harness can count mismatches and still leave go test green. Its wrapper logs a mismatch status without failing the test. If I deliberately corrupt three outputs, that arrangement can give me this:
3 compared, 0 identical, 3 mismatched
--- PASS: TestSelfhost
The referee sees the foul. The whistle is decorative.
Disagreeing outputs have to make the wrapper fail, otherwise a green result can let me accept a patch that never required agreement.
Someone using Pipit may suspect their own code before suspecting mine. They spend time reducing a program and learning enough of the engine to explain what failed, and I don't want my patch to save me time by handing that work to them.