release

Abbreviatory 0.2 Tells You What Changed Since the Last Test Run

The first release made one run’s output short. This one compares it with the run before. Abbreviatory v0.2.0 keeps a history per command, and every report now opens with what changed: which failures are new, which got fixed, which are still failing and for how long, and which tests are flaky.

Most CI runs aren’t news. The same three tests failed yesterday and they fail today. Reading the full report again, or having a coding agent read it again, mostly confirms what everyone already knows. What matters is the difference. One new failure among five old ones is the line you want, and in a plain report it looks like the other five.

Six tests over the last ten runs, with passed and failed runs marked. One test failed for the first time in run 13 and is reported as new, one passed after nine failed runs and is reported as fixed, one has failed four runs in a row, and one changed between passing and failing seven times and is reported as flaky. Next to it, the report for run 13, and below it the one-line report when nothing changed.

What a Report Looks Like Now

Here’s a real go test run of 1,213 tests, compared with the run before it:

[abbreviatory] go test -json output, 4,882 lines (808.5 kB), ref 3e35712ad529
run 2; since run 1: 1 new, 1 fixed, 6 still failing (longest for 2 runs)
new: TestFieldSpaces (example.com/gosample/parse, 0.00s) [ref 3e35712ad529:g1205]
fixed: TestLegacyFormat (example.com/gosample/parse, 0.00s) [ref 6f92e70f603c:g1213]
tests: 1,205 passed, 7 failed, 1 skipped
FAILED (7):
- TestFieldSpaces (example.com/gosample/parse, 0.00s) (new) [ref 3e35712ad529:g1205]
    parse_test.go:33: Field("width = 12"): unexpected error: strconv.Atoi: parsing " 12": invalid syntax
- TestPickPast (example.com/gosample/crash, 0.00s) (failing for 2 runs) [ref 3e35712ad529:g1]
...

The new failure sits at the top of the list, marked. The fixed one points into the run where it last failed, so its old output is still one abbreviatory expand away. Run the same thing again and the whole report is one line:

[abbreviatory] go test -json output, 4,882 lines (808.5 kB), ref 3e35712ad529; run 3, identical to run 2: the same 7 failures

That one line is the whole point for a coding agent in a loop. It reruns the tests after a change and learns in one line that it fixed nothing and broke nothing.

Flaky Tests Don’t Count as News

A flaky test fails one run and passes the next. Compare only two runs and it shows up as new, then fixed, then new again, every other day, and people learn to ignore the list. So Abbreviatory looks at the last 10 runs. A test that changed between passing and failing 3 or more times in them is listed as flaky, not as new or fixed. It still shows in the failures when it fails, marked (flaky), after the new ones.

Tests that didn’t run are kept apart too. When the output names the tests that passed, as go test -v, go test -json and JUnit reports do, a failing test that was skipped or left out this time isn’t called fixed. It’s listed as not run. Plain go test and pytest only name the failures, so there a failing test that disappears counts as fixed.

Logs Work the Same Way

For a log, the history tracks error kinds instead of tests. Two errors are the same kind when they match with numbers, IDs, addresses and timestamps aside, the same rule that folds repeats within one log. So a nightly job’s report says which errors are new tonight, which are gone, and which have been there for 30 nights. A JSON response works too: its error fields are compared by path and value.

Run It in Place of the Command

abbreviatory run -- go test -json ./... runs the tests, reduces what they print, compares the run with the last run of the same command in the same project, and exits with the test command’s own status. A CI step that used to run the tests can run this and still fail when the tests fail. If the job times out or gets cancelled, the signal reaches the tests and the report still comes out. A run killed with the same output as last time isn’t folded into “no change” either. The changed exit status is news, so it’s shown. abbreviatory history lists the runs, with the counts and what changed in each.

CI machines start empty, so the history needs the store kept between runs, with the cache saved even when the tests fail. The quick start has a GitHub Actions example.

How It Was Tested

The plan set the bar for this release before any code was written: 500 simulated CI runs with known changes, and the reported changes exact on every one. The test does that three times: 500 runs of a 40-test suite printed by go test -v, 500 of the same suite printed by plain go test, and 500 runs of a log with 25 error kinds. Tests break, get fixed, flake, get skipped and go missing at random. The same test names turn up in two packages, exit statuses change, and some runs change nothing at all. Each run goes the whole way, its history written to disk and read back for the next run, at a random byte budget.

A separate piece of the test works out from the known statuses what each report should say. For every one of the 1,500 runs, the new, fixed, not run, still failing and flaky lists matched, the run streaks matched, and the one-line report came exactly when nothing changed. To make sure the check can fail, the flaky threshold and the streak counting were broken on purpose, one at a time. Both times the test caught it within a few runs.

Then an independent review went over the new code, as one did before v0.1.0. It found eight problems. The worst: with plain go test or pytest a fixed test was reported as not run, and plain go test mixed up tests of the same name in different packages, so a new failure could hide behind “no change”. The simulation only used go test -v and one package, so it couldn’t see either. Both are fixed, and the simulation now covers both cases. The others were a change in exit status counted as no change, run not passing a CI cancel on to the tests, two runs of one command at the same moment mixing up their run numbers, and smaller ones. A second pass over the fixes found five more, three of them caused by the fixes. One would have called every failing test fixed after a broken build. A third pass found one more corner of the same thing, a package that failed to build. Those are fixed too, each with a test.

The four promises of v0.1.0 still hold with a history: the report never goes over the budget, the same input and history give the same bytes, and every reference expands, the ones into the previous run included. In all, 74 Go tests pass, with the race detector too. The results are in test/results in the code repository.

Get It

Download v0.2.0 for Linux, macOS or Windows. It’s still one static binary, open source under Apache 2.0, with no language model and no network. Next comes an HTTP service and an MCP server, so CI systems and coding agents can call it directly.