testing

How Abbreviatory Was Tested Against Its Four Promises

A tool that leaves things out has to be careful about what. So the testing for v0.1.0 is built around four promises, checked over and over: the report never goes over the budget, nothing important gets lost, the same input gives the same bytes, and every reference expands back to exactly the original.

Every number in this post comes from the result files in the code repository, in test/results. They were recorded on October 18, 2026, at commit d0d473f, on a Linux machine with an Intel Xeon processor at 2.10GHz and 2 CPUs, with Go 1.24.7. One script, scripts/record-tests.sh, runs everything and writes the files, so anyone can run it again.

The four promises: within budget, never a byte over or a clear refusal. Nothing important lost, each failure, error and warning shown or counted and referenced. Deterministic, same input and options give the same bytes. Reversible, every reference expands back to the original byte for byte.

Real Output to Test On

The fixtures are real. Two small sample projects in the repository fail on purpose: two Go packages with 1,212 tests between them (subtests included), seven of which fail, one by panicking, and a pytest file with 910 tests, five failures and an error in a fixture. Their output was recorded as go test plain, verbose and -json, pytest plain and verbose, and pytest’s JUnit report. A generated application log of 63,369 lines and a generated API response of 2,500 items round it out. The log has repeated warnings, Java stack traces, Python tracebacks and Go panics mixed into a lot of noise.

The Promise Checks

Every fixture is reduced at seven budgets from 256 bytes to 16 kB, with four preserve settings and with and without keep rules: 448 runs. Each one is checked on all four promises:

  • Budget. The report is never longer than the budget. When the budget is too small to account for everything, the run must refuse and say how many bytes it needs.
  • Nothing lost. Every failure, error, warning and keep-rule match is either shown with its own reference, or counted on a “not shown” line whose reference covers it.
  • Same bytes. A second run gives the same report and the same reference ID.
  • Reversible. Every reference in the report expands, the whole-input reference gives the fixture back byte for byte, and for line formats every input line belongs to exactly one group.

Seeded random logs get the same checks, at four budgets each. All of it passes. Then the whole suite runs again under Go’s race detector, which finds nothing.

What the Reports Look Like

At the default budget of 4,096 bytes, the 5.6 MB log came down to 3,075 bytes, 0.05% of its size. The go test -json run of 808 kB came down to 2,383 bytes, with all seven failures and their messages. The API response of 719 kB became a 3,185-byte outline that is still valid JSON, with the three items that carry an error shown in full.

Small inputs are left alone. The plain go test output is only 1,735 bytes, so reducing it wouldn’t save anything, and it’s shown whole with a one-line header. Every before-and-after report is in test/results/examples.

Memory

Logs and test output stream through. A generated log of 1,107 MB, 12.5 million lines, went through in 32.7 seconds with a peak heap of 15.1 MB. Memory doesn’t grow with the input. JSON documents are the exception: they’re read whole, so they’re capped at 16 MB, and anything bigger is read as a log.

The Review

Before the release, an independent review went looking for inputs that break the promises. It found 17 bugs. A Maven log, whose lines start with [, was mistaken for JSON and refused. Reducing the same input with different keep rules could make an old reference point at the wrong lines. A two-line pytest summary claiming two billion passed tests hung the tool. A crashed pytest run looked clean. Every one is fixed, and every one has its own test, so it stays fixed.

The Smoke Check on Release Binaries

Tests run against source. People run binaries. So the release binary is built the way the release workflow builds it, and a smoke check runs on it: the version is right, every fixture reduces within budget at two sizes, is detected as its format and gives the same bytes twice, every reference in the output expands, the whole input comes back exactly, and bad input fails cleanly. The release workflow runs the same check on Linux, macOS and Windows binaries before anything is published.

What the Tests Don’t Show

They show that what v0.1.0 claims, it does, on these inputs. They don’t make it read formats it doesn’t know, or judge free prose. The reference lists every limit. Run it on your own CI output and you’ll have numbers of your own.