heibench

Open adjudications

Real conflicts between shipped members, recorded rather than resolved.

Resolving one of these changes released behaviour, and that is a decision with an owner. They are written down so a consumer knows what it is reading, not so someone quietly harmonises them.

Do not "fix" a member to match this page without an issue and a decision entry. The vocabulary follows the tools; the tools do not silently follow the vocabulary. Decisions live in each repo's docs/DECISIONS.md, numbered, with the reasoning that produced them — do not relitigate a numbered decision; if it is wrong, add a superseding entry.

Shared vocabulary

Statuses, per check

Common core, in every member that adjudicates: pass · fail · unsupported · skipped. Only pass is green.

approximate is a domain-gated extension, for when a measured error interval straddles the threshold. partspec has it; netspec deliberately does not, because connectivity is discrete and importing interval epistemics into an exact domain adds concepts without adding truth (D9, guarded by test_report_carries_no_tolerance). Absence is a decision to record, not an omission to fix.

Exit codes

Agreed across partspec and netspec today:

0    satisfied
1    violated
4    environment fault / error — not a verdict on the design

The adjudications

A1 — exit code 2 means two different things resolved

verified against source 2026-09-05 · netspec#20 · settled in netspec 0.9.0, 2026-09-06 (D26)

Resolved: netspec adopted partspec's split — 2 is could not tell, usage moved to 64 (EX_USAGE). The two tools now agree on 0, 1, 2, 4 and 64. netspec does not adopt partspec's EMPTY = 3: a contract asserting nothing is a defect in the contract, and its synthetic result already says so.

partspec maps Verdict.INCOMPLETE to exit 2 (src/partspec/status.py:107) and puts usage at 64 (EX_USAGE). netspec uses EXIT_USAGE = 2 (src/kicad_netspec/cli.py:32), with no code for "could not evaluate". A consumer branching on 2 across both tools is reading two different facts.

Recommendation: adopt partspec's split.

A2 — netspec cannot report "could not tell" at the verdict level resolved

read from source 2026-09-05, not reproduced · netspec#19 · settled in netspec 0.9.0, 2026-09-06 (D26)

Resolved: Verdict is now pass · fail · incomplete · error. A skipped or unsupported rule yields incomplete at exit 2, and fail outranks it so one unsupported rule cannot mask a real finding. Decided with A1, because exit 2 had to stop meaning usage first. Half of D10 stays open: error is in the vocabulary and the exit map, but nothing returns it — an environment fault raises before a netlist is read, so there is no report to carry the field D10 asks for.

Verdict = Literal["pass", "fail"] (src/kicad_netspec/check.py:39), green only when every rule is green — so an unsupported rule collapses into fail, saying the board is wrong when the truth is the tool could not check it. D10 also documents a verdict: "error" the type does not contain; the environment fault rides on the exit code, not on a report field as D10 requires.

Also case 2 in the record of the core defect.

A3 — gerberdiff has no third state resolved

reproduced 2026-09-05, v0.29.1 · gerberdiff#17

has_changes: boolean (docs/schema.md), layer status matched | added | removed. For a differ, too, silence must never read as no difference.

Written up in full as case 1, which is now resolved — gerberdiff#17 was fixed in d71f6ce (2026-09-06), so an undefined aperture is an Error and exits 2 rather than dropping the flash in silence.

Resolved 2026-09-06 in 0685cde. The differ now has its third state: a comparison reports identical | different | indeterminate, and every operation the geometry engine could not model is counted in summary.unrepresented and named on stderr. different outranks indeterminate, so one unmodellable stroke cannot mask a trace that moved. gerberdiff's exit codes now match partspec and netspec — 0 · 1 · 2 · 4 · 64 — which was the other half of A1 and left gerberdiff the last member out of step.

The instance that motivated it is case 10: a trace stroked with a macro aperture read as no change at exit 0 while the raster engine reported it. Note the shape — has_changes was a boolean, so it was false for two identical boards and for a board whose geometry could not be modelled. Two verdicts could not carry three answers, exactly as in A2.

A4 — orlab's version fallback was log-only resolved

reproduced 2026-09-05 · fixed in 6ebbaad, orlab#61 closed

Resolved: profile_exact is now a public attribute on both OpenRocketInstance and SimulationPool, so a caller can branch on whether it is running against an exact profile or a nearest-older fallback. Retained as case 3 in the record, where it is the first instance found in the drive layer rather than the verify layer.

The rule that produced them

Every adjudication here is a disagreement about how to say could not tell. That is not a coincidence — it is the one rule everything else follows from, showing up at the seams between tools that each solved it slightly differently.