SWE-bench: a PASSED line is not a passing test

A coding-agent evaluator needs evidence that the tests actually passed. If the candidate can influence the text being parsed as a test result, for example by printing its own forged PASSED lines, that text needs an independent check. This was the problem behind my accepted contribution to SWE-bench PR #620.

I reported the grading failure and contributed the initial fix under the AUTHENSOR handle. The maintainer refined and merged the implementation. The final change records the test command’s exit status and uses it to detect a disagreement with apparently successful test output.

The evidence an evaluator trusts

Test logs are useful because they identify which cases passed or failed. But they are still text produced during execution. When the program being evaluated can affect that output, a parser can receive pass-looking lines even though the real test command failed.

The question is whether the grader has evidence beyond the candidate-influenced text. In this case, the test command’s own exit status supplied a second signal.

An illustrative inconsistent result is:

Recorded test exit status: 1
Parsed test results: all pass
Grading decision: reject the result

This disagreement is the condition the accepted check addresses. Genuine failure output already follows the ordinary failure path.

Capture the status before cleanup changes it

The subtle implementation detail is where the exit status comes from. An evaluation script can run the tests and then execute a successful cleanup or repository-reset command. Reading only the wrapper script’s final status can therefore lose the earlier test failure.

The accepted implementation captures the status immediately after the test command, then emits a dedicated result marker after the end of the parsed test-output region. The grader reads that recorded value when deciding whether the log is consistent.

This is a useful pattern beyond one benchmark: identify the operation whose success matters, preserve its result before another operation overwrites it, and carry that evidence through to the final decision.

What the accepted check does

The final grading change rejects a nonempty, all-pass parsed result when the recorded test exit status is nonzero. Its behavior distinguishes several cases:

Tests succeed and the parsed output reports success: the new inconsistency check does not reject the result.
The command fails but the parsed output reports only passes: the result is rejected.
The output already contains failures or errors: ordinary failure handling remains relevant.
An older log has no exit-status marker: the change retains compatibility with that log format.

These distinctions matter when describing the result. This is a specific accepted repair to grading evidence, rather than a claim that every way of manipulating a benchmark has been eliminated. Test isolation, permissions and the integrity of result transport remain part of the broader design.

Why this belongs in AI security research

AI capability claims depend on the instruments used to measure them. When an evaluator trusts an artifact the candidate can influence, that becomes a security boundary in the experiment.

My research on the Evaluator Trust Boundary examines related failures in judges, parsers and scoring pipelines. The MLflow CVE case study concerns a different boundary, network validation at connection time, but follows the same method: inspect the real path, reproduce the mismatch and work toward an upstream repair.

Source record

This note explains the published implementation. The examples above are illustrative; no new benchmark-wide experiment is claimed here.