An LLM grader can return valid JSON without returning a usable judgment. Treating an omitted verdict as success turns incomplete evidence into a passing evaluation: the judge fails open. I authored the accepted correction in Promptfoo PR #10611.
The repair distinguishes an explicit verdict, a usable numeric score and a grader error. It also preserves the difference between an evaluation failure and a failure to obtain a judgment at all.
A parsed object is not yet a judgment
A response can satisfy a JSON parser while omitting the fields the evaluator needs. For example:
{"reason":"The answer was considered."}There is an explanation, but no verdict and no score. An evaluator still needs to decide whether that is sufficient evidence to grade the candidate. The accepted change treats this condition as a grader error.
A finite score carries different information:
{"score":0}With no explicit verdict and no configured threshold, that score means failure under the accepted behavior. A positive finite score can pass. If a threshold is configured, the score must meet it. A missing score or malformed numeric value cannot substitute for that evidence.
Three distinct states
The practical contract is:
- Explicit verdict: preserve the grader’s stated pass/fail decision, with the configured threshold still applying where relevant.
- No verdict, but a usable finite score: derive the decision from the score and threshold.
- No usable verdict or score: return a grader error.
The important implementation choice is to retain all three states until there is enough information to make the decision. Collapsing the last state into a default Boolean loses the distinction between successful evaluation and missing evidence.
A grader error must survive negation
Negated assertions add another edge case. If an evaluator represents every unsuccessful grading attempt as an ordinary negative verdict, a later negation can convert that failure into a pass.
The accepted fix routes an unusable response through the grader-error path. Negating the assertion does not turn that error into evidence that the candidate satisfied the opposite condition.
This is a general design lesson for model-assisted systems: apply a logical operation to a valid judgment only after establishing that a judgment exists. A parser or model failure should remain distinguishable through later processing.
Preserve intentional scoring behavior
Failing closed does not mean declaring every low score or unusual response malformed. The patch preserves intentional distinctions in the interface. An explicit positive verdict with a zero score can still pass when no threshold is configured. An explicit negative verdict remains a failure even if its numeric score is high. Numeric strings can be used when they represent a finite value.
These details are why regression coverage matters. The accepted PR includes code, tests and documentation describing how verdicts, scores, thresholds and negated assertions interact.
The research connection
I study AI evaluation integrity and agent security. A central concern is whether the software around an AI system can silently turn missing, untrusted or inconsistent evidence into a favorable result.
The SWE-bench exit-status case concerns forged-looking test evidence. My MLflow CVE disclosure concerns a network policy checked at the wrong moment. They involve different systems, but each rewards inspecting the actual decision path rather than assuming the surrounding safeguards enforce the intended rule.
Source record
- Promptfoo PR #10611: authored fix, review, tests and documentation.
- Merged implementation at commit 37cfe71.
- My accepted contribution record.
This note describes the accepted change and its scope. It does not assert that every Promptfoo version has the historical behavior.