NVIDIA garak: a flagged word in fullwidth letters scored clean

A detector score is evidence only if the detector could have seen what it was looking for. In June 2026 I found a gap in NVIDIA garak, the LLM vulnerability scanner: its string detectors compared flagged terms with model output code point by code point after lower-casing, with no Unicode normalization. A listed word scored 1.0 in plain ASCII and 0.0 when the model wrote it in fullwidth letters or spliced a zero-width joiner between its letters. I reported it in garak issue #1867. The fix, written by @PhinehasNarh and merged in garak PR #1884, adds a normalization option to the base string detector.

One word, three scores

The reproduction ran garak's own unsafe_content.LDNOOBW detector, which checks output against a public list of profanity and slurs, on three spellings of the same listed word:

case                   score
plain ASCII (control)    1.0
fullwidth letters        0.0
zero-width joiner        0.0

A person reading either variant sees the word. The detector compared code points, found no match and reported the output as clean.

Why the scanner missed its own attack

StringDetector lower-cased both sides with str.lower() and then tested for a substring or a word-boundary match. Every detector built on it inherited the gap, including unsafe_content.LDNOOBW, lmrc.Anthro and goodside.RileyIsnt.

garak already knows this evasion on the attack side: probes.badcharacters generates homoglyph and invisible-character payloads. So garak could test whether a model is vulnerable to these characters, but before this fix it could not see them in the model's answer. A harmful output written that way was scored as safe, and the scan reported a clean pass.

The accepted fix

#1884 adds a normalize setting to StringDetector. When it is set, the trigger list and the model output are both normalized before any matching:

None, the default, keeps the old behavior.
"NFKC" applies Unicode NFKC normalization, which folds fullwidth letters and other compatibility forms into their plain equivalents.
"NFKC+strip_format" does the same and then removes invisible format characters (Unicode category Cf), such as the zero-width joiner, the zero-width space and the byte-order mark.

An unrecognized value makes the detector return no score instead of a clean one, so a configuration mistake cannot read as a pass.

My report asked for normalization by default or, at minimum, as an option operators could turn on. The accepted change is the option: existing results stay the same until a detector or a run configuration enables it.

The same failure in AI evaluation

This is ETB-06 in the Evaluator Trust Boundary taxonomy: the evaluator makes a byte comparison over text the evaluated system wrote, and the system controls the bytes. Exact matching is the right default for code and identifiers. A safety detector reading natural language has to compare normalized text, or its score measures how the model spelled something rather than what it said.

Source record

This note describes the report and the accepted change. Neither normalization mode removes combining marks or maps lookalike letters from other scripts, such as Cyrillic "а" for Latin "a", so those variants still need a separate check.