Software testing is a mature discipline with settled methods. Probing a language model for harmful behaviour borrows the vocabulary and almost none of the underlying structure, and the difference explains why coverage claims should be read sceptically.
There is no specification to test against
A conventional test asserts that a function returns a defined value for a defined input.
A model has no such contract. Its intended behaviour is described in policy documents written in natural language, using terms like harmful, misleading and inappropriate.
Those terms require judgement, and judgement varies between reviewers, which means the same output can pass one assessment and fail another.
Testing therefore begins with an argument about what counts as a failure, and that argument is never fully resolved.
The input space cannot be enumerated
Software has a finite interface: defined parameters with defined types, and equivalence classes that make exhaustive reasoning possible.
A model accepts any text. The number of ways to phrase a request is unbounded, and a refusal triggered by one phrasing may not be triggered by a near-identical one.
Testers therefore sample rather than cover, and the sample reflects what the testers thought to try.
A finding of no failures means nobody found one, which is a much weaker statement than a passing test suite.
This is why published safety results are usually framed as reductions in observed rates rather than as guarantees.
Failures are contextual rather than local
A model may answer a question safely in isolation and unsafely after twenty turns of conversation that gradually shifted the framing.
The same request embedded in a fictional scenario, a translation task, or a request to critique a draft can produce entirely different behaviour.
Each of these is a separate axis, and they combine, so the practical space is far larger than the list of prompts anyone writes.
Reproducing a reported failure often fails, because the surrounding conversation was not captured or the sampling landed differently.
The system is stochastic
Running the same input twice can produce different outputs, since generation involves sampling from a distribution.
A failure observed once may appear in a small fraction of attempts, which makes it easy to dismiss and impossible to ignore at scale.
A behaviour occurring rarely still occurs constantly across millions of daily requests, and rarity is not safety.
Testing therefore requires repeated trials per case, multiplying an already infeasible search by another factor.
Fixes are not surgical
Repairing software means changing a line that caused a defect, with effects bounded by the call graph.
Changing model behaviour means additional training or altered instructions, and both shift behaviour across unrelated tasks.
Suppressing one category of response commonly makes the model refuse legitimate requests that resemble it, which users experience as the system becoming less useful.
Every fix therefore requires re-running the whole evaluation set, and the outcome is a trade rather than a repair.
This is why the same complaints alternate over successive releases: one version is criticised for permissiveness, the next for excessive caution.
Adversaries adapt after release
Internal probing is conducted by people who are paid to find problems and who share assumptions with the builders.
Public users number in the millions, coordinate openly, and iterate faster than any internal team.
Techniques that defeat a filter circulate within days of a release, and the defence has to be rebuilt against a moving target.
Treating a pre-release exercise as the security review is the same error as testing a website once before launch and never again.
Automated probing has its own blind spot
Using one model to generate attacks against another scales the search enormously, and it is now standard practice.
The attacking model produces variations of what it has seen, so it explores densely near known techniques and sparsely elsewhere.
Genuinely novel approaches tend to come from humans with unusual domain knowledge, which is precisely what automation does not supply.
The combination is stronger than either, and the automated portion should not be mistaken for coverage.
Disclosure norms have not settled
Security research has decades of convention around reporting a flaw privately and publishing after a fix.
Model failures do not fit that pattern, because there is often no fix, only a mitigation that reduces frequency.
Publishing a technique that cannot be closed spreads a capability rather than pressuring a repair, and withholding it leaves users unaware of a limitation affecting them.
The field is arguing this out in public, and the absence of a settled norm means the same disclosure is praised as accountability by one group and criticised as recklessness by another.