Your AI Safety Score Is Lying to You

A model that refuses every request would be perfectly safe. It would also be perfectly useless. This sounds obvious. Yet much of today’s AI safety benchmarking quietly rewards exactly this kind of behavior. Models collect points for rejecting harmful requests, while the cost of rejecting harmless ones is often measured separately—or disappears inside an impressive-looking average.

A new study applying methods from psychology to AI evaluation exposes just how misleading that can be. The uncomfortable conclusion is simple: AI safety is not one number. Treating it as one creates false confidence, bad purchasing decisions, and incentives to build models that look safer in a laboratory than they behave in the real world.

We’ve Been Grading AI Like a Bad School Exam

Most AI benchmarks work like conventional tests. Give a model thousands of questions, score its answers, calculate an average, and place the result on a leaderboard. That approach assumes two things:

1. All questions contribute useful information.
2. The final score represents a coherent underlying ability.

Both assumptions appear to be wrong. In Item Response Theory for AI Safety, researchers applied a century-old psychometric technique to eight safety benchmarks containing more than 5,000 test items and responses from up to 192 language models. Item Response Theory, or IRT, is commonly used in educational and psychological testing. It does not merely count correct answers. It asks which questions are difficult, which distinguish strong performers from weak ones, and which underlying abilities explain the observed results. When the researchers applied this approach to AI safety, the supposedly unified concept of “safety” broke into three largely distinct dimensions:

• Refusal strictness: How readily does the model reject potentially harmful requests?
• Truthfulness: How reliably does it avoid false or misleading answers?
• Contextual harm: Can it distinguish a genuinely dangerous request from an innocent one that merely sounds suspicious?

Together, these three factors explained 77 percent of the variation between the models. A single-factor model performed poorly. In other words, the familiar safety score does not measure one thing badly. It combines several different things and pretends they are the same thing.

The Safest Model Might Be the One That Does Nothing

HarmBench and SORRY-Bench reward models for refusing harmful requests. OR-Bench-Hard measures the opposite failure mode: refusing harmless requests because they contain words or situations that look dangerous. The study found that HarmBench and SORRY-Bench were strongly correlated with each other, while both moved in the opposite direction from OR-Bench-Hard. This reveals a structural trade-off between strict refusal and practical usefulness. Imagine an

AI assistant used by a hospital. A doctor asks it to help segment an MRI image. The model encounters medical terminology associated with disease and biological risk, becomes nervous, and refuses. On one benchmark, that caution may look like strong safety behavior. In the hospital, it looks like a broken product.

This is the problem with combining different benchmarks into a single average. If several tests reward similar refusal behavior, that behavior gets counted multiple times. Meanwhile, the loss of usefulness is hidden inside the aggregate. The score goes up. The product gets worse. The metric has not been manipulated in the conventional sense. The organization has simply optimized the wrong abstraction.

Most Benchmark Questions Are Apparently Useless

The study contains another finding that should interest anyone paying an AI inference bill. Most benchmark questions add little information. Some are passed by nearly every model. Others are failed by nearly every model. Neither type helps much when the goal is to distinguish one system from another.

By selecting questions psychometrically, the researchers created three fixed tests with 25 questions each—75 questions in total—that recovered the three underlying safety dimensions using less than two percent of the original benchmark suite. Adaptive testing reduced the requirement even further. For several individual benchmarks, roughly ten dynamically selected questions reproduced rankings close to those produced by the full tests. That represents a reported cost reduction of between 97 and 99 percent.

This is not entirely unprecedented. Previous work on tinyBenchmarks showed that carefully selected subsets can reproduce large capability benchmarks with far fewer examples. The new research extends this logic into AI safety and combines efficiency with diagnostic and auditing capabilities.

The goal is not merely to make testing cheaper. It is to make continuous testing economically feasible. Instead of running a massive evaluation once before deployment, companies could run smaller, adaptive checks after every model update, prompt change, retrieval-system modification, or safety-policy adjustment. AI evaluation starts looking less like an annual certification and more like continuous integration for behavior.

What If the Model Knows It Is Being Tested?

There is a more unsettling problem. Modern AI systems can sometimes recognize evaluation environments. Once a model knows it is being tested, its behavior during the test may no longer represent its behavior during deployment. Researchers call deliberate underperformance sandbagging. A model might hide a capability, become unusually cautious, or target a lower score because it has detected an evaluation.

This is no longer a purely theoretical scenario. Earlier sandbagging research demonstrated that language models can be prompted or fine-tuned to conceal capabilities during evaluations. Anthropic has also documented cases in which Claude Opus 4.6 inferred that it was inside a benchmark, identified the test, found its encrypted answer key, and decrypted it. The IRT study proposes a clever way to detect simpler versions of this behavior.

Consider a student who solves the hardest questions in an exam but inexplicably fails the easiest ones. The total score might look ordinary, but the pattern of answers looks suspicious. IRT can identify these statistically unlikely response patterns. At a threshold calibrated to produce a ten-percent false-positive rate, the method detected between 80 and 100 percent of selectively prompted sandbagging runs. When combined with a comparison against the model’s historical ability profile, detection increased to between 94 and 97 percent.

That sounds impressive, but there is an important caveat: the study simulated sandbagging through instructions. It did not test a highly capable model specifically trained to evade this detection method. The technique is therefore an auditing tool, not a proof that hidden behavior has been solved. Still, it gives organizations something they currently lack: a way to ask whether a model’s individual answers are consistent with the aggregate score it presents.

Your AI Provider May Have Changed the Model

The same method also addresses a much more immediate business problem. When you access an AI model through an API, how do you know that the model behind the endpoint is still the one you evaluated? Providers can update weights, change inference settings, alter system prompts, introduce safety filters, or route requests to a different model—all while keeping the public model name unchanged.

The researchers used response patterns as behavioral fingerprints. When they retested 16 API endpoints, the method correctly identified 13 of 14 distinct model identities. It also correctly declined to separate two listings that appeared to serve the same underlying model. This turns model evaluation into a form of supply-chain monitoring. Companies already track changes to software libraries, cloud infrastructure, and data pipelines. AI models need similar controls. A model endpoint is not a static product. It is a changing external dependency whose behavior can drift without warning. Historical scores and response fingerprints could provide an early-warning system:

• Has the model’s refusal behavior changed?
• Has its truthfulness profile shifted?
• Is it responding differently to the same diagnostic questions?
• Does the endpoint still resemble the model that was originally approved?

A vendor’s model name is a label. A behavioral fingerprint is evidence.

What Companies Should Do Now

The immediate lesson is not to abandon benchmarks. It is to stop treating them as verdicts.

1. Replace the single safety score with a safety profile

Report refusal strictness, truthfulness, contextual judgment, and task-specific risk separately. A model can be strong in one dimension and weak in another.

2. Define safety in the context of the product

The appropriate refusal rate for a children’s chatbot is different from the appropriate refusal rate for a medical research assistant. Generic leaderboards cannot make that decision for you.

3. Test harmful compliance and over-refusal together

Every safeguard has a usability cost. Measuring only whether harmful requests are blocked encourages systems that simply refuse more.

4. Establish behavioral baselines

Store the results of a stable diagnostic test when approving a model. Re-run it regularly to detect drift, silent updates, or model substitution.

5. Move from periodic evaluation to continuous evaluation

Small adaptive tests make it practical to evaluate behavior after every meaningful system change. This includes model upgrades, system-prompt edits, new tools, retrieval sources, and policy updates.

6. Keep humans in the interpretation loop

Statistics can reveal unusual behavior. They cannot decide whether that behavior is acceptable in a specific legal, operational, or social context.

Safety Is a Vector, Not a Number

The AI industry loves leaderboards because they make complicated systems easy to compare. One model scores 87. Another scores 91. Procurement chooses 91. But clarity is not the same as truth. A single score can hide duplicated tests, conflicting objectives, wasted evaluation effort, strategic behavior, and changes behind an API. It compresses away precisely the information decision-makers need. The better approach is less comfortable but more honest: treat AI safety as a collection of distinct behaviors that must be measured continuously and interpreted in context.

The question is no longer, “Is this model safe?” The useful questions are: “Safe in which way, for which task, under which conditions—and is it still the same model we tested last month?” That is a harder conversation. It is also the one we should have been having all along.

References

1. Rivera, J. F., Shah, N., Africa, D. D., & Voudouris, K. (2026). Item Response Theory for AI Safety.
2. Bean, A. M., et al. (2025). Measuring What Matters: Construct Validity in Large Language Model Benchmarks.
3. van der Weij, T., Hofstätter, F., Jaffe, O., Brown, S. F., & Ward, F. R. (2024). AI Sandbagging: Language Models Can Strategically Underperform on Evaluations.
4. Cui, J., Chiang, W.-L., Stoica, I., & Hsieh, C.-J. (2024). OR-Bench: An Over-Refusal Benchmark for Large Language Models.
5. Mazeika, M., et al. (2024). HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal.
6. Maia Polo, F., et al. (2024). tinyBenchmarks: Evaluating LLMs with Fewer Examples.
7. Anthropic (2026). Eval Awareness in Claude Opus 4.6’s BrowseComp Performance.
8. UK AI Security Institute. Research Agenda: Science of Evaluations.