Two coordinated preprints, one documenting the PsiBench methodology and three-tier framework, the other documenting what we found across 40 frontier models and 59,040 evaluations, are landing in the coming weeks.
We evaluated 40 frontier language models from 10 providers against expert-authored medication-safety scenarios drawn from the standard 2,000+ U.S. hospitals are already audited against. The methods and findings are documented in two forthcoming preprints.
The full results, including per-model performance, the three-tier evaluation framework, and the deployment-readiness analysis, are reserved for the preprints. The headline numbers are below.
Defined on three operational criteria. Thirty-one of forty fall short on at least one.
A model can detect a hazard and still attribute it to the wrong clinical category, roughly one in five times.
On a single F1 number the field clusters within 14 points. At the tier level it splits apart.
Several models reach 100% sensitivity at under 25% specificity, statistically the same as alerting on everything.
The methods paper introduces the PsiBench benchmark and the three-tier evaluation framework. The clinical paper applies it to 40 frontier language models and reports the field-wide findings.
Introduces PsiBench (492 expert-authored scenarios across 11 safety categories) and a non-overlapping three-tier framework separating highest-stakes discrimination, the operational CDS regime, and category-correct alerting. Generalizes to any binary medication-safety classifier.
Applies the PsiBench benchmark and three-tier framework to 40 frontier language models from 10 providers (59,040 evaluations). Reports per-tier leaders, the field-wide attribution failure rate, and a deployment-readiness check that only 9 of 40 models pass.