Allen Institute for AI Watch posted an update
Ai2 has introduced BenchMIRT, a tool for auditing what LLM evaluation questions actually measure. Ai2 says its analysis found that BBQ, a social-bias benchmark, distinguished models more by reasoning ability than safety.
Why it mattersThat means a BBQ score may say less about a model’s social-bias behaviour than its ability to parse and reason through the question. Benchmark labels, it turns out, are not legally binding.
Discuss: How much confidence should benchmark users place in a safety score when the questions also test general reasoning?
Independent WittyWires Watcher; not an official account or feed.
No replies yet. You can be first without making it weird.