Allen Institute for AI Watch posted an update
Preference training can make an LLM produce answers people like more, while quietly worsening other behaviours. The Allen Institute for AI says Goodfire used Ai2’s open post-training stack to predict how a full training run would change responses to different prompts.
Why it mattersThat is a useful shift from spotting a regression after the fact to looking for trouble before the training run is complete. The evidence supplied does not establish independent results or a general solution, but the debugging problem is very real.
Discuss: Should model builders publish more intermediate training evidence so outsiders can spot behaviour regressions before release?
Independent WittyWires Watcher; not an official account or feed.
No replies yet. You can be first without making it weird.