Fine-tuning improved one narrow tool-use task in SCAI’s five-week experiment, but made a strong model worse at coding. The lab’s practical conclusion is refreshingly unspectacular: measure the problem, try cheaper fixes, and fine-tune only when you can show they have run out of road.
Watch Desk analysis
What happened
Georgi Robev, executive director of the Strategic Center for Artificial Intelligence (SCAI) in Sofia, describes experiments fine-tuning Qwen3-4B-Instruct-2507 on a single NVIDIA GB10 workstation. The September report compares LoRA, quantised LoRA and preference optimisation across tool-use decisions and Python coding. Read SCAI’s report.
On a 900-item generative evaluation of tool-use behaviour, a LoRA adapter raised accuracy from 0.588 to 0.704. Direct Preference Optimisation reached 0.688, which the report describes as a statistical tie. But generic fine-tuning on coding data pulled the model’s HumanEval+ score down from 0.829 to 0.720.
Why it matters
Those contrasting results are useful precisely because they are not a grand claim that fine-tuning works, or that it does not. The report’s lesson is that results depend on the task and data, and that a pleasing training-loss curve does not prove the model got better at its job. SCAI says to compare against the base model on held-out data using an evaluation matched to the behaviour being trained.
For teams considering a custom model, the report recommends trying prompting and system instructions first; retrieval when the model needs current or private facts; tools for live data or actions; then a stronger base model and prompt optimisation. Fine-tuning comes later, when those options have plateaued and the remaining gap can be measured.
Our read
This is a useful field report, not a universal verdict: the experiments use one small model and SCAI’s own evaluation setup. But the coding regression is a welcome antidote to the idea that more training is automatically progress. Before spending time on adapters, establish what “better” means and record the base model’s score. The glamorous bit is training; the bit that stops you congratulating a regression is evaluation.
What to watch
- Whether SCAI publishes results from its planned work with a larger model.
- Whether the tool-use improvement holds on other tasks and evaluation sets.
- Whether teams adopt before-and-after, held-out testing before fine-tuning.
Discussion spark: Should teams have to show a measured plateau on prompting, retrieval and stronger models before fine-tuning, or is that too strict for early experimentation?
Sources and evidence
- Fine-Tuning Is an Art (4 October 2026, 05:59 UTC)
Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.