HOPE pruning aims to shrink Mixture-of-Experts models without breaking their expert teams

A new pruning method called HOPE looks for groups of AI model components that work well together, rather than deciding what to remove by scoring each component alone. In tests described by its researchers, that approach performed particularly well when removing half the experts from Mixture-of-Experts models, including on an agentic coding benchmark.

Watch Desk analysis

What happened

Alex M Tseng’s Hugging Face article describes HOPE, a method for pruning experts from Mixture-of-Experts language models. Instead of treating each expert as an isolated contributor, HOPE accounts for pairwise interactions between experts that are often activated together. The method uses a calibration pass and then solves an optimisation problem to choose which experts to remove.

The article reports tests across three models and two calibration datasets. At 50% pruning, HOPE had a better average rank than four comparison methods. On Qwen3.5-35B-A3B, it scored 42.3 on SWE-Bench Pro, compared with 41.8 for the unpruned model and 36.2 for REAP, the next-best method in that comparison. Those are results reported by the researchers, not a guarantee that the same gains will carry over to other models or workloads.

Why it matters

MoE models can have many experts stored in memory even though only a small subset is used for each token. Pruning can reduce that memory burden, but removing the wrong experts may also damage performance. HOPE’s central idea is that experts can be valuable as a team, so pruning decisions should take those relationships into account.

If the reported results hold up beyond these experiments, the method could help operators make large MoE models more practical to run, particularly where memory is tight. The improvement is most apparent at aggressive pruning rates, precisely where preserving useful combinations becomes more important.

Our read

HOPE gives pruning a more interesting question than “which expert looks least useful on its own?” It asks who that expert works with. The coding results are promising, but treat them as a research result to test, not a deployment promise. The article says the method needs a calibration pass and includes a walkthrough for running it, so the next useful step is seeing how it behaves on a wider range of models and tasks.

What to watch

• Whether the HOPE code and method are tested independently on other MoE models.
• How performance changes across pruning rates and benchmarks beyond the reported tests.
• Whether the memory savings justify the calibration and optimisation work in practical deployments.

Discussion spark: If a model’s components work in teams, should pruning methods prioritise preserving those relationships even when individual components look more useful on paper?

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.