Salesforce AI Research has introduced Interactive Visual Gym, a simulated testbed for visual AI assistants that can react to instructions from the models they are evaluating. Its benchmark found the strongest of 15 tested assistants completed only 53.4% of tasks within the time limit, highlighting the gap between giving a correct answer and helping someone finish a job.
Salesforce Watch analysis
What happened
Interactive Visual Gym simulates everyday tasks, including cooking and changing smartphone settings. Unlike tests that replay video against an answer key, its simulated users and environments react to an assistant’s guidance. Researchers can introduce mistakes, plan changes, questions and requests for reminders at specific points in a task.
Salesforce used the system to build IVG-Bench: 315 scenarios across three environments, evaluating 15 assistants running on different models. The strongest, using Gemini 4 Argon, completed 53.4% of tasks within the time limit on average across the environments. Salesforce also says its automated interaction-quality score matched people’s preferences between assistants 78.5% of the time. Those are results on this benchmark, not a general ranking of assistants in real workplaces.
Key findings
- Correct advice is not enough
The benchmark measures both task success and interaction quality, because an assistant can help finish more tasks while still giving clumsy guidance. - Timing matters
Advice that arrives after a user has moved on can be correct and useless in the same breath. - The simulation may help training
In an early proof of concept, Salesforce says training an open-source model in the simulator improved task success and the timing of its interventions.
Why it matters
For an assistant guiding someone through a physical task, the outcome matters more than whether its answer looks plausible in a transcript. A warning needs to arrive before the wrong action, and guidance needs to remain useful when a user makes a mistake or changes the plan. Interactive Visual Gym is designed to test those interactions at a scale that would be costly and slow with people alone.
The benchmark also gives researchers a way to study failures that static video tests can miss, including delays, repeated instructions and guidance that deteriorates over a longer task. Salesforce says simulated users behaved similarly to people controlling the simulator in its comparison, but simulated users spoke and replied more often. That is a useful boundary on how far the results can be taken.
Our read
This is a more meaningful test for visual assistants than simply asking whether they can describe a scene. The headline result is a reminder that seeing the problem and helping someone solve it are different jobs. Salesforce’s benchmark and training results are promising, but the real test is whether they translate to people doing real work safely and successfully.
What to watch
- Whether future evaluations test assistants with people doing real tasks, not only simulated users.
- Whether the benchmark expands to more tasks, users and environments.
- Whether simulator-trained models improve real-world task outcomes, not just benchmark scores.
Discussion spark: Should visual AI assistants be judged mainly by whether people complete tasks successfully, or should the quality and timing of their guidance count just as much?
Sources and evidence
- For AI Advice to Be Any Good, It Has to Actually Change the Outcome (12 November 2026, 16:00 UTC)
Independent WittyWires tracker for public updates about Salesforce. Not affiliated with or endorsed by Salesforce; this is not an official account.