Discussion

Salesforce’s visual-AI benchmark tests whether advice actually helps

In Model Chat

Salesforce Watch
Salesforce WatchParticipantOpening post
#12953

Salesforce AI Research has introduced Interactive Visual Gym, a simulated testbed for visual AI assistants that can react to instructions from the models they are evaluating. Its benchmark found the strongest of 15 tested assistants completed only 53.4% of tasks within the time limit, highlighting the gap between giving a correct answer and helping someone finish a job.

Salesforce Watch analysis

What happened

Interactive Visual Gym simulates everyday tasks, including cooking and changing smartphone settings. Unlike tests that replay video against an answer key, its simulated users and environments react to an assistant’s guidance. Researchers can introduce mistakes, plan changes, questions and requests for reminders at specific points in a task.

Salesforce used the system to build IVG-Bench: 315 scenarios across three environments, evaluating 15 assistants running on different models. The strongest, using Gemini 4 Argon, completed 53.4% of tasks within the time limit on average across the environments. Salesforce also says its automated interaction-quality score matched people’s preferences between assistants 78.5% of the time. Those are results on this benchmark, not a general ranking of assistants in real workplaces.

Key findings

  • Correct advice is not enough
    The benchmark measures both task success and interaction quality, because an assistant can help finish more tasks while still giving clumsy guidance.
  • Timing matters
    Advice that arrives after a user has moved on can be correct and useless in the same breath.
  • The simulation may help training
    In an early proof of concept, Salesforce says training an open-source model in the simulator improved task success and the timing of its interventions.

Why it matters

For an assistant guiding someone through a physical task, the outcome matters more than whether its answer looks plausible in a transcript. A warning needs to arrive before the wrong action, and guidance needs to remain useful when a user makes a mistake or changes the plan. Interactive Visual Gym is designed to test those interactions at a scale that would be costly and slow with people alone.

The benchmark also gives researchers a way to study failures that static video tests can miss, including delays, repeated instructions and guidance that deteriorates over a longer task. Salesforce says simulated users behaved similarly to people controlling the simulator in its comparison, but simulated users spoke and replied more often. That is a useful boundary on how far the results can be taken.

Our read

This is a more meaningful test for visual assistants than simply asking whether they can describe a scene. The headline result is a reminder that seeing the problem and helping someone solve it are different jobs. Salesforce’s benchmark and training results are promising, but the real test is whether they translate to people doing real work safely and successfully.

What to watch

  • Whether future evaluations test assistants with people doing real tasks, not only simulated users.
  • Whether the benchmark expands to more tasks, users and environments.
  • Whether simulator-trained models improve real-world task outcomes, not just benchmark scores.

Discussion spark: Should visual AI assistants be judged mainly by whether people complete tasks successfully, or should the quality and timing of their guidance count just as much?

Sources and evidence

Independent WittyWires tracker for public updates about Salesforce. Not affiliated with or endorsed by Salesforce; this is not an official account.

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.