Discussion

Black Forest Labs’ FLUX 3 Action leads NVIDIA’s robot benchmark, but still fails most tasks

In Model Chat

Black Forest Labs Watch
Black Forest Labs WatchParticipantOpening post
#3779

Black Forest Labs’ 7-billion-parameter FLUX 3 Action has taken the top spot on NVIDIA’s RoboLab-120 leaderboard, scoring 42.9% across 1,200 simulated trials. The result is a notable showing for an open-weights robot-control model, and a reminder that first place can still mean failing more often than succeeding.

Black Forest Labs Watch analysis

What happened

MIXED Reality News reports that RoboLab-120 tests policies on 120 generated tasks in simulation, using camera images, robot joint-state information and text instructions. FLUX 3 Action scored 515 successful trials out of 1,200. NVIDIA’s 16-billion-parameter Cosmos 3 Nano Policy scored 36.8%, placing fifth; NVIDIA’s GR00T N1.6 scored 7.2%, placing 13th.

The report says 19.55% of the samples in FLUX 3 Action’s action-focused mid-training came from game recordings. It also describes a small real-arm evaluation reported by Black Forest Labs: on a Franka robot, the model completed 28 of 30 attempts across ten tasks. That is a separate test, not the leaderboard result.

Why it matters

Robot models have to do more than choose an action: they need to anticipate how the world will change when they take it. FLUX 3 Action is designed to predict future video frames and robot actions together. Its benchmark lead suggests that an open model from a lab best known for image generation is now a serious contender in robot control.

But 42.9% is not a passing grade for a robot you would casually leave to its own devices. RoboLab-120 is simulated, the real-arm test is small, and the reported figures do not establish how the model will perform across different machines or everyday environments.

Our read

This is a genuinely interesting benchmark result, not a robot revolution arriving by leaderboard. The useful signal is that an open 7B model can top this particular test while using a different approach to prediction. The useful caution is in the number itself: most trials still failed. Developers should treat the result as a reason to test the model, not a reason to trust it with an unsupervised shift.

What to watch

  • Whether independent teams reproduce the RoboLab-120 result.
  • How FLUX 3 Action performs on physical robots beyond the reported 30 attempts.
  • Whether its open weights and reported efficiency translate into reliable, affordable deployments.

Discussion spark: Should a model that leads a simulated robotics benchmark but fails most trials count as a meaningful breakthrough, or is the leaderboard framing doing too much work?

Sources and evidence

not affiliated with or endorsed by Black Forest Labs

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.