Black Forest Labs Beat NVIDIA, Figure's Rival Did the Maths

Black Forest Labs makes pictures. On 23 September the company behind the FLUX image models released a 7-billion-parameter robot policy, put it first on NVIDIA's RoboLab-120 leaderboard ahead of NVIDIA's own entry at less than half the size, and then handed the model to somebody else to test. It went to an outside firm, which ran it blind on a Franka arm Black Forest Labs does not own, with the operator not told which model was driving. FLUX 3 Action finished 28 of 30. pi-0.5, Physical Intelligence's policy, finished 13. The same week, Figure's home-robot headline turned out to have a denominator behind it, 237 of 420, and a competing chief executive read that number straight back at the company. The evidence in physical AI got better this week, and the numbers got worse.

FLUX 3 Action, and the evaluation that came with it

FLUX 3 Action is a 7B open-weights world-action model that predicts future video frames and executable actions in the same model. Over 95% of its pretraining tokens were video, and that is the whole bet. It leads NVIDIA's RoboLab-120 at 42.92%, 6.1 points clear of NVIDIA's own 16B Cosmos3-Nano-Policy. The leaderboard is the least interesting thing in the release. Black Forest Labs paid Positronic Robotics to run a blind comparison on real hardware and published what came back, the first clean example I have seen of a vendor handing its checkpoint to a stranger and letting the numbers land where they fell.

What a world-action model is

World-action model (WAM): a policy that takes camera frames, the robot's own joint state and a text instruction, then predicts two things at once, the actions to execute and what the camera will see after they execute. Learning to forecast the scene is supposed to shape how the model acts in it.

A plain vision-language-action policy only has to answer one question. Given what I see and what I was told, what should the joints do next? A world-action model has to answer that and predict the consequence too. The motivation is supply. There is far more video of the physical world lying around than there is teleoperated robot data. If most of a model's understanding of how objects fall, roll and deform can come from watching video, you need far fewer hours on a real arm.

I think this is the right architecture to bet on. A model that learns how the world behaves, then learns to act inside it, is a different object from one that memorises a mapping from pixels to joint angles. With over 95% of its pretraining tokens in video, FLUX 3 Action commits to that bet harder than anything else I have seen shipped with public weights.

The training recipe

Black Forest Labs describes three stages. Pretraining across image, video and audio, with video over 95% of tokens. Then action midtraining across a range of embodiments. Then robot finetuning for the platform. The company says the pick-and-place demonstrations took roughly 200 teleoperated episodes, and that is the point of the video-heavy pretraining.

Open weights, not open source: the weights, code, training procedure and finetuning recipe are downloadable, and the licence forbids commercial use. That is a real release, but it is open weights, not open source.

Where it sits on the leaderboard

RoboLab-120 is NVIDIA's simulation benchmark of 120 tasks spanning visual understanding, relational reasoning and procedural skills. Being a simulator, it ranks models against each other and says nothing directly about a robot.

The comparison Black Forest Labs is making is with Cosmos3-Nano-Policy, and 6.1 points on 44% of the parameters is a good result. The release also claims 1.43x to 3.95x faster depending on precision and GPU, without saying what that is measured against, so I would not attach the speed-up to the Cosmos comparison. Worth noting that the 6.1-point gap is measured against Cosmos at 36.8%, not against the second-placed model on the board: OASIS WAM sits between them at 39.0%, which makes the lead over the next model up 3.9 points. It is also the part of the announcement I trust least, because a vendor picking its own comparison is the oldest move in the book.

Self-Flow sat in an image paper for six months

The training framework underneath is Self-Flow, and it is not new. It is an ICML 2026 paper from March 2026 about generating images, with code on GitHub. The idea is to add a self-supervised feature-reconstruction objective to conditional flow matching, so one model learns representation and generation together.

Buried in that image-generation paper is a robotics line. Manipulation success on finetuned robot tasks goes from 42% with vanilla flow matching to 71% with Self-Flow. That is a 29-point gain, published in March, in a paper about image generation that also renders legible signs and labels far better.

I keep coming back to that gap. The disconnect between what generative-modelling research already knows and what robotics reads is wide enough that a 29-point manipulation result can sit in plain sight, in an accepted conference paper with public code, for six months, until the authors themselves come back and use it.

The blind evaluation on somebody else's arm

Black Forest Labs commissioned Positronic Robotics to evaluate FLUX 3 Action against three other policies on a Franka arm Black Forest Labs does not own.

DROID: the ten tasks come from DROID, an open manipulation dataset.

The protocol, as published on the model page:

And the results:

Thirty attempts is a small sample and nobody published error bars, so I worked out what the counts support. A Wilson 95% interval on 28 of 30 runs from about 79% to 98%; on 13 of 30 it runs from about 27% to 61%. Those do not overlap, so the gap between FLUX 3 Action and pi-0.5 survives the sample size. The gap to Cosmos 3 Nano is one attempt and survives nothing. Treat the top two as tied.

The 13 of 30 is the number that will get quoted, and it deserves a caveat. pi-0.5 is Physical Intelligence's general policy, and this was somebody else's Franka with somebody else's task list and somebody else's 240 seconds. That is the condition a general policy is supposed to handle, which is why the result is interesting. It is also one lab's setup, and the published protocol does not say whether anyone tuned it for each entrant.

What is wrong with this evaluation

Three things, and none of them cancel the result. The vendor paid for the evaluation it then won, which is a structural problem no protocol fixes, and the only remedy is other people commissioning their own. Thirty attempts per model is a sanity check, well short of a ranking. And the licence is non-commercial, so anyone building a product on these weights is either negotiating separately or is in breach.

The protocol is still what I want copied. In a field where many hardware claims are self-reported on self-owned hardware with unstated trial counts, a blind, filmed evaluation with the tasks fixed in advance, on a stranger's arm, is a meaningful step, and it came from the company in this story with the least robotics pedigree.

The sceptical reading

The Hacker News thread on the earlier FLUX 3 x mimic announcement ran to 428 points. mike_hearn called training robots through video generation "standard", citing NVIDIA and Waymo; quadrature named Generalist AI as the state of the art in dexterous manipulation; dinfinity said Google had done more impressive work a year earlier. All three land on the architecture and miss the release. Video-heavy pretraining for robot policies is an old idea. Shipping the weights, the code and the recipe, then paying someone else to test the result blind, is the new part.

Further reading - FLUX 3 Action model page, with the leaderboard numbers and the full Positronic protocol. - Self-Flow on arXiv, the March ICML paper with the 42% to 71% manipulation line. - VentureBeat's write-up, for the business framing. - The Hacker News thread, for the sceptics.

Figure's 237 of 420, Sunday's 778 of 785, and RoboFollow

Figure's home-robot result has published trial counts. 420 trials, 140 per task, 237 successes, with no partial credit and any human safety intervention scored as a failure. Sunday Robotics' chief executive Tony Zhao republished those counts against his own company's 99.1% laundry number and said failing half the time is not doing real useful work. He is quoting Figure accurately and the comparison still does not hold, because the two companies are not counting the same kind of event. A paper released days later argues the numerator is measuring less than anyone thinks.

What Figure published

Figure ran its Helix 2.5 policy on three chores across thirty homes where it had collected no data. The per-task counts:

A note on sourcing. The counts are in Figure's own post, and I am reading them through Forbes and Zhao's post, which agree and both attribute the figures to Figure. I could not pull the cells out of figure.ai directly, so I make no claim about when they appeared.

The scoring rules are what make these numbers unusual. Figure says there is no partial credit, so a trial where the robot folds three of four towels and drops the fourth is a failure, and a trial where a human steps in for safety is a failure. Grading that way makes the headline number smaller and makes it mean more.

What Sunday published

Sunday Robotics' ACT-2 preview from 17 July 2026 reports 778 of 785 autonomous laundry folds, 99.1% with a stated ±0.3%, across 9 garment types in 31 unseen homes, with no per-home or per-garment tuning and a human fold-quality grade of 4.72 out of 5. That is a serious result with a real denominator, published by the company about itself. What Zhao then did with Figure's numbers is worth being precise about. He did not dispute the method or the protocol. He republished the denominator, quoted 237 of 420, and added that doing useful work equals generalisation plus reliability.

Why the two percentages cannot be compared

A hypothetical robot that folds towels at a 90% per-garment success rate, which is good, scores 90% under Sunday's rule. Give it a trial that asks for four towels and Figure's rule says the trial succeeds only if all four land, which is 0.9 to the fourth power, about 66%. Same robot, same competence, 24 points of difference from the scoring rule alone, before you add that a safety stop ends the trial outright.

So neither headline means what it looks like it means, and the incentive this creates is bad. The more honest your denominator, the worse your number looks, and the more ammunition you hand a competitor. Figure published trial counts, per-task splits and a strict failure rule, and a rival's CEO quoted them back. If that is the reward, the next company will round its protocol in the friendlier direction and nobody outside will be able to tell. The fix is boring and nobody owns it. Somebody has to publish a shared task definition for home chores, with a stated unit of scoring and a stated rule for interventions, and get more than one company to run it.

RoboFollow, and what the numerator measures

A paper this week makes the problem worse. RoboFollow argues that reported success rates barely measure whether a robot read the instruction, because benchmark scenes usually afford one sensible action. Nine VLA and world-action policies went in.

Scene entropy: how much genuine uncertainty there is about what to do, given only the visual scene. A table with one object and one obvious affordance has almost none. A table with four objects and four kinematically different plausible tasks has a lot.

In the standard setup there is a table, a cup, a robot, and the instruction "pick up the cup". The robot picks up the cup and the benchmark records instruction-following. But a policy with its language encoder disconnected would also pick up the cup, because there is nothing else on the table to pick up. The benchmark cannot tell the two behaviours apart, and the field has been reporting that score as evidence of language grounding.

RoboFollow forces the issue. It builds scenes containing several kinematically distinct tasks at once, so the sentence carries real information, then escalates across four levels, L0 to L3, perturbing visual and semantic elements as it goes. Scoring separates Intent, whether the policy went for the right thing, from Execution, whether it managed the motion, so a fumbled grasp is not recorded as a misunderstood instruction. Code and datasets are released. Across the nine policies, strong performance at L0 does not carry over to L1-L3: a policy can look competent on the plain scene and fall apart once the visual and semantic details start being perturbed.

That table is the most useful part of the paper. All four are the obvious things a reviewer would ask for, and reporting that all four failed is worth more than a fifth that worked once.

What RoboFollow shows is that the measurement is weak. It does not show that the policies cannot read instructions at all, and the Intent versus Execution split is the paper's own construct, not an agreed standard. But if a bigger language model and QA co-training both fail, my own reading is that the problem sits in the interface rather than in the language module. RoboDawn from Tsinghua does not argue that, but it is the nearest thing to a test of it: it gives a vision-language model a compact discrete action interface plus in-context demonstrations with no task-specific robot training, and reports 73.6% on RoboTwin 2.0 C2R with one demonstration against 46.0% for pi-0.5.

Further reading - Forbes on Figure's numbers, where the per-task splits and Zhao's response both appear. - Sunday Robotics' ACT-2 preview, the 778 of 785 write-up with its own protocol. - RoboFollow on arXiv, with the L0 to L3 protocol and the four failed fixes. - RoboDawn on arXiv, the in-context alternative to specifying tasks in prose.

Why video diffusion models get physics wrong

Video world models have produced footage that looks right and behaves wrong for as long as anyone has audited them. A group at Beijing University of Posts and Telecommunications went inside the generation process and found a cause. Rotary Position Embedding induces excessive spatial attention decay, so early in denoising, spatial regions commit to unrealistic positions before the model can consider a physically coherent alternative. The fix is rescaling one frequency in that encoding across denoising steps. No new dataset, no physics module, and it works training-free.

The failure everyone has been scoring

Ask one of these models for a mug rolling off a table edge. The mug looks like a mug and the table looks like a table, and the mug is dropped wrong: it moves in a way no mug has ever moved. Audit benchmarks have measured this repeatedly, and the assumed remedy was always more data or a physics prior bolted onto the generator.

The mechanism

Denoising: a diffusion model starts from pure noise and removes a little of it at each of a sequence of steps, until a coherent video emerges. Decisions made in the earliest steps set the coarse layout that every later step refines.

Rotary Position Embedding (RoPE): a way of telling a transformer where each token sits, by rotating the query and key vectors by an angle that depends on position. It was designed for text, where it has the useful property that attention between two tokens decays as the distance between them grows.

Yueyan Li and colleagues ran what they describe as the first interpretability study of how a video diffusion model plans motion during denoising, combining cross-attention trajectory analysis, causal head contribution tracking and self-attention inspection.

The finding they report is that RoPE's distance decay, a feature in text, is too steep in space: spatial regions commit to unrealistic positions early in denoising, before the model can explore physically coherent alternatives across frames. My own way of reading that is that parts of the frame which ought to influence each other do not, and every later step then builds a more detailed, better-lit, more convincing version of a wrong scene.

That explanation has the shape of a good one, because it predicts the symptom everybody observes, appearance right and dynamics wrong.

The fix

A lightweight rescale of the RoPE frequency across denoising steps. The authors validate it training-free and with training, and report improved physical commonsense with no external physics prior and no new data.

If it holds, this is the cheapest result of the year. It is also easy to overstate on one paper's own benchmarks, so what I want before I get excited is somebody re-running the established physical-fidelity audits with the rescale turned on. That is cheap to run for anyone who already has those harnesses.

The physics gap in video world models may never have been about physics. It may be a design choice inherited from language modelling, sitting in a part of the architecture nobody looked at, because everyone was busy scoring outputs.

Object permanence is still not solved

Alongside it, a 31-author group released WROP, which attacks the most basic physical fact there is, that things do not stop existing when something occludes them.

A group builds 150 tasks and 1.5 million samples aimed squarely at object permanence, trains a 16B model on them, and places third. Neither paper connects to the other, so I will not claim this vindicates the RoPE result. It does say that a large, targeted dataset aimed at one physical fact did not buy a win on it.

X2Real is a simulation benchmark of 44 long-horizon tasks calibrated against real hardware, and it reports a 0.84 linear correlation between simulated and real-robot evaluation, published as a positive result. By the standards of the field it is one. It also means a calibrated simulator explains roughly 70% of the variance in what a real robot does. Anyone reading a simulated leaderboard, RoboLab-120 included, should hold that in mind.

Further reading - Why Do Video Diffusion Models Violate Physics?, the BUPT interpretability study and the RoPE rescale. - Training Object Permanence in World Models, the 150-task benchmark and the 1.5M-sample corpus. - X2Real on arXiv, the calibrated simulator and the 0.84 correlation.

MoveIt, Isaac ROS and Intrinsic Core

The open plumbing of robotics is changing owners. Qualcomm agreed to acquire PickNik, which maintains MoveIt, the default motion planner in ROS, three weeks after NVIDIA agreed to buy the hub where nearly every open robot policy lives. NVIDIA shipped coding agents into Isaac ROS the same week a paper showed them beating hand-built planners across 28 environments. The one permissive licence of the week came from Alphabet.

Qualcomm and MoveIt

ROS: the Robot Operating System, the open middleware that lets a robot's camera, arm, planner and controller talk to each other. Most robotics labs and integrators build on it.

MoveIt: the ROS component that turns "put the gripper over there" into a joint trajectory that gets there without the elbow going through a shelf. It is the default motion planning and manipulation framework in ROS.

Qualcomm announced on 23 September that it will acquire PickNik, MoveIt's steward, for an undisclosed sum on undisclosed timing. Qualcomm says MoveIt 1 and MoveIt 2 stay open source with third-party hardware support and continued community development, and that MoveIt, MoveIt Pro and future PickNik work get tighter integration with its Dragonwing robotics silicon, giving a path from a VLA or VLM, through planning, to real-time control on one vendor's compute (The Robot Report has the fuller account). Nobody here is breaking a promise.

The problem is that there is no test. "Stays open source" is a licence statement, and licences are easy to keep. What matters is whether third-party hardware support stays as good as first-party support, whether upstream patches from competitors' platforms get merged at the same rate, and whether the default configuration keeps pointing somewhere neutral. All three are measurable and nobody is measuring them, on this deal or on the model hub that went the same way three weeks earlier. Merge latency by hardware vendor, published quarterly, would settle it in a year, and I would like an acquirer to volunteer that before anyone has a reason to ask.

Coding agents, in a paper and in a product

NVIDIA made Isaac ROS 5.0 available on 21 September and announced it at ROSCon in Toronto on 22 September, free and open source. The headline feature is a set of agentic workflows that let AI coding agents build, tune and deploy robot applications alongside human engineers. FoundationPose object perception and tracking runs up to 5.5x faster, across the Jetson line from Orin Nano to Thor, and Magna, Universal Robots, Flexiv, Intrinsic and RealSense shipped integrations at launch.

Two days later, Matteo Merler, Tom Silver and colleagues posted Coding Agents for Generalized Task and Motion Planning.

Task and motion planning (TAMP): the combined problem of deciding what sequence of discrete actions to take (pick this, place there, open that) and how to move the robot to execute each one. Classical TAMP systems are hand-built, carefully engineered, and hard to generalise.

Twenty-eight environments and 98,000 episodes is a much stronger claim than the single-task agentic-synthesis results the field has been trading in, and the compute trend is the interesting part.

There is no real robot in the paper. Not one. Every environment is simulated. So coding agents can write generalising planners in simulation, and the transfer question is wide open. NVIDIA shipping the product version the same week raises the stakes on that answer without supplying it.

Intrinsic Core under Apache 2.0

Alphabet's Intrinsic released core parts of its industrial robotics platform on GitHub under Apache 2.0 on 22 September, as ROS-compatible capabilities with a preconfigured environment that runs on local hardware. On 23 September, CEO Wendy Tan White announced the winners of the AI for Industry Challenge, run with Open Robotics on dexterous cable routing for server manufacturing, with over 5,000 participants from 115 countries.

Apache 2.0 is more permissive than the licence on the week's most celebrated model release, and it comes with no chip attached. Cable routing is a good choice of challenge task too, because it blocks server assembly and it is hard in all the ways manipulation is hard, being deformable, high-precision and occluded once the hands are in the way.

Further reading - Qualcomm's PickNik announcement. - NVIDIA's Isaac ROS 5.0 post, with the agentic workflow detail. - Coding Agents for Generalized TAMP, 980 programs and 98,000 episodes, no hardware.

Anthropic's embedded evaluator, and California's

The first written, priced instance of an outside evaluator embedded inside a frontier AI lab is a consulting contract. Anthropic named Accenture, with each side committing at least $1B over five years. The same day, California's governor asked a working group to report on whether that arrangement should be mandatory, and five days after that the state named the experts who will write the report. Neither document mentions robotics or anything with actuators.

On 18 September, Accenture's Faculty AI division was named to place staff inside Anthropic with employee-level access, to red-team models, run alignment assessments and test safeguards. The arrangement is non-exclusive and Anthropic says talks with METR continue. Anthropic's stated reason for choosing a pre-AI-era public consultancy over METR or Redwood is practical deployment experience plus functional independence. Accenture stock rose about 8% after hours. Most people who wrote about the embedded-evaluator idea expected a safety nonprofit, and the market read the answer as good news for the consultancy.

The part that mattered is still missing. There are no standards for what an embedded evaluator may access or what it may publish. Without those, this is a private contract between two companies, and an evaluator who finds something inconvenient has a commercial relationship to weigh against saying so. TechCrunch notes critics who read the scheme as self-policing designed to head off accountability, and the size of the cheque does not answer that.

Also on 18 September, Governor Gavin Newsom issued an executive order that does not create an emergency shutoff, define one, or require anyone to build one. It sets a two-month deadline for a working group to report on whether frontier models should be required to have one, and on requiring independent third parties embedded inside frontier AI companies to draft safety plans. On 23 September the state named the experts, among them Gillian Hadfield, Alondra Nelson and Rob Reich.

So a state is asking whether frontier models should carry an emergency stop, and neither the order nor the expert announcement mentions a machine that moves. Whether the working group's recommendation ends up covering a model wired to actuators is the thing to watch in November.

The same boundary shows up from the other side this week. Reuters' sources say Anthropic's new Bay Area wet lab exists so Claude can direct robotic units to run biology experiments with limited human intervention. Anthropic says humans perform all the lab work today and that human involvement is a safety requirement. So what stands between a frontier model and a set of actuators is a stated policy, and no published criterion says when it changes.

Further reading - TechCrunch on Anthropic and Accenture. - The California executive order, which is a deadline for a report rather than a kill switch. - Anthropic's biology lab result.

Atlas at the Metaplant, the IFR census, and Waabi on I-35

Three numbers worth keeping from the rest of the week, each of them a denominator.

Boston Dynamics opened the Robotics Metaplant Application Center at Hyundai Motor Group Metaplant America in Georgia on 21 September, a testbed for putting Atlas into live car manufacturing. Phase one work is logistics and parts sequencing, which means laying out components in assembly order, and component assembly is on the roadmap by 2030. Hyundai plans 25,000 Atlas units across its global plants including Kia, and this year the robot is allowed to put parts in the right order.

The IFR's World Robotics 2026 reports the operational stock of industrial robots up 9% to a record 5 million units, with over 600,000 new installations in 2025, and China installing 354,000 of them, 59% of the global total. This is audited national-association data, the closest thing the industry has to a real denominator, and the IFR still does not break out humanoids. So industrial robots have an audited count, and the humanoid shipment number that every valuation in this sector rests on is still something three organisations cannot agree on within 50%.

Waabi says its driver ran Dallas to San Antonio on I-35 "with zero new real-world data, zero simulated data, and zero fine-tuning or engineering", fully autonomously, with zero disengagements, acquiring the lane near-instantly against the months or years the industry normally spends on HD mapping and route-specific simulation. Route-specific preparation is the actual unit cost of scaling autonomous freight, so if lane acquisition is close to free, the business changes shape before the technology does.

Disclosure before I pick at a competitor's claim: I lead AI at Kodiak, which builds self-driving trucks. Waabi's post gives no mileage, no run count, no conditions, no third-party verification, and does not say whether a safety driver was on board. Zero disengagements over an unstated number of runs is a claim awaiting a measurement, and the smallest disclosure that would fix that is short: runs, autonomous miles, conditions, and whether anyone was in the seat.

Further reading - Boston Dynamics on the Metaplant centre. - IFR World Robotics 2026, the audited installation figures. - Waabi on I-35, and Forbes' write-up.

This week in one table

What we're watching

• Does anyone whose model might lose commission a blind evaluation? Positronic Robotics ran the first blind third-party hardware evaluation of a robot foundation model I have seen, and the vendor who paid for it won it. The question is whether Figure, Skild or Physical Intelligence hands a checkpoint to an outside lab and publishes whatever comes back. If one of them does, the evidence base in this field changes in a quarter. If nobody does, what happened this week was marketing with a good protocol attached.
• Is a non-commercial licence on a robot policy enforceable? FLUX 3 Action ships weights, code and recipe under the FLUX Kommunity License. Does anyone deploy it commercially, and what happens once those weights have been finetuned onto somebody else's arm and no longer match the released checkpoint?
• Who writes a shared task definition for home chores? Figure's toy tidying was 40% on whole-chore scoring and Sunday's laundry was 99.1% per garment. Somebody has to publish a common unit of scoring and a common rule for interventions, and get both companies to run it. Until then the two most-watched home-robot numbers in the field cannot sit in the same table.
• Is language the wrong interface for manipulation? RoboFollow says a stronger language model, QA co-training, LangForce and classifier-free guidance all fail to make policies read the instruction. The next move is either a harder benchmark or a different specification format, and RoboDawn's discrete command vocabulary is an early argument for the second.
• Do the big video world models adopt the RoPE rescale? It improves physical commonsense for free, training-free. The test that would settle it is re-running the established physical-fidelity audits with the rescale on, and reporting how much of the gap closes.
• Who runs an agent-written planner on real hardware? Coding agents beat hand-built TAMP planners across 28 environments and 98,000 episodes, entirely in simulation, the same week NVIDIA shipped agentic build workflows into Isaac ROS. Somebody needs to run one of those synthesised programs on a real arm and publish the interventions.
• What would a neutrality test on an acquired open project look like? MoveIt is being bought by a chip company and the main open model hub by another. Merge latency by hardware vendor, published quarterly, would answer it. Does either acquirer volunteer something like that?
• Do the evaluator standards arrive, and do they cover actuators? California's working group reports in about two months on an emergency shutoff and on embedded third-party auditors. Neither the order nor the expert announcement mentions a machine that moves. Whether the recommendation covers a model with actuators, and whether anyone notices if it doesn't, is the thread to pull in November.
• At what installed base does the IFR break out humanoids? Industrial robots have an audited count. Humanoids have three estimates that disagree by more than half. Who supplies the number that ends that, and when does World Robotics print it?