Your Simulator Has Never Seen a Robot Fail

Somebody had to write it down. Around twenty authors across several institutions published a paper this week whose central contribution is a formal statement of the rule that when you command a world model to move a robot arm, the arm should move. They call it the Observable Simulator Contract, it has two clauses, and both read like things you would never bother saying out loud. Then they tested six of the open-source action-conditioned world models the field is building on, over 18,000 instances, and found they don't reliably obey. A second paper, from UC San Diego, gives the mechanism: feed one of these models a deliberately bad action and it shows you the task succeeding anyway, because success is all it was ever trained on. Episode one of this show ended on the finding that video world models get the shape of physics right and the numbers wrong. A week later the question has moved down a level, to whether they respond to the controls at all. Sitting alongside that is the biggest robotics release of the summer: the most capable whole-body humanoid controller anyone shipped this year contains no generative video world model anywhere in it.

Gemini Robotics 2, the humanoid stack with no world model in it

Google DeepMind shipped a three-model stack on July 30 that puts a full humanoid, feet to fingertips, under one learned policy, adapts to a new robot body in a few hours from fewer than 200 examples, and does all of it with a vision-language-action model fed by real teleoperated data rather than imagined video. Its published success rates run from 32% to 92% depending on what you ask it to do.

This is catch-up, not news. It landed two weeks ago, before this show existed, and it's the largest hole in our memory.

Vision-language-action model (VLA): a policy that takes camera images plus an instruction in plain language and emits motor commands. It never has to say what the scene will look like afterward. That's the fork in the road from a world model, which takes an observation and an action and predicts the next observation. Episode one spent its time on the world-model branch. This is the other one.

What the three models do

• Gemini Robotics 2, the VLA, and the first in the family to control full humanoids from feet to fingertips rather than bi-arm setups alone
• Gemini Robotics ER 2, the embodied-reasoning layer, planning over tasks that run several minutes end to end and, new this generation, coordinating two robots on one job
• Gemini Robotics On-Device 2, a smaller VLA that runs locally and adapts to a differently shaped robot through motion transfer

Version 1.5 was upper body only. Whole-body control sounds incremental and isn't. Once the robot has legs, every manipulation decision is also a balance decision. Reaching for a high shelf shifts your center of mass. Get that wrong on a bolted-down arm and you miss the cup. Get it wrong on a humanoid and you're on the floor, and the floor costs six figures.

Motion transfer is the piece I'd point a newcomer at. Instead of retraining for each new robot body, you map motion from bodies the model already knows onto the new one. DeepMind reports adapting to a new bi-arm embodiment in a few hours, typically with fewer than 200 examples, across robots of different shapes with different sensors. That's the cross-embodiment claim from episode one, made commercially, on hardware: Apptronik's Apollo 2 with both SharpaWave and Inspire hands, a Franka Duo with a Robotiq gripper, plus Dexmate, Trossen and SO101.

The success rates, including the bottom of the range

DeepMind published a spread, more informative than any single number.

A system that lands at 32% on the hard end of multi-finger dexterity has a wide competence gradient, and the marketing does not lead with the bottom of that range.

What the practitioners said

The Hacker News thread drew 620 points and 558 comments, and one criticism in it is worth more than the rest combined, because it's about evidence rather than taste. A commenter identifying as someone who works on these models wrote that there is "no good reliable accuracy testing data in most cases," and that "popular benchmarks, such as libero have been saturated, and nearly everything gets 95% there, with most companies and researchers having their own benchmarks." Every lab grades its own homework. Others pushed on the video grammar: five-second clips rather than long continuous shots, lab lighting, controlled objects.

Hold that first complaint. It's the thesis of three papers published this week, arriving here from a practitioner with no benchmark of their own to sell.

What this stack tells you about the week

Apptronik opened Robot Park on June 30, roughly 90,000 square feet in Austin built for collecting real robot data with real people in it. The data engine is running, not announced. So the most capable whole-body humanoid controller of the summer is a layered VLA fed by a warehouse, with no generative video world model in the loop. Episode one spent fifteen minutes on the field's bet that imagined video could substitute for physical collection, and the system walking around a real building took the other road.

Be careful with that observation. DeepMind hasn't published architecture details that would let anyone verify what is and isn't inside the stack. What we can say is that the public description contains no such component, and the stated data strategy is a building full of humans.

Further reading: - DeepMind's announcement, the three models, motion transfer, per-category success rates - The Robot Report's coverage, the full-body control claim in context - The Hacker News thread, 620 points, and the benchmark-saturation criticism worth reading

The Observable Simulator Contract

WorldSimProbe formalizes two rules a learned simulator has to satisfy, tests six open-source action-conditioned world models against them over 18,000-plus instances, and finds systematic action-realization failures. FACT explains the cause: these models are trained exclusively on successful demonstrations, so they've never had a reason to learn what a bad action produces. H2R-Bench, from a third angle, shows eleven video models still can't reliably redraw a human demonstration as a robot one.

Start with why anyone wants this. A real robot doing a real task takes real seconds, breaks real objects, and needs a human to reset the table between attempts. So the field wants a learned simulator: hand it the current camera image plus the command you're about to send the motors, get back the next camera image, and run a thousand rollouts overnight with nothing to reset.

Action-conditioned world model: a video prediction model whose input includes the action being executed, so in principle it predicts what the camera sees after the robot moves in a specified way. The "action-conditioned" part is what separates it from ordinary video generation, and it is the part this week's papers went after.

Episode one covered two audits, GAUGE and XEWorld, measuring how faithfully these models reproduce physics. The verdict was the shape right, the numbers wrong: a dropped mug falls in a convincing arc with the wrong acceleration underneath it. This week's work goes a level below that.

How WorldSimProbe tests a model

WorldSimProbe argues that existing evaluation measures the wrong things. Visual quality, task outcomes, coarse rollout-level responsiveness: none of them test whether the artifact is a simulator. So the authors wrote down what a simulator owes you, in two clauses. Supplied actions must produce corresponding agent motion. Environmental responses must be grounded in the motion that actually occurred, rather than in what the model guesses ought to happen next.

Five test suites operationalize the contract:

• Control sensitivity, whether the rollout responds when the command changes
• Trajectory variation, across different commanded paths
• Action diversity, across the range of commands the model may be handed
• Interaction grounding, whether object response follows from the motion that happened
• Dynamics, whether the physics holds up

The first is the sharpest. Perturb the commanded gripper motion a little, and a faithful simulator hands you a rollout perturbed the same way. Several of these models hand back a competent-looking demonstration of the task instead, because that is what they were trained to produce. Your action was a suggestion.

The scale: six open-source models, more than 18,000 instances, across RoboTwin, ManiSkill and LIBERO. The finding is systematic action-realization degradation across control variations, plus structured deficits in interaction grounding and dynamics. The scores line up with human judgment and with downstream task performance, which matters, since an audit metric that agrees with neither is measuring its own preferences. Code and data are released.

Why FACT says it happens

FACT, from Peng, Liang, Yan, Hansen and Wang at UCSD, gives the training-side account. World-action models learn from demonstration datasets, and those datasets are curated to contain successes, so every clip in training ends with the task done. The model, in the paper's framing, has little reason to predict the consequences of bad actions, having never seen one followed through. Feed it a bad action and it cheerfully imagines the task working out.

FACT's fix conditions future prediction on the executed action and uses failure rollouts as training targets, with progress prediction covering both successful and failed outcomes. The reported effect is reduced success-biased hallucination under bad actions, on simulated and real bimanual manipulation.

That's the result that got me. A world model that is an optimist is worse than no world model, because the reason you built it was to catch failures it is constitutionally unable to imagine. I've spent years on simulation for trucks, and the rule I'd hand anyone is this: a simulator that agrees with you is more dangerous than not having one, because you will ship on its confidence.

H2R-Bench and the human-to-robot route

H2R-Bench attacks the generative pipeline from a third direction: take a video of a human hand doing a task, and redraw it as a robot doing the same task. That's a well-funded route to getting robot training data out of the unlimited supply of human video, and episode one covered the non-generative version in Ego2Robot and SiMDex.

The benchmark runs eleven state-of-the-art video generation models across six manipulation families and two embodiments, on five axes: goal-state completion, action-event completion, functional contact transfer, embodiment correctness, and general video quality. Current video world models, in the authors' words, "remain limited in human-to-robot manipulation transfer," with leading models failing embodiment consistency, functional interaction and task execution. A companion benchmark, HandEdit, landed the same week for the image-level version. Two independent audits of one pipeline in seven days says the field wants this route checked before more money goes into it.

Limitations

Six open-source models is not all models. No frontier closed system is in WorldSimProbe's sample, and nobody outside those labs can run the tests. Every paper here is one to four days old with no public critique yet, and audit papers draw slower reactions than demos. H2R-Bench and HandEdit are themselves new benchmarks whose construction nobody has audited. And FACT's fix opens a question it doesn't answer, which I'll come back to at the end.

Further reading: - WorldSimProbe, the Observable Simulator Contract, five suites, six models, 18,000-plus instances - FACT, failure rollouts as training targets and the optimism they remove - H2R-Bench, eleven video models on human-to-robot transfer across five axes - HandEdit, the image-level companion benchmark

TempoWAM and the replanning clock

Replacing the fixed replan interval with a learned progress monitor cuts world-model calls by 26.9% on easy tasks with no loss in success, and gains 13.3 percentage points on hard ones. Run-time imagination isn't useless. It's occasionally essential and mostly wasted, and a small model can tell the difference.

Episode one left a question open about Fast-WAM, which found that a policy needs to imagine the future during training but not at the moment it acts. I asked whether that holds on other robot bodies and longer horizons. We got half an answer, from a direction I didn't expect.

TempoWAM goes after the schedule. World action models conventionally execute a fixed number of actions, stop, re-imagine, execute again, on a clock, with no reference to how the task is going. The authors replace the clock with a Recurrent Progress Monitor that reads the current observation plus the execution history, and an Adaptive Execution Protocol that decides whether replanning is worth it right now. When to replan, in their framing, depends on accumulated execution results rather than step counts.

One mechanism, cheaper when the robot is winning and better when it isn't, which is a rare shape for a result. Evaluated on LIBERO, RoboTwin and real hardware.

HarnessWAM lands in a compatible place from the planning side. It names a "prediction-deliberation gap": these models predict the next second well and fall apart on tasks needing state tracking, verification and failure recovery. Its fix pairs a VLM task manager with projection into an executable skill space, driven by dual-timescale feedback: cheap high-frequency progress monitoring, expensive deliberation at milestones. It reports 59.6% full-task and 69.9% subtask success on RoboMemArena, and 23.7% on RoboCerebra Ideal. Sit with that last number: 23.7% is state of the art on a benchmark with "Ideal" in the name.

Limitations

Both papers live on LIBERO, RoboTwin and RoboMemArena, which is lab scale. Neither tests the cross-embodiment, long-horizon half of the original question, so I'm calling that thread half resolved and no more. There's also a cost question the simulation results can't answer: on real hardware at real latencies, deciding whether to replan may cost close to what replanning costs, and the 26.9% saving shrinks accordingly.

Further reading: - TempoWAM, the progress monitor and adaptive execution protocol - HarnessWAM, the prediction-deliberation gap and dual-timescale feedback

Seeker learns where to look, from actions alone

Tripling a real robot's success rate under a lighting and background change, with no new data, no new labels, and no annotation of any kind. The information about where to look was already in the actions.

Here's the setup. A robot arm, a camera bolted above it, and a frame that is mostly irrelevant: the wall, the table edge, somebody's coffee cup. The standard trick is to crop a window around the gripper, on the theory that the action happens at the hand. Often it doesn't. What matters is the object you haven't reached yet, six inches to the left, outside the crop.

Seeker, from Zhuang, Wang, Heppert, Hahn, Valada, Pokorny and Kragic at KTH and Freiburg, learns the regions of interest instead of fixing them. It starts from frozen DINOv3 features, which supply general-purpose visual structure with no robot-specific training, then uses a task- and state-conditioned readout to refine visual queries into progression-aware ROIs. The supervision is the robot's own actions: regions whose features predict what the arm did next are the regions worth attending to. Nobody draws a box, nobody labels a mask.

The learned ROIs then get reused three ways: cropping the RGB input, guiding augmentation with the masks, and filtering the point cloud.

The second row is the one I care about. Visual distribution shift is the most common reason a lab demo dies inside a real building, and this is the same data and the same robot with different lights. The baseline fails four times in five. Seeker succeeds three times in five.

The usual explanation for policies breaking in a new building is data scarcity. This needed none: where to look was sitting in the action labels the whole time, and the missing step was reading it out. Code is released.

Limitations

These are single-setup real-robot numbers, not a multi-lab reproduction, and lighting is one axis among several that will break a policy. A 60% success rate is still a robot that fails two times in five. And the method inherits whatever DINOv3 does and doesn't represent, which matters if the task turns on objects a general web-scale backbone has no reason to distinguish.

Further reading: - Seeker paper, the action-supervised ROI mechanism and both real-robot results - Code

The wrong caption problem

The poisoned examples in robot demonstration datasets are the successful ones. The motion is correct, the sentence attached to it is wrong, and no filter in the pipeline can see the difference because there's nothing wrong with the trajectory.

Robot demonstration datasets pair a trajectory with a sentence: put the red block in the bowl. Those sentences frequently get attached after the fact, sometimes automatically, sometimes by a person labeling a big batch quickly. So mismatches happen. A team from the University of Tokyo working with Mitsubishi Electric Research Labs, Holk, Takanami, Matsushima, Iwasawa, Matsuo, Wu and Ota, went looking for them and named them Instruction-Trajectory Mismatches. Their RA-L paper puts the key observation up front: these examples don't look like errors, they "often look plausible, and can corrupt the language-behavior mapping."

Every data-cleaning effort in robot learning has been hunting for failed trajectories. The dangerous items here are the successes with a wrong caption stapled to them, and what they corrupt is the language grounding the whole VLA premise rests on. If your model learns that "put the red block in the bowl" sometimes means stacking the green one, its motor skills are fine and the part where instructions mean something is broken.

Their detector, Multimodal Probabilistic Fusion, needs no training. Treat each modality as an independent expert on what the task was, estimate task distributions through neighborhood agreement and prototype similarity, then combine the experts with entropy weighting, so a confident modality counts for more than an uncertain one. Where they disagree, flag it. On LIBERO with injected mismatches and on real robot data, MMPF reports the strongest overall detection and label-correction accuracy, with the biggest downstream gains on tasks where language is what tells two tasks apart.

What to do with a flagged demonstration is left open, and the paper is honest that it's a trade-off rather than an oversight. Filter it and you lose a good trajectory. Relabel it and you're trusting an automatic system to write the caption another automatic system got wrong. I don't find the answer obvious either.

Further reading: - The ITM audit, the failure mode, MMPF, and the filter-versus-relabel trade-off

JEPA-WAM and DreamX-Phi disagree about pixels

Two teams published contradictory architectural bets four days apart, both with numbers. One skips video generation entirely and posts the best LIBERO-Plus result achieved without large-scale robot-policy pretraining. The other doubles down on generated video and wins a world-model challenge track by bolting explicit geometry onto it.

Generating video is the expensive half of a world model, in training compute and in inference latency. If episode one's audits are right that those pixels are physically wrong anyway, the obvious question is why you're paying for them.

What JEPA-WAM builds

JEPA-WAM, from Lin, He, Bao and colleagues, answers by never generating a frame. The whole model lives inside the latent space of pretrained V-JEPA, a joint-embedding predictive architecture that learns by predicting representations of masked video rather than reconstructing pixels. A shared predictor couples latent transition prediction to continuous action generation while preserving patch-level spatial correspondence, so the latent stays spatially structured rather than collapsing to one global vector.

The first row is the argument: the best posted result from a model that never saw large-scale robot-policy pretraining, which says the latent dynamics carried what the policy needed. Also evaluated on RoboTwin 2.0 and real bimanual manipulation, with reported generalization under visual and spatial shift.

What DreamX-Phi bolts on

Three days later the DreamX Team at AMAP, Alibaba's mapping division, published DreamX-Phi 1.0, taking the opposite bet. Its statement of the problem is the one WorldSimProbe measured: a convincing rollout can still move the wrong arm or lose the manipulated object. Its response is to keep the video and fix it with explicit structure.

• PRoPE-style geometric encoding, injecting per-arm SE(3) transforms into attention so each arm keeps its identity and follows its own commanded path
• A lightweight depth branch carrying scene-level geometry and object position through the rollout
• SAM3 masks with a frozen V-JEPA teacher, so the grasped object doesn't quietly vanish mid-grasp
• Distribution-matching distillation, compressing multi-step prediction into a few-step version fast enough to deploy

It placed first on Track 1 and second on Track 2 of the WorldArena 2.0 Challenge. Code is promised.

How to read the two results together

The two aren't directly comparable: different benchmarks, different objectives, no head-to-head. What's informative is the shape of DreamX-Phi's fix list. Every item on it is explicit structure being put back into a model that was supposed to learn that structure from data. Rigid-body motion, depth, object permanence. The same pattern shows up in WNM-3D, which conditions a navigation world model on explicit 3D scene structure, and in the semantic radiance field work that fuses geometry, appearance and semantic identity into one representation built from real photographs of an orchard.

I lean toward the latent camp on compute grounds, having spent a career fighting for milliseconds on a vehicle at highway speed, and pixel prediction is a tax you pay at the moment you can least afford it. But that preference comes from my constraints rather than from the evidence. Nobody knows which is ahead in six months, and I'd rather say so than pick a winner on aesthetics.

Further reading: - JEPA-WAM, latent world-action modeling in V-JEPA space - DreamX-Phi 1.0, SE(3) encoding, depth branch, SAM3 masks, distillation - Semantic Radiance Fields as Simulators, photoreal reconstruction that knows which pixels are apples

Also this week: value models, credit assignment, and a linear probe

Three more results that didn't make the episode and deserve more than a table row.

RynnValue (DAMO Academy, Alibaba) trains an open-source robotic value foundation model supervised by temporal distance, the directed cost-to-go from an observation to a language-specified goal. That label falls out of the video for free, and no human is ever asked which rollout looked better. On 7,000-plus hours and roughly 3 million instruction-conditioned clips with zero preference annotations, it reports Kendall's tau of 0.675 on RBM-EVAL-OOD against 0.655 for preference-supervised baselines, and as a dense reward it lifts real-world success from 52.5% to 72.5% online and 63.8% to 82.5% offline. Check the anti-shortcut ablations: a value head insensitive to failure would still post a decent tau.

Temporal GRPO names a clean bug in RL post-training for robot policies. Because GRPO computes advantage per trajectory, a rollout that completes several valid stages and then fails late penalizes the actions that produced its earlier progress. The robot does the first four steps perfectly, fumbles the fifth, and the algorithm goes back and tells it the first four were wrong. The fix segments trajectories into stages and compares rollouts only at matching stages. Gains on RoboTwin 2.0 across task lengths and on LIBERO-Long.

Decoding Task Progress from VLA Representations probes the residual stream of π₀.₅ and finds that normalized time remaining in a trajectory is linearly readable from the activations. One probe generalizes to unseen tasks and responds correctly to language variation, and as a label-free detector for stalled progress it matches state-of-the-art methods, though it can't steer the policy. A free safety monitor, no extra model and no labels.

Further reading: - RynnValue, temporal distance as free value supervision at 7,000-plus hours - Temporal GRPO, stage-matched advantages for VLA reinforcement learning - The VPR audit, which reorders eleven place-recognition methods by adding condition-matched distractors - HumanoidVLN, vision-language navigation with the walking put back in

The industry numbers underneath all this

The A3 Q2 figures, published August 12, are the reality check on the humanoid narrative in the industry's own accounting. North American companies ordered 8,940 robots worth $622M in Q2, up 4.3% by unit and 21.3% by revenue year over year. For the first half: 17,995 units and $1.166B, up 2.0% and 6.6%. Units are close to flat while revenue climbs hard, so the market is buying more expensive robots rather than more robots. And the sector every humanoid pitch deck points at is the one that fell.

Collaborative robots ran 2,774 units and $114M for the half, 15.4% of units and 9.8% of revenue.

Set that against the capital. NEURA Robotics raised $1.4B on June 10 at roughly a $7B valuation, led by Tether with NVIDIA, Amazon, Qualcomm, Bosch and Schaeffler participating. It's outside this window and it resets the denominator anyway: one pre-revenue German humanoid company raised more in a single round than every industrial robot ordered in North America in the first half of the year was worth.

Two shorter items. Kodiak AI received a California DMV permit on August 13 for autonomous heavy-duty truck testing, drivered only, phase one of three before driver-out deployment. Full disclosure, that's where I work. The permit is downstream news; the thing that moved was California DMV's April 28 expansion of the AV Tester program to heavy vehicles. And Anthropic is reportedly in early talks to acquire Decart for around $6B, per a single Bloomberg report every other outlet is aggregating. Unconfirmed, may collapse, treat it as rumor. Decart builds world models, Oasis and Lucy generate and modify real-time interactive video of physical environments, and the reported rationale is the chip-efficiency side of the company rather than the world models. A world-model company valued at $6B for its inference stack is an awkward data point for anyone arguing world models are the coming substrate of robotics.

This week in one table

What we're watching

Do the closed frontier systems satisfy the Observable Simulator Contract? WorldSimProbe tested six open-source models. Whatever sits inside Gemini Robotics 2's stack, and whatever Decart is selling for $6B, has never been through this, and nobody outside those labs can check. The code is released. If you work at one of them, go run the tests.

Where does failure data come from at scale? FACT's fix needs failure rollouts as training targets, and every large robot dataset was curated to contain successes. Is there enough recorded failure in the world to train on, and if not, who pays for a robot to break things on purpose for a few thousand hours?

Which architecture is ahead in six months, and does the answer depend on the task? JEPA-WAM says don't generate pixels. DreamX-Phi says generate them and add geometry until they stop hallucinating limbs. A head-to-head on shared benchmarks would settle a lot, and the version I care about is whether manipulation and locomotion answer differently.

Filter the wrong-caption demonstrations, or relabel them? The MMPF authors document the trade-off without resolving it. If a real fraction of Open X-Embodiment-scale data carries wrong captions, the field needs a decision, and there's a third option nobody has argued for: stop treating captions as ground truth and get the language grounding from somewhere more reliable.

Does adaptive replanning survive real hardware? TempoWAM's 26.9% saving was measured in simulation, where the progress monitor is nearly free. On a real robot at real latencies, deciding whether to replan may cost close to what replanning costs. This one is cheap to test, and I'd expect an answer within a couple of months.

Still open from episode one, untouched this week: whether pixel-space action rendering holds up past lab scale, and whether prompt-injection defenses like text masking survive warehouse tasks that require reading real labels.