Every AI lab seems to be building a "world model" right now, and almost none of them mean the same thing by it. A text-to-video tool that paints a gorgeous drone shot gets called a world model. So does a physics engine running a warehouse simulation, and so does the navigation stack inside a delivery robot. That looseness is fine at a dinner party. It is not fine when you are choosing what kind of synthetic visual data will train a computer vision model that has to work outside a demo reel, in robotics, SAR, or maritime detection.
In June 2026, Stanford researcher and World Labs co-founder Fei-Fei Li published an essay that finally gave the term some structure. She split "world model" into three functional categories based on what each one actually outputs: pixels, state, or actions. It's a useful lens, and it maps almost perfectly onto a question we get from engineering teams building synthetic data for computer vision pipelines: why does a dataset that looks convincing in a demo still fail in the field?
The answer usually traces back to which category of world model actually produced the data, and whether the team knew that at the time they picked a vendor.
The confusion has a real source. Reinforcement learning textbooks have used the same diagram for decades to describe how an agent interacts with its environment: an agent takes an action, that action changes the state of the world, and the agent receives an observation, a partial, imperfect glimpse of that state, before choosing its next action. Formally this is a partially observable Markov decision process. Informally it's just how perception and control work, whether the agent is a person, a robot, or a piece of software.
State is the tricky word here, and it doesn't mean what a chemist means by it. In this context, state is the complete underlying description of a scene at a given instant: every object's position, every velocity, every material property, all of it. No sensor sees state directly. A camera captures photons. A radar antenna captures returned pulses. A sonar array captures acoustic reflections off a hull. All of these are observations, compressed and partial views of a state that stays hidden no matter how good the sensor is.
The concept itself is old. Psychologist Kenneth Craik proposed in 1943 that minds reason by running small-scale models of reality, and the phrase carried into early neural networks by the late 1980s. What's new is the pressure to be precise about it, now that computer vision, robotics, reinforcement learning, and generative AI are all shipping products under the same three-word label.
Once you separate state from observation, the different things people call "world models" stop looking like competitors and start looking like three different projections of the same underlying loop.
Li's taxonomy names them by what they produce, not by what architecture they use underneath. That distinction matters more than it sounds like it should.
A renderer's job is to output an observation, almost always an image or a video, that looks right to a human eye. Text-to-video generators are renderers. So are the new wave of interactive world generators that produce a playable scene in real time from a prompt. The bar for success is visual plausibility, and today's best renderers clear that bar with startling consistency: tools like Google's Nano Banana have put convincing image generation in front of hundreds of millions of users. What they carry no explicit model of is three-dimensional structure. A renderer can produce a flawless aerial shot of a city and still fail to keep a doorway the correct width from one frame to the next, because nothing inside it is tracking geometry. It's optimizing for what a viewer sees, not what is actually there, and that gap rarely shows up until someone tries to build on top of the output.
A simulator's contract is different, and stricter. Instead of outputting pixels, it outputs state: a representation of the scene that respects geometry, obeys physics, and stays internally consistent enough that a program can compute on it, not just look at it. Two very different audiences depend on that contract. Architects, filmmakers, and product designers need dimensional accuracy that goes past "looks convincing." Reinforcement learning agents, robot controllers, and perception models need a training ground where they can act, fail, and act again at a scale and speed real-world data collection can't match.
Li makes a point in her essay that's easy to miss on a first read: "the simulator gets the least public attention, and is the most consequential" of the three categories. Renderers get the funding rounds and the viral demos. Simulators do the unglamorous work of producing data that a model can actually trust.
The third function is the inverse of a renderer. Instead of taking an action and producing an observation, a planner takes an observation and a goal, and produces the next action. Vision-language-action models and the current wave of robotic foundation models are attempts at building planners. They're also, by most honest accounts from the field, the least mature of the three. Lab demos of robots folding laundry or sorting parts look impressive in a thirty-second clip. Almost none of them have been tested at the object variety, task duration, or environmental unpredictability that a real deployment demands.
This is where the taxonomy stops being an academic exercise and starts explaining why some computer vision projects stall for months after the demo looked great. Three domains illustrate the pattern especially clearly, and all three sit close to what our team at Vivid 3D builds for robotics and Physical AI customers.
A robot arm that needs to grasp, stack, or navigate around clutter is constantly making decisions that depend on exact geometry: how far is that edge, will this object tip if pushed here, does this gap actually clear the gripper. A renderer can produce a photorealistic frame of a warehouse shelf and get every one of those questions wrong, because visual fidelity and physical fidelity are simply not the same property. Training data for robotic perception needs object placement, camera viewpoint, occlusion, and contact behavior that hold up under simulation, not just under a glance. That's the difference between a dataset a model can generalize from and one it quietly overfits to.
Real-world capture makes this worse before it makes it better. Collecting enough footage of a warehouse floor to cover unusual lighting, rare product placements, and near-miss collisions can take months, and by construction it underrepresents exactly the situations that cause failures later. A simulator sidesteps that by letting a team generate the rare case on demand instead of waiting for it to happen in front of a camera.
Synthetic Aperture Radar imagery is arguably the clearest case for the renderer-versus-simulator distinction, because SAR isn't even visual in the way cameras are. It's a coherent reconstruction from reflected radar returns, shaped by surface roughness, incidence angle, and material properties that have nothing to do with how a scene appears to the human eye. A generative model trained mostly on photographs has no grounding for any of that. What SAR-focused computer vision teams actually need is synthetic SAR imagery generated from controlled scene and target conditions, useful for building and stress-testing detection and classification pipelines before committing to expensive real-world radar collection. That's a meaningfully different, and more honest, claim than saying the data is physically identical to a live sensor capture. Teams evaluating a vendor's synthetic radar data should ask directly how the imagery was generated and what it has and hasn't been validated against, rather than taking "SAR" as a label that implies sensor-level accuracy on its own.
Detecting boats reliably sounds simpler than it is. Vessels appear at wildly different scales, distances, and orientations, against backgrounds that range from open water to cluttered harbors to fog and glare. A renderer, asked to produce "more boat images," will happily generate photorealistic frames that are all subtly similar: same lighting logic, same implicit assumptions about what a boat scene looks like. A simulator, by contrast, lets a team dial in the variables that actually matter for detection performance: vessel type, heading, distance, sea state, occlusion by other vessels or structures. That control is what turns a dataset from "more of the same" into genuine coverage of the edge cases a deployed model will eventually meet.
It also matters for how the resulting model gets evaluated. A benchmark built from a narrow slice of real-world footage can make a vessel detector look strong right up until it meets a hull shape, wake pattern, or lighting condition that never showed up in training. Simulator-generated scenes make it possible to test against exactly those gaps before a model ships, instead of finding them after deployment.
If the renderer-versus-simulator line matters this much, it should show up in how a team evaluates synthetic data, not just in how they talk about it. A few concrete checks separate a real simulator-class dataset from a renderer wearing a simulator's name tag
None of this is exotic. It's the same diligence a data science team already applies to a real-world dataset, just redirected at a synthetic one
Vivid 3D sits squarely in the simulator category in Li's taxonomy, not the renderer one. The platform builds controllable 3D scenes and generates labeled visual data from them, which is a different exercise from prompting a video model and hoping the physics holds together. For robotics and Physical AI teams specifically, that means object placement, camera viewpoints, and environments a team can vary on purpose, with ground truth that comes from the scene itself rather than a manual labeling pass. We've written before about how this plays out for world models versus 3D simulation in Physical AI, and why world models need a physics teacher to stay grounded instead of imagining scenarios that quietly violate physics. This piece is the missing middle layer: it's the taxonomy that explains why that distinction exists in the first place, and why it shows up so consistently across robotics, SAR, and maritime work specifically.
None of this makes renderers useless, to be clear. They're the right tool for a concept video or a quick visualization. They're just the wrong tool for a computer vision model that has to make a correct call in the real world, and increasingly, teams evaluating world models for computer vision projects are learning to ask which of the three categories they're actually buying before they commit a training budget to it.
"World model" is the broader, overloaded term that gets applied to renderers, simulators, and planners alike. A simulator is one specific function within that category: a system that outputs state, meaning a geometrically and physically consistent representation of a scene, rather than just pixels that look convincing or a chosen next action.
It can help with tasks where visual plausibility is what matters most, such as pretraining on general scene diversity or generating quick visualizations. For tasks that depend on precise geometry, physics, or ground truth, like object detection, segmentation, or grasp planning, renderer output alone tends to produce models that look fine on a benchmark and struggle on deployment.
Treat any claim of "physically accurate" or "sensor-perfect" SAR simulation with caution unless a vendor can show the validation behind it. What's realistic to expect from most synthetic SAR platforms today is controlled, synthetic radar imagery useful for detection and classification experimentation, not a guaranteed stand-in for live sensor data. Ask specifically what the generation method captures and what it doesn't.
Sim-to-real transfer is how well a model trained on simulated or synthetic data performs once it meets real-world sensor input. It matters because it's the actual test of whether a simulator's state representation was faithful enough to be useful, not just visually or statistically similar to reality.
A video generation model, a renderer in this taxonomy, learns to produce plausible sequences of pixels from examples, without an explicit internal model of geometry or physics. Simulator-class data comes from a scene that's actually being computed: object positions, physical constraints, and camera parameters are all known and controllable, which is what lets it produce structured, trustworthy ground truth alongside the imagery.
