I was thinking about the state of AI since everyone became familiar with LLMs, and trying to draw a contrast in their underlying architecture, bottlenecks, and operation requirements with world models (for consistency, I will be calling them LWM, Large World Models). And how we can get LWMs to operate at the scale and efficiency, or more, of what LLMs are doing now.
The first thing that struck me was how long the data required to train both model types have existed. To train LLMs, the required data type — written text — has existed for close to 5,000 years, with cuneiform and Egyptian hieroglyphics going back to roughly 3400–3200 BCE, while for LWM, spatial/embodied action-consequence-reaction has existed for as long as anything with a nervous system has been moving through space and learning consequence — which predates written text by a wide margin. Every animal that ever avoided a predator or caught prey was doing world-modeling. Text is just the young, weird, recent invention. Physics-literacy is the ancient default.
Well, this isn't necessarily a fair comparison, because we're comparing a recorded communication primitive (text, when it's written) in a form we can parse and use for LLM training, to an unrecorded primitive (spatial communication, even when it is recorded) in a way we can reuse and parse to train LWMs.
If we consider this, we can conclude that LLMs truly have had a head start with written literature ported online and scraped for free, where LWMs can't do the equivalent move, because there's no pre-existing "internet of physical consequence" sitting around to scrape.
Interestingly, this is just Moravec's Paradox — it reaffirms that the things that seem hardest for humans (chess, math, writing) turned out easiest for machines, and the things humans do without thinking (grasping a cup, walking on uneven ground, catching a ball) turned out hardest.
Aside from the data for training, one other thing we should consider is the mechanism for training, or the architecture on which the models are built. The breakthrough in transformers led to the acceleration of what we have as modern LLMs. This caused a paradigm shift — from RNNs and LSTMs, which processed sequences step-by-step (one token at a time, in order) and struggled to hold onto long-range context, to transformers, which process a whole sequence in parallel using self-attention — changing how we reason about building AI. So the question is: can we rely on this mechanism — breaking everything down into tokens and passing them through the transformer network — to get results, is that the best approach for LWMs? And if it is, for now, how do we handle the eventuality that it might scale for this use case? I honestly don't have enough knowledge to answer these yet but I'll definitely come back to this.
To get action/consequence/reaction data at text-like scale, our options are limited to a few:
- -Real robots, real time: Real robots recorded in real time to capture this action/consequence are genuinely bottlenecked — a robot arm takes actual seconds to grasp an object, can't be sped up the way a GPU processes tokens, and data doesn't transfer cleanly across different robot bodies. This is the slow-but-real path.
- -Simulation, run in parallel at superhuman speed: This is the "efficient mechanism" used and pushed by many companies working on this, like World Labs, Google, etc.
I have always been of the opinion that we should experiment with the different modalities or primitives of communication between humans, and not restrict everything to text — which is really just one instance of a broader thing I keep coming back to (notes). We have had some of our big wins with text with chatbots and AI coding agents, then image and video with frontier models like Nano Banana, Veo, and Seedance, audio (which is closer to how we communicate by default), and now we have the chance to recreate how we interact with the environment. Note to self: start experimenting with this. I'm excited to see how this all plays out.