In one sentence
ChatGPT works by repeatedly predicting a plausible next token with a very large trained neural network. Its surprising coherence suggests that human language and ordinary thought contain more compressible structure—and perhaps more discoverable “laws”—than previously assumed. Yet next-token prediction alone does not guarantee factual correctness or deep computation; reliable systems must use external computational tools.
Overview
Published February 14, 2023, the essay explains ChatGPT from the outside in. Wolfram begins with the simple operational picture: given the text so far, estimate probabilities for possible next tokens, sample one, append it, and repeat. He then explains why naïve word-frequency or n-gram methods fail, introducing neural networks as flexible models that interpolate beyond examples. The discussion progresses through training, embeddings, transformers, attention, post-training human feedback, and the limits imposed by feed-forward computation. The final sections interpret ChatGPT’s success as evidence that meaningful language has hidden regularities that might eventually be expressed through semantic grammar or computational language.
Core ideas
The core loop is next-token prediction
ChatGPT does not compose an entire essay at once. It repeatedly estimates a probability distribution over possible next tokens, selects one—sometimes using controlled randomness—and feeds the growing text back into the same process. “Token” matters because the unit may be a word fragment rather than a whole word. This simple loop can produce large-scale coherence when the probability model is good enough.
Why simple lookup tables are insufficient
A model based only on observed word or phrase frequencies would run out of data: the number of possible long sequences vastly exceeds the amount of text available. An LLM therefore learns a compact, parameterized model that estimates probabilities for sequences it has never literally encountered. The important achievement is generalization, not memorization of every passage.
Neural networks learn functions, not explicit rules
A neural network is a large mathematical function built from simple weighted operations and nonlinear activation functions. Training adjusts the weights to reduce a loss function on examples, usually by gradient-based optimization. The network is not explicitly programmed with rules such as “look for whiskers” or “apply grammar”; useful intermediate features and procedures emerge through training.
Embeddings turn language into geometric representations
Words, tokens, and longer passages are represented as high-dimensional arrays of numbers. Items used in similar contexts tend to occupy nearby regions in an abstract meaning space. These embeddings let the network work with relationships among words and contexts rather than treating every word as an unrelated symbol. The geometry is useful, but its exact semantic interpretation remains unclear.
Transformers provide selective memory through attention
A transformer processes token embeddings while allowing each position to “attend” selectively to earlier positions. This gives the model a way to connect a current word with relevant material much farther back than a simple local n-gram could. ChatGPT’s operation consists of embedding tokens and positions, passing them through many attention blocks, and decoding the final representation into next-token probabilities.
Training has two importantly different stages
First comes large-scale pretraining on human-produced text, where the model learns to predict missing or subsequent tokens without manually labeled answers. Then human feedback is used to make outputs more conversational and less likely to wander. Human judgments train a separate preference model, which helps tune the original system toward responses people rate as better.
Coherence is not the same as truth or computation
The model is optimized to continue text plausibly, not to verify claims against the world. It can produce locally convincing language while failing at global consistency, formal logic, arithmetic, long-range counting, or computationally irreducible tasks. Wolfram’s practical conclusion is that language models should call external tools—such as symbolic or numerical systems—when correctness requires explicit computation.
Computational irreducibility sets a ceiling
Some processes cannot be shortcut: to know their outcome, one must effectively carry out the steps. Wolfram argues that trainability depends on compressing regularities, whereas powerful general computation often resists compression. This creates a tradeoff: a network that is easy to train may be poor at deep algorithmic computation unless it can invoke an external computational system. Wording such as “computational irreducibility” reflects Wolfram’s broader theoretical framework, not an uncontested explanation of all LLM limitations.
Practical takeaways
- Use the next-token model as a mental model for why fluent output can arise without an explicit internal database of facts.
- Treat fluency as evidence of learned linguistic regularities, not as proof of understanding, truth, intention, or reliable reasoning.
- Expect performance to be strongest on tasks resembling patterns in the training distribution and weaker on exact, long-range, or novel algorithmic procedures.
- For mathematics, data lookup, code execution, planning with hard constraints, and factual verification, connect the language model to specialized tools or independently check its output.
- Prompting works partly because a prompt steers the model through a learned space of patterns and relationships; examples and framing can activate useful behavior without changing the underlying weights.
- When reading explanations of LLMs, separate three levels: the mechanical operation of the model, empirical observations about what it can do, and speculative claims about what its success reveals about human language.
Caveats and counterpoints
- The essay is an accessible conceptual account, not a complete description of ChatGPT’s exact production system. Wolfram often uses GPT-2 or GPT-3 examples and simplified diagrams rather than documenting every detail of the ChatGPT product available in February 2023.
- Several numerical descriptions—such as 175 billion weights and particular layer or token counts—refer to the GPT-3-era systems discussed in the essay and should not be generalized to every later language model.
- Wolfram’s claim that ChatGPT reveals that language is fundamentally simpler or more law-like is an interpretation and research hypothesis, not a demonstrated scientific conclusion. Other views emphasize that linguistic competence may arise from massive optimization and exposure without implying a compact, human-readable theory of meaning.
- The account understates some practical sources of model behavior, including data filtering, architecture-specific details, evaluation design, safety training, retrieval, tool use, and system-level orchestration. Its “just next-token prediction” framing is operationally accurate but can obscure the sophistication encoded in the learned representations.
- The discussion of humanlike language does not establish that the model has humanlike consciousness, grounded understanding, intentions, or world experience. The essay itself stresses that plausible continuation can diverge from correct computation or globally meaningful claims.
Questions worth revisiting
- Which parts of language competence can be explained by next-token prediction alone, and which require grounding in perception, action, or external verification?
- Does a model need an explicit semantic grammar, or can sufficiently rich learned representations continue to improve without one?
- How should “understanding” be distinguished from successful prediction, flexible in-context adaptation, and tool-assisted reasoning?
- What architectural changes could add loops, memory updates, or computation without making training impractical?
- If language models implicitly learn regularities of thought, which of those regularities can be extracted into transparent symbolic rules?
- Where does the essay’s analogy between neural networks and brains illuminate the issue, and where does it risk misleading equivalence?
Return to this when…
Return to this note when you need a compact conceptual map of LLMs: next-token prediction → learned neural-network function → embeddings and attention → training and human feedback → coherent but non-guaranteed output → need for external tools. Revisit the caveats before using Wolfram’s broader claims about language, thought, or semantic laws as established conclusions.