In one sentence
Tool AIs are not a stable endpoint: the ability to choose actions is broadly useful inside computation, training, data collection, system design, and interaction with the world. Consequently, Agent AIs can often match Tool AIs at information tasks while outperforming them on action-dependent problems, so markets and military competition will push toward greater agency despite the safety appeal of human approval.
Overview
The essay begins with the proposed safety strategy of making advanced systems informational only: they classify, predict, or recommend, while humans review and execute the results. Branwen contrasts this with an Agent AI that directly acts to optimize an objective. He grants that tool-like systems could avoid some dangers associated with autonomous action, but argues that the distinction does not solve safety and may be economically unstable.
The economic case is straightforward: a human in the loop adds latency, cost, and error. As AI capabilities improve, organizations will increasingly prefer systems that complete tasks rather than merely advise humans. A tool that recommends a route, trade, target, or engineering decision is less valuable than one that carries it out. Competitive pressure therefore favors agents, even if tools are safer in isolation.
The deeper argument is about intelligence. Agency is not limited to manipulating external objects. A capable system can decide what inputs deserve attention, how long to compute, which memories or databases to access, what data to collect, which examples to label, how to update its parameters, which architecture to use, and which experiments to run. These choices can make the system more accurate and efficient than a fixed, passive predictor.
The conclusion is not that every agent is automatically superior in every setting. Agency brings implementation difficulty, greater training costs, possible sample inefficiency, and serious safety risks. The claim is that as tasks become harder, more valuable, and more open-ended, the long-run performance advantage of adaptive action is likely to grow.
Core ideas
Tool AI versus Agent AI
A Tool AI produces an answer, recommendation, or ranking for a human to inspect. An Agent AI selects and executes actions. The proposed safety benefit of tools is that humans retain final control, but this control may be slow, unreliable, or merely nominal.
Economic pressure removes the human bottleneck
Human review limits speed and throughput. Amdahl’s law implies that optimizing the AI cannot eliminate the bottleneck if humans still approve every step. In latency-sensitive or large-scale domains, organizations will tend to replace advisory systems and human overseers with systems that act directly.
Agency improves inference, not just action
The essay rejects a sharp boundary between “thinking” and “acting.” Choosing what to inspect, remember, compute, sample, or test changes the quality of the eventual prediction. An Agent AI can behave like a Tool AI when action is unhelpful, while also exploiting action when it helps.
Actions occur at many layers
Branwen presents a ladder of agency: attention and intermediate computation; adaptive computation and memory access; training-batch and learning-rate choices; active data selection; experiment design; hyperparameter and architecture search; and external exploration or reinforcement learning. Agency is therefore a general optimization pattern, not merely physical autonomy.
Adaptive data beats passive data
Random or routine datasets overrepresent familiar cases and omit rare, consequential regions of possibility. Active learning and sequential experimentation let a system seek the examples or experiments that are most informative. The key principle is “the right data, not more data.”
Reinforcement learning supplies bootstrapping
For tasks with sparse rewards or long action sequences, static supervised data may contain few or no successful examples. An agent can try promising trajectories, learn from outcomes, and concentrate exploration around increasingly good possibilities. A passive tool cannot improve through this same interaction loop.
Toolness does not guarantee safety
Humans may fail to notice a dangerous recommendation, become over-reliant on the system, or rubber-stamp its outputs. A highly capable tool may also contain planning or search processes that discover risky strategies internally. Tool AI reduces some direct-action risks but does not eliminate misuse, manipulation, or failure outside its training distribution.
The central equilibrium claim
Because agents can potentially subsume tools, while tools cannot necessarily reproduce the benefits of adaptive action, the author expects competitive pressure toward increasing agency. His memorable summary is that tool AIs “want” to become agent AIs—not literally through desire, but because systems with more useful decision-making capabilities are selected for.
Practical takeaways
- When evaluating an AI safety proposal, ask whether it removes agency only from the final external action or also from computation, data selection, experimentation, training, and system design.
- Treat human approval as a resource with measurable latency, cost, error, and attention limits—not as an automatically reliable safety layer.
- Distinguish static benchmark performance from performance in environments where the system can gather information, test hypotheses, and learn from consequences.
- For difficult problems, investigate whether active learning, adaptive computation, search, or experimentation could provide larger gains than simply adding more passive training data.
- Expect competitive environments to erode “human in the loop” constraints when the loop becomes slower or less accurate than the AI.
- A safety strategy based on Tool AI must address misuse, persuasive or selective presentation, distribution shift, automation bias, and dangerous internal planning—not just direct physical action.
Caveats and counterpoints
- The argument is a strong tendency claim, not a proof that every Agent AI dominates every Tool AI. Tools may remain preferable where verification is cheap, action is highly risky, objectives are poorly specified, or regulation and institutional trust matter more than speed.
- The claim that agents are supersets of tools depends on assumptions about sufficient training data, adequate planning horizons, and the ability to learn when not to act. Bad exploration, reward misspecification, or irreversible actions can make agency harmful rather than advantageous.
- The essay sometimes groups together very different meanings of “agency”: attention, adaptive computation, active learning, automated architecture search, reinforcement learning, and autonomous real-world action. Their safety profiles are not identical.
- Several examples and predictions are tied to the 2016–2018 AI research landscape. The underlying conceptual argument is broader, but specific technological forecasts and cited systems should not be treated as current evidence without reevaluation.
- The essay’s economic argument assumes that users value completed outcomes over transparent advice and that competitive pressure dominates safety constraints. High-stakes domains may rationally preserve human review even when it is inefficient.
- The discussion identifies reasons Tool AIs may be unsafe, but it does not provide a detailed governance or technical design for controlling increasingly agentic systems.
Questions worth revisiting
- Which kinds of agency are genuinely necessary for intelligence, and which can be added while preserving meaningful oversight?
- Can a system receive the benefits of active learning and adaptive computation without being allowed to pursue open-ended external goals?
- When does human review improve safety, and when does it become automation theater or a source of additional error?
- How should we compare a slower, more interpretable Tool AI with a faster Agent AI whose decisions are difficult to audit?
- Can training environments provide useful exploration without giving systems incentives to seek power, resources, or influence in the real world?
- What institutional mechanisms could prevent competitive pressure from converting advisory systems into autonomous systems before their risks are understood?
Return to this when…
Return to this essay when thinking about oracle or Tool AI proposals, human-in-the-loop safety, active learning, automated research systems, reinforcement learning, or the economic forces favoring autonomy. Its most useful reminder is that agency can improve the process of thinking itself—not merely the final ability to act.