← All links
Theme
Using automatic theme
Back to top

Article notes

Why Tool AIs Want to Be Agent AIs

By Gwern Branwen

Read the original

Listen

Audio version

A direct reading of the notes, with clickable timestamps throughout the article.

Total length: 9:44
9:44 remaining

In one sentence

Tool AIs are not a stable endpoint: the ability to choose actions is broadly useful inside computation, training, data collection, system design, and interaction with the world. Consequently, Agent AIs can often match Tool AIs at information tasks while outperforming them on action-dependent problems, so markets and military competition will push toward greater agency despite the safety appeal of human approval.

Overview

The essay begins with the proposed safety strategy of making advanced systems informational only: they classify, predict, or recommend, while humans review and execute the results. Branwen contrasts this with an Agent AI that directly acts to optimize an objective. He grants that tool-like systems could avoid some dangers associated with autonomous action, but argues that the distinction does not solve safety and may be economically unstable.

The economic case is straightforward: a human in the loop adds latency, cost, and error. As AI capabilities improve, organizations will increasingly prefer systems that complete tasks rather than merely advise humans. A tool that recommends a route, trade, target, or engineering decision is less valuable than one that carries it out. Competitive pressure therefore favors agents, even if tools are safer in isolation.

The deeper argument is about intelligence. Agency is not limited to manipulating external objects. A capable system can decide what inputs deserve attention, how long to compute, which memories or databases to access, what data to collect, which examples to label, how to update its parameters, which architecture to use, and which experiments to run. These choices can make the system more accurate and efficient than a fixed, passive predictor.

The conclusion is not that every agent is automatically superior in every setting. Agency brings implementation difficulty, greater training costs, possible sample inefficiency, and serious safety risks. The claim is that as tasks become harder, more valuable, and more open-ended, the long-run performance advantage of adaptive action is likely to grow.

Core ideas

Tool AI versus Agent AI

A Tool AI produces an answer, recommendation, or ranking for a human to inspect. An Agent AI selects and executes actions. The proposed safety benefit of tools is that humans retain final control, but this control may be slow, unreliable, or merely nominal.

Economic pressure removes the human bottleneck

Human review limits speed and throughput. Amdahl’s law implies that optimizing the AI cannot eliminate the bottleneck if humans still approve every step. In latency-sensitive or large-scale domains, organizations will tend to replace advisory systems and human overseers with systems that act directly.

Agency improves inference, not just action

The essay rejects a sharp boundary between “thinking” and “acting.” Choosing what to inspect, remember, compute, sample, or test changes the quality of the eventual prediction. An Agent AI can behave like a Tool AI when action is unhelpful, while also exploiting action when it helps.

Actions occur at many layers

Branwen presents a ladder of agency: attention and intermediate computation; adaptive computation and memory access; training-batch and learning-rate choices; active data selection; experiment design; hyperparameter and architecture search; and external exploration or reinforcement learning. Agency is therefore a general optimization pattern, not merely physical autonomy.

Adaptive data beats passive data

Random or routine datasets overrepresent familiar cases and omit rare, consequential regions of possibility. Active learning and sequential experimentation let a system seek the examples or experiments that are most informative. The key principle is “the right data, not more data.”

Reinforcement learning supplies bootstrapping

For tasks with sparse rewards or long action sequences, static supervised data may contain few or no successful examples. An agent can try promising trajectories, learn from outcomes, and concentrate exploration around increasingly good possibilities. A passive tool cannot improve through this same interaction loop.

Toolness does not guarantee safety

Humans may fail to notice a dangerous recommendation, become over-reliant on the system, or rubber-stamp its outputs. A highly capable tool may also contain planning or search processes that discover risky strategies internally. Tool AI reduces some direct-action risks but does not eliminate misuse, manipulation, or failure outside its training distribution.

The central equilibrium claim

Because agents can potentially subsume tools, while tools cannot necessarily reproduce the benefits of adaptive action, the author expects competitive pressure toward increasing agency. His memorable summary is that tool AIs “want” to become agent AIs—not literally through desire, but because systems with more useful decision-making capabilities are selected for.

Practical takeaways

Caveats and counterpoints

Questions worth revisiting

Return to this when…

Return to this essay when thinking about oracle or Tool AI proposals, human-in-the-loop safety, active learning, automated research systems, reinforcement learning, or the economic forces favoring autonomy. Its most useful reminder is that agency can improve the process of thinking itself—not merely the final ability to act.

References

  1. Original Why Tool AIs Want to Be Agent AIs