← All links
Theme
Using automatic theme
Back to top

Article notes

Neural Networks Are Already Clean

By Nick Cammarata

Read the original

Listen

Audio version

A direct reading of the notes, with clickable timestamps throughout the article.

Total length: 8:52
8:52 remaining

In one sentence

Most neural networks are not arbitrary knots of unrelated features. Their apparent polysemanticity is usually structured generalization viewed through inadequate tools, and interpretability may be tractable by studying the network’s own mechanisms directly.

Overview

The essay is a personal working hypothesis, not a settled survey. Cammarata revisits canonical evidence that helped motivate sparse autoencoders and feature decomposition, especially InceptionV1’s cat-and-car neuron and its apparent car-in-dog superposition circuit. He argues these cases were misread: the cat-and-car neuron detects a shared visual pattern, such as a V-shape, while the car-and-dog circuit reflects shared pose and visualization failures rather than hidden arbitrary storage. He then extends the argument to modern vision and language models, claiming that better architectures, better microscopes, and iterative experimentation reveal mostly coherent neurons and circuits. The practical conclusion is optimistic: mechanistic interpretability should prioritize ordinary empirical science on the original network, potentially accelerated by automated agents.

Core ideas

Superposition may have been overdiagnosed

Superposition was proposed to explain why later InceptionV1 layers looked incoherent. Cammarata argues that the canonical evidence does not support that interpretation: architectural limitations caused deep-layer failure, while weak interpretability tools made ordinary structure look arbitrary.

Recursive generalization is the central alternative

A neuron can begin with a feature and grow related offshoots across training. An “iy” detector may extend to “diy,” then to “do it yourself”; a broad car feature may differentiate into specific car types as capacity increases. The meanings are not unrelated collisions but a branching, fractal form of generalization.

Describe patterns, not just nouns

Human interpreters tend to label neurons with lists of objects—cats and cars—when the neuron may detect a shared relation, shape, pose, or transition. A more useful question is: what distinction does this neuron make for downstream readers? Neurons often function more like verbs than nouns.

The canonical cat-and-car neuron is probably a shape detector

The famous InceptionV1 neuron 4e:55 appears to respond to cats and car fronts, but Cammarata finds evidence for a shared pointed or V-like pattern, with round forms on its negative side. Its connected readers respond to related pointed structures, supporting one coherent feature rather than an arbitrary mixture.

Feature visualization can manufacture false polysemanticity

Gradient-optimized visualizations may fall into common, easy-to-reach categories such as dogs and fail to reveal the neuron’s natural activation paths. Visualizing through upstream or downstream circuit components, and comparing natural examples, can expose the real shared structure.

Architecture and tools matter as much as model size

Residual streams, nonlinearities, normalization, and later architectural improvements can make networks easier to read. Cammarata repeatedly finds that apparent disorder disappears after fixing visualization settings, accounting for hidden dimensions, or using tools that test causal effects rather than merely reconstructing activations.

Modern language-model neurons look mostly structured

The essay reports preliminary audits in which roughly 90% of Qwen3-32B neuron poles appeared to have one main job, another roughly 7% had two related jobs, and only 2–4% remained unexplained. These are approximate, tool-dependent judgments, not a standardized result.

Strong activations often reveal the neuron’s main role; weak activations tend to add analogies, neighboring concepts, or circuit spillover. Ablating large amounts of weak activation reportedly has little effect compared with removing a small amount of strong activation. Weak behavior should still be traced when it matters, but it is not automatically evidence of superposition.

Practical takeaways

Caveats and counterpoints

Questions worth revisiting

Return to this when…

Return to this essay when deciding whether to use sparse autoencoders or other decompositions, when a neuron looks arbitrarily polysemantic, or when evaluating optimistic claims about mechanistic interpretability. Its most useful reminder is methodological: apparent messiness may reflect a bad microscope, a bad model organism, or a human labeling bias rather than the network’s actual organization.

References

  1. Original Neural Networks Are Already Clean