The LLM black box is a practical problem: a model can answer a difficult question in seconds while leaving us unable to explain which internal computations produced it. On October 15, 2026, Nick Cammarata, a coauthor of OpenAI's 2023 neuron study, published Neural Networks Are Already Clean as a Goodfire researcher. He argues that this "black box" may be easier to study than the field assumed. The claim deserves attention. It also needs a careful reading: the essay is a working hypothesis, and its companion technical paper on language-model neurons is still forthcoming.
The useful question is whether researchers can move from an interesting description of a neuron to a reliable explanation of a model's behavior. Cammarata offers new evidence for the first step and several causal demonstrations of the second. Neither amounts to a complete map of a large language model.
For a plain-English summary, jump to the simple version.
Why does the LLM black box persist?
Researchers know the mathematical operations in a trained network. The hard part is explaining how millions of learned components combine to produce a particular answer. A single artificial neuron may activate for examples that appear unrelated. If it responds to both a cat's face and the front of a car, what does it actually represent?
One influential answer was superposition: a model might pack unrelated features into overlapping internal directions because it has limited space. Researchers then train a second model, often a sparse autoencoder, to separate those features.
Cammarata argues that some of the classic evidence for this picture was misread. The apparent mixture may instead reflect a shared pattern, an imperfect microscope, or a network architecture that made later layers harder to train. His definition of superposition is specifically the packing of unrelated computations into the same mechanism; other researchers use the word more broadly for representing more concepts than there are dimensions. His essay explains that distinction.
What changed in the new research?
The familiar cat-and-car neuron came from a vision model, not an LLM. Cammarata now describes it as responding to a shared V-shaped visual pattern. In his account, the neuron groups related visual structure rather than hiding two arbitrary concepts in one slot. He also reports that after architectural changes to InceptionV1, a vision model, 99–100% of neurons in its final layer became easy to interpret while model size and training data stayed fixed. That is a claim about a retrained vision-model layer. It is not a statistic about modern chat models.
The language-model evidence is more relevant to LLMs and more preliminary. Goodfire's team estimates that in an initial audit of Qwen3-32B, about 90% of neuron poles have one main job, often with small related offshoots. Another roughly 7% have two related jobs, and about 2–4% remain unexplained.
A "pole" refers to one side of a neuron's positive or negative response; the percentage does not mean that 90% of an LLM's behavior has been explained. Cammarata says the classification changes with the threshold for weak activations, model depth, circuit density, and what counts as one job. Those qualifications appear next to the estimate.
The team offers more than labels for activations. Its interactive essay includes 1,000 randomly selected neuron case studies from Qwen3-32B and Qwen3.6-35B-A3B. In one test, an agent used written explanations to invent examples that should activate a neuron and near misses that should not. Among 183 neurons with explanations of their positive side, the new examples activated strongly 76% of the time, against 11% for near misses. That is a useful prediction test within the team's setup. It does not establish that the descriptions capture every important activation or downstream effect.
Can the researchers change an LLM's answer by intervening?
Yes, in selected examples. Goodfire traces groups of Qwen3-32B neurons and then removes them to see what changes. In a prompt whose clues point to Santa Claus, removing eight selected neurons associated with the answer lowered the model's reported probability of "Santa" to 0.059; removing the same number of random neurons from matching layers left it at 0.997. This is evidence that those selected neurons mattered for that prompt and next-word prediction. It is not a measured success rate across arbitrary prompts or models. The intervention and control are shown in the original essay.
Cammarata also shows a neuron-based attribution graph for Kimi K3, a model with multiple trillions of parameters. The wallet example follows some paths from the prompt toward an answer about returning lost property. The essay calls the displayed groups a hand-picked slice and says full audits at that scale have not begun. A readable slice of a very large model is promising, but it cannot tell us what every omitted path did.
How are other researchers responding?
On October 17, researcher Leo Gao published Neural networks are not already clean. He agrees that earlier work may have called neurons polysemantic too quickly.
His objection is that a tidy description of the strongest examples can still miss inputs that should activate the neuron, weak responses that matter, or the importance of activation strength. Gao proposes a tougher test: replace the neuron's values with values predicted from the explanation, then measure whether the rest of the model still behaves the same way. If behavior degrades, the explanation has missed something important.
Neel Nanda's direct response makes a related distinction. He sees practical value in using native neurons and attribution graphs to investigate a particular suspicious behavior. He remains unconvinced that this proves ambitious reverse engineering is close, and points especially to weak activations in LLMs and information in the residual stream. Cammarata himself says attention is less understood in this work and that enormous language models are far from fully mapped.
The disagreement is productive. Goodfire is asking whether the network's own components can be the starting point for interpretation. Gao and Nanda are asking how much explanation must predict before we can trust it. A neuron can look legible, participate causally in one answer, and still have important behavior we have not captured.
Two other reactions sharpen the burden of proof. In the same public discussion, John Wentworth asks for a neuron-by-neuron explanation of the showcased ConvNeXt vision model, with the evidence for each interpretation. He says the public microscope alone did not make those neurons' meanings immediately clear to him. That is a request for an inspectable artifact, not a test that disproves the LLM findings. Alex Gibson points to a different gap: the essay calls its largest conclusion highly speculative, while some short social posts sound more certain. These are early individual responses, not a field-wide verdict.
Does this make an AI system safe or transparent?
No safety certificate follows from these results. An explanation of one neuron is a local claim. An attribution graph is a partial account of one computation. A system-level assurance claim would need coverage of important circuits, reliable tests against missed behavior, and evidence that the method works beyond the examples chosen for inspection. For a tool-using system, AI agent permissions→ still need to limit what the model can do, regardless of how readable its internal circuits become.
Goodfire's language-model technical paper and independent replication will matter here.
There is also an access boundary. These experiments inspect internal activations and intervene on components. If your product only calls a hosted chat API, you cannot run the same tests on that model through ordinary prompts.
The simple version
A question goes into an AI model, and an answer comes out. The hard part is knowing what happened inside.
What happened: Goodfire examined small parts of a language model called neurons. In selected tests, researchers could describe what some did and change an answer by removing them.
Why it matters: The model's inner workings may be easier to investigate than many researchers expected.
What it does not mean: We cannot yet explain every answer or call the model safe. If you build with ordinary AI APIs, keep testing outputs and limiting what the AI can do. Later studies may lead to better tools for debugging and monitoring models.
What does this mean for AI products today?
For the agent and content workflows I build, this result does not give us a new switch to turn on. The immediate work is to test the tasks users actually run, check sources and factual claims, log model and tool behavior, restrict tool permissions, and require human approval before consequential external actions. Those controls address observable failures whether or not the model's neurons are legible.
You can still evaluate a hosted model's external behavior. My earlier guide to LLM confidence calibration→ covers one such task: checking whether a score tracks correctness on your own work. That measures a different question from mechanistic interpretability, and both require clear limits.
Before adopting an internal monitor, I would ask what behavior it detects on held-out work, how often it misses harmful cases or interrupts benign ones, whether it acts before a tool call, and whether the provider actually exposes the required activations. Ordinary API outputs do not answer those questions.
What could change longer term?
If other teams reproduce the neuron findings and can trace the important paths between neurons, model developers could diagnose failures at the mechanism level rather than only studying outputs. Cammarata also sketches automated research agents and carefully changing a training environment that proves harmful as possible future steps. Those are proposals to test, not capabilities established by this essay.
A related direction already has a concrete example. In a separate Goodfire study of Kimi K3, an activation probe feeds a second model that judges whether to interrupt a risky agent interaction. Goodfire reports promising results in its own evaluations. This monitor needs access to internal activations, and the study does not validate Cammarata's neuron explanations. It shows what model-level signals might eventually offer a provider or an open-weight deployment; an API customer would need an exposed service and its own task-specific evaluation.
Better internal visibility could help safety reviews and also make model development faster. The balance is unknown. A useful future standard would ask providers to show which behaviors their internal explanations cover, where they fail, and whether a monitor improves on output-only checks in realistic workflows.
What should we watch next?
I would look for five pieces of evidence in the promised technical paper and follow-up work:
My reading is that Goodfire has made a credible case for looking harder at a model's native neurons. It has also given researchers better experiments to argue about. As of October 17, 2026, the strongest conclusion is narrower than "the LLM black box is solved": some parts may be more readable than expected, and the route from readable parts to reliable whole-model explanations remains open. For teams building on hosted APIs, behavior tests and permission boundaries remain the practical starting point.
Primary sources: Cammarata's research essay, Gao's response, the direct researcher discussion, and Goodfire's separate activation-monitor study.
