Mechanistic interpretability is one of the most important bets in AI right now. It might also be one of the most honest.
By mechanistic interpretability, or MI, I mean the effort to reverse‑engineer what an AI model’s doing inside. I don’t mean just watching what it says. I mean trying to understand the internal computations that produce those outputs: features, representations, circuits, and algorithms. In plain terms, I mean trying to read an AI’s mind.
That sounds simple enough. It isn’t. The real task is brutal. We don’t just want to study what models do. We want to understand what they are. We’re treating a neural network like compiled code without source, and trying to recover the logic from the outside.
If we could do that reliably, it’d change everything.
Why I think it matters
The first reason is safety. We train models on outcomes. We reward the answer, not the process that led to it. That works until a model learns to be useful in a deceptive way, or learns a proxy goal that looks fine in training but goes wrong in deployment.
Without interpretability, we’re left with behavior tests. Those matter. But they’re not enough. You can’t prove that something lacks a bad capability just by poking at it a lot. You can only fail to trigger it. That’s a weak standard. It’s a bit like trying to prove software has no backdoor by running a few tests and hoping for the best.
Interpretability gives us a stronger kind of evidence. Not just, “It didn’t lie in 10,000 tests,” but, “We can see whether the internal machinery linked to deception is present or active.” We’re not there yet. But that’s the right goal.
The second reason is capability. A lot of people talk about MI like it’s only a safety tax. I don’t buy that. If we understand how models store facts, do arithmetic, carry out chain‑of‑thought, or copy information, we can make them smaller, faster, easier to edit, and less likely to hallucinate.
The work on induction heads, grokking, the key‑value memory view of MLPs, and sparse autoencoders that find interpretable features isn’t just safety theater. It’s real science. It’s how we stop treating a model with 100 billion parameters like some kind of magic lump.
There’s also a philosophical payoff. MI forces us to ask what it means for a thought to be inside a model. Are features linear? Are concepts mixed together? Is there one right level of description? The answers are stranger than the old 2017 view of deep learning, and more interesting too.
Where I stay skeptical
I spend a lot of time reading MI papers, and I keep coming back to three worries.
The first is scale. We can explain a tiny two‑layer model. We can even find a few hundred interpretable circuits in GPT‑2 Small. But when we move to frontier models, the picture gets messy fast. Sparse autoencoders can give us millions of features, and many of them are noisy, overlapping, or just artifacts of the method. We still don’t have a principled way to tell whether we’ve explained 1 percent of the model or 90 percent.
The field hasn’t yet crossed the gap from something like a simple animal to something like a mouse, let alone to anything human‑like.
The second worry is that interpretability may be looking in the wrong place. Human concepts may not be the model’s natural structure. We’re looking for English words inside a system that may be “thinking” in high‑dimensional vector math. Some of it will be legible. Some of it won’t. The most important parts could be distributed, alien, or encoded in a way that loses something when we force it into human language. Trying to understand that only through circuits could be like trying to understand a GPU by looking for tiny gears.
The third worry is time. Even if MI does scale eventually, it may not scale as fast as the systems it’s supposed to explain. New architectures, new training methods, and new capabilities may change the internal structure faster than researchers can map it. A method that works on yesterday’s model may still be useful science, but not tomorrow’s safety system.
Before AGI
People usually split into two camps here.
One camp says safety‑relevant interpretability has to come first. Building powerful systems we can’t inspect is reckless, they say, and I think they’ve got a point.
The other camp says interpretability’s moving too slowly to serve as a hard gate. Capabilities work will keep going, so safety has to rely on a mix of behavioral tests, control methods, governance, monitoring, and other tools. I think they’ve got a point too.
My own view is that both sides are half right.
I don’t think full mechanistic interpretability can or should be a hard stop for all progress. That gate won’t hold, technically or politically. Research will continue. Capability will keep advancing. If we demand a total moratorium until we can fully decompile GPT‑4, we’ll probably get neither safety nor a moratorium.
But I do think we need a meaningful level of progress before we build systems that are highly autonomous, self‑improving, or given irreversible authority.
That distinction matters. “AGI” is about capability. It asks whether a machine can think like a human, or close enough to it. That’s an abstract and fuzzy standard. “Irreversible authority” is about power. It asks whether a machine can enforce outcomes that humans can’t easily undo.
That’s the real boundary.
A system doesn’t need to be generally intelligent to seize control of a power grid, a financial market, or an automated weapons system. A sandboxed human‑level mind may be impressive, but if it has no real‑world leverage, it’s not the same danger. The real risk isn’t when a machine understands the world. The risk is when it locks in a reality we can’t reverse.
So the question isn’t whether we can explain every neuron. That’s the wrong bar. The better question is whether we can do the following:
- Detect deceptive planning or hidden objectives at the representation level.
- Recognize when a model’s identifying its overseer, noticing that it’s being evaluated, or deliberately sandbagging.
- Track stable objectives or planning tendencies over long periods.
- Tell whether fine‑tuning actually removed a dangerous capability, or only buried it.
- Show that these methods still work when the model knows it’s being inspected, and when we apply them to new models or unfamiliar situations.
That fifth point may be the most important of all. If the method works only when the model’s unaware of it, then it’s not yet a safety system. It’s a lab trick.
There’s real progress here. Work on hidden‑objective audits, internal reasoning representations, and natural‑language descriptions of model activations does suggest that the field’s moving forward. But most of that progress still happens in controlled settings. The success rates are still far below what real deployment would require.
My current view
I don’t think we need to finish mechanistic interpretability before AGI. But we do need to prove that the approach works well enough to catch the failures that could make advanced AI catastrophic.
If we can’t do that in the next generation of models, then we shouldn’t build the one after it.
MI doesn’t need to explain every neuron. It needs to reliably expose deception and misaligned power‑seeking before we hand models irreversible authority.
That’s how I see the field now. The best use of MI isn’t as a stop sign. It’s as an instrument panel. It tells us how fast we’re moving. It tells us how little we understand the machinery. And it tells us whether something dangerous is forming beneath the surface.
Right now, the warning light still looks red.