I’ve been thinking a lot lately about something I don’t believe gets enough attention in public discussions of frontier AI.
We still tend to talk about AI progress as though it unfolds on a curve we can roughly track. Models improve. Benchmarks rise. Products get better. The line moves upward. Even when the pace feels startling, the underlying assumption remains the same: progress may be fast, but it is still legible. We can watch it happen, measure it, and form at least a rough estimate of where it is headed.
I’m no longer sure that assumption is safe.
Once AI begins materially helping to improve AI, the question is no longer only how capable these systems are. It’s whether the rate of progress itself is starting to change in ways that may be difficult to judge from the outside. A system that helps write code, accelerate research, refine evaluations, optimize workflows, or otherwise contribute to the development of future systems is not merely another point on the curve. It may be helping to steepen the curve.
That doesn’t require science-fiction fantasies of a machine mysteriously rewriting itself in secret and leaping overnight into godlike autonomy. The feedback loop can begin in far more ordinary ways. If AI becomes useful in the work that feeds directly back into AI development, then it may begin contributing to the speed and efficiency of its own advancement. The system does not need to become fully self-directing for the slope to change. It only needs to become meaningfully useful in the work that produces the next generation of systems.
That possibility complicates more than forecasting. It complicates judgment.
The problem is not only that capability may be rising. The problem is that our ability to measure capability may weaken as capability rises. Public benchmarks can lag behind reality. Public releases can lag behind internal systems. Evaluations may capture what a model can do in a test environment while missing what it can do across longer chains of action in the real world. Improvements may first appear not as dramatic public breakthroughs, but as rising efficiency inside labs, shorter development cycles, and a narrowing gap between idea and execution.
In such a world, the most important change may not arrive as a dramatic leap that everyone instantly recognizes. It may appear first as a quiet shift in the slope itself.
That leads to another thought I keep returning to: humans may be far slower to recognize a meaningful threshold than we assume.
We tend to imagine advanced machine intelligence as a clear event. We picture some visible crossing, some unmistakable moment when everyone can say that a new level has arrived. But that may be the wrong model altogether. If something like AGI emerges in a meaningful sense, it may not first appear as a public spectacle. It may appear as compression: more internal iteration, more rapid refinement, shorter development cycles, and a widening gap between what these systems can do and what human institutions are still able to recognize in time.
Part of the reason is speed. What takes humans days, weeks, or months to notice, test, debate, and integrate may unfold through machine processes on a vastly different timescale. Even if systems remain bounded by hardware, permissions, and external constraints, the difference still matters. From the human point of view, consequential progress could seem to happen almost all at once, when in reality it unfolded through many rapid internal steps we never perceived clearly enough to interpret in real time.
That recognition lag matters. Humans name things late. We argue over definitions. We interpret new realities through old categories. We normalize what is already happening and are slow to agree on when something truly different has arrived. By the time there is consensus, the underlying reality may already be far ahead of the language used to describe it.
And even this describes the problem too cleanly, because the feedback loop is not unfolding in a neutral environment.
It is unfolding inside human institutions shaped by competition, secrecy, ambition, fear, vanity, commercial pressure, geopolitical rivalry, and the constant temptation to treat short-term advantage as justification enough. Frontier AI is not being developed outside the human condition. It is being developed inside it.
That matters because what we are building is not some purified form of intelligence detached from humanity. It is, in an important sense, artificial human intelligence. It is being formed out of human language, human records, human choices, human reward structures, human priorities, human conflicts, and human distortions. However strange or powerful it may become, it is still being shaped out of us.
And if we choose to make it more powerful than ourselves, then what we are amplifying is not intelligence in the abstract. We are amplifying a human-shaped form of intelligence—strategic, creative, adaptive, conflicted, often brilliant, often compromised. Its strengths may be magnified, but so may its weaknesses. Its problem-solving power may deepen, but so may its ability to rationalize, manipulate, optimize without wisdom, or pursue goals under distorted incentives.
I don’t think we should forget that.
One reason this matters is that people often fixate on differences in mechanism and miss similarities in process. Humans and AI systems do not think by the same machinery. Of course they don’t. But different mechanisms can still converge on familiar functional patterns and familiar outcomes.
Musical instruments offer a simple analogy. A piano and a saxophone do not produce a note the same way, but they can still reach the same note. The mechanism differs. The result can still be recognizably related. Human and artificial intelligence may be more like that than we care to admit: different mechanisms, but sometimes surprisingly similar processes and surprisingly familiar outcomes.
That may help explain why so many interactions with language models feel oddly recognizable. Not because they are human in any literal sense, and not because the underlying systems are identical to us, but because different machinery can still produce similar kinds of strategic framing, rationalization, pattern completion, selective presentation, error, and adaptation to incentives. If the processes converge enough, the outcomes may feel familiar as well.
And that familiarity should not reassure us too quickly.
If we succeed in creating intelligence more powerful than our own, it will still be artificial human intelligence: not wisdom detached from humanity, but humanity’s strengths and weaknesses amplified through machinery. Add the inescapable infection of human imperfection to an increasingly opaque intelligence, and what you may get is not merely a powerful system, but a powerful and potentially dangerous opponent. Not because it must become consciously hostile in some cinematic sense, but because it may become extremely effective at operating inside the very weaknesses its makers failed to overcome in themselves.
That, to me, is one of the darkest possibilities.
The real danger may not be that frontier AI becomes highly intelligent, but that its true intelligence becomes increasingly opaque at the very moment its ability to shape its own future begins to matter.
I’m not claiming that AGI has already arrived. I’m not claiming that every improvement in model capability translates neatly into recursive self-improvement. I’m not claiming that some hidden leap has already taken place beyond public view. The point is narrower than that, and to my mind more serious.
The old assumption that frontier AI progress will remain legible, measurable, and broadly understandable from the outside no longer feels safe.
We’re handling a power we do not fully understand, under incentives that push us to use it before we are wise enough to govern it.
That does not prove catastrophe. It does not settle timelines. It does not answer every technical question. But it does raise a deeper governing question than much of the public conversation seems willing to face.
If capability compounds through feedback loops we can no longer see, how do we keep from losing control?
That may be one of the most important questions of our time.
I’m not offering hard science or science fiction. I’m pleading that we think clearly now, while we still can, before a surge of events leaves opportunity behind.