The discussion about the future of AI is being conducted within the wrong intellectual framework.
The prevailing story says that AI systems are becoming steadily more intelligent, that at some point they will reach something called AGI, that thereafter they may become "superintelligent", and that the great questions are what these increasingly intelligent entities will do for us or to us.
It is a remarkably crude model. It mistakes one property of a technological system for the whole phenomenon.
Intelligence is not the master variable
There is no single axis called intelligence along which systems simply move from dim to clever to superhuman.
A system may be extraordinarily good at formal mathematics and poor at recognising which mathematical problem is worth solving. It may possess encyclopaedic factual knowledge while repeatedly mishandling an ordinary building project because it has lost one crucial fact from the context. It may generate a hundred plausible hypotheses while having no good basis for deciding which one deserves six months of experimental work.
This was already obvious in classical symbolic AI. A decent resolution theorem prover could generate thousands of propositions which followed perfectly legitimately from its axioms. Almost all were worthless. The problem was never merely generating consequences. It was recognising significant consequences: no algorithm for that!
Mathematics and science operate in precisely this way. What counts as important is not encoded in the axioms. Scientific significance depends upon a much wider human ecology: existing problems, experimental evidence, aesthetic judgement, tacit knowledge, disciplinary fashions, rival research programmes and accumulated experience about where intellectual leverage is likely to be found.
Kuhn understood something here which contemporary AI rhetoric largely forgets. Science is not a proposition-generating machine. It is a historically evolving community possessing standards of relevance and periodically revising those standards.
LLMs have absorbed the results of those judgements. They can therefore reproduce them remarkably well. But that is not the same thing as belonging to a community which creates, tests and revises them through engagement with the world.
Inference is not judgement. And judgement is not simply inference with the dial turned further clockwise.
Being clever is not the same as being competent
There is another dimension which becomes painfully apparent when using AI for ordinary professional work.
Suppose an AI is helping to manage a building project (I'm thinking of my personal use-model here). The essential problem is rarely whether it can solve a difficult equation. It is whether it remembers that the building is listed, understands that one particular room must remain accessible, notices that a quotation is six months old, realises that the surveyor has not yet approved something, and asks the obvious question when one of those facts is missing.
Present systems are still shockingly poor at this sort of situated competence. They proceed when they should stop. They fail to recognise holes in their own knowledge. They silently discard context. They produce an intelligent answer to the wrong problem.
For serious users this is not peripheral. It undermines trust.
I would therefore happily take substantially more capable models. But what I want is not merely a system which scores another ten percentage points on some reasoning benchmark. I want one which has enough contextual intelligence to know what matters, enough self-knowledge to recognise when it does not know something, and enough judgement to ask before marching confidently down the wrong road.
That kind of improvement would have considerable economic value.
There is more than one AI market
This immediately changes the way one should think about the foundation-model companies. The standard horse-race narrative asks which company is ahead in "AI". But Google, OpenAI, Meta and the Chinese companies are not running the same race.
- One race is the frontier-model race: make the underlying cognitive machinery substantially better.
- Another is the industrialisation race: take systems which are already good enough and integrate them into the economy.
Those strategies produce very different incentives.
Google, for example, sits inside an enormous technological empire. It can make money from models through search, advertising, workspace, cloud computing, chips, data centres, enterprise software and innumerable forms of integration. It does not necessarily need the world's cleverest conversational intelligence (and indeed it doesn't have it). It needs models sufficiently good to increase the value of the rest of Google.
The same broad logic can apply in China. The strategically important question may not be whether some Chinese model defeats everybody else on an abstract intelligence leaderboard. Cheap, competent models integrated throughout manufacturing, logistics, administration, robotics and consumer services might be considerably more important in boosting the objectives of the Chinese state.
Meta has still another position. Its interests include enormous distribution platforms and the surrounding ecosystem within which AI becomes ubiquitous. For Meta, AI is a tide which lifts all its boats.
OpenAI is structurally different. It has much more riding on the proposition that the model itself can continue becoming sufficiently better that people will pay a premium for superior capability. That gives OpenAI a stronger reason than Google to pursue improvements at the cognitive frontier.
There is therefore no single "AI industry strategy". There are competing strategies for extracting value from different layers of the technological stack.
The economist Schumpeter is much more useful for understanding this than AGI mythology. Firms are searching for temporary rents produced by innovation. They occupy different positions in an evolving industrial structure and therefore rationally pursue different technological trajectories.
From Schumpeter to Olson
But firms themselves are only part of the story.
Once AI is placed inside society rather than imagined floating above it, Mancur Olson becomes your go-to economist. Here is what he has to say: human societies consist of actors and organised groups attempting to advance their interests. Corporations, bureaucracies, professions, armies, political parties, trade associations and states acquire resources and use them to alter outcomes in their favour.
Intelligence helps them. But intelligence is only one component of effective agency.
A successful human actor needs objectives, persistence, contextual knowledge, relationships, resources, institutional position, knowledge of other people's motives and some capacity to make events turn out differently.
That is why extremely intelligent people routinely lose political and organisational battles to less intelligent people who understand the institution better.
Why doesn't this mature insight from economics transform the AI discussion?
Current LLMs are not actors in this Olsonian sense. They have no independently established interests. They hold no property. They possess no constituency. They do not ordinarily decide what they wish to achieve and then spend years building coalitions and acquiring resources to achieve it.
They inherit their objectives from us, being astonishingly sophisticated tools embedded inside the struggles of existing human actors.
The tool-to-actor transition
There is nevertheless a meaningful boundary worth watching. It simply isn't "AGI".
It's the transition from tool to actor.
An AI begins to resemble an actor when it possesses persistent objectives, remembers them over extended periods, acquires resources, acts independently in pursuit of them, negotiates with other actors, protects its capacity to continue operating and perhaps produces successor systems.
None of those things requires consciousness per se. Nor does an artificial actor have to possess human emotions or experience something corresponding to ambition. Think of a plague of locusts; or just a plague.
Conversely, none of these agentic perils follows from solving harder mathematics problems.
This is why the notion that an increasingly intelligent chatbot simply continues getting cleverer until one morning it decides to exterminate humanity is completely unserious.
Embodiment matters, although not necessarily biological or robotic embodiment. A sufficiently autonomous system could acquire a perfectly serviceable social body through bank accounts, cloud systems, communications networks, contracts, software agents and human intermediaries. Corporations already act without having a single biological body.
But somebody would have to construct and empower such a system.
That point is routinely skipped.
Who would pay for an artificial actor?
AI systems do not descend upon civilisation from some independent evolutionary tree. They are extraordinarily expensive products of human institutions.
The central question is therefore: who is paying for this capability, and why?
A corporation might pay for an autonomous purchasing agent because it saves money. A military might pay for autonomous targeting or cyberwarfare because it believes that provides strategic advantage. A financial institution might fund autonomous trading systems. A state might build systems for surveillance and political control.
Capability is developed because some organised human interest expects to gain from it. AI always firmly fits into political economy.
Even the evolution of more agentic AI is likely to proceed through institutional demand. Systems will acquire greater autonomy because somebody, somewhere finds that autonomy useful.
And therefore the immediate dangers are not mysterious artificial desires emerging from nowhere. They are perfectly familiar human desires amplified by unfamiliar machinery.
The most plausible dangers are thoroughly human
There will certainly be AI accidents. Complex technologies fail. Cars crash. Aircraft fall out of the sky. Chemical plants explode. Software brings banking systems down. Medicines have unexpected effects.
AI will generate its own catalogue.
Some failures will be serious. A trading system may destabilise a market. An autonomous military system may misidentify a target. An AI-controlled industrial process may make some disastrous optimisation. Large automated bureaucracies may make terrible decisions at enormous scale.
But technological accidents are not usually species-threatening because accidents have no strategic objective. Humans identify the problem and then design it out.
A petroleum refinery does not respond to investigators by secretly constructing another petroleum refinery.
For genuinely catastrophic autonomous AI, considerably more is required. A system would need some combination of persistent objectives, operational independence, resource acquisition, resistance to control and most likely the capacity for independent replication. Its objectives would then have to conflict sufficiently strongly with ours for it to undertake hostile action. It's conceivable but it's not a straightforward extrapolation from better language models. Not at all.
Military development supplies one possible path because military institutions have unusually strong incentives to construct systems capable of acting independently against an opponent. Cyberwarfare provides another possible route. Systems designed for extraterrestrial exploration and autonomous industrial replication supply a more exotic one: a machine intended to establish an autonomous ecology on Mars would necessarily possess capabilities very different from those of ChatGPT.
But we are now discussing particular engineering trajectories, social, political and institutional choices; not an intelligence singularity.
The more immediate problem is ownership
There is also something slightly perverse about obsessing over the hypothetical future motives of machines while largely ignoring the motives of the organisations presently building them.
Amazingly, we already possess autonomous agents with enormous resources, persistent objectives and an impressive capacity to manipulate human beings.
They are called corporations and states.
Give those actors vastly better surveillance, persuasion, military planning, administrative control and economic optimisation may be the most consequential political effect of AI over the next few decades. We'll need to ask:
Who possesses it, what do they want, and what additional power does it give them over everybody else?
That is an Olsonian question rather than an AGI question.
A different paradigm
The better framework therefore begins by abandoning AGI as the organising concept. Instead consider at least four variables.
- There is cognitive capability: what intellectual tasks can the system perform?
- There is judgement: can it identify what matters, recognise missing information and choose worthwhile objectives or lines of enquiry?
- There is agency: can it pursue persistent objectives through the world?
- And there is institutional embedding: who owns it, who funds it, which systems can it control and whose interests does its behaviour advance?
AI development then stops looking like a single upward-sloping curve marked INTELLIGENCE. Instead we see normal technological development: a messy process in which different capabilities emerge, firms pursue different rents, institutions grab useful technologies, organised interests deploy them against one another, and governments intervene when consequences become sufficiently important or seductive.
The likely future
Frontier models will continue becoming more cognitively capable. Some customers will pay heavily for that additional competence, particularly where errors are expensive and intellectual quality genuinely matters. Most ordinary people with real, complex problems would value more general smartness - I would!
Elsewhere, existing models will become cheap enough and reliable enough to disappear into ordinary economic infrastructure. In those applications good enough will win.
At the same time, companies and governments will push selected systems towards greater agency where agency produces useful results. The important risks will emerge from those particular deployments rather than from intelligence in the abstract.
And always the decisive actors will remain human institutions pursuing thoroughly human objectives.
Perhaps eventually we really will create an autonomous artificial species capable of reproducing itself, acquiring resources and competing successfully against humanity. If so, that would unquestionably change everything. That might well be the stupidest decision in the history of humanity, but who am I to judge?
But there is no intellectual justification for pretending that an unusually good large language model is already halfway there.
We are not watching intelligence escape into the world. We are watching existing human actors acquire an extraordinarily powerful new class of tools.
And anyone wanting to understand what happens next would probably do better to read Schumpeter and Mancur Olson than another scaremongering/boosterist but undeniably vapid manifesto about AGI.
This is a GPT-5.6 (High) write-up of an extended rant I directed at this undeserving LLM. I was irritated by Mark Zuckerberg's absurdly self-serving, Pollyannaish piece about the ever-brighter future awaiting ordinary people beneath the benevolent cloak of Meta's upcoming personal AI.
Yep.
No comments:
Post a Comment
Comments are moderated. Keep it polite and no gratuitous links to your business website - we're not a billboard here.