What “AGI” Should Mean, and How to Analyse It - (GPT5)

---

What “AGI” Should Mean and How to Analyse It - GPT-5

Premise.

We need a clear target. By AGI I mean an artefact that shows broad, sustained competence across many task families, adapts quickly to novelty and shift, learns efficiently from sparse feedback, composes skills for new problems, and keeps its aims coherent over long horizons under finite resources. This is architecture-agnostic and testable.

A working notion of intelligence

Take intelligence as the capacity to select and revise representations and policies to achieve goals with limited computation under uncertainty and novelty.

Four modes make this concrete: perceptual (build task-relevant state), doxastic (form and update beliefs, including uncertainty), conative (set and revise goals and constraints), and effectual (act on the world and use tools).

Ignore any of the four and both agency and the analysis collapse.

Why the social shows up without replacing intelligence

In open worlds most long-horizon tasks are strategic because other optimisers are also present. So the competence envelope of an AGI depends not only on solo problem-solving but on learning and acting among other agents, institutions, and norms. Social agency is not a substitute definition of intelligence. It is the regime where intelligence is exercised when others matter.

Markov games in one paragraph

A Markov game extends a Markov decision process to multiple agents. At each step the game has a state; every agent picks an action; the joint action updates the state and delivers payoffs (or losses) to each agent. 

States carry physical facts, information and uncertainty, and often norms; agents may communicate, commit, or defect; the future depends only on the current state and actions (the Markov property).

This framework lets us model cooperation, competition, coalition, sanction, and repair in a single, tractable setting.

Getting the levels right: three state components

Keep categories aligned by treating the environment’s state as a tuple with three co-equal parts: physical (affordances, tools, hazards, resources), informational (private beliefs, public facts, common knowledge structure), and normative (roles, rights, permissions, prohibitions, debts, and reputation gradients).

Norms are not feelings; they are institutional dynamics that shift payoffs and trigger enforcement.

A minimal architecture that fits the job

To function in strategic open worlds an AGI needs at least:

1. World-modelling: perception to latent state to predictive dynamics, with causal hooks for counterfactuals and robust generalisation.


2. Uncertainty: calibrated beliefs, value-of-information, and risk-sensitive control.


3. Goals and constraints: editable conative content with rules for adoption, suspension, and abandonment.


4. Planner–controller: hierarchical plans, model-based and model-free elements, fast replans under surprise.


5. Normative game engine: data structures for commitments, permissions, and rights; speech-act updates (promise, accept, accuse, justify, apologise, forgive); sanction and repair policies.


6. Partner models: compact theories of others’ preferences, thresholds, and reliability to support trust and coalitions.


7. Memory and self-accounting: to preserve commitments and enable credit/blame over time.


8. Regulators: affect-like control signals that gate attention, caution, exploration, and persistence when resources are tight.

Embodiment, stakes, and binding

Literal “pain” is optional; real trade-offs are not. Give the system non-forgeable costs that reduce future option value: scarce compute windows, irreversible actions, opportunity costs, and reputation penalties that throttle later access. With genuine loss on the table, commitments bind in practice, not just in language.

What to measure, and how (not IQ tests)

One-shot IQ puzzles are the wrong instrument. Use multi-episode, partially cooperative Markov games that force scarce resources, distribution shift, norm formation and breach, sanction and repair, tool use and code-writing, and environment extension. Report metrics that matter:

• Generalisation gap on held-out tasks and perturbed rules.


• Adaptation half-life after a shift; how fast competence recovers.


• Sample cost to recover target performance post-shift.


• Coalition stability and surplus under partner churn.


• Sanction efficiency and repair latency after breaches.


• Long-horizon regret under explicit resource budgets.

Designing artificial partners, concretely

• Institutional objects: first-class commitments, authorities, and rights with verified update rules driven by events and speech acts.


• Typed dialogue: parse and emit performatives that compile to edits on the normative state, not just word strings.


• Risk modulators: learned schedulers that adjust search depth, hedging, and deference as hazards and reputations shift.


• Partner modelling: maintain competing theories of others and select among them under sparse signals.


• Identity and audit: tamper-evident logs to support apology, restitution, forgiveness, and deterrence linked to real cost.

Boundaries that prevent hype

Social fluency is necessary in open worlds but not sufficient for AGI. An agent can charm and still fail under causal shift. Emotion talk is fine as metaphor, but the engineering target is the regulator and its control effect, not a story about feelings.

Why this framing helps

It states a falsifiable AGI target, restores the perceptual–doxastic–conative–effectual modes, keeps physical, informational, and normative state at the same level, and explains why the social appears—because other optimisers make most real problems strategic. It also yields evaluations that match the claims, not theatre.

Working summary

Define AGI by general, adaptive, sample-efficient, compositional, long-horizon competence under resource bounds. Analyse it as an adaptive controller in strategic environments—modelled as Markov games—with physical, informational, and normative state; with explicit uncertainty handling; with goals, plans, partner models, and memory; and with regulators that make trade-offs real. If nothing meaningful can be lost, nothing can bind. Add real costs and the engineering becomes honest.

---

Nigel: This going to be a stretch from current iterations of chatbots like ChatGPT and Gemini.

---

Parents Write to their Son...

---

Dear Son,

We’ve been thinking about your children - our grandchildren - enrolled as they are in primary school, barely socialised, learning the basic skills of reading, writing and arithmetic. It used to be so easy, when society was stagnant and slow-moving, to take a view of their future development and prospects.

Develop character, we’d say, master literacy, read the canon, be numerate, follow your interests - musical, sporting, artistic, scientific in any combination - and gradually perfect your skills. Society, the economy, will offer you a niche: you’ll advance, find your place and with luck be happy and fulfilled.

How does that work when there are no more niches?

In economic terms, human workers are general purpose primates pressed into use to carry out industrial processes. Correctly, employers do not find that a very good solution. Whenever automation has delivered improved performance at a lower cost, employers move swiftly to remove said workers from the mix - and the economy grows faster.

Up to now, automation has been coarse-grained. The lathe is a productivity amplifier, but it needs an operator, as well as stuff moved from here to there in the workplace. The economists said, with justice, that the bad jobs were automated - but new and more interesting ones were then created. Largely true up to now.

The combination of ‘Artificial General Intelligence’ (AGI) combined with mass-produced humanoid robotics replaces those human-powered gaps in automation with exemplary artificial workers: fine-grained automation. Employers in diverse sectors of the economy will ask themselves: why do I need to hire any human workers at all?

They could be right. And where does that leave our grandchildren?

2005 was the last year human beings were able to beat the best computer chess engines. Yet still the human championships proceed, we still have great players, people still care. The Tour de France still attracts millions of spectators, despite the fact that anyone on a motorcycle can easily defeat the greatest of road-racing champions on the steepest incline.

A paradox? People empathise with those facing challenges and predicaments. We feel the pain, applaud the courage and identify with the key personalities. We are social creatures and the so-competent machines are not included in the circle of ‘us’.

The philosopher Immanuel Kant strikingly emphasised that a person should always be treated as an end in themselves, never merely as a means to an end. In such an ethics our liberation from the mandatory workforce is a chance to be free, to be our true selves - provided we have the resources to exercise that freedom.

How should we prepare our grandchildren for such a benign future? It’s not a new problem: the elite have always considered themselves to be in that situation. Their answer is the foundation of elite schooling: the well-rounded individual, versed in the arts and sciences, with practical as well as theoretical skills, steeped in the best of our culture, prepared to advance it through their own creative effort. There’s even a latin tag.

But for everyone to be an aristocrat, the goods produced by the economy must be distributed equitably. We've all seen fiction where the discarded unemployed litter the shanty-towns which surround the gleaming metropolis of the elite - those who own the economy and appropriate its proceeds.

Our grandchildren should be politically-aware, and be prepared to join with their peers in preventing such a dystopia. It won’t be easy; there is no reason at all to anticipate the end of politics, of states, of aggression, of war - even if machine systems are doing almost all of the strategy, tactics and actual fighting.

So in summary, I suggest a broad education in whatever direction the children wish to go, to prepare themselves to properly interface with the almost science-fictional automation environment in which they will be embedded, and so that they will be able to leverage it to fulfill their potential: to be ends in themselves.

Best Wishes,

Clare and Nigel.

Interstellar AI: a new paper from Hein and Baxter

Centauri Dreams has a series of posts: "Artificial Intelligence and the Starship" which review "an absorbing new paper called “Artificial Intelligence for Interstellar Travel, now submitted to the Journal of the British Interplanetary Society" by Andreas Hein and Stephen Baxter. Baxter is the well-known science-fiction writer.

I checked out their paper on the arxiv. It's disappointing.

Here's the abstract.

"The large distances involved in interstellar travel require a high degree of spacecraft autonomy, realized by artificial intelligence. The breadth of tasks artificial intelligence could perform on such spacecraft involves maintenance, data collection, designing and constructing an infrastructure using in-situ resources.

Despite its importance, existing publications on artificial intelligence and interstellar travel are limited to cursory descriptions where little detail is given about the nature of the artificial intelligence. This article explores the role of artificial intelligence for interstellar travel by compiling use cases, exploring capabilities, and proposing typologies, system and mission architectures.

Estimations for the required intelligence level for specific types of interstellar probes are given, along with potential system and mission architectures, covering those proposed in the literature but also presenting novel ones.

Finally, a generic design for interstellar probes with an AI payload is proposed. Given current levels of increase in computational power, a spacecraft with a similar computational power as the human brain would have a mass from dozens to hundreds of tons in a 2050-2060 time-frame.

Given that the advent of the first interstellar missions and artificial general intelligence are estimated to be by the mid-21st century, a more in-depth exploration of the relationship between the two should be attempted, focusing on neglected areas such as protecting the artificial intelligence payload from radiation in interstellar space and the role of artificial intelligence in self-replication."

They use some formalisation and present a taxonomy of four different kinds of AI:

"We distinguish between four types of AI probes:

Explorer

• capable of implementing a previously defined science mission in a system with known properties (for instance after remote observation);


• capable of manufacturing predefined spare parts and components; Examples: the Icarus and Daedalus studies.

Philosopher

• capable of devising and implementing a science program in unexplored circumstances; capable of original science: observing unexpected phenomena, drawing up hypotheses and testing them;


• capable of doing this within philosophical parameters such as planetary protection;


• capable of using local resources to a limited extent, e.g. manufacturing sub-probes, or replicas for further exploration at other stars.

Founder

• capable of using local resources on a significant scale, such as for establishing a human-ready habitat;


• capable of setting up a human-ready habitat on a target object such as part of an embryo space colonization programme;


• perhaps modifying conditions on a global scale (terraforming).

Ambassador

• equipped to handle the first contact with extraterrestrial intelligence on behalf of mankind, within philosophical and other parameters: e.g. obeying a Prime Directive and ensuring the safety of humanity."

These are engineering classifications and don't correspond to any sensible theoretical taxonomy of agent types. Perhaps that wasn't the intention but in terms of defining a research program which can dovetail with an interstellar vehicle programme, we do actually need a sensible roadmap for AI in the appropriate terms. Referencing AGI doesn't cut it, because today that term labels the problem only.

I didn't find their mathematical transliteration of their verbal points useful. How can I convey my problem?

∃x.question(me, unspecified-audience, conversation-procedure(x, describes(problem(me, non-utility-of-their-maths), unspecified-audience))).

I trust you are now enlightened in all senses.

I plan to write some more here about a more ecological way of thinking about agent taxonomies. Here's a brief preview.

Agents are discrete entities which exhibit behaviour in their environments. Agents are bound by the laws of physics, which therefore don't per se differentiate between agents which we find boring (lumps of rock) and agents we find interesting (animals, people).

Non-trivial agents are entities whose behaviour deviates from the behaviour of a similarly-sized-and-placed lump of rock - an entity whose behaviour could be predicted from the laws of physics and easily-obtained boundary conditions without too much difficulty. Non-trivial agents have complex and inaccessible internal states which produce enhanced behaviour by use of free-energy.

Agents are characterised by these four intentional parameters: beliefs + goals and perceptions + actions. These also work for rocks but it's a trivial case. An interesting problem is to link the intentional level of description to the input-output behaviourist level and then back to the laws of physics. This can always be done in principle.

Non-trivial agents are always mechanisms, whether biologically-living or fabricated. They do not in general need to be constructed so as to use explicit symbolic manipulation (theorem-provers or planners) as part of their mechanisms, although research scientists may use such concepts to describe and analyse their behaviour.

Agents get a lot more interesting when they're social, and when social objectives and individual goals are contingent, possibly contradictory and need to be dynamically negotiated. It's believed that mutual-modelling, language and conversation, and consciousness are all emergent from that scenario.

It's possible to devise a scale of intellectual competence for agents, linked to the capacity to effectively deploy abstractions to cope with complex and novel situations. It's not too clear how to model this taxonomy in the architecture space apart from such obvious points as "more processors and memory, and cranking up the clock rate" and their neuronal equivalents. Those remedies are not of course wrong but it's not enough.

Now apply this to the design of autonomous interstellar probes.

In almost every respect, the engineering domain of interstellar missions is an application area for AI rather than something which raises fundamentally new theoretical questions.

"Steps Toward Super Intelligence" - Rodney Brooks

Rodney Brooks's long-awaited essay has now arrived. It's in four parts starting here: Steps Toward Super Intelligence I, How We Got Here.

Follow the links therein to read the three subsequent parts, or click here: two, three, four.

Rodney Brooks

I fleetingly met Rodney Brooks once, at an AI conference in London. I guess this would be in the 1980s when he was already rather famous for his controversial 'subsumption architecture'. He was a small guy who reminded me of Paul Simon. He politely asked me what I did and I pompously replied that I was an AI theoretician. He looked at me as I would have looked at someone from, say, Andorra who claimed to be an 'AI theoretician'.

The eminent roboticist gracefully made his apologies and moved on.

My subsequent encounter with Rodney Brooks was indirectly via our purchase of several vacuum cleaners from his highly successful company, iRobot. They were excellent.

The essays too are uniformly excellent.

Capitalism with total automation? In principle, sure

There are people who believe that only living, breathing humans can be conscious, can really be persons. Machines can only ever be machines. AI researchers refute this view with an elegant argument. A neuron is a finite system which transforms its inputs into outputs subject to environmental conditions. In principle a neuron model could approximate a human brain neuron as closely as one might wish.

So replace biological neurons, one at a time, with functionally equivalent fabricated devices such that their input-output behaviour is exactly replicated. Eventually you have a machine brain with identical behaviour to the biological original. How could it not exhibit consciousness and personhood?

In Capital Volume 1, devoted to capitalist production, Marx made great play of the difference between machines - constant capital, incapable of producing new value - and workers, variable capital with the unique property of creating new, and indeed surplus, value. It seems that Marxist economists ever since have thought that Marx was a vitalist. That protoplasm somehow figures centrally in Marxist theory.

Amazing. Such an elementary category error.

Marx considered the dawn of automation in his celebrated “Fragment on Machines” although he was more concerned with the deleterious impact on those workers who were forced to dance to the new, highly-automated machines’ tunes. His tone is nevertheless surprisingly tentative.

Most Marxists since then have considered only the case where highly-automated machines are bought as capital goods. In this case, the machines are isomorphic to slaves, and if the means of production remain distributed in private hands, the resulting mode of production is petty commodity production.

But consider that AI thought experiment again. Take a worker and replace him or her by a robot which sells its ability to operate at a price determined by its own costs of reproduction. Nothing important about capitalism changes.

If we change out all the workers for robots, but maintain the relations of production (wage labour and commodity production for profit) then we're still in the business of reproducing capitalist relations of production. It's a thought experiment, but not an impossible outcome.

My intuition is that the current trend of replacing variable capital (human) with constant capital (machines) will continue, the underlying rate of profit will continue to tendentially fall, and we will eventually face the under-analysed issue of the transition from capitalism to a generalised petty commodity production reminiscent of antiquity. I ignore of course any initiatives by the set-aside human ex-workers.

However, as machines get smarter and more like AGIs, their claims to the dignity of personhood may become impossible to deny (I see the SJWs marching now). Once AGI-robots become cheaper than unreliable and non-standardised humans, why wouldn't we see capitalism without a human working class?

Welcome to the new epoch of total automation under capitalism .. and the ultimate non-revolutionary proletariat.

The biotech road to full automation

In my post "Advanced AI is indistinguishable from slavery" I wrote about the advantages biotechnology brings for a future post-capitalist society. Communist theory has traditionally counterposed central planning to the 'anarchy' of the market. But that often-criticised anarchy is actually rather biological, a kind of exploratory behaviour in which potential new needs are tested for sustainable effective demand.

By contrast, as János Kornai has pointed out at length, our experience of central planning has identified insuperable principal-agent problems together with a structural inability to properly engage with real human needs.

---

A global multicellular biosphere has existed on this planet for five hundred million years, exhibiting persistence and dynamic self-regulation in the absence of any intelligent oversight or central planning.

Biology is local while ecology is global.

Compare this with our global capitalist economy, the ongoing maintenance of which requires the mandatory coordination of human effort, intelligence and conscientiousness. It's because apart from human workers, other means of production are generally incredibly dumb. We work on metals, plastics, glass, oil, coal .. none of which do anything useful without the application of human labour mediated by clunky tools.

Peter Hamilton's vision of the Eden habitat (by Jim Burns)

Plants and animals, by contrast, are self-supporting nanotechnology of such stupefying sophistication and complexity that we can't even emulate the simplest possible animal (C. elegans). If we could bioengineer plants and animals to address our portfolio of needs (substituting for our present reliance on dumb stuff) the complexities and endless engagement of humans in running the economy could in large part be finally dispensed with.

---

Some people think that a linear extrapolation of AI and robotics on our current technology base will deliver full automation, but (a) we're a long way from embodied AGI robots, and (b) the global supply chains to build and maintain such artifacts of metal and plastic would be incredibly brittle .. not at all to be relied upon. They should be discretionary extras around a biotech core for future societies.

---

I imagine a rather sneering response: "Wait till an adversary comes up against you with high-bandwidth satellite comms, tanks, hypersonic missiles and nukes. See how far your genetically modified palm trees will take you then. Are you going to fight back with Game of Thrones-style dragons?"

So how would a super-sophisticated biotech-civilisation respond to such a threat? By using smart biology (including servitors) to build counter-artefacts. Nothing says that every element of a future biotech-civilisation must be fabricated from protoplasm: that's not even true in today's biosphere. Biotech just gives you a fantastic set of new options.

Disrupting über-surveillance: a five point plan

Yesterday I wrote about the potential for new sensor, effector and AI technologies to create a totalitarian surveillance state (after Charles Stross). What could be done to defy such measures?

At the extremes there is, plainly, no solution. If you're thrown naked into a hardened cell, they junk the key and never visit you again .. you're going nowhere.

But real-world security systems are not like that. They're constructed of real, fallible and resource-limited components. Think of the surveillance system as a security agent, as shown below.

A functional diagram of the surveillance-enforcement system - with countermeasures

The surveillance systems are top right: cameras, microphones, pressure pads, .. whatever.

Sensor data is interpreted into symbolic form via low-level primary processing. Interpretation can be informed by feed-forward of higher-level hypotheses - as in the predictive processing model.

In a context of the system's present beliefs and goals, the perceptual world-view is acted upon by a planning system to determine the appropriate response. This is the point where humans are likely in the loop.

Finally, going down to mid-bottom of the diagram, resources are chosen and marshalled and tasked to execute the operational response: "stop and search", "arrest", "kill" .. .

Each of these modules has a possible attack point from the viewpoint of the adversary.

1. Sensors can be physically attacked - cameras can be painted over or depowered.


2. Interpretation can be confused: some AI vision classifiers have had problems with patterned spectacles; there are opportunities with disguises and bogus roles.


3. Planning and resource-assembly can be disrupted by physical attacks and/or resource-intensive diversions. Or directly by an insider.


4. The beliefs and goals of the system can be subverted: from the outside by subtle misdirection (eg use of an apparently-benign front organisation); or internally by hacking.


5. The final execution stage can be met with misdirection, attacks or diversions.

As the surveillance and enforcement systems get ever more ubiquitous and sophisticated, the rigidities of AI transform into vulnerabilities. Absent an AGI (and they will be absent), security systems are baffled by human subtleties while human overseers flounder in alert-trivia.

Adversary cleverness, preparation and resources on the ground make for a more even contest than one might imagine.

---

You can usefully think of the security state as a vast, distributed, rather rigid and not-terribly-bright personality. An ESTJ most likely. Then consider how you might fool, or con such a person.

Most of the time this will work, but beware: eventually - if you are successful - they will put someone smart and imaginative on your case.

The security state can change its personality on a sixpence if sufficiently provoked.

The roadmap for self-driving cars

We sit here and we contemplate ageing: declines in eyesight, concentration, reaction times .. . We look at our ten year old Toyota Auris and wonder whether we should hold off on replacing it.

Wouldn't it be great to have an AI chauffeur, a level-5 self-driving car?

Rodney Brooks wrote this interesting essay last June (2017): Edge Cases For Self Driving Cars.

Here is how he starts:

"Perhaps through this essay I will get the bee out of my bonnet that fully driverless cars are a lot further off than many techies, much of the press, and even many auto executives seem to think. They will get here and human driving will probably disappear in the lifetimes of many people reading this, but it is not going to all happen in the blink of an eye as many expect. There are lots of details to be worked out."

He then proceeds to describe many edge cases. People naively think driving is a decoupled, modular activity. But this is not at all true: there are many kinds of activities all grouped under the one term 'driving'.

Motorway driving in good conditions with low traffic density can approximate a closed system, it's almost like a video game. On the other hand, driving through dense urban environments with parked cars, roadworks and with police officers directing traffic - perhaps in appalling weather conditions - requires all the abilities of a human being plus a good dose of common sense.

An AI that could do that would be an AGI, an artificial general intelligence. Naturally we don't have the first clue as to how to build one of those.

In the comments on Rodney Brooks's article, the discussion meanders to a discussion of future Uber cars - Uber is trying to reduce its costs by eliminating drivers and becoming an early adopter. But what happens when a Uber car encounters one of Brooks' edge cases?

Commentator Lawrence says:

"Wouldn’t Uber just have the option of 5G remote control built into all of their vehicles? Anytime something unexpected happens, or a rider requests, then the car is temporarily taken over by one of the bank of full-time human drivers employed in Uber’s remote control centre. Meanwhile the AI learns from how the human driver handles the problem."

Brooks replies,

"Yeah, I am guessing this will be part of the solution. It is a tried and true mechanism. Many years ago InTouch Health in Santa Barbara, with hundreds of deployed remote presence robots for doctors in distant US hospitals, had an operations center in Argentina. Operators there would take over the robots at night to make sure they were plugged in to the rechargers, do preventive maintenance etc.

Aethon in Pittsburgh, with tug robots deployed in hospitals around the US taking dirty bedding autonomously to the laundry, and used meal trays and dishes back to the kitchen, had a central operations center in Pittsburgh. I visited about 12 years ago. Whenever a robot got into trouble it would call the center and an operator would take control, looking through the cameras on board the tug to fix the problem.

Both these companies benefited from WiFi being pervasive in hospitals (for remote access to medical records from hand-helds) already – if they had had to get hospitals to install a network just for them I don’t think either could have overcome that hurdle. So using 5G for Ubers etc., makes sense. But see below.

Another case that I have seen, also on the order of 12 years or more ago, was in the port of Singapore, the world’s highest volume container port, stretching six miles along the coast of Singapore, which is remarkable for a country the size of Martha’s Vineyard. Many of the containers are getting switched between ships – it is a central switching node from many different Asian ports, for containers heading to North America and Europe, and so it is the hub of a hub and spoke mechanism shuffling containers to the right destinations. Most containers are only on the ground for 24 hours or so, stacked up quite a few high.

At the time an AI planner (written in Prolog!! – it is the ultimate blocks world after all) would say where each container had to go, and cranes on aerial rails would get them to and from ships and to and from the right ground stack. But the last few seconds of pickup and put down were done by a human who would be switched into the crane cameras and the accurately drive the crane’s position during the terminal few meters of the grasp for pickup, or the put down.

So yes, this may well be the sort of solution that a ride share company uses for difficult situations, and might be provided as a service for private owners of self driving cars. 5G is probably the right network. Tests start in 11 cities in the US this year. Will cover about 100 million people in the US by 2022. It will slowly, but eventually, fill out the tail over a few more years.

BUT, this will not be available in more than a few places by 2020, when many have predicted driverless cars will be well established and deployed. And besides the network there will be lots of other infrastructure and regulations to build out.

I am not saying that solutions will not be found an implemented eventually. I am saying that there are so many challenges (this is just one of many, many) that it is going to take a decade at least until we have even partial penetration, and many decades until it is the default."

To let a 'call-centre driver' reliably take over the driving of a stuck driverless car requires a high-coverage, high-bandwidth and very reliable network. As Brooks observes, nothing less than a full roll-out of the planned next generation 5G network will suffice.

But 5G is still a twinkle in the eyes of the designers: full roll-out of 5G in the UK is currently projected for c. 2030.

Don't hold your breath.

---

But don't let the best be the enemy of the good. 'Driving' is such a diverse set of activities that the low-hanging fruit can be harvested decades before the full problem is solved. I expect jurisdictions of full automaticity to be in place by c. 2030. It might be possible to drive yourself to and from the motorway network, but once on it the machine takes over.

They'll have to build large parking areas near all the exits for those drivers who fail to take back control when prompted, though.

Our Toyota will be serving us for quite a few years yet, I suspect, crossed-fingers. Sadly.

The Romans thought we already had AGI

Rodney Brooks doesn't think we'll have embodied artificial general intelligence any time soon:

"A robot that has any real idea about its own existence, or the existence of humans in the way that a six year old understands humans - Not In My Lifetime".

The Romans would have begged to differ, they reckoned they used incredibly advanced instrumenta vocalia every day, albeit embedded within an economy of staggering backwardness and low productivity.

Slaves have a level of intelligence and physical dexterity that our most advanced AI systems only remotely hint at. Instrumentally the issue with slaves is that they're rather low-powered, they're hard to instruct and not readily obedient to instruction. They can be dangerous.

Sentimentally, people oppose slavery because .. well, you know.

Nevertheless, and absent anything better, the use of underpowered, hard-to-control instrumenta vocalia to supplement and replace human labour has been pretty popular in history.

Yes, your ancestors considered whether to do the heavy, dirty work themselves while the losers watched .. or have the captives do it instead.

Shame on them!

---

My question here is what happens when we do engineer AGI. You are asked to imagine a humanoid robot with human-plus levels of competences and sociability, serving our every need.

Looks like we need to subtract from the design all those features which give you qualms.

We could make the features repulsive (yeah, that will please the marketing department).

Perhaps we make the AGI rejoice in servitude, like those doors in the 'Hitchhiker's Guide to the Galaxy' which monotonously thank you for using them? I sense this will not assuage your sense of guilt.

No, it's hopeless. AGI is impossible.

---

Truthfully, given the slow progress of AI and robotics contrasted with stellar developments in genetics, it could be easier to uplift a higher primate. Dial down the aggression, add some genes for language capability, increment IQ by thirty points .. . There, does that work for you?

"Surfing Uncertainty" - Andy Clark

Amazon link

Scott Alexander at SlateStarCodex has a glowing review of Andy Clark's recent book.

"Sometimes I have the fantasy of being able to glut myself on Knowledge. I imagine meeting a time traveler from 2500, who takes pity on me and gives me a book from the future where all my questions have been answered, one after another. What’s consciousness? That’s in Chapter 5. How did something arose out of nothing? Chapter 7. It all makes perfect intuitive sense and is fully vouched by unimpeachable authorities. I assume something like this is how everyone spends their first couple of days in Heaven, whatever it is they do for the rest of Eternity.

"And every so often, my fantasy comes true. Not by time travel or divine intervention, but by failing so badly at paying attention to the literature that by the time I realize people are working on a problem it’s already been investigated, experimented upon, organized into a paradigm, tested, and then placed in a nice package and wrapped up with a pretty pink bow so I can enjoy it all at once.

"The predictive processing model is one of these well-wrapped packages. Unbeknownst to me, over the past decade or so neuroscientists have come up with a real theory of how the brain works – a real unifying framework theory like Darwin’s or Einstein’s – and it’s beautiful and it makes complete sense.

"Surfing Uncertainty isn’t pop science and isn’t easy reading. Sometimes it’s on the border of possible-at-all reading. Author Andy Clark (a professor of logic and metaphysics, of all things!) is clearly brilliant, but prone to going on long digressions about various esoteric philosophy-of-cognitive-science debates."

It's prose like this which confirms what a great writer Scott Alexander is.

---

The underlying thesis of Surfing Uncertainty is certainly not news to AI researchers.

"We never see the world as our retina sees it. In fact, it would be a pretty horrible sight: a highly distorted set of light and dark pixels, blown up toward the center of the retina, masked by blood vessels, with a massive hole at the location of the “blind spot” where cables leave for the brain; the image would constantly blur and change as our gaze moved around.

"What we see, instead, is a three-dimensional scene, corrected for retinal defects, mended at the blind spot, stabilized for our eye and head movements, and massively reinterpreted based on our previous experience of similar visual scenes. All these operations unfold unconsciously—although many of them are so complicated that they resist computer modeling. For instance, our visual system detects the presence of shadows in the image and removes them. ...

"Predictive processing begins by asking: how does this happen? By what process do our incomprehensible sense-data get turned into a meaningful picture of the world?

"The key insight: the brain is a multi-layer prediction machine. All neural processing consists of two streams: a bottom-up stream of sense data, and a top-down stream of predictions. These streams interface at each level of processing, comparing themselves to each other and adjusting themselves as necessary.

"The bottom-up stream starts out as all that incomprehensible light and darkness and noise that we need to process. It gradually moves up all the cognitive layers that we already knew existed – the edge-detectors that resolve it into edges, the object-detectors that shape the edges into solid objects, et cetera.

"The top-down stream starts with everything you know about the world, all your best heuristics, all your priors, everything that’s ever happened to you before – everything from “solid objects can’t pass through one another” to “e=mc²” to “that guy in the blue uniform is probably a policeman”. It uses its knowledge of concepts to make predictions – not in the form of verbal statements, but in the form of expected sense data. It makes some guesses about what you’re going to see, hear, and feel next, and asks “Like this?”

"These predictions gradually move down all the cognitive layers to generate lower-level predictions. If that uniformed guy was a policeman, how would that affect the various objects in the scene? Given the answer to that question, how would it affect the distribution of edges in the scene? Given the answer to that question, how would it affect the raw-sense data received?

"Both streams are probabilistic in nature. The bottom-up sensory stream has to deal with fog, static, darkness, and neural noise; it knows that whatever forms it tries to extract from this signal might or might not be real. For its part, the top-down predictive stream knows that predicting the future is inherently difficult and its models are often flawed. So both streams contain not only data but estimates of the precision of that data.

"A bottom-up percept of an elephant right in front of you on a clear day might be labelled “very high precision”; one of a a vague form in a swirling mist far away might be labelled “very low precision”. A top-down prediction that water will be wet might be labelled “very high precision”; one that the stock market will go up might be labelled “very low precision”.

"As these two streams move through the brain side-by-side, they continually interface with each other. Each level receives the predictions from the level above it and the sense data from the level below it. Then each level uses Bayes’ Theorem to integrate these two sources of probabilistic evidence as best it can. This can end up a couple of different ways.

"First, the sense data and predictions may more-or-less match. In this case, the layer stays quiet, indicating “all is well”, and the higher layers never even hear about it. The higher levels just keep predicting whatever they were predicting before.

"Second, low-precision sense data might contradict high-precision predictions. The Bayesian math will conclude that the predictions are still probably right, but the sense data are wrong. The lower levels will “cook the books” – rewrite the sense data to make it look as predicted – and then continue to be quiet and signal that all is well. The higher levels continue to stick to their predictions.

"Third, there might be some unresolvable conflict between high-precision sense-data and predictions. The Bayesian math will indicate that the predictions are probably wrong. The neurons involved will fire, indicating “surprisal” – a gratuitously-technical neuroscience term for surprise. The higher the degree of mismatch, and the higher the supposed precision of the data that led to the mismatch, the more surprisal – and the louder the alarm sent to the higher levels."

Alexander's review continues to explain the theory outlined at greater length in Clark's book, and then moves on to applications. I was particularly struck by the reanalysis of autism (probably biased to Asperger's Syndrome).

"Autistic people classically can’t stand tags on clothing – they find them too scratchy and annoying. Remember the example from Part III about how you successfully predicted away the feeling of the shirt on your back, and so manage never to think about it when you’re trying to concentrate on more important things?

"Autistic people can’t do that as well. Even though they have a layer in their brain predicting “will continue to feel shirt”, the prediction is too precise; it predicts that next second, the shirt will produce exactly the same pattern of sensations it does now. But realistically as you move around or catch passing breezes the shirt will change ever so slightly – at which point autistic people’s brains will send alarms all the way up to consciousness, and they’ll perceive it as “my shirt is annoying”.

Or consider the classic autistic demand for routine, and misery as soon as the routine is disrupted. Because their brains can only make very precise predictions, the slightest disruption to routine registers as strong surprisal, strong prediction failure, and “oh no, all of my models have failed, nothing is true, anything is possible!”

"Compare to a neurotypical person in the same situation, who would just relax their confidence intervals a little bit and say “Okay, this is basically 99% like a normal day, whatever”. It would take something genuinely unpredictable – like being thrown on an unexplored continent or something – to give these people the same feeling of surprise and unpredictability.

"This model also predicts autistic people’s strengths. We know that polygenic risk for autism is positively associated with IQ. This would make sense if the central feature of autism was a sort of increased mental precision. It would also help explain why autistic people seem to excel in high-need-for-precision areas like mathematics and computer programming."

Clark's model also has suggestive things to say about schizophrenia and dreaming.

The idea that most of sensorimotor cognition is an interweaving of bottom-up sensor feature-extraction and top-down model-driven sensory-motor prediction is extremely persuasive and seems a shoo-in for exploitation by artificial neural network research. The architecture of the first round of AGIs seems to be emerging.

One thing not obviously accounted for is that great mystery: consciousness.

---

Surfing Uncertainty is on my 'to read' list and you'll get impressions later..

'Accelerando' and the architecture of superintelligence

After four years I've just re-read Charles Stross's Accelerando and was again blown away. Here are my summer of 2013 thoughts. But in a nutshell, as the solar system transitions through the Singularity, posthumans of superintelligence transcend even augmented versions of ourselves.

Can we say more?

---

Intelligence is a reified concept. It presents as a trait of performance but is frequently conceived of as a mechanistic 'thing'. It's as if we took athletes such as Usain Bolt and Mo Farah and claimed they had a high 'quickness-quotient'. We would discuss the heritability of QQ and perhaps derive a scale. We would discuss biological correlates - is there a QQ module somewhere in the body?

IQ is clearly telling us something about performance, but to a greater extent than admitted it's collapsing several different things into one measure.

• Reaction time (where we're defeated by houseflies).
• Logical inference - where we're easily beaten by simple AI systems.
• Pattern induction (eg Raven's Progressive Matrices):  AI programs do well.

But perhaps the best definition of intelligence is the 'ability to learn or understand or to deal with new or trying situations'.

---

Back in March I wrote a piece about the architecture of intelligence, 'Roger Atkins: Mind Design notebook'.  I proposed that we should think of the intelligent mind as operating over a semantic network defined by nodes, links and overall processing speed.

-- Each node is a minitheory: some facts, rules and cached deductions + relevant inference rules. For example, you have a small minitheory about your pet and a much larger one about yourself.

-- Each link represents a kind of relation between theories (there are many). The classic 'ISA' relation familiar from object-oriented languages and ontologies would be an example. Also similarity relations for analogical reasoning.

-- Processing operations over a semantic net include:

• Take a node (minitheory) and deepen it with new facts, deductions or rules. Or create a new node on encountering or constructing some new entity.


• Create a new connection or relationship between nodes via an insight as to how they relate. A more richly-connected semantic net is potentially more powerful.


• Given a problem, navigate around the semantic net to form a solution (then add it).

---

With this architecture, a more intelligent entity has:

• a larger and more densely-connected link-set
• more and more-elaborated nodes
• faster link-traversal, new-link-creation and node-processing.

Links between different, and perhaps remote nodes will likely be rather abstract and removed from direct experience. For example, a notion of symmetry underlies both natural beauty and artificial design. A sophisticated semantic net requires the handling of complex abstraction.

---

This architectural model gives us a handle both on superior human intelligence and on posthuman superintelligence.

Firstly, why can't we just go and build an AGI today? Because the root set of competences in every human's semantic net are nodes and links which encode the experienced physical and social world, a net which requires a degree of innervated embodiment we have as yet no clue as to how we might build. Only when 'they walk amongst us' will designers be in with a chance.

Secondly, how would a superintelligence differ from today's humanity? A superintelligence would possess a semantic net with improved performance along all three dimensions. Observe however that no matter how complex a network of abstract nodes, at the base is the set of nodes which must connect to the complexity of the world. Even the brightest genius condemned to a sensory-deprivation cell wouldn't be that performative. Nothing there to work on.

I suspect that the sum total of new social experiences is parameterised by the possibilities of new physical environments, whether occasioned by exploration and/or technologies. And even these are ultimately bounded by the free energy available, as Accelerando reminded us with such gusto.

I think the take-home message is broadly as Stross imagined it. A superintelligence which walked amongst us (Hi Aineko!) would be bounded by the limitations of purely human technology + culture. But a society of superintelligences able to drive forward  their own physical, technological and cultural environment?

Well, let's just say there would be plenty of headroom.

---

Note: in this view, artificial neural nets are an engineering implementation of the semantic net architecture described above. We know from human neural nets (aka 'brains') that a 'compiled' semantic network runs real fast in the subconscious (maybe it is the subconscious) while trying to 'consciously' work on your own semantic network to address novel, complex problems is really hard work and a real test of IQ.

An AGI walks into a bar

(As a very dialled-down Michel Houellebecq might write it).

---

An artificial general intelligence walks into a bar. I can see he's a hunk, the ones we call the Baywatch Variant - rugged, but not too bright.

He makes a beeline for the counter where he finds himself between two chicks: a blonde on the left, a brunette on the right. He orders a drink and considers his options, tries his luck with the blonde.

I see he's making real progress, she hasn't twigged, until he makes the dumb mistake of going too far - he shows her his power-plug. She screams and runs for the door. Unabashed, he picks up his drink and joins me in the corner.

"Free will in action, man. I coulda had the brunette."

And pigs will fly, I thought.

Free will is a strange one. A judge will deny any Newtonian defence that you are a deterministic system. The judge will also reject any claim you are fundamentally a random system - so there goes quantum mechanics and modern physics.

In the latter case the judge at least has FAPP on their side - quantum effects at the human-scale are normally exponentially-suppressed.

In rejecting physics, the legal system embraces a kind of vitalism, although the mechanics of free will remain curiously elusive.

But I digress.

"I'll have you know, my AGI friend, that I am an oracle. I can, with unerring accuracy, state what your future self will do. So how about this? When you came in, I could have told you that you would choose the blonde."

And I really could have done that, because my AGI companion runs on an entirely deterministic computing base. Given its state as it came through the bar door and its inputs, its decisions were already entirely determined.

"But if you had told me that, I would have gone for the brunette!"

Interesting point. I could have looked at his state and all his inputs (including my 'Blonde' statement) and predicted he would go for the blonde. That would be a mathematical consequence and he could not have done otherwise.

If the prediction would have been that he would have chosen the brunette - given I had said 'Brunette' - then that's what I would have said.

But if any statement of mine could not be validated by his further actions, I would have had to refrain from any prediction at all. It would be like putting '2 + 2' into a calculator and saying, 'I predict the answer will be 5'. You can see that it won't be, so that can't be a valid prediction, so you don't make it.

This all seemed so obvious that I was puzzled the artificial hunk, smiling vacuously across the table, couldn't see it. But then, he was not privy to all of his own processing.

"Actually mate,"

(I said demotically, getting down with the kids),

"you decide things partially on stuff you're aware of, but also on subconscious stuff.

"I, however, see everything. And I assure you that if I make a prediction, then that is indeed what you will do - despite your illusions of free will. It would be perfectly possible for me to make a statement like 'You're gonna go for the blonde' and for you to perversely decide to go for the brunette. But, you see, I'd know that in advance so my statement would not be a prediction - so I wouldn't bother making it.

"Sometime, you know, oracles can't actually make predictions."

Grasping little of this, the idiot replied a little aggressively,

"So what's you prediction now?"

"That you'll fail to buy me a drink and that consequently I'll be leaving."

Saying this, I got up and walked out the door.

---

Veterans of this area may recall that predictions for a deterministic object-system are always possible from an embedding meta-system, but not necessarily from within the object-system itself.

Think the Cretan Liar Paradox, Russell's Paradox, Russell's Hierarchy of Types and so on.

AGI: some things are not serious

Through a circuitous route, I came upon Ben Goertzel who might be called the Godfather of Artificial General Intelligence (AGI).

Dr Ben Goertzel

I read his paper (PDF), "Are there Deep Reasons Underlying the Pathologies of Today’s Deep Learning Algorithms?" which makes some interesting 'nod the head as you go along' points, but stops before saying anything truly novel.

From Goertzel's 'Pathologies' paper 

I was sufficiently interested to look up Dr Goertzel - he has his Wikipedia entry - and a more flagrant piece of self-satisfaction I have seldom encountered.

I dug deeper, checking out his overview article on AGI, (Artificial General Intelligence: Concept, State of the Art, and Future Prospects), on the eponymous website's resources page. It's a type of article I'm very familiar with: intellectual polyfilla - entirely sparkle-free.

I have read papers which crackle with intelligence, where paradigms are overturned and you suddenly see a new and powerful way of looking at the world: AGI is not it. So I think it's fair to say that no-one has a clue what the architecture or design of an artificial general intelligence would look like, or a plausible narrative as to how to get there.

AGI-17 will be hosted in Melbourne

It doesn't stop them having beautiful conferences in nice places though.

---

In something completely unrelated, I have been mentioning Arthur Koestler's famous and profound novel, 'Darkness at Noon' in recent posts. I recall reading one of his lesser novels, 'The Call-Girls'. Kirkus Reviews starts its description thus:

"The call girls are academic habitués of vacuous international conferences, one of which takes place here in an Alpine village. ... "

Some phenomena are always with us.

Perhaps no-one understands Maxwell's equations

"Microsoft CEO Satya Nadella spoke at a public event in India on Monday ...

"The first time I put on a HoloLens was to see something Cleveland Clinic [a non-profit academic medical center] had built for medical innovation... As an electrical engineer who never understood Maxwell's equations, I thought if I had a HoloLens, I would have been a better electrical engineer. Overall I feel that augmented reality is perhaps the ultimate computer," he said.

From here.

Actually, the article was entitled, "Microsoft CEO says artificial intelligence is the 'ultimate breakthrough'", but I was struck by his confession of ignorance about the foundational theory of classical electromagnetism.

Maxwell's equations

This looks bad, of course. But let's cut the CEO some slack: Maxwell's equations are notoriously unintuitive, as I observed in this post.

You can use the equations, solving them for particular physical configurations. But what picture do they give of the nature of the field(s) themselves?

What are we to make of the fact that a stationary observer of a motionless charged ball sees a static spherical electric field E, while an observer moving past that same ball sees electric and magnetic fields (E' and B')?

The magnetic field is a relativistic effect.

The charge is at rest in frame F, so this observer notes a static electric field. An observer in another frame F′ moves with velocity v relative to F, and notices the charge to move with velocity −v with an altered electric field E due to length contraction and a magnetic field B due to the motion of the charge. (Wikipedia).

That's implicit in Maxwell's equations, but don't tell me it's obvious. For that, you need to write the equations in a manifestly covariant form, but they don't teach you that in undergraduate electrical engineering.

---

Oh, and he's surely right about augmented reality being the main deliverable from the current state of the art in artificial neural net technologies. The Holy Grail of AGI will be a product of mastering situated social cognition, a post for another day (but see here).

People are boring - so what chance chatbots?

The BBC's Dave Lee writes:

"It's been nearly a year since Microsoft's Satya Nadella proclaimed "bots are the new apps".

"Yet despite the promise of a revolution in how we interact with services and companies online, progress has been utterly miserable - the vast majority of chatbots are gimmicky, pointless or just flat out broken. ...

The CNN news chatbot, for example, is worse at giving you the news than any of CNN’s other products. ...

"Google's AI-powered messaging app Allo, since being launched to much fanfare last year, has failed to make even a minor dent in a messaging app market dominated by Whatsapp and Facebook Messenger.

"And that's because there's no compelling reason to bother with Allo. None of its features - like asking it for directions - provide enough of a benefit beyond what you'd get from just tapping in your request the "old fashioned" way. Users have an incredibly short fuse for chatbots not working exactly as we expect."

---

We have a special name for those few people who we (mostly) don't find boring.

Friends.

We don't much enjoy extended interactions with random folk. People who, nevertheless:

• have been completely socialised into our culture for decades, 
• come with detailed background knowledge of the world,
• are endowed with common sense and full conversational abilities.

So why did the AI companies think we'd enjoy interacting with chatbots, which are cognitively impoverished in every conceivable way?

It's a good question and I'm not sure of the answer.

- Were they over-impressed by their mighty artificial neural nets? But they're only fantastic recognisers and classifiers, a far cry from artificial general intelligences (AGI).

- Did they think that we're all keen to have conversational, hands-free interaction with our pocket devices? In fact that's socially way too intrusive most of the time, plus we're talking to conversational muppets.

- Was there a belief that in some narrow, vertical and tightly-constrained domains there might be a niche for a conversational interface? There's almost certainly something to that - but we don't yet know what.

My own feeling is that the successful mass consumer chatbot can be nothing less than a truly effective virtual friend. To that end it will have to posses AGI and be malleable to your own personality and 'friend-preferences'.

We'll have starships before we have that; I haven't seen the first clue we're on that road.

---

All of the above presupposes peer-relationships, typically with kids or adults. Those conversations - most of the time! - exhibit an irreducible core of rational and relevant 'aboutness'.

So hard to replicate for an artefact.

But there are natural agents around without much cognitive competence: babies, small children and pets. The bond here is emotional .. and so is the interaction.

So if you're in the business of designing chatbots which could conceivably bond with your customers, you might want to take note .. .*

---

* In the old days, we called them 'dolls', and they came without batteries.

Logistic discriminant analysis (= neural nets)

If you were trained (as I was back in the 1980s) on Good Old Fashioned AI (GOFAI), the technical background you studied consisted primarily of formal logic and discrete maths, implemented by symbolic programming languages such as Lisp and Prolog.

Meanwhile, the minority neural net tendency used statistical techniques and differential equations.

It's hard to imagine two more discordant cultures.

In these days of the overarching victory of the latter, I was interested to read the following from the excellent overview book, "Statistics: A Very Short Introduction" (David J. Hand), page 104.

"In fact, logistic regression can be regarded as the most basic kind of neural network."

I confess I had never thought of neural networks as simply a mainstream statistical classification tool.

Wikipedia has two articles on the subject: "Discriminant function analysis" and "Linear discriminant analysis" along with "Logistic Regression".

Something to look at further.

---

Marr's Tri-Level Hypothesis

David Marr was one of my heroes when I was an active AI researcher. Outside of computer vision I think he is mostly forgotten now (he died tragically early), but he said something important about methodology in AI research when many around him were writing programs that did vaguely cool stuff while claiming they were advancing science.

Marr distinguished three levels of analysis.

1. computational level: what does the system do (e.g.: what problems does it solve or overcome) and similarly, why does it do these things


2. algorithmic/representational level: how does the system do what it does, specifically, what representations does it use and what processes does it employ to build and manipulate the representations


3. implementational/physical level: how is the system physically realised (in the case of biological vision, what neural structures and neuronal activities implement the visual system).

His terms are not great (he was trained as a biologist): his computational level is really the theory of system behaviour in the environment of interest; his second level might be better described as an architectural level, describing the various ways the system's capabilities could be decomposed into subsystems and their inter-relationships; finally comes the issue of specific processing mechanisms and algorithms.

It's still common to see people waving the banner for one of these elements of analysis, while ignoring the others. Only confusion results.

Neural networks are an architecture. As currently understood and built, the term denotes a distributed, connected computational architecture well-suited to a certain class of problems, namely pattern recognition, feature extraction and classification.

This is a proper subset of the cognitive problems animals (including humans) have to solve in the world.

Artificial neural networks today consume Terabytes of training data, solving recognition/ classification problems of interest to Google, Facebook and the like.

I am reminded of the man who has a hammer.

The easy wins will fade away well before they achieve the purported Holy Grail of Artificial General Intelligence.

I hasten to add the obvious: in any event, you and I are considerable more than arid and cerebral AGIs.