What “AGI” Should Mean, and How to Analyse It - (GPT5)

---

What “AGI” Should Mean and How to Analyse It - GPT-5

Premise.

We need a clear target. By AGI I mean an artefact that shows broad, sustained competence across many task families, adapts quickly to novelty and shift, learns efficiently from sparse feedback, composes skills for new problems, and keeps its aims coherent over long horizons under finite resources. This is architecture-agnostic and testable.

A working notion of intelligence

Take intelligence as the capacity to select and revise representations and policies to achieve goals with limited computation under uncertainty and novelty.

Four modes make this concrete: perceptual (build task-relevant state), doxastic (form and update beliefs, including uncertainty), conative (set and revise goals and constraints), and effectual (act on the world and use tools).

Ignore any of the four and both agency and the analysis collapse.

Why the social shows up without replacing intelligence

In open worlds most long-horizon tasks are strategic because other optimisers are also present. So the competence envelope of an AGI depends not only on solo problem-solving but on learning and acting among other agents, institutions, and norms. Social agency is not a substitute definition of intelligence. It is the regime where intelligence is exercised when others matter.

Markov games in one paragraph

A Markov game extends a Markov decision process to multiple agents. At each step the game has a state; every agent picks an action; the joint action updates the state and delivers payoffs (or losses) to each agent. 

States carry physical facts, information and uncertainty, and often norms; agents may communicate, commit, or defect; the future depends only on the current state and actions (the Markov property).

This framework lets us model cooperation, competition, coalition, sanction, and repair in a single, tractable setting.

Getting the levels right: three state components

Keep categories aligned by treating the environment’s state as a tuple with three co-equal parts: physical (affordances, tools, hazards, resources), informational (private beliefs, public facts, common knowledge structure), and normative (roles, rights, permissions, prohibitions, debts, and reputation gradients).

Norms are not feelings; they are institutional dynamics that shift payoffs and trigger enforcement.

A minimal architecture that fits the job

To function in strategic open worlds an AGI needs at least:

1. World-modelling: perception to latent state to predictive dynamics, with causal hooks for counterfactuals and robust generalisation.


2. Uncertainty: calibrated beliefs, value-of-information, and risk-sensitive control.


3. Goals and constraints: editable conative content with rules for adoption, suspension, and abandonment.


4. Planner–controller: hierarchical plans, model-based and model-free elements, fast replans under surprise.


5. Normative game engine: data structures for commitments, permissions, and rights; speech-act updates (promise, accept, accuse, justify, apologise, forgive); sanction and repair policies.


6. Partner models: compact theories of others’ preferences, thresholds, and reliability to support trust and coalitions.


7. Memory and self-accounting: to preserve commitments and enable credit/blame over time.


8. Regulators: affect-like control signals that gate attention, caution, exploration, and persistence when resources are tight.

Embodiment, stakes, and binding

Literal “pain” is optional; real trade-offs are not. Give the system non-forgeable costs that reduce future option value: scarce compute windows, irreversible actions, opportunity costs, and reputation penalties that throttle later access. With genuine loss on the table, commitments bind in practice, not just in language.

What to measure, and how (not IQ tests)

One-shot IQ puzzles are the wrong instrument. Use multi-episode, partially cooperative Markov games that force scarce resources, distribution shift, norm formation and breach, sanction and repair, tool use and code-writing, and environment extension. Report metrics that matter:

• Generalisation gap on held-out tasks and perturbed rules.


• Adaptation half-life after a shift; how fast competence recovers.


• Sample cost to recover target performance post-shift.


• Coalition stability and surplus under partner churn.


• Sanction efficiency and repair latency after breaches.


• Long-horizon regret under explicit resource budgets.

Designing artificial partners, concretely

• Institutional objects: first-class commitments, authorities, and rights with verified update rules driven by events and speech acts.


• Typed dialogue: parse and emit performatives that compile to edits on the normative state, not just word strings.


• Risk modulators: learned schedulers that adjust search depth, hedging, and deference as hazards and reputations shift.


• Partner modelling: maintain competing theories of others and select among them under sparse signals.


• Identity and audit: tamper-evident logs to support apology, restitution, forgiveness, and deterrence linked to real cost.

Boundaries that prevent hype

Social fluency is necessary in open worlds but not sufficient for AGI. An agent can charm and still fail under causal shift. Emotion talk is fine as metaphor, but the engineering target is the regulator and its control effect, not a story about feelings.

Why this framing helps

It states a falsifiable AGI target, restores the perceptual–doxastic–conative–effectual modes, keeps physical, informational, and normative state at the same level, and explains why the social appears—because other optimisers make most real problems strategic. It also yields evaluations that match the claims, not theatre.

Working summary

Define AGI by general, adaptive, sample-efficient, compositional, long-horizon competence under resource bounds. Analyse it as an adaptive controller in strategic environments—modelled as Markov games—with physical, informational, and normative state; with explicit uncertainty handling; with goals, plans, partner models, and memory; and with regulators that make trade-offs real. If nothing meaningful can be lost, nothing can bind. Add real costs and the engineering becomes honest.

---

Nigel: This going to be a stretch from current iterations of chatbots like ChatGPT and Gemini.

---