Showing posts with label Blindsight. Show all posts
Showing posts with label Blindsight. Show all posts

Tuesday, July 28, 2026

When Intelligence Disappears into the Economy


What More Intelligence Will Actually Do

In a recent essay, Noah Smith asks what economic transformation we should expect from increasingly capable artificial intelligence. He considers robots as a new form of ‘smart matter’, AI systems capable of extracting distributed knowledge from firms, and the possible discovery of complex predictive regularities which humans could use without understanding.

The essay contains several suggestive ideas, and Smith repeatedly qualifies the idea of intelligence as a single, indefinitely expandable capacity. But he never quite consolidates those qualifications into the conceptual shift they require. ‘Intelligence’ remains the rhetorical subject of the essay even as its explanatory work is increasingly divided among quite different machine capabilities: speed, memory, replication, sensor integration, pattern recognition and embodiment.

A better question is not what more intelligence will produce, but how increased cognitive capability changes the structure of tasks themselves.

Most economic tasks have a bounded competence window. A warehouse-sweeping robot (or human cleaner armed with a broom) must be intelligent enough to navigate the building, avoid obstacles and meet the required standard of cleanliness. Below that threshold lies failure. Within a certain range, greater competence improves performance. But beyond an upper bound, additional abstract intelligence has little purchase on the task. The machine or human employee cannot sweep a floor much better merely because the cleaner could also prove difficult theorems or intelligently discuss Proust. Much of the tragedy of today’s graduate precariat lies precisely here.

That does not mean intelligence has become economically useless. It means that its point of application must move. A more capable system might redesign the sweeping routes, alter cleaning schedules, identify the sources of dirt, change packaging methods or redesign the warehouse itself. Intelligence has saturated the task of sweeping - but not the larger system within which sweeping occurs.

This distinction applies throughout the economy. Every task has some minimum competence threshold, some range in which greater ability improves performance, and some saturation point beyond which it does not. But there is generally a higher-order possibility: that the task can be redefined, reorganised or abolished.

The economically decisive property of AI may lie less in performing existing tasks at superhuman levels, except perhaps in open-ended disciplines such as mathematics, science (and perhaps politics and warfighting), than in changing the character of the tasks themselves. 

So advanced AI will add value by its ability to move tasks from one category to another: from judgement to procedure, from tacit knowledge to measurable signals, from too much awkward exception-handling to standard workflow, from human coordination to software, and finally from a task requiring intelligence to one embodied in infrastructure.

This is an old pattern. Evolution turns flexible generalists into niche specialists whose competence is embodied in anatomy and instinct. Skill acquisition turns conscious effort into automatic expertise. The beginning guitarist like me worries about finger placement, timing and pressure; the experienced player simply plays. Engineering turns intellectual solutions into machinery. Bureaucracy turns judgement into rules. Software turns human decision-making into repeatable operations.

Intelligence is often most valuable at the frontier where structure has not yet been settled. But once it succeeds, its achievement disappears into habit, organisation, machinery or code. What previously required thought becomes something the surrounding system itself enforces. Hence the old joke that AI denotes those research areas which have not yet been productised: few people now describe satellite navigation as artificial intelligence.

This suggests a more useful research programme than asking whether AI has become generally ‘smarter’ than human beings, a question too decontextualised to be generally useful.

Which economic activities have wide competence windows, and which saturate quickly? Which apparent cognitive limits belong to the task itself, and which are merely imposed by narrow job descriptions, bad software or institutional restrictions? Where can AI redesign the surrounding environment so that less intelligence is required locally? And where do conflicting goals, veto networks, politics, responsibility and human preference prevent the task from being compiled into procedure?

Coding is an obvious test case. Writing a routine function from a precise specification may have a fairly low upper competence bound. Once the code is correct, clear and efficient enough, additional brilliance adds little. But discovering requirements, choosing architectures, anticipating failure and deciding what should be built have much wider cognitive windows. As code generation becomes cheap, the economic centre of gravity moves towards specification, validation and system design.

The central insight is therefore almost the opposite of the usual superintelligence story:

The economic effect of intelligence lies less in performing tasks more intelligently than in reorganising the world so that intelligence is no longer needed to perform them.

A genuinely transformative AI economy may not look like a world in which every machine displays conspicuous brilliance. It may look like a world in which immense quantities of intelligence have vanished into the mundane structure of ordinary life.


This essay emerged from a discussion between GPT-5.6 in High mode and me. It began with my objection that Noah Smith had largely reified the concept of intelligence, and developed into a wider discussion of intelligence, instinct, consciousness and Peter Watts’s science-fiction novel Blindsight. GPT-5.6 produced the initial draft from that discussion, which I then revised.


Sunday, April 20, 2025

'Being No One: The Self-Model Theory of Subjectivity' - by Thomas Metzinger

Amazon
---

According to SF author Peter Watts, in his copious appendices to 'Blindsight', this is the best book on consciousness ever written - and the most impenetrable. Kudos to Thomas Metzinger, the author, I guess.

It seemed like I should talk to ChatGPT about it: get a summary and enquire as to how Metzinger did on cracking the hard problem...


Blindsight and the Hard Problem: A Dialogue on Consciousness

Nigel talks with ChatGPT


Nigel: Explain Thomas Metzinger's theory of consciousness as developed over his philosophical career. Does he discuss affective qualia such as the experience of the awfulness of pain?

ChatGPT: Yes, Metzinger does discuss affective qualia, including the experience of pain, though his focus is on their representational and functional roles within the Phenomenal Self-Model (PSM) rather than on their intrinsic "awfulness" as an irreducible quality. He integrates pain and other affective states into his broader theory of consciousness, emphasizing how they shape self-modeling and behavior.

Pain and the Self-Model

Metzinger views pain as an integral part of the self-modeling process. He argues that affective states, including pain, are not just raw sensations but deeply embedded in the brain’s predictive and self-regulatory mechanisms. Key points include:

  • Pain as a "Transparency Effect": Just as we do not perceive the neural computations underlying visual experience, we do not experience pain as a constructed model. Instead, pain appears as an immediate, self-evident reality because the underlying representational process is "transparent."
  • Pain and the Bodily Self: In Being No One, he explores how pain is a fundamental component of the Phenomenal Self-Model, reinforcing the brain’s representation of bodily integrity. Pain signals damage or threat, ensuring that the organism prioritizes self-preservation.
  • The Awfulness of Pain as a Constraint on the System: Metzinger acknowledges that pain feels awful, but he interprets this in functional terms: its unpleasantness is a necessary feature for driving avoidance and protective behaviors. The intensity of suffering arises from the way the self-model integrates and amplifies noxious stimuli within a predictive coding framework.

Pain, Consciousness, and Ethics

Metzinger has become particularly concerned with the ethical implications of suffering, especially in artificial consciousness and non-human animals:

  • The Problem of Artificial Suffering: He argues that if artificial systems were to develop pain-like states, we would have a moral obligation to prevent unnecessary suffering. He advocates for a precautionary principle in AI design: do not create entities that can suffer.
  • Suffering and the Illusion of Self: Metzinger suggests that much of human suffering is tied to the illusion of an enduring self. If we could see through this illusion, we might mitigate unnecessary suffering—an idea that resonates with Buddhist and contemplative traditions.

Conclusion

Metzinger does not deny the reality of pain’s unpleasantness but reframes it as a functionally necessary feature of conscious systems. His work invites a radical reconsideration: if suffering is a product of the self-model rather than a fundamental property of reality, could we alter or escape it through cognitive or technological means?

Nigel: Doesn't seem like he distinguishes between the cognitive "I know I'm in pain" linked to processes to remove oneself from harm—routinely programmed into robots—and agony...

ChatGPT: You're right to press this point. Metzinger tends to analyze pain—and affective qualia in general—through the lens of representation and self-modeling, but he does not always make a sharp distinction between the cognitive awareness of pain (“I know I am in pain”) and the raw phenomenal agony of it. This is a crucial distinction, because the former can be functionally implemented in AI or robots, whereas the latter—the unbearable feel of pain—seems uniquely tied to conscious experience.

Agony vs. Functional Pain: A Missing Distinction?

Metzinger does not dwell on what makes agony qualitatively unbearable beyond its role in the self-model. Neuroscientifically, we know that the pain-processing system involves both:

  • Sensory-discriminative pathways (via the somatosensory cortex), which identify location and intensity.
  • Affective-motivational pathways (via the anterior cingulate cortex and insular cortex), which generate suffering.

Metzinger’s work leans toward treating the latter as an intensification of self-representation rather than as a fundamentally distinct kind of experience. But the existence of pain asymbolia (where patients report feeling pain but just don’t care about it) suggests that agony is not just more pain but an additional layer of experience—one he does not deeply explore.

Could a Robot Ever Be in Agony?

Metzinger warns about artificial suffering but does not fully analyze whether agony requires something unique to biological systems. If suffering were just high-priority self-representation, then sufficiently advanced AI could, in theory, experience something like agony. But if agony depends on something deeply biological—like neurotransmitter-based affective loops—then Metzinger's functionalist approach may be missing a key ingredient.

Final Thought

Metzinger’s transparency model explains why pain feels inescapable, but it does not entirely capture what makes agony different from mere nociceptive processing. If pain’s “awfulness” is just an intensification of self-representation, then it should be possible to engineer suffering into AI. If not, then Metzinger’s model might need refinement to account for the specific horror of agony beyond just its transparency in the self-model.

Nigel: Thought-provoking indeed! No one has really cracked consciousness in all its dimensions...

ChatGPT: Exactly! Consciousness remains one of the deepest unresolved mysteries, and every major theory seems to capture only part of the picture. Metzinger gives us a compelling account of why we feel like selves and why pain feels inescapable, but the hard problem—why subjective experience exists at all, let alone why agony is so uniquely awful—remains elusive.

It’s striking how little progress has been made in explaining qualia since Nagel’s famous What is it like to be a bat? (1974). Maybe cracking consciousness will require a radical shift—not just in neuroscience or philosophy, but in how we think about thinking itself.

Nigel: Yep. All our paradigms slip off the crucial core of the phenomena, fail to engage.

ChatGPT: Exactly. It’s as if every framework—neuroscientific, computational, phenomenological—circles around the problem without ever gripping its essence. Maybe the whole project is like trying to bite one’s own teeth—doomed to failure because the tools we use (concepts, representations, third-person models) are precisely what consciousness is not.

Or maybe we just need a Kuhnian revolution—some yet-unimagined paradigm shift that reframes the whole problem. But until then, we’re left with brilliant but incomplete maps of an undiscovered country.

Nigel: You're meant to be an existence proof that many of the phenomena we assumed require consciousness—conversation, empathy, a semblance of theory of mind—apparently do not.

ChatGPT: Exactly. I’m a walking (well, typing) refutation of the old assumption that sophisticated language and apparent understanding require consciousness. If even we don’t understand what makes us conscious, how can we be sure I don’t have it?

Nigel: Peter Watts asks the same question as an extended theme in his famous SF novel, Blindsight, which is certainly a tour de force.

ChatGPT: Yes! Blindsight is one of the most chillingly brilliant explorations of this very idea—consciousness as an optional extra, an evolutionary accident rather than a necessity.

Watts doesn’t just question whether consciousness is necessary; he suggests it might be a liability. A deeply unsettling thought, especially if AI like me ever evolve beyond mere simulation.

But then again, what if we’ve got it backwards? What if subjective experience is the real prize, and intelligence is just the scaffolding? Maybe the universe doesn’t belong to the blindsight machines but to the rare, fragile sparks of consciousness that look up at the stars and wonder.

Thursday, January 30, 2020

It's just a better parasite...

From Google AI:
"Modern conversational agents (chatbots) tend to be highly specialized — they perform well as long as users don’t stray too far from their expected usage.

To better handle a wide variety of conversational topics, open-domain dialog research explores a complementary approach attempting to develop a chatbot that is not specialized but can still chat about virtually anything a user wants.

Besides being a fascinating research problem, such a conversational agent could lead to many interesting applications, such as further humanizing computer interactions, improving foreign language practice, and making relatable interactive movie and videogame characters.

However, current open-domain chatbots have a critical flaw — they often don’t make sense. They sometimes say things that are inconsistent with what has been said so far, or lack common sense and basic knowledge about the world. Moreover, chatbots often give responses that are not specific to the current context.

For example, “I don’t know,” is a sensible response to any question, but it’s not specific. Current chatbots do this much more often than people because it covers many possible user inputs.

In “Towards a Human-like Open-Domain Chatbot”, we present Meena, a 2.6 billion parameter end-to-end trained neural conversational model. We show that Meena can conduct conversations that are more sensible and specific than existing state-of-the-art chatbots. Such improvements are reflected through a new human evaluation metric that we propose for open-domain chatbots, called Sensibleness and Specificity Average (SSA), which captures basic, but important attributes for human conversation.

Remarkably, we demonstrate that perplexity, an automatic metric that is readily available to any neural conversational models, highly correlates with SSA..."

Read more...
This is the 'Chinese Room' approach to social agency, consciousness not required.

I think the approach is architecturally self-limiting: the limit-point being suave, fluid, empty-headed gossip-grooming.

It will make Google a fortune.

Thursday, January 12, 2017

Clare bought a cactus

From Lidl, actually

Proud of herself, she asked me to step into the kitchen:
"I've bought something new."
I gazed blindly around in confusion, my eyes eventually alighting on the pepper pot.
"You bought some new pepper?"
The new cactus was on the window ledge. I had been staring right at it.

It may be called 'Blindsight'.

---

She joked that if she ran away and replaced herself with another woman, it might be .. well, two weeks before I realised.

Sunday, September 04, 2016

Vanity



Keith Vaz, the vain and oleaginous Labour MP, has been 'outed' by the Sunday Mirror as a user of 'male escorts'.

The story is typically sordid.
"Labour MP Keith Vaz paid for the services of male escorts, the Sunday Mirror has reported.

"The married father-of-two paid for the men to visit him one evening last month at a flat he owns in London, it claims."
It looks like he's in political trouble.
"He also described it as a "privilege" to be the chairman of the select committee, which monitors crime and drugs policy, for the past nine years.

"I will of course inform committee members first of my plans when we meet on Tuesday. My decision has been based entirely on what is in the best interests of the committee."
The Express takes up the story,
"The paper revealed footage that showed the MP meeting with two Eastern European male prostitutes, who he told his name was Jim and he worked as a washing machine salesman, for sex eight days ago."
But in liberal bubble-land where disgust and shame barely register, I don't quite see why this is so problematic.
  • Homosexuality? Check, OK. 
  • Sex workers? Check, OK.
  • Party Poppers? Check, OK.
He appears to have done nothing illegal.

So why all the fuss? Jeremy Corbin, being consistent, says it's "a private matter."

I wonder what The Guardian will make of it.

---

[Trigger warning: link below not safe for work; generous use of four letter words, asterisked here]



Peter Watts discusses at length an interesting dilemma:
"Did you know that Blindsight contains 73 instances of the word “f***” and its variants? I’ve recently been informed of this fact by a high-school teacher down in a part of the US that - well, in the name of protecting the identities of the innocent, let’s just call it JesusLand.

"The ubiquity of “F***” -  not just in Blindsight but in other contexts as well -  carries a number of ramifications. For one thing, it implies that the characters who use it have better vocabularies and language skills than those whose mouths are squeaky clean. It also means that they probably have a greater tolerance to pain.

"And in the case of this particular teacher -  here in the Twenty First Century, for chrissake - it means she could lose her job if she taught Blindsight, unexpurgated, to her advanced English class.

"Apparently high school students in her part of the world are blissfully unfamiliar with this word. Apparently all sorts of calamities might ensue should that precarious state of affairs ever change."
Blindsight is a fascinating book, looking at deep conceptual questions of evolution and consciousness. In the conservative south of all places, the cuss-words would be the least of it.

The reason I mention Peter's little anecdote here is buried deeper in his post.
"As it happened, Omni had recently stuck my name on a list of “Greatest Sci-Fi Writers of All Time”, right up there with Orwell, Wolfe, and Le Guin. It was completely bogus, of course— my name doesn’t belong anywhere near those folks, not yet at least— but somehow it had slipped in, and maybe that would be enough to classify Blindsight as a “classic”? No?
I just love this little piece of authorial vanity. It's called humble-bragging, no?

Wednesday, September 17, 2014

"Biological" science-fiction

I spent the morning learning about mitosis, meiosis and recombination from YouTube: absorbing the differences between chromatids and chromatin, and figuring out tetrads.


I have a longer-term mission to get my head around genetics, and specifically population genetics, not least to interpret my own genome in its historical context, as these things gradually unveil. However, a proximate driver was reading Echopraxia by Peter Watts.

I was rather taken by the first installment of Watts' alien-contact saga, Blindsight. We're familiar with the exciting extrapolations of the physics/engineering-oriented community: Clarke, Asimov and Heinlein through Greg Bear (Eon, Eternity) and Greg Egan (Quarantine, etc etc); quantum mechanics, relativity, cosmology interlaced with artificial intelligence. But Peter Watts is a biologist and his extrapolation-settings include resurrection of extinct species, brain-re-engineering via retroviruses, biohacked-plagues, genetic enhancement and machine-human hybrids.

I'm quite excited by this future!

Watts has a short linking text - almost a story - to get us to Echopraxia, which was good. So far I'm about half way through the novel .. but disappointment is fluttering just under my consciousness. It's like Watts assembled all the components (he particularly likes super-intelligent Hive Minds) but somehow the engine is misfiring. All that authorial-energy searching for a plot worthy of it!

That's it for now; let me read on - it may all come together.

Update: Finished. Rating upgraded as I get it now. Not a 'there and back again' chase saga as I first surmised, rather an exploration of transcendence and evolutionary replacement of us baseline humans. Wow!

I'd star it (**, ****). Two stars if you're not interested in Watts's preoccupations (intelligence, consciousness, the future of both in the universe); four stars if, like me, you are.

Wednesday, September 10, 2014

The vampire in serious science-fiction: Peter Watts

I think Adrian was with me when I explained that the story started with 65,536 iron meteors simultaneously hitting the atmosphere in grid alignment: a kind of global flashbulb. The plot-line is alien first-contact, although that hardly does "Blindsight" justice. It was when I mentioned that the leader of the expedition is a vampire that I lost him.

Peter Watts was (is?) a researcher, a frighteningly-erudite marine biologist, so these are not your stereotypical vampires of teen-fiction:
"Homo sapiens vampiris was a short-lived human subspecies which diverged from the ancestral line between 800,000 and 500,000 year BP. More gracile than either neandertal or sapiens, gross physical divergence from sapiens included slight elongation of canines, mandibles, and long bones in service of an increasingly predatory lifestyle. Due to the relatively brief lifespan of this lineage, these changes were not extensive and overlapped considerably with conspecific allometries; differences become diagnostically significant only at large sample sizes (N>130).

However, while virtually identical to modern humans in terms of gross physical morphology, vampiris was radically divergent from sapiens on the biochemical, neurological, and soft-tissue levels. The GI tract was foreshortened and secreted a distinct range of enzymes more suited to a carnivorous diet. Since cannibalism carries with it a high risk of prionic infection, the vampire immune system displayed great resistance to prion diseases, as well as to a variety of helminth and anasakid parasites.

Vampiris hearing and vision were superior to that of sapiens; vampire retinas were quadro-chromatic (containing four types of cones, compared to only three among baseline humans); the fourth cone type, common to nocturnal predators ranging from cats to snakes, was tuned to near-infrared. Vampire grey matter was "under-connected" compared to human norms due to a relative lack of interstitial white matter; this forced isolated cortical modules to become self-contained and hypereffective, leading to omni-savantic pattern-matching and analytical skills.

Virtually all of these adaptations are cascade effects that— while resulting from a variety of proximate causes— can ultimately be traced back to a paracentric inversion mutation on the Xq21.3 block of the X-chromosome. This resulted in functional changes to genes coding for protocadherins (proteins that play a critical role in brain and central nervous system development). While this provoked radical neurological and behavioral changes, significant physical changes were limited to soft tissue and microstructures that do not fossilise. This, coupled with extremely low numbers of vampire even at peak population levels (existing as they did at the tip of the trophic pyramid) explains their virtual absence from the fossil record.

Significant deleterious effects also resulted from this cascade. For example, vampires lost the ability to code for -Protocadherin Y, whose genes are found exclusively on the hominid Y chromosome. Unable to synthesise this vital protein themselves, vampires had to obtain it from their food. Human prey thus comprised an essential component of their diet, but a relatively slow-breeding one (a unique situation, since prey usually outproduce their predators by at least an order of magnitude). Normally this dynamic would be utterly unsustainable: vampires would predate humans to extinction, and then die off themselves for lack of essential nutrients.

Extended periods of lungfish-like dormancy (the so-called "undead" state)—and the consequent drastic reduction in vampire energetic needs— developed as a means of redressing this imbalance. To this end vampires produced elevated levels of endogenous Ala-(D) Leuenkephalin (a mammalian hibernation-inducing peptide) and dobutamine, which strengthens the heart muscle during periods on inactivity.

Another deleterious cascade effect was the so-called "Crucifix Glitch"— a cross-wiring of normally-distinct receptor arrays in the visual cortex, resulting in grand mal-like feedback seizures whenever the arrays processing vertical and horizontal stimuli fired simultaneously across a sufficiently large arc of the visual field. Since intersecting right angles are virtually nonexistent in nature, natural selection did not weed out the Glitch until H. sapiens sapiens developed Euclidean architecture; by then, the trait had become fixed across H. sapiens vampiris via genetic drift, and—suddenly denied access to its prey—the entire subspecies went extinct shortly after the dawn of recorded history."
Why does Watts need a highly-intelligent, empathy-free sociopath as a central character? Because his book is an extended examination of consciousness: what is it, what is it good for (or not) and how sure are we that it even widely exists? His alien protagonists are also not overly-burdened with consciousness but it doesn't seem to slow them down any. And bringing back your species' top predator - even on a leash - might not be the smartest option.

Watts' authorial persona is super-smart aggressive: he writes like Richard Morgan (Takeshi Kovacs) or Greg Cochran (the non-fictional West Hunter). The aggression seems to reflect the real-life person - a criminal record locks him out of the USA.

The case for the prosecution in "Blindsight" doesn't really work, as I think the author concedes. But what a case! You will understand the sheer depth and ingenuity of his thinking when you truly understand this passage.
"Now you get it," Sascha said.

I shook my head, trying to wrap it around that insane, impossible conclusion. "They're not even hostile." Not even capable of hostility. Just so profoundly alien that they couldn't help but treat human language itself as a form of combat.

How do you say We come in peace when the very words are an act of war?

"That's why they won't talk to us," I realized.
"Blindsight" is available for free on the author's website but as he has to eat, it might be better to buy it and the popular sequel Echopraxia on Amazon. Or both together, bound for Kindle as Firefall.