An upper Bound to Intelligence?

---

This a fragment in my conversation with my not-so-super-intelligent GPT-6 Astra Extra High. No publishable essay was forthcoming.

Somehow, from this large assembly of interesting points, I would like to pull together some kind of essay about the upper bound to intelligence in the context of the frontier labs striving for ever-increasing intelligence on the part of their systems, and the doomsters' idea of the singularity and some kind of unbounded exponential takeoff of intelligence.

All of these ideas under-theorize, I think, the concept of intelligence and the theory of it. So, in the first place, you've looked up quite a lot of things. That's good, but I think you also have to do some thinking about what we've managed to unearth and try to put some kind of order into it. So I would like you to just list the five or seven or nine points that represent the argument of an essay. 

So to get started let me throw a few ideas in, but I don't think they're very good. There would be something about the initial framing, the belief that many people appear to have that intelligence is something which has no upper bound, that we can build systems that are smarter and smarter by improving the reasoning architecture and making the chips run faster and having more of them.

And I think the next point would be to say that the problem with that is that it under-theorizes what intelligence might be. So here we have the idea that intelligence is what you said it was before: the ability to handle unfamiliar situations in a competent and rapid way.

The third point, I think, will be to say, well, we need a better model of what those situations might be and what actually it is that the intelligent entity is doing when it's trying to cope with novel problems in the context of its abilities to perceive and act. So we might here revert to the language of logic, formal logic and models, and say if we conceptualize the world via some kind of super-accurate mathematical structure, then there could be a 'correct ultimate theory' which that mathematical model of the world (which is meant to be isomorphic to reality) actually instantiates.

Then to know that theory and to be able to do things with it, is to be effectively completely and correctly informed about all the aspects of the world, subject perhaps to a stochastic model, but there are ways to deal with that.

We could, if we wished, methodologically equate that view to a God-like view, and we can discuss what a God like that would actually be able to accomplish and in what sense we could talk about its intelligence, because, by definition, it has the correct theory of the world and the correct facts of the world in their totality and can therefore draw conclusions which would always be validated inside the world. And it could be both correct and complete, we could say, for this particular model, which happens to coincide with reality. Does this kind of God have maximal intelligence - or none at all because there are no surprises?

Then, moving on, we could talk about the problem of agents situated within subsets of the world, effectively-closed subsets or partially-open subsets, where the observable facts are a subset of the totality of facts of the world and therefore do not comprise all of the things which are true, and in which the theories which are induced by those agents, based on partial evidence, would only ever be partial theories, which they could never be sure were generalizable to the entirety of reality, because they had not experienced the entirety of reality.

So we could argue that for agents like these there is the question of to what extent their conceptualization of the state of affairs they encounter - maybe individually or collectively - and the problems that they face and the remedies available to them through actions, themselves provide bounds which essentially saturate the concept of intelligence for them.

That once one has the ability to perceive and conceptualize (given the theories which have so far been adequate and cannot be improved upon in the current situation) and one has access to the actions that are available, essentially that's as good as it gets.

So that's an upper bound to intelligence (or at least the utility of intelligence) right there, and beyond that, whatever engineering is applied to increasing, say, raw computation speed, or the ability to generate new concepts in an ungrounded way, it doesn't really help, because we already have an optimal perceive-think-do loop, which is as effective as it's going to be.

And then we can say something about that might apply to a mostly-closed/decoupled subset of the universe, but it's possible to modify that in a couple of ways. Sometimes there are better theories, even within a closed system, which allow for new kinds of solutions, and those better theories could be something which a more intelligent system could come to realise.

And secondly, by increasing the domain of perception, essentially by doing new kinds of experiments or by exploring, one generates new facts, which may contradict existing theories, and thereby generate new ones which are more adequate, and therefore the ability to increase the space of conceptual possibilities would also be something which would reward a concept of higher intelligence.

But again, it would be bounded largely by other things than simply the ability to generate concepts, as, say, a random axiom-producing system might generate concepts, or one might just play around inferentially with a given axiomatic base in a fairly random way to generate new concepts and see if they fit, all of which could be done, but: it wouldn't necessarily get you anything other than enhanced power bills. Generally progress towards greater competence in the world is constrained by more things than lack of good ideas.

Well, I mean, if it was done sufficiently rapidly and pruned sufficiently rapidly, it might very well look like a higher intelligence, but in a bounded situation which all situated agents find themselves in, it seems that there would be an upper bound simply given by the true nature of reality defining the best theories that could ever be found. Further search would be a waste of time, although you could never know that for sure.

And then to finish there would be some kind of conclusion about, given the current state of humanity and the current state of technology and the current state of what we're asking AI systems to do, we might find saturation almost everywhere - so without needing too much incremental intellectual abilities on the part of our AI systems. Most jobs in our bureaucratic political-economic societies are bounded by process and technology and do not reward intelligence beyond that of playing your role competently.

Only in areas where those conditions do not apply, where the situations were unsaturated, for example in mathematics and some areas of science, might we find true rewards for increasing our novel conceptual manipulation abilities, which we otherwise would call intelligence. That's a separate discussion - see my post coming here on Wednesday.

---