"Superintelligence" by Nick Bostrom: a terrible book

Derivative, dreary, uninspiring and not very original

As promised a few days back, I have now force-marched my way through this tedious book, admittedly skimming the last few chapters. There has been a fair amount of hype about the alleged dangers of future-AI. I could never see it myself, but thought it fair to give the proponents of these scare-stories their best shot .. which seems to be Nick Bostrom.

Alas, Professor Bostrom from Oxford University says nothing in this book which hasn't been dealt with more profoundly in a dozen science-fiction stories. I learned literally nothing whatsoever of value. The book is so badly written - endless lists of not-very-insightful conceptual categories & pointlessly over-abstract language - that my eyes glazed over at the prospects of writing a detailed review. Thankfully, someone else with a PhD in artificial intelligence has done the job over at Amazon and I can quote their thoughts verbatim in this vitriolically-amusing review.

"As someone who holds a PhD in AI, I was super-excited to get Nick Bostrom's book "Superintelligence". Finally, mainstream discussions on a topic I care about from a highly ranked academic institution, with a cool looking owl to boot. Yea, owls! Fantastic, right? Wrong. After the first few chapters I had to force myself to finish it; it's miserable.

"This book is a 260 page tribute to Eliezer Yudkowsky that does not appreciate concepts at all in the entire field of constraints nor does it take into account common sense or any practical basis in AI. At best, this is a grammatically well-written sensationalist book designed to inspire irrational fear of a fictional form of AI, but more likely it is but one of many examples of the distilled essence of naivety of people writing on a topic they know nothing about.

"At one point Nick quotes an idea postulated by Eliezer as a practical / credible scenario in the rise of malevolent AI. Eliezer suggests an advanced AI could reassemble a biomimetic computing device using a naive human and a stereo speaker to force certain chemical reactions to create a 'nanosystem' threat inside a glass beaker (p. 98). Oh, but first it needs to 'crack the protein folding problem'. Never mind that problem has been shown to be at least np-hard, and possibly np-complete - a clear demonstration of a lack of understanding of the protein folding problem in the first place, but the notion of basic physics to assemble nano-particles using a liquid substrate through non-uniformed shaped vessel made of glass seems "far-fetched", to put it mildly. I've read more plausible science fiction.

"At one point Nick states that by adding more money, growth in computational capabilities increase linearly. I guess he's never heard of order of operation -- sometimes referred to associativity. You see, we have things like division and subtraction that take precedence in computer science. (Most people are introduced to these concepts before 5th year maths, folks.) This means we cannot just pass along as many computations as we'd like evenly across a giant grid of processors as Nick seems to assume.

"This book is riddled with purple prose and denotatively 'weird' use of terms -- such as "recalcitrance". Ever been next to that person at the party who uses unnecessarily large words when a simplistic explanation will do? You'll get a lot of that sensation in this. You'll also see the occasional spelling error, which is nice -- like at the bottom of page 56, amongst others.

"In addition the citations are dubious at best - e.g 'world robotics 2011'. Was there some big consensus at a conference regarding all of one topic in AI? That'd be a first in history I'm pretty sure for any conference on any topic; I guess I missed one hell of a conference.

"The head of one of Oxford's philosophy departments might be an intelligent person but he's absolutely unqualified to speak on any topic in practical AI - by his own demonstration. I can honestly say that purchasing this book was literally the worst money I've ever spent - and I've bought an Oasis album.

"There is one redeeming quality of this book: The survey of scientists in AI-related fields who believe SGI might become a reality that describes when, at a 50% threshold, that might occur (p. 19), and the general description of the 'types' of SGI and the mediums in which those have been thought to occur in (Chapter 2) -- e.g. whole brain emulation versus evolutionary models, etc, but these come with serious errors in assumptions. Basically, if you had a PhD student and this was their thesis, it'd be s*** except for the second chapter of their work.

"TL;DR: If you want to be told that AIs will take over space and destroy humanity using a stereo speaker then this is the book for you. If you like reason and don't want to waste £20, then may I suggest something of equivalent intelligence such as Where's Wally, or maybe a nice collection of toilet roll? At least the latter would be fit for a higher use. Oxford should be ashamed to have given this guy his own department, but proud he's free of any teaching responsibilities. Who knows how much damage he'd do to graduates."

I like to think that if this were my review, I would have been just a tad politer.

The topic (and indeed danger) of 'superintelligence' is not stupid per se. Evolution tells the story of species being wiped out by changes in their niches or by invaders who can exploit their niche more effectively. Nothing says that humans can't design agents which will out-compete us, even if bio-hacked viruses seem more of a risk than super-AIs.

Nevertheless we will no doubt design at some point very competent competitors. That competence is not reducible to 'superintelligence' however, which properly only refers to a general competence and efficiency of concept management (see here).

The road to systems with the overall competence to take us all out is so complex and multifaceted that the idea that some hacker with a PC is going to come up with the killer algorithm in their bedroom, or that it could all happen over the weekend, is completely fanciful. Might as well write a book about the dangers of black magic.

Sherlock at Tyntesfield, criminals at the Probate Registry

Today we visited Tyntesfield to explore the Sherlock trail. This National Trust property was used to film scenes for the Sherlock Christmas Special, "The Abominable Bride".

And you could dress up.

Clare considers the clues (look to your left!)

Your author cogitates - to no avail

With a brisk, cold wind from the north, we skipped clues 3-7 and failed to solve the puzzle (to be fair, by that stage we had forgotten what the mystery was). We did not collect a prize.

---

Earlier we had driven to the Bristol Probate Registry near Bristol Bridge. The Probate Registry was not where the satnav had said it was, and when we eventually found it (thanks, Google Maps!) there was no parking. I abandoned the car to Clare, who set off (in central Bristol) to find somewhere to park, while I entered the building, or tried to.

Thinks: 'Why won't this door open?'

A few minutes pass while I try to figure it out.

Passer-by: "That's the exit, try the door to your left."

Having negotiated the policeman searching belongings and the security scanner, I entered the vestibule and found a deserted Probate Office window. I used the phone provided to call down an official, a pleasant and brisk young man who quickly took me through the script. Five minutes later I had affirmed my Oath, signed where necessary and left the building.

Outside on the Plaza ("No skateboarding: the full force of the law will be brought to bear.") I waited for Clare in the chill northerly wind. I was soon joined by a muscular young man in tee-shirt and cords, leather boots and tattoos who was taking a fag break. He glanced-without-glancing at me, a casual superiority already classifying me as prey. Skin prickling, I moved out of his way and inspected the board by the door. Probate was the least of the Justice functions housed in the building - most prominent were the large number of Courts.

Right.

Clare arrived in the car after a further few minutes. How pleased I was that she had failed to find a parking space. We departed forthwith for Tyntesfield.

For once, the yeuk factor is real

From the MIT Technology Review today.

" ...  MIT Technology Review estimates that about 20 pregnancies of pig-human or sheep-human chimeras have been established during the last 12 months in the U.S."

So how does that work?

"By modifying genes, scientists can now easily change the DNA in pig or sheep embryos so that they are genetically incapable of forming a specific tissue. Then, by adding stem cells from a person, they hope the human cells will take over the job of forming the missing organ, which could then be harvested from the animal for use in a transplant operation.

“We can make an animal without a heart. We have engineered pigs that lack skeletal muscles and blood vessels,” says Daniel Garry, a cardiologist who leads a chimera project at the University of Minnesota. While such pigs aren’t viable, they can develop properly if a few cells are added from a normal pig embryo. Garry says he’s already melded two pigs in this way and recently won a $1.4 million grant from the U.S. Army, which funds some biomedical research, to try to grow human hearts in swine."

OK, there's a yeuk factor there, but surely if the objectives are good .. ?

"The worry is that the animals might turn out to be a little too human for comfort, say ending up with human reproductive cells, patches of people hair, or just higher intelligence. “We are not near the island of Dr. Moreau, but science moves fast,” NIH ethicist David Resnik said during the agency’s November meeting. “The specter of an intelligent mouse stuck in a laboratory somewhere screaming ‘I want to get out’ would be very troubling to people.”

Oh yes, the screaming mouse.

"... Desperately ill people on organ waiting lists might someday order a chimera and wait less than a year for their own custom organ to be ready. “I really don’t see much risk to society,” he says.

"Before that can happen, scientists will have to prove that human cells can really multiply and contribute effectively to the bodies of farm animals. That could be challenging since, unlike rats and mice, which are fairly close genetically, humans and pigs last shared an ancestor nearly 90 million years ago.

"To find out, researchers in 2014 decided to begin impregnating farm animals with human-animal embryos, says Pablo Ross, a veterinarian and developmental biologist at the University of California, Davis, where some of the animals are being housed. Ross says at Davis he has transferred about six sets of pig-human embryos into sows in collaboration with the Salk Institute and established another eight or 10 pregnancies of sheep-human embryos with Nakauchi. Another three dozen pig transfers have taken place outside the U.S., he says.

"These early efforts aren’t yet to make organs, says Ross, but more “to determine the ideal conditions for generating human-animal chimeras.” The studies at Davis began only after a review by three different ethics committees, and even then, he says, the university decided to be cautious and limit the time the animals would be allowed to develop to just 28 days (a pig is born in 114 days).

"By then, the embryonic pig is only half an inch long, though that’s developed enough to check if human cells are contributing to its rudimentary organs.

“We don’t want to grow them to stages we don’t need to, since that would be more controversial,” says Ross. “My view is that the contribution of human cells is going to be minimal, maybe 3 percent, maybe 5 percent. But what if they contributed to 100 percent of the brain? What if the embryo that develops is mostly human? It’s something that we don’t expect, but no one has done this experiment, so we can’t rule it out.”

So run that past me again ...

 “My view is that the contribution of human cells is going to be minimal, maybe 3 percent, maybe 5 percent. But what if they contributed to 100 percent of the brain? What if the embryo that develops is mostly human? It’s something that we don’t expect, but no one has done this experiment, so we can’t rule it out.”

If a pig-embryo can have its kidney developmental genes knocked out, so growing kidneys constructed from stem cells from a specific person - so that less than a year later there's a non-rejectable transplant available. Well,  it's going to happen.

My question: are we happy to eat the bacon sandwich made from the rest of the pig?

Funding the Police

My Speed Awareness course has now been booked for the afternoon of Thursday, February 4th 2016. So that's £80 for a four hour course, paid upfront. We've been warned that if anyone is abusive or non-cooperative, they will be thrown out and will incur the full force of the Force's displeasure: naturally I have no thought but to be fully-engaged and completely docile.

--

Books I'm about to read.

Is Nick Bostrom really the smartest man in Britain?

Good reviews from people who know about this stuff

Hard to tear myself away from Patrick Lee's Travis Chase trilogy though. Who knew books could be this exciting, this page-turning?

Alex and Adrian departed to resume their quotidian lives on Sunday. On Monday all the Christmas decorations were packed away, candle wax removed from the carpet and the house comprehensively vacuumed. It's now sparse, uncluttered and clean while the mountains of laundry have been reduced to mere foothills.

As 2016 gets underway, I have residual probate duties as executor of my mother's Will. The Probate Registry cashed my cheque this morning so I know they're now on the case. Beyond that, no major issues currently loom in the middle distance. I rather like a lack of drama in the personal - the better to engage with the extra-ordinary in the world.

Art

In the great traditions of Clare's Galleria we bring you .. art!

The Male Patriarchy. punching down onto the Oppressed

The Wax Horse

---

Sometimes we get visitors here from the United States. No-one could be more opposed to stereotype threat than myself, but it has to be said that Americans sometimes don't 'get' British humour. So let me clarify.

• The top picture, sketched by myself on my Nexus 10, is utterly without merit.


• A child could do better.


• The Wax Horse demonstrates my wife's extraordinary skill in 'remote art'. In a typical moment of casual inattention, the pink candle on the hearth went into malfunction spilling hot wax across the tiles (and indeed, onto the carpet where it currently remains). The shape, solidified, seems indeed to resemble an equine quadruped.

To be fair, Clare has more success when she is actually in the control loop .. .

---

Did any of that change your view on the quality of the art portrayed above?

Sixty five

My 65th birthday lunch with the family, in The Swan at Wells.

At one level, nothing changes when the dial of the year encounters your 65th birthday. But it makes a difference: my father, at 65, was in good health and had almost twenty years of life ahead of him. He was delighted to wave goodbye to work he hated, and welcomed his new autonomy.

Life for the 'getting-on' is an instance of the 'bathtub model' of engineering failure.

Everything is fine-ish until the last few years, at which point hospital appointments crowd the diary. One hopes for calmness and serenity.

Still, with luck, that's a way yet. Plenty of time to comment on the idiocies of public policy, the hubris of engineers .. and hopefully the wonders of new science and great writing.

'Our Robots, Ourselves' by David Mindell: a review

Link to Amazon page here

David Mindell trained as an electronics engineer, where he worked on deep-water submersibles, and as a social scientist where his ethnomethodological studies have looked in detail at how people construct and rework social relationships to integrate new technologies. If there’s one take-home message here, it’s that people create artefacts – technologies – to solve problems, and that all such tools, even those with substantial AI in the control loop, have to operate successfully embedded within a matrix of human processes.

Mindell’s case studies include underwater crewed and remotely-controlled vehicles; advanced aviation automation and military RPVs; space systems such as Mars rovers and the ISS, and finally innovative proposed systems such as driverless cars. I read one comment on his work which argued that there could be no issue in principle with fully autonomous systems: the fact that existing ‘autonomous’ systems always sit in a loop with human specialists is simply that it hasn’t yet proven cost-effective to automate said specialists. This is to spectacularly miss the point, that all such ‘autonomous’ systems are in fact embedded within human systems; all such artefacts are tasked, their operations monitored and their results delivered into wider human systems and contexts. You can never escape the issue of the human-machine interface.

And this issue has been around for a long time. A generation ago people worried about the inscrutability of expert systems, the bafflement humans felt when a  system veered off into some unexpected course of deduction or action. Was this intended or is it a glitch? In either case, how can we humans continue to engage with the ‘rogue’ system? Mindell has many examples of autonomous subsystems (autopilots and autolanders come to mind) where the automation fails inscrutably, throwing control back to the unprepared user/driver/pilot with consequent disaster. This is the stuff of newspaper headlines.

The automation which seems to work best is augmented reality. Here the automation systems map a high-complexity environment (eg engines and system status, outside terrain) into a visualisation which makes task performance easy (eg keep the craft icon on the guidance icon as you come onto target). Here the human stays in the loop with enhanced powers.

Autonomous systems work where environmental behaviour is largely predictable and there are few (and acceptable) negative consequences to unanticipated failure modes. In military affairs, guided missiles and mines come to mind, although unintended consequences in the latter case have proven controversial. Put a putative autonomous system into an open environment with a great deal of under-constrained human interaction and unpredictable environmental variation (snow, weather, building work, crashes, diversions) and no AI system currently foreseeable can cope. The system requirement would be emulation of the full common-sense and conversational capabilities of a socialised adult human. No, we are nowhere near that point.

These seem to be good reasons to be sceptical about prospects for fully-automated driverless cars, despite their great desirability and corresponding levels of hype.  David Mindell finishes his book with a well-researched assessment of prospects for the Google car and similar automotive innovations. I can summarise by saying, don’t hold your breath.

The author has written an excellent and thought-provoking book, a welcome attempt to see beyond the limitations of a pure systems engineering approach to advanced AI automation: essential reading.

The Predator and the Google car

[From "Our Robots, Ourselves: Robotics and the Myths of Autonomy" by David A. Mindell .]

The Predator presents an image of an autonomous ethereal presence, loitering with murderous intent over the distant battlespace. The trailer view? It's a pig to drive.

"Predator remains prone to computer crashes and lockups during critical periods. Crews learn through difficult experience not to press certain key combinations, not to issue commands too quickly, not to confuse this button with the one next to it. It takes several steps to do simple things. It takes more than twenty keystrokes, for example, simply to turn on the aircraft's autopilot. "The conjecture among us pilots," observes one operator, "is [that] the engineers thought we were too stupid, and we would be idiots and be hitting buttons all the time and doing dumb things, so they tried to put a two-step process into everything that we do."

"Manuals are lengthy and unclear. Some important features are hidden in the system's code and documented nowhere, passed down through word of mouth between the operators in a kind of oral tradition of stories of workarounds to make the system perform. No small part of the operators' skill is simply making the system do things it was not designed to do."

Meanwhile Google has much of the world believing that autonomous cars will take over the world in just a few years. But this is part utopian engineering optimism and part smoke-and-mirrors.

"Google has been testing self-driving cars on California roads since 2009, claiming hundreds of thousands of miles of accident-free highway driving. They travel routes mapped with great precision by Google's human-driven survey cars; the maps serve as virtual railway tracks for the cars (indeed, they are as yet unable to drive on roads without these detailed maps). The drives have included human safety drivers and software experts who can turn the autonomy on and off. "The idea was that the human drives onto the freeway, engages the system, [and] it takes them on the bulk of the trip—the boring part—and then they re-engage," said Google engineer Nathaniel Fairfield.
...
"Google's rhetoric around the project has the kind of Silicon Valley optimism that typically surrounds software systems. Roboticist Sebastian Thrun, lead engineer for the project, envisions a future of utopian autonomy "without traffic accidents or congestion."

"A number of critics of Google's approach have pointed out its limitations. Most of the work has been done in northern California or other western states. The Google car's successful driving tests in Nevada were run under tight constraints from the company for good weather and simple routes (the company also sought to avoid disclosure of details of the safety drivers' disengagements of the autonomous operation). The vehicle's algorithms had difficulty negotiating construction sites, requiring the safety driver to take control. Technology journalist Mark Harris has recently shown that becoming a safety driver for one of these cars can require weeks of training, suggesting the computer to human hand-offs remain complex and risky.

"In contrast to the wide-open West of car commercials and Google's trials, urban driving entails a great deal of social interaction, as we drive through a messy, complicated, and dynamic physical and social landscape. Google admits this problem is ten to one hundred times harder than driving on highways. Once again, autonomy within a human context proves much tougher than the abstracted technical problem.

"MIT's John Leonard, who helped develop some of the basic algorithms that driverless cars use for localization and mapping, points out how much driving depends on social interaction. My late friend Seth Teller, formerly an MIT roboticist, perceptively observed that urban driving consists of hundreds of "short-lived social contracts between people," as we scan the streets, make eye contact, let people in and wave "thank you." Computers are slowly getting better at assigning labels to the physical world and to different kinds of objects. Yet as Predator pilots can attest, techniques to similarly interpret human identities and intentions remain primitive.
...
"Only half joking, Leonard contends that driving in Boston can be considered operating in an extreme environment. He put a video camera on the dashboard of his car and is collecting examples of driving situations that are difficult for algorithms to handle: merging onto a busy road at rush hour; staying within road lines obscured by dust or snow; turning left across several lanes of traffic. In the snowy Boston winter of 2015, the three-dimensional landscape of urban driving would change overnight, as snow piles nine feet high narrowed the roads and altered traffic patterns.
...
"What have we learned from extreme environments that might shed light on possible futures for autonomous cars? We know that driverless cars will be susceptible to all of the problems that surround people's use of automation in the environments we have examined—system failures, variability of skills among users, problems of attention management, the degradation of manual skills, and rising automation bias as people come to rely on automated systems."

The classic failure mode: some unexpected human or otherwise unpredictable intervention occurs in the driving environment; the automation gives up, handing control to the disengaged and inattentive driver; the situation rapidly spirals to disaster; and the verdict is - driver error.

So don't hold your breath.

David Mindell's book is, by the way, excellent.

Sherlock: "The Abominable Bride"

Increasingly in this series, the puzzle for 'Sherlock' is taken to be Sherlock himself.

The ostensible plot is a 'bride' who fires wildly at men in the Victorian street from a balcony, then shoots herself fatally in the head - this is the abominable bride. Later, however, she seems to have returned from the dead to shoot her husband, before dying again. And then the 'bride' keeps returning and killing men: the signature is the word 'YOU' sketched in blood near the scene of the crime.

Sherlock is a certain archetype of twenty first century intellectualism: super-smart and asexual. Along with Watson, we are appalled at his arrogance, his disregard for women (except for Irene Adler, the one woman whom he acknowledges as an intellectual equal - and even then there is no sexual frisson), and his lack of interest in anything other than solving puzzles.

It's time to probe deeper into the Holmes subconscious.

In homage to Inception, the Victorian setting appears to be a deep simulation, a kind of super-memory-palace into which Holmes has retreated under the influence of drugs in an attempt to solve a historical mystery. Within the Victoriana, Holmes and Mycroft occasionally insert discordant 21st century references to jets, computer viruses and the like which they explain by the sheer 'anticipatory power' of their minds.

Sherlock eventually solves the nuts-and-bolts of how the 'abominable bride' accomplished her apparently-impossible feats. With only moderate spoilers, it comes down to the coordinated action of a secret society of militant vote-seeking feminists. But since this is a deep-dream sequence, the realisation that the super-smart crime was the action of .. mere women, can't be allowed - how could they be smart enough? So Moriarty has to be invoked: no-one less could have discomfited Sherlock's intellect.

In the end, it always comes down to Sherlock vs. Moriarty - Sherlock can never escape the limitations of his own archetype. At the Reichenbach Falls (within the deep dream-simulation) Watson intervenes to break the fearful symmetry and free Sherlock to wake (nice homage to 'The Matrix' there).

At which point, unlike the viewer, Sherlock has learned precisely nothing about himself.

We are thus cued to the next series. Excellent!

Google Fetch now in Beta

Android tablets and phones have that Google box at the top of the screen. Say "OK Google," or tap on the microphone icon and you can speak your query. If it's straightforward, a synthesised voice will reply. It's remarkably effective.

This morning I noticed an additional function on my Nexus 10. If you say 'Google Fetch', the search box turns black.

On a whim, I said, "Google fetch roadkill." The black surface began to ripple like heavy, viscous oil. I pushed my fingers in - the consistency was that of thick treacle. The borders of the search box wavered and expanded to make room for my hand, extending down the screen. Soon, I felt something! My hand came out bearing the squashed and mangled remains of a kit - a baby rabbit.

I was at a loss what to do with it, now I had it. In the end, we boiled it and fed it to the cat.

There had been rumours that Google^[x] was involved in research on Einstein-Rosen Bridges so I suppose this new feature is no surprise but like so much from Google, it's well-engineered and quite awesome in practice. It's certainly a novel alternative to Amazon's idea of using drones for delivery.

I think that to be truly useful, however, Google are going to have to deliver tablets with bigger screens.