5: The Ethics of Utilitarianism

AI in the Battlespace: Chapter 5

5: The Ethics of Utilitarianism

It’s a scene different in detail but the same in essentials. This time a different girl meets the soldiers; there is a titanic detonation. The drone rocks in the shock wave, the camera tilts to the sky and then to the ground before reverting to the American patrol. The smoke clears to reveal bodies in pieces bleeding in the dust. Of the girl there is no sign at all.

Ursula's hand is over her mouth. Her slender body tightens with revulsion.

“I’m sorry to show you scenes this distasteful - both are genuine by the way, from the early days before the rules of engagement were changed - but they show the kind of dilemma faced by autonomous-soldier developers.”

She looks at him, still emotional, her mind still locked to the carnage they've just viewed.

“The AIs are trained on large datasets, operational clips like those we just saw. Specialised tacticians carefully annotate the clips with ground truths such as:

“If the girl is bearing a gift she should be allowed to deliver it.

“If she is delivering a bomb she needs to be stopped.

“What’s the difference, though? We can’t see the bomb, it’s under her robes. The intentions of the village leaders? Those we can’t see either. The AIs are stuck.”

He pauses, takes a calming sip of water and continues at a slower pace.

“You know, the Pentagon ran simulations. When they applied the rule: Never shoot a child, little girls and boys carried bombs more than 50% of the time - the insurgents had figured the cost-benefit. So then - in the simulation -  they changed the rule: always shoot the child. Soon no child ever appeared bearing gifts or bombs. Once the message hit home very few kids subsequently died. Ironic, isn’t it, that the more bloodthirsty policy saved many more simulated lives in the end?”

Ursula does not find this ironic or even thinkable. Her face mirrors her utter distaste.

“German soldiers will never shoot children. As a first resort or as a last resort.”

Alain sighs inside. He rather fears Ursula may be correct.

“Minister,” Alain says quietly, “let me tell you what US CENTCOM policy actually is. Firstly there's an obvious cost to not stopping the bomber-child: our people die, the child dies and the bad guys redouble their efforts.”

Ursula is now looking bored, wishing the meeting could just end.

“But there's a cost to shooting children too. The villagers are mostly related, all part of the clan. We kill one of their kids, they vow eternal vengeance. It's gasoline on the insurgency.”

Caron is silently pleading for the Minister's attention. He's coming to the important bit.

“So here's what happens in the field today. Analysts make an estimate of insurgent sympathies for each village based on recent events. Our commanders use that in selecting tactics. They make their best call - but sometimes a kid gets killed and in hindsight it was a terrible mistake.”

Ursula has a stubborn expression on her face. She doesn't want to hear any of this. Her ethics are absolute. She won't like what comes next, Alain thinks, and she'll draw the wrong conclusion.

“We tested the Russian software in a series of benchmark-scenarios, many with ethical dilemmas. The best way to describe the Russian soldier-drone is psychopathic.”

Ursula's harsh grin says it all: that’s exactly what she expected.

“Hold on, this is not a reflection on Russian military doctrine - don't be misled by our own propaganda. It turns out that if you only have local, tactical information, a psychopathic response usually is the best tactical response. Stalin said it best: no man, no problem.”

Ursula shakes her head in denial.

“You pay for local savagery with collateral damage - reputational hits, third-party criticism, increased hatred all round. But for the squaddies on the ground, those guys in the situation, it's usually better to go in hard. That's a ground-truth down the ages. The Roman Legions were seldom known for nuance. The Russian robots are locally optimising because their developers don't know how to do any better - and neither do we.

“Incidentally, when we ran the battle-sim between Russian autonomous soldiers and Bundeswehr troops operating under your rules of engagement, your guys got slaughtered. Every single time.”

The camera lingers on Ursula’s face: determined, not giving an inch. Unbowed.

---

The video comes to an end and Colonel Liu returns to the lectern, calm and competent, ready to draw lessons.

“Let me highlight the important issues from that sequence,” he says, “Firstly, are we being too ambitious? There's an analogy with self-driving cars where tech companies are spending billions. Their cars are way too cautious because they can't safely take the initiative, even in a defined situation like driving.”

Colonel Liu can’t see the audience, they’re lost in the blackness, but he can sense their interest.

“If we can't even get AI cars to work what chance do we have with AI soldiers? They're in an unstructured, complex and largely unknown terrain - where we want them to selectively deploy lethal force and meet ill-defined objectives.”

The auditorium is silent.

“Secondly, training,” he continues, “Our staff colleges don't show recruits endless YouTube battle clips. We teach tactics, strategy, case-studies; the controlled use of force for political ends.”

He waves an arm for emphasis, almost in exasperation.

“How would we do that with AIs today? In our incompetence we oscillate between ineffectual pacifism - the German idea that a soldier-drone is just an AI-car in camo - and the Russian model of counterproductive murderous lethality.”

He pauses, takes another sip of water.

“The Pentagon believes that the autonomous AI soldier is far from ready to rock and roll. We don’t know how to build it and neither does anyone else. There is, however, a place for sophisticated AI in military operations - and that will be the subject of our final session after lunch.”

---

Next     Previous.