During a general election campaign in the late 1990s, I accompanied the candidate whose campaign I was managing on a visit to a highly successful technology company in a prosperous constituency in the East of England.
We were ushered into a smart boardroom and treated to the usual corporate presentation, complete with projected slides explaining the company’s latest technological triumphs.
Among other things, they were developing technology that would allow passengers on the London Underground to receive electronic messages even while travelling through tunnels. The executives were understandably pleased with themselves. Here were clever people solving difficult technological problems and, no doubt, making a great deal of money in the process. Fair enough.
But as the presentation continued, a rather different thought occurred to me. The technology that made it possible to reach people wherever they went could also make it possible to track their movements. And once you could do that, you were creating something rather more interesting and dangerous than a convenient communications system.
“Good God,” I said. “You’re inventing Nineteen Eighty-Four.”
They looked astonished. It had apparently never occurred to them that technology designed to make our lives more convenient might also make it considerably easier for somebody to keep an eye on us.
I have often wondered what the Stasi, East Germany’s notoriously efficient secret police, would have made of such technology. Think of the trouble they went to. The informers, the intercepted letters, the bugged apartments, the surveillance photographs, the mountains of paper files. At its peak, the Stasi employed approximately 91,000 full-time staff and maintained a vast network of informers, all to keep watch over a population of barely 17 million.
Imagine giving them smartphones, facial recognition, location tracking and artificial intelligence capable of analysing millions of people’s communications and movements. They would have thought they had died and gone to heaven (which they officially didn't believe in).
Recently, I came across what appeared to be a fascinating piece of black-and-white archive footage on YouTube of Hannah Arendt, the German-born political philosopher who fled Nazi Germany and became one of the twentieth century’s most important thinkers on totalitarianism.
Arendt, whose books include The Origins of Totalitarianism (1951), The Human Condition (1958) and Eichmann in Jerusalem (1963), devoted much of her intellectual life to understanding how supposedly civilised societies could descend into barbarism.
In the video, she was explaining the difference between ordinary tyranny and totalitarianism. An ordinary dictator, she argued, wants your obedience. He doesn’t particularly care what you think, provided you keep your mouth shut. The totalitarian dictator wants something altogether more ambitious. He wants to control what you believe to be true. And how does he achieve that? Through terror, which isolates people from one another, and ideology, which offers an explanation of everything so comprehensive that inconvenient facts can simply be dismissed.
Arendt’s most disturbing observation was that the ideal subject of totalitarianism is not the fanatical believer, but the person who has lost the ability to distinguish between fact and fiction, truth and falsehood. TikTok anyone?
I found the argument fascinating. But something struck me as odd. Arendt spoke with extraordinary fluency. No hesitations, no pauses, none of the little interruptions that normally accompany spontaneous speech. It was almost as though she were dictating from a perfectly prepared manuscript. Which, in a manner of speaking, she was.
When I checked, I discovered that the video was made by artificial intelligence. The woman I had been watching was not Hannah Arendt at all, but a digital recreation delivering words she had never actually spoken. To be fair, the argument was a faithful representation of her philosophy. But the performance itself was a fabrication.
What an extraordinary irony.
Here was Hannah Arendt, apparently warning us about a world in which people could no longer distinguish fact from fiction, while the very woman delivering the warning was herself a fiction.
Manufacturing reality is only one of the possibilities offered by AI. Another is the ability to monitor, analyse and ultimately control the behaviour of entire populations. The Stasi could only have dreamed of such powers. The Chinese authorities are already putting some of them into practice.
The Surveillance State
The Chinese authorities can potentially discover where individuals go, whom they meet, what they buy, what they say online and which websites they visit. Computers can combine this information, identify patterns and flag behaviour the authorities consider suspicious.
In Xinjiang, Human Rights Watch documented a policing system that collected personal information and identified people for investigation on the basis of such activities as unusual electricity consumption, using certain messaging applications or associations with people who had changed telephone numbers. But this technology isn’t confined to monitoring suspected terrorists or political dissidents. China also operates official blacklists that can prevent ordinary citizens who fail to comply with court judgments from buying airline or high-speed railway tickets or enjoying certain business opportunities.
Some restrictions have even extended to parents paying for expensive private education for their children. And in some Chinese cities, the authorities have experimented with systems awarding and deducting points for everyday behaviour, from breaches of pandemic regulations to misconduct by government employees.
There is no single national computer assigning every Chinese citizen a score for good behaviour. Yet. The reality is more fragmented. But the underlying principle is unmistakable. It reminds me of a sheepdog managing a flock. The dog doesn’t have to savage the sheep. A bark here, a nip there, and the sheep soon learn where they are supposed to go. The modern surveillance state doesn’t necessarily need to arrest everyone who steps out of line. It can simply make life progressively more harsh for those who do.
The Dangers of Joined-Up Thinking
There is a fashionable expression in management circles: joined-up thinking. In government, it means getting departments to talk to one another, share information, and stop behaving as though the people in the office next door were working for a different country.
In Britain, we have been trying to achieve this for decades, with decidedly mixed results. One department doesn’t know what another is doing, databases refuse to communicate, and citizens are repeatedly asked to supply information the government already possesses somewhere else.
China has taken the idea rather further. A court judgment can affect your ability to buy a railway ticket. Information collected for one purpose can be used by another department to impose consequences you never anticipated.
China’s systems are not all connected, yet the direction of travel is towards greater integration. And that is when joined-up government becomes something altogether more sinister. The tax authorities know what you earn. The health service knows your medical history. The police may know where you have travelled. Join all that information together, add artificial intelligence, and place the resulting machinery at the disposal of a government prepared to use it against its citizens. The secret policeman no longer needs to knock on your door. The computer can simply decide that you aren’t going anywhere. And all this could be presented as a triumph of administrative efficiency. After all, who could possibly object to joined-up government?
Who Protects the Chinese?
Of course, there is one rather important difference between China and Britain. In Britain, our ultimate protection against an overmighty government is supposed to be Parliament. We elect its members, who can question ministers, investigate abuses, change the law and ultimately bring down a government that has exceeded its authority.
We also have independent courts, a free press and an opposition whose principal ambition is to put the government out of business. None of these safeguards is perfect. But at least they exist.
So who protects the Chinese people? China has a parliament, grandly called the National People’s Congress, together with courts, a constitution and an elaborate legal system. Unfortunately, they all operate under the leadership of the Communist Party, which has no intention of allowing itself to be voted out of office.
In other words, the people who possess these extraordinary powers of surveillance are ultimately answerable to the same political organisation that authorises their use. And that is precisely why the Chinese example should concern us.
The Perfect Business Opportunity
It would be comforting to dismiss all this as simply another example of the wicked Chinese Communist Party doing what wicked communist parties do. But that rather misses the point. The Chinese have assembled and adapted technologies developed around the world, many for perfectly respectable commercial purposes.
And why should we imagine that in the future democratically elected governments won't be tempted to do the same? Indeed, they are already purchasing some of the same capabilities.
Take Palantir, the American company whose software brings together and analyses enormous quantities of information. Its technology is used by intelligence agencies, police forces and, in Britain, the NHS. The applications may be entirely legitimate, but the ability to combine information from different sources is equally useful to anyone wishing to monitor a population.
Or consider Pegasus, the sophisticated Israeli-developed spyware capable of secretly infiltrating mobile telephones. There is nothing inherently sinister about technology that helps police investigate criminals or doctors treat patients. But the same tools can be used for purposes their inventors never intended.
And there is a danger even greater than the misuse of technology by the government that acquires it. What happens when a perfectly respectable democratic government assembles an enormous database of its citizens in the interests of administrative efficiency, and a rather less respectable government inherits it? History provides an uncomfortable warning.
In 1940, the German occupation authorities ordered Jews in occupied France, whether French citizens or foreigners, to register with the police. The following year, the Vichy government imposed its own census of Jews throughout France. Names, addresses and nationalities were meticulously recorded, creating files that subsequently helped the authorities organise arrests and deportations. And all they had were index cards.
Imagine what they could have done with artificial intelligence.
Hannah Arendt died in 1975. She never saw the internet, the smartphone or the extraordinary surveillance machinery that governments can now deploy against their own citizens.
She understood what totalitarian governments could achieve with secret policemen, informers, propaganda and filing cabinets, but she could scarcely have imagined the technological resources available to them now.
I wonder what she would make of China today.
Or, for that matter, what she would make of us, and our extraordinary capacity to recognise the dangers, shrug our shoulders and carry on regardless.
PS Once, when I was in Hong Kong training the group board of a very large company in crisis communications, I was asked, during a private conversation in a bar, what the West thought of China. Normally, I avoid generalisations like the plague (and clichés too), but in the interests of international friendship, I made an exception.
“Peasants ruled by gangsters,” I replied.
She looked rather shocked.
Later, sitting in Cathay Pacific’s First Class lounge, eating a small bowl of bird’s nest soup made from the saliva of Indonesian swiftlets (which I was assured cost more per mouthful than half a glass of pre-phylloxera Château Lafite Rothschild 1869), while a young woman massaged the backs of my ears with an Imperial Tibetan unguent containing extracts of a rare Himalayan snow lotus, long thought to be extinct, a thought occurred to me.
Just how private had that 'private' conversation been?
I’ve never been back.