Written by Miroslav Tomoski

Illustration by Deshi Deng

In the late 18th century, British philosopher Jeremy Bentham devised what might be considered the perfect prison system. What Bentham called the Panopticon was a circular structure in which the prison cells line the outer walls facing inward toward an opaque guard tower. The guard in the centre could see every cell from his perch, while the prisoners could never know whether they were being watched or not and, therefore, would always behave as if they were.

“He who is subjected to a field of visibility, and who knows it, assumes responsibility for the constraints of power,” said French philosopher Michel Foucault, who later described the concept in his book Discipline and Punish, “He makes them play spontaneously upon himself; he inscribes in himself the power relation in which he simultaneously plays both roles; he becomes the object of his own subjection.”

In essence, surveillance, whether real or not, is a powerful motivator. It’s a concept the Chinese government has decided to take to its most extreme end with the help of modern technology in it’s newly proposed Social Credit System.

The system is one in which the government will track the online activity of its citizens – their social media posts, their online shopping, and even the people they associate with, to determine the trustworthiness of its population.

“It will forge a public opinion environment,” a 2014 government document titled Planning Outline for the Construction of a Social Credit System states, “where keeping trust is glorious. It will strengthen sincerity in government affairs, commercial sincerity, social sincerity, and the construction of judicial credibility.”

Though the idea has been floating through the halls of power in Beijing for years, and a voluntary system already exists, this comprehensive system will be mandatory for all Chinese citizens by 2020. The program is ideologically inspired by the lofty and unifying goals of General Secretary Xi Jinping. It’s another Great Leap Forward that’s meant to modernize a society in which contracts are routinely violated and counterfeit products are mass produced. Yet its dangerously totalitarian nature is best described by a September 25 document titled, “Warning and Punishment Mechanisms for Persons Subject to Enforcement for Trust-Breaking.”

Outlining a series of punishments for those with a low social credit score, the document begins with the broad statement that, “if trust is broken in one place, restrictions are imposed everywhere.”

Just what are those restrictions? They can range from restricted travel to hurting an individual’s job prospects and their ability to receive a loan. Beyond the professional repercussions, a citizen’s score will also be affected by that of their family, friends, and acquaintances, essentially threatening to ostracize anyone who doesn’t lead a government approved life.

As for those with higher social credit scores, the system rewards these citizens with fast track visas to Europe, government granted loans, and the higher end of everything to which their less trustworthy neighbors have limited access.

The exact point system has yet to be devised, but a number of pilot systems currently exist that offer incentives for better scores, but do not yet require punishments on the lower end. It’s currently a voluntary system that is reminiscent of Amazon Prime discounts and Facebook Likes.

The People’s Banks of China initially planned to license eight private companies to compete for a pilot program that will be taken over by the government in 2020. That licensing plan has since been delayed, but it’s interesting to note the kind of companies the government has chosen to design its new system.

Among the two most capable candidates are China Rapid Finance (the owners of the social media platform Tencent and developer of the massive messaging app, WeChat), and Ant Financial Services Group, which has access to resources of online retail giant Alibaba (the current program even offers credit for Alibaba’s website as a reward).

There is one thing these companies have in common which makes them ideal candidates to develop a social credit system. They both have access to a massive trove of personal information, from shopping habits to one-on-one interactions, known as Big Data.

To Western observers, a social credit system is understandably an alarming leap toward dystopia. Yet in China, the difference between the current highly-censored and widely-monitored system is a matter of visibility. Where once the government kept a close eye on its citizens from behind the curtain, its citizens will now police themselves in a nationwide social panopticon.

What ought to concern those on the outside looking in, is how closely the structure of this Social Credit System resembles our own online community. If the private companies primed to create a dystopian surveillance system in China are that country’s equivalent of Facebook and Amazon, it’s worth considering if Big Brother is a little closer to home than we think.

Would companies that rely so heavily on voluntary consumers be able to get away with participation in wide-scale government surveillance? After all, while Edward Snowden showed us that tech giants like Google routinely engage in government contracts, they are also built on altruistic slogans like, “Don’t Be Evil.”

It’s likely that the political culture of the West and public image-based marketplace limits the most visible tech companies from descending into an Orwellian nightmare. In Western democracies, the more likely candidates, as Amnesty International suggested in early 2017, “are companies you probably haven’t heard of  – although it’s their business to know you.”

These companies fall through legal loopholes, which allow them to sell private information to anyone who can afford it. “It’s a little bit of the Wild West over here,” an employee who wanted to remain anonymous told Amnesty International during an investigation.

This includes companies like Cambridge Analytica, a data mining and analysis firm employed by the Brexit and Donald Trump campaigns. “We have somewhere close to four or five thousand data points on every adult in the United States,” CEO, Alexander Nix, boasted at the 2016 Concordia Summit.

Palantir Technologies, which keeps the names of its customers confidential and has received over a billion in federal contracts since 2009, would also fall under the same umbrella. As The Intercept reported in early 2017, those investments have given Palantir a key role in providing data to an alliance of intelligence agencies – from the US, Canada, the UK, Australia, and New Zealand – known as Five Eyes.

In standard techie fashion, Palantir derives its name from an all-seeing stone in J.R.R. Tolkein’s The Lord of the Rings. In fiction, it’s used by an evil sorcerer to spy on and manipulate people from afar, but the company’s website says that its focus is, “protecting our fundamental rights to privacy and civil liberties.”

It’s founder, Peter Thiel, has said otherwise, “I no longer believe that freedom and democracy are compatible,” Thiel wrote in the journal Cato Unbound in 2009, explaining why he’s chosen to stay out of politics. Still, Thiel spoke at last year’s Republican National Convention and is currently an advisor to President Trump.

At a time when the leader of the free world is openly discussing immigration bans and a “Muslim registry,” the ease of access governments and Big Data have to citizens’ most private information is worrying and go beyond shadowy companies like Palantir.

Little-known firms like ExactData.com are just a Google search away and offer information on nearly 2 million American Muslims, whose data could be purchased and registered without their knowledge.

Though companies like ExactData never sell the names of individuals, the data provided can be narrowed to a level of detail that would make identifying any individual as easy as a Facebook search. For a category they call “Unassimilated Hispanic Americans” they boast having over 600,000 email records, access to 1.7 million social media accounts and 3.5 million postal records.

Only a few versions of Bentham’s panopticon ever made it into the real world, and fewer still hold prisoners today, but the idea of a self-imposed prison has endured. With the comforts free market democracies have to offer, the dangers of dystopia can seem like they are a world away. But with an ever-connected online community and our personal lives etched daily in permanent ones and zeroes, Big Data has made for a small world.