A few months ago, you accidentally defaulted on a phone bill. The mistake affects your credit score: It’s hard to get a loan. You can no longer make jokes about Marco Rubio on Twitter; such remarks will algorithmically define you as a libertarian loon—another sort of person likely to default on social obligations. After a couple of close friends miss their student loan repayments, you can’t even travel: your social circle is now all “discredited, unable to take a single step.”

This is the incipient scenario in China, whose state-backed “social credit scheme” will become mandatory for all residents by 2020. The quoted text is from a 2014 State Council resolution which promises that every involuntary participant will be rated according to their “commercial sincerity,” “social security,” “trust breaking” and “judicial credibility.”

Some residents welcome it. Decades of political upheaval and endemic corruption has bred widespread mistrust; most still rely on close familial networks (guanxi) to get ahead, rather than public institutions. An endemic lack of trust is corroding society; frequent incidents of “bystander effect”—people refusing to help injured strangers for fear of being held responsible—have become a national embarrassment. Even the most enthusiastic middle-class supporters of the ruling Communist Party (CCP) feel perpetually insecure. “Fraud has become ever more common,” Lian Weiliang, vice chairman of the CCP’s National Development and Reform Commission, recently admitted. “Swindlers must pay a price.”

The solution, apparently, lies in a data-driven system that automatically separates the good, the bad, and the ugly. But with President Xi Jinping, China’s most authoritarian leader since Mao Zedong, at the helm, much English-language coverage of the plan so far predicts “unprecedented” levels of dictatorial surveillance.

Commercial versions of the nascent national program are already in operation. Ant Financial, the finance arm of e-commerce giant Alibaba, is piloting Sesame Credit, which offers a range of perks, such as travel upgrades and deposit-free car rentals, to top scorers. But Sesame’s system, which assigns a rating between 350 and 950, is murky and complicated. The company says even innocuous activities, like late-night web browsing or buying video games, could see one’s rank downgraded for “irresponsible” behavior. One undergraduate saw her score plummet to 350 after being named in an unresolved civil suit: Sesame had automatically listed her as a laolai, a deadbeat, “subject to enforcement for breaking trust.”

The worst-case scenario is a form of high-tech Stalinism for our brave new world, in which those who toe the line are kept doped with rewards like fast-track visas for countries with compliant customs (developing regions deeply indebted to China via its “One Belt, One Road” could end up having to align themselves with Beijing’s emigration and other policies). Meanwhile, those whom the system considers dissenters, dropouts or deadbeats would be effectively excommunicated from mainstream society. How these miscreants are defined is one of the most worrying aspects. “I have no doubt that the current efforts are intended to produce a more authoritarian state,” Stanley Lubman, a Chinese law specialist at UC Berkeley, tells me.

“Good” behavior is equally subjective. Sesame Credit automatically upgrades customers who purchase curtains or diapers, for example—items which suggest a certain middle-class stability. This is partly because Sesame “is designed to incentivize behaviors that drive profits for Alibaba,” explains Mark Natkin, managing director of Beijing-based Marbridge Consulting, such as “heavier online and offline use of Alibaba’s payment tool, Alipay, and the user’s ability to recruit more friends to join their Alipay [social] circle.” Mrs. Chu, a middle-class working mother in her early 30s, tells me she finds Sesame “very convenient... because I have a high score, I can get refunds [online] quicker, without having to wait to return the items.”

But once compulsory state “social credit” goes national in 2020, these shadowy algorithms will become even more opaque. Social credit will align with Communist Party policy to become another form of law enforcement. Since Beijing relaxed its One Child Policy to cope with an aging population (400 million seniors by 2035), the government has increasingly indulged in a form of nationalist natalism to encourage more two-child families. Will women be penalized for staying single, and rewarded for swapping their careers for childbirth? In April, one of the country’s largest social-media companies banned homosexual content from its Weibo platform in order to “create a bright and harmonious community environment” (the decision was later rescinded in favor of cracking down on all sexual content). Will people once again be forced to hide non-normative sexual orientations in order to maintain their rights? An investigation by the University of Toronto’s Citizen Lab also warns that social credit policies would be used to discourage protest.

State media has defended social credit against Orwellian charges, arguing that China’s maturing economy requires a “well-functioning” apparatus like the U.S.’s FICO credit score system. But, counters Lubman, “the U.S. systems, maintained by three companies, collect only financially related information.” In the UK, citizens are entitled to an Equifax report itemizing their credit status. In China, only the security services have access to an individual’s dang’an, the personal file containing every scrap of information the state keeps on them, from exam results to their religious and political views.

While outside observers agree that the situation likely bodes ill for many unwitting citizens, few have considered how vulnerable the system is to the corruption, con artistry, and incompetence that plagues much of Chinese society. Who will have access to the data, and how will they be able to use or abuse it? Will it be shared between ministries and departments, or jealously guarded? Can it be manipulated, altered, faked—or stolen?

Private data in China is already openly (and cheaply) available on eBay-like platforms such as Alibaba’s own Taobao, making the company indirectly responsible for both harvesting and selling its customers’ data. Scams and identity theft are infuriatingly common. Sesame Credit requires highly sensitive personal information, such as degree certificates and title deeds, to be uploaded to its cloud to enhance users’ credit scores—cybersecurity experts say such a centralized digital database would be a treasure trove for hackers.

Meanwhile, reports in China’s financial media suggest the commercial systems are already being abused, with micro-lenders using it to scam clients. “Sesame Credit... is still unable to control the quality of the data reported by partner lenders,” observed a Caixin article. “Information often includes errors like mistaken user identity, and some lenders deliberately misrepresent user information… they will actually put their favorite customers on their blacklist shared with other lenders, so that other platforms will reject the customer, allowing the original lender to have exclusive access.”

And it’s not just businesses and crooks looking to game the latest gimmick: Already accustomed to having their data mined and lives surveilled, tech-savvy Chinese are wondering how they can rig their scores—and entrepreneurial hackers will be more than willing to oblige. On the popular Q&A site Zhihu, users constantly wonder how to boost the numbers: “Can I click-farm this?” many ask. The top-rated answer skewers the system mercilessly:

With my countrymen’s knowledge for seizing every opportunity, and penchant for taking shortcuts, it won’t be long before we’ll have plenty of companies willing to farm your score. What’s that? Sesame scores are connected to the frequency you use your credit cards? Simple—my company will help swipe and repay your card for a year, then charge you for how many points your score accumulates.
What? Sesame points are related to the scores of your circle of friends? Simple. I’ve got plenty of high-score friends: I’ll bring you in. What’s that? You’re afraid of bringing down everyone else’s scores? Don’t be. Using some bullshit card-swiping method, we’ll aggregate the IDs of all your parents, relatives and friends in the countryside to bring up your score, and when the time comes, divide up the points evenly. Don’t think it’s not possible.

The Zhihu user explains that he is simply applying a pattern of past behavior to the new model: “Some things that you can’t do in other countries, you can do in China, like fake divorces to get around housing purchase limits, or driving restrictions in Beijing. I’ve a friend who got a dozen buddies to help him enter the license-plate lottery... Of course I think there’s an urgent need for a credit rating system. But I really don’t have faith that they’ll do it well.”

Marbridge’s Natkin acknowledges some dangers and drawbacks, but suggests social credit ratings “will also create a greater disincentive to engage in anti-social behavior, like a landlord capriciously deciding not to return a security deposit, or a shared-bike user parking in the middle of the street.” These are everyday grievances in China’s scofflaw society that many will be glad to see gone, or at least punished.

Riding the country’s flagship high-speed rail this year, I overheard an announcement warning passengers that bad behavior on board “could affect your personal credit”—now it’s been revealed that a whole range of infractions, from smoking to having the wrong tickets, could land citizens on a “deadbeat blacklist.” (So, too, will offering “insincere” apologies for defaulting on loans; one must not only learn to grovel, but like it.)

To work effectively, social credit requires Chinese citizens to place complete trust in both their unaccountable government and vast cartel-like corporations. And therein lies the problem: A secretive scheme that proposes to (literally) codify credibility within a society that inherently lacks any is more likely to undermine public trust that instill it. Few would knowingly risk signing up for such a scheme; unfortunately, by 2020, no one living in the People’s Republic of China—foreign or Chinese—will have a choice.

How Do You Control 1.4 Billion People?

By Robert Foyle Hunwick

Page 1