On March 1, the Provisions on the Administration of Internet Information Service Algorithmic Recommendations, jointly issued by the Cyberspace Administration of China and three other departments, formally went into effect. Requirements such as the one to “provide users with a convenient option to turn off algorithmic recommendation services” are tailored to meet the actual needs of users and set a global regulatory precedent with regard to specific algorithms. They have therefore received widespread attention. This article analyzes the ins and outs of the relevant regulations and possible future developments.

That mobile phone apps can “read your mind” has long ceased to be a secret. Many people have experienced the following: having just mentioned in a chat with a friend that they wanted to buy something, moments later they’ll see that very product appear in an app; or they browse a few news stories on a certain topic, and suddenly some app is filled with responses on that same issue.

Many internet users have long been unhappy with, or even a bit scared of, this kind of situation.

More recently, a switch has “finally appeared after a chorus of requests.” On the “Settings” page of many frequently used apps you should now be able to find something like a “turn off personalized recommendations” option.

The author found the relevant switch on “Zhihu” and “Taobao” and personally tested it out.

Why are all the familiar apps quietly giving an option to turn off personalized recommendations? The reason is that the Provisions on the Administration of Internet Information Service Algorithmic Recommendations (hereinafter “Provisions”) announced on January 4 contain the following article:

Article 17 Algorithmic recommendation service providers shall provide users with an option of not having their personal characteristics targeted, or provide users with a convenient option to turn off algorithmic recommendation services. If a user chooses to turn off an algorithmic recommendation service, the algorithmic recommendation service provider shall immediately stop providing such a service.

These Provisions, which were jointly issued by the Cyberspace Administration of China, the Ministry of Industry and Information Technology, the Ministry of Public Security, and the State Administration for Market Regulation, will officially go into effect on March 1.

To the best of the author’s knowledge, this is the first law in the world to impose specific constraints on algorithmic recommendation-related conduct.

Europe’s Digital Markets Act (DMA) also contains a stipulation on “not permit[ting] the exploitation of data advantages to show users targeted advertisements without obtaining explicit permission from users.” But the DMA was only passed by the European Parliament’s Internal Market Committee last November and negotiations with European governments have yet to begin. There is a way to go before it becomes law.

In China, however, the major internet enterprises have already taken action to implement the Provisions.

Potential Hazards of Algorithmic Recommendations

In his 2006 book Infotopia, Harvard University Professor Cass Sunstein introduced the notion of a kind of “information cocoon” in human society. He argued that in information dissemination, people’s own demands are not comprehensive, and instead they will just pay attention to selecting the information that they want and that can make them happy. Over time, the information they come into contact with becomes increasingly restricted. They become tightly wrapped up as though by the silk spat out by a silkworm, and ultimately shut themselves up inside an “information cocoon,” thus losing opportunities to encounter and understand different viewpoints.

Algorithmic recommendation may actually reinforce the information cocoon effect: the more interested you are in something, or the more inclined you are to a certain viewpoint, the more likely an algorithm is to recommend materials concerning that thing or materials supporting that viewpoint so that you are constantly reinforcing your own interests and inclinations. Moreover, algorithmic recommendations may also be used to purposefully guide populations and thereby influence public thinking and even influence political decisions. U.S. mathematician Cathy O’Neil thus refers to recommendation algorithms as “weapons of math destruction” in her book of the same name. These “weapons of mass destruction” have had a real-world impact many times in the past few years.

In 2016, consultants from the main social-networking platforms such as Facebook, Google, and Twitter worked together in a San Antonio office as part of Project Alamo, which was supporting Trump’s re-election campaign. They put roughly 90 million U.S. dollars into digital ads. Project Alamo made use of subtle algorithmic recommendation techniques to influence voters: when an internet user was recognized as a “pivotal voter” (e.g., a swing voter in a swing county of a swing state), the social networks would direct content of a leading nature to such a voter and in this way influence the election outcome at relatively little cost.

A few months before the U.S. general election, Cambridge Analytica in the United Kingdom used Facebook user data to manipulate the U.K.’s Brexit referendum, with the result that the pro-Brexit camp unexpectedly won—exactly as Donald Trump was unexpectedly elected.

Prior to their official use in influencing U.K. and U.S. politics, similar methods had already been tested in many developing countries. In 2010, in Trinidad and Tobago, a “Do So” movement originating from Facebook led a large number of voters of African descent to refuse to vote, thereby aiding the ethnic Indian-dominated United National Congress in the general election. In 2015, some Nigerian users saw short bloody and violent anti-Muslim videos on Facebook. Their purpose was to frighten voters and manipulate the election. Once algorithmic recommendations are abused, they truly can become a “weapon of mass destruction.”

Even if not intentionally abused, algorithmic recommendations may implicitly contain social bias and discrimination. In October last year, Twitter’s recommendation algorithms were found to have “unintentionally amplified the dissemination of right-wing group content”: Tweets posted by elected right-wing officials were algorithmically amplified more than those of their left-wing counterparts; the influence of right-wing media was greater than that of left-wing media.

Earlier still, the search algorithm of the professional networking website LinkedIn (which can be seen as a form of recommendation algorithm: “most suitable” content is recommended based on search keywords) was found to entail sex discrimination: male job seekers would be placed in higher positions. Google’s advertising platform AdSense was discovered to be racially biased. If a search keyword seemed like a black person’s name, AdSense would be more likely to recommend advertisements relating to criminal record checks.

Because algorithmic recommendations come with such risks of potential harm, some European and American researchers have long proposed subjecting recommendation algorithms to controls. The measures required by these Provisions, such as algorithmic mechanism auditing, science and technology (S&T) ethics reviews, and the granting of permission to users to turn off algorithmic recommendations, have all already been proposed and recommended outside of China. However, the international giants of the internet never put these suggestions into practice. They still commonly argue that “deep learning-based algorithms cannot be audited.” To help readers comprehend the importance of the Provisions, the author will concisely introduce the technical principles behind algorithmic recommendation.

Technical Principles of Algorithmic Recommendation

The various forms of algorithmic recommendation include the forms enumerated in the Provisions, namely “generative or synthetic, personalized pushing, ranked selection, search filter, and decision-making and scheduling.” All the current mainstream methods of implementation use machine learning, and the principles behind them involve predictions based on Bayesian statistics—which sounds deep and unfathomable, but is in fact easily understood through a simple example.

Suppose that you cast a dice that has never been used. What do you think the probability is of getting a 6? Of course, in the absence of any additional information, your prediction would be “1 in 6.” Then you throw the dice 20 times in a row. Each throw results in a 6. At this point, what would you think the probability would be of getting a 6 on the next throw? Classical probability theory holds that each throw of the dice is an independent, random event—the results of previous throws do not affect the results of future throws—so your forecast should still be “1 in 6.” But obviously a normal person would not think in this way.

The information that “this dice came up a 6 in 20 straight throws” would clearly affect future decisions (for example, it might indicate that this was a leaded dice) and so you would predict a high probability of a 6 on the next throw. Simply put, Bayesian statistics “predict events that are to occur in the future based on events that have already occurred in the past.” All algorithmic recommendations are making these kinds of predictions:

Zhihu’s personalized recommendations predict what questions and answers users might like to see.

Baidu’s search filter predicts which search results users might be interested in.

Taobao’s ranked selection predicts which products users might purchase.

The “events that have already occurred in the past” on which these predictions are based are a very broad, user-related dataset, which not only contains direct user actions such as “which answers the user saw/liked/bookmarked,” but also contains a large volume of the user’s own attribute information: age, sex, region, level of education, occupation, internet connection device, things they’ve purchased, remarks they’ve made, the size of home they live in, how many people are in their household, that they like Jeff Chang, that they don’t like Cai Xukun…All this information can be used to predict the user’s preferences.

Each piece of attribute information like this is also called a “feature.” An internet company generally possesses thousands or tens of thousands of pieces of feature information on a single ordinary user. Some of this feature information comes from the company’s own business, but more feature information comes from other platforms. Enterprises such as the three main operators [China Mobile, China Telecom, and China Unicom], Weibo, Tencent, Alibaba, and mobile phone manufacturers all share users’ personal feature information with other internet apps in the form of software development kits.

Given a specific prediction, some of these pieces of feature information will have higher relevance to the prediction while other features will be of less relevance. If it is possible to trace back from the prediction result to those features that had an important effect, we can say that this algorithm is “auditable.” For example, the principle of a linear regression, which is the simplest, most basic machine learning algorithm, is to assign a weight to each feature based on past events and then to predict future events based on these weights. From a linear regression prediction model, it is possible to directly see the weight of each feature in the prediction. Therefore, a linear regression is an algorithm that is especially easy to audit.

Of course, the simplest, most basic algorithm suffers from the problem of insufficient predictive ability. Figuratively speaking, it is not possible to squeeze out all the information implicitly contained in feature values just by using a simple linear regression and so prediction results might not be especially good. Hence, scientists and engineers have thought up many methods for squeezing information from feature values. One method is called “feature engineering,” which put plainly means deriving new feature values from known feature values. For example, assigning a new label of “considerable buying power” or “fad follower” to a user based on the user’s mobile phone model or shopping list would be a simple instance of feature engineering.

Another way to squeeze feature values is to regard initial feature information as a “layer” of input and then to use various mathematical methods to convert the input layer into new information nodes and in this way form a multi-layer “network.” This process of conversion may be repeated, and the greater the number of conversion layers, the “deeper” the network is said to be—this is where the term “deep learning” comes from.

Although scientists often use “neuron” and “neural network” to analogize the results of these mathematical conversions, the information nodes resulting from these conversions often have almost no real world meaning, and are purely a product of mathematical tools. So there is a saying in the profession: deep learning is like alchemy (or, in China, “like concocting immortality pills”). You toss data into a neural network, and out come the results for reasons unknown to you. If the results are less than ideal, just add a few layers to the neural network.

Because deep learning is often cloaked in an alchemic-like aura of mystery, the engineers who use it often don’t know themselves why an algorithm is effective. For example, Google published a paper introducing their image recognition algorithm, which used a 19-layer-deep neural network (VGG19). However, Google’s photo service (Google Photos) was exposed on several occasions as having implicit racial bias. It even identified black people as “gorillas.” Subsequently, Google found it impossible to locate the problem in the algorithm. All it could do was delete the “gorilla” labeling action.

In spite of the warning offered by Google’s embarrassing failure, similar problems continue to pop up in the products of all the internet giants. In 2020, some Facebook users, while watching a video that featured a black protagonist, received a recommendation prompt asking them whether they wished to “continue watching videos about primates.” In 2018, Joy Buolamwini, a researcher in the MIT media laboratory, discovered that Microsoft, IBM, and Face++ facial recognition algorithms had a much higher gender identification error rate for black people than for white people. Moreover, the darker the skin color was, the lower the identification rate. The identification error rate for black females was as high 35%. An over-reliance on alchemic-type deep learning algorithms is the reason that these internet giants have a lackadaisical attitude towards algorithm audits. It also makes the job of amending implicit systemic discrimination more difficult.

China’s Provisions: Significance and Concerns

It is precisely due to the industry’s reliance on algorithmic recommendations and deep learning technology that these Provisions appear to be especially important. The author believes that the announcement of the Provisions has, on the one hand, compelled internet enterprises to constrain their own behavior and use algorithmic recommendations for good purposes, i.e., upholding a mainstream value orientation and actively spreading positive energy, and not building information cocoons or inducing user addiction. On the other hand, it has compelled internet enterprises to strengthen the building of internal capabilities, i.e., establishing algorithm auditability and proactively selecting and optimizing recommendation algorithms that are comprehensible and auditable while avoiding “technology-only-ism” and over-reliance on alchemic-type recommendation algorithms.

However, being that these Provisions are a world first, the author still has some specific concerns relating to their implementation.

First, the question of how to implement algorithmic mechanism auditing and S&T ethics reviews may be a new challenge for regulators. Although the Provisions require that “algorithmic recommendation service providers shall regularly audit, evaluate, and verify algorithmic mechanisms, models, data, and application outcomes,” there may be a lot of space for fuzziness as to whether this requirement is actually put into practice, whether internet enterprises have really carried out audits, evaluations, and verifications, and whether algorithm results meet requirements. After all, an algorithmic recommendation audit is not like an illegal harmful information audit, where the discovery of illegal harmful information is sufficient to know immediately that there is a problem in the auditing process. The effects of algorithmic recommendations are manifested statistically over a long period of time and across a broad scope. How to determine whether an audit was truly implemented might in itself be a technical puzzle.

Second, on explaining the provision of algorithmic recommendation services and giving options for users to turn off personalized recommendations, although all the main internet enterprises have already created these functions, it is hard to say that that they are informing users “in a conspicuous manner.” The author, who is an IT professional and who was purposefully searching, still spent a fair amount of time before managing to locate the different places where the “turn off algorithmic recommendations” option was hidden in several of the major apps.

Of course, internet enterprises definitely hope to hide these functions where the majority of users cannot find them. After all, a function that the vast majority of users can’t find might as well be a non-existent function. If regulators are to prevent the right of “users to turn off algorithmic recommendation services” from becoming hollow words, shouldn’t they consider something along the lines of the [General Data Protection Regulation] (GDPR) and require that personalized recommendations be provided only after “express permission” from the user?

The GDPR requires that websites obtain users’ express permission before recording user information via cookies. It compels websites to solicit user permission in a truly conspicuous manner.

Lastly, though internet enterprises will not be able to forcibly provide personalized recommendations under the Provisions, they may still coerce users into turning on (or being unable to turn off) personalized recommendations by engaging in “slowdowns.”

To take Zhihu as an example, once one has selected the “turn off personalized recommendations” option in the privacy center, information cocoon-type recommended information will indeed stop, but there will also be a sudden drop in the volume of information that appears in the app. After turning off personalized recommendations, the author made a rough count and found that no more than three new items appeared daily on the Zhihu app’s “Select” page, while responses from many days earlier were still continuously appearing. On multiple occasions, two pieces of duplicate information appeared on the first page, yet content from the “Zhihu Hot List,” which gets the most traffic, was never pushed to the “Select” page. Is it possible that such a large company as Zhihu has no content to push to users after they turn off personalized recommendations? It seems likely that they are planning to use reduced content volume to present users with a combination of incentives and disincentives to keep them from turning off personalized recommendations. I believe that internet enterprises are capable of coming up with many more such slowdown methods. In implementing the Provisions, regulators will be faced with the new challenge of how they should discover and cope with these tricks.

These practical concerns notwithstanding, the Provisions have, after all, set a new precedent by regulating specific internet algorithms. Internet technology has long been regarded by the government and the public as a mysterious black box where only the externally visible results could be regulated. Capitalists and technologists were given too much space in which to play little games. I hope that the Provisions prove to be the first step in opening the black box and that, together with other future regulatory policies and implementation measures, they rip away the veil covering internet technology and expose the games hidden in the box to reasonable, regulatory sunlight. I also hope that the relevant regulatory authorities establish sufficient technological capabilities as soon as possible so as to truly put these regulatory measures into practice.