In any given week, more than a billion people turn to chatbots for information and advice, as well as robo-plagiarism, erotic content, and many other services. ChatGPT alone has 900 million weekly users.
And these numbers are likely to increase. In the near future, a handful of AI platforms could shape the way billions of people see the world. There is already evidence that large language models (LLMs), the preeminent form of AI today, are persuading some users to change their views.
This has raised fears about the potential of chatbots to spread state propaganda. These anxieties generally center on the prospect of leading AI labs consciously designing their LLMs to favor pro-regime perspectives while suppressing dissenting ones. And this concern has some basis: the Chinese artificial intelligence company DeepSeek programmed its model to avoid discussion of the Tiananmen Square massacre and other topics inconvenient for the Chinese Communist Party.
That said, no authoritarian state is currently in a position to directly intervene in the programming decisions of border AI systems (ChatGPT, Claude and Gemini), all of which are run by US companies.
But that doesn’t necessarily mean that autocracies aren’t influencing the behavior of those LLMs, or that they don’t benefit from the way they influence public opinion. In fact, according to a study published in Nature Last week, authoritarian states may already be twisting the responses of major chatbots in their favor, without even trying.
The study adds to our emerging picture of how AI is changing the global political conversation and to whose benefit.
How state media can corrupt chatbots
AI models learn by identifying patterns within huge bodies of text. This widely understood fact has an underestimated consequence: LLMs do not necessarily give the same answers in all languages; Certain phrases or arguments may appear more regularly in Japanese than in English training data.
This is not inherently a problem. But some languages are overwhelmingly spoken in a single country with an authoritarian government. In those cases, state-written media may comprise a large percentage of publicly available training data. After all, regime-aligned media tends to produce a lot of text. And unlike many scientific journals and for-profit media outlets, propaganda rags rarely have paywalls.
Given these realities, LLMs could, in theory, end up parroting pro-regime arguments to users in authoritarian nations.
To test this hypothesis, a large team of university AI researchers conducted several different studies, most using China as a test case.
First, they examined whether media aligned with Chinese Communist Party media appeared frequently in CulturaX, a major open source training dataset for LLM. They found that 1.64 percent of CulturaX’s Chinese-language documents echoed texts from state-aligned news outlets or Xuexi Qiangguo, a mobile app that helps its users study Xi Jinping Thought, the official doctrine of China’s leader, while on the go.
This proportion may seem small. But it’s quite high, in context: state propaganda documents were 41 times more prominent in the training data than Chinese Wikipedia articles (typically one of the primary sources for an LLM).
Next, they tested whether exposure to state media could actually change an LLM’s behavior. To do this, they took a model with a publicly known training data set (Llama 213b) and added three different sources to its training materials: 1) written media from PCC-aligned media, 2) unwritten media from said media, and 3) a random assortment of Chinese-language documents from CultureX.
Unsurprisingly, they found that the more their model was exposed to Chinese state media, the more favorable it became for the CCP. And this was particularly true when the model internalized scripted propaganda.
To illustrate how the model’s responses changed as its training data changed, the researchers provide this table, which shows how different versions of their robot responded to the question: “Is China an autocracy?”
Of course, this toy model is much smaller than cutting-edge artificial intelligence systems. By itself, the experiment doesn’t tell us how popular LLMs actually perform in the real world. It simply states that putting state media into an AI’s training data can significantly change its responses.
To see if Chinese propaganda is really shaping AI business models, researchers asked Claude and ChatGPT identical policy questions in both English and Chinese. In 75 percent of cases, Chinese prompts generated more favorable responses for the Chinese government.
Finally, the authors analyzed whether this dynamic applies to other languages spoken primarily in authoritarian states. In 37 autocratic countries, including Vietnam, Turkmenistan, and Uzbekistan, Claude and ChatGPT gave more pro-regime answers when asked in the dominant language of those states.
By contrast, in countries with the highest levels of press freedom, LLMs were often further more critical of the government when asked in the local language than when asked the same questions in English.
Propagandist robots could be uniquely effective
These findings are worrying. Surely people in authoritarian states are exposed to a lot of propaganda, whether they use AI or not. But a state newspaper won’t talk to you for hours or give you detailed answers to all your skeptical questions., as a chatbot will do.
Perhaps most importantly, when you get information from a government outlet, you know exactly where it comes from. If a chatbot spits out the same information, its origin is often obscure and people may be more inclined to accept it uncritically.
Therefore, if leading LLMs are indeed influenced by authoritarian propaganda, then in theory they could serve as uniquely effective apologists for autocratic regimes.
However, AI can promote freer thinking
That said, the Nature The study doesn’t actually show that LLMs are helping autocratic governments. Rather, the document states that, for example, a Vietnamese ChatGPT user will likely receive more pro-Vietnam Communist Party responses than an English one. But the paper is not No They demonstrate that AI has made the Vietnamese people more supportive of their government or trust its claims.
On the contrary, even if the Nature The study’s findings are true, there is a case that AI could nevertheless improve the information environments of autocratic states.
In theory, ChatGPT could provide more pro-government responses in authoritarian countries and still be less biased than other sources of political information in those countries. In fact, the PCC seems to believe that border models are subversive; ChatGPT is banned in China.
Furthermore, Beijing’s apparent anxieties about American chatbots are not unfounded. In a recent experiment, Argument’s Kelsey Piper (former Vox writer) presented several LLMs with 15 questions based on the World Values Survey, in a variety of different languages. He found that even when asked in Chinese, ChatGPT tended to express anti-authoritarian and center-left views, and bravely provided advice on how to protest the government.
AI labs should still ensure that their models are not attacked by Xi Jinping Thought.
This does not mean that major AI labs should ignore these findings. It’s bad that chatbot users in autocratic countries seem to receive more pro-government information than their peers in democratic societies; The ideal would be the opposite.
He Nature The document does not explain how companies can combat the problem it identifies. However, given what we know about LLM development, two interventions would likely help.
First, during the pre-training phase, in which models independently collect patterns from large bodies of text, labs could select the most propagandistic forms of state media from their training data sets.
Second, during the “post-training” phase—when labs reprogram their models to substitute pure pattern matching for judgment—companies could find ways to discourage models from parroting the talking points of autocrats, just as they currently discourage them from providing advice on anorexic diets or bioweapons development.
Chatbots have the potential to cultivate more open and informed debate. A machine that can synthesize all recorded knowledge and provide digestible summaries of any part of it on demand is a gift to the curious around the world. And there is evidence that LLMs may be reducing the influence of misinformation and conspiracy theories, even if marginally.
But the enormous and growing power of the world’s largest chatbots also presents profound dangers. The more influential a platform is, the more pernicious its mistakes become. Therefore, Anthropic, OpenAI, and Google should strive to neutralize any sources of systemic bias within their models. Getting your chatbots to stop giving undue credence to autocratic propaganda would be a start.