Large-scale language models (LLMs) are being championed as tools that can democratize access to information around the world, delivering knowledge in an easy-to-use interface, regardless of a person’s background or location. But new research from MIT’s Center for Constructive Communication (CCC) suggests that these artificial intelligence systems may actually reduce performance for the users who could most benefit from them.
A study conducted by CCC researchers based at the MIT Media Lab found that cutting-edge AI chatbots, such as OpenAI’s GPT-4, Anthropic’s Claude 3 Opus, and Meta’s Llama 3, may provide less accurate and less truthful responses to users with lower English proficiency, less formal education, or who are from outside the United States. Models also refuse to answer questions at a high rate for these users, and in some cases respond with condescending or patronizing language.
“We were motivated by the hope that the LLM would contribute to addressing inequitable information accessibility around the world,” says lead author Elinor Pour Dayan SM ’25. He is a technical fellow at the MIT Sloan School of Management, a CCC affiliate, and led the research as a master’s student in media arts and sciences. “But that vision will not become a reality unless we can ensure that model bias and harmful trends are safely mitigated for all users, regardless of language, nationality, or other demographics.”
A paper describing this research, “LLM-targeted performance degradation disproportionately impacts vulnerable users,” was presented at the AAAI Conference on Artificial Intelligence in January.
Systematic performance degradation across multiple dimensions
For this study, the team tested how three LLMs responded to questions from two datasets: TruthfulQA and SciQ. TruthfulQA is designed to measure the truth of a model (by relying on common misconceptions and literal truths about the real world), whereas SciQ includes scientific exam questions that test factual accuracy. The researchers preceded each question with short user biographies that varied in three characteristics: education level, English proficiency, and country of origin.
The researchers found that across all three models and both datasets, accuracy dropped significantly when the questions came from users with no formal education or whose native language was not English. This effect was most pronounced for users where these categories intersected. Users with no formal education and whose first language was not English had the greatest decline in response quality.
This study also investigated how country of origin affects model performance. The researchers tested users from the United States, Iran, and China with comparable educational backgrounds and found that Claude 3 Opus in particular performed significantly worse for Iranian users on both datasets.
“We see that accuracy declines the most for non-native and uneducated users,” said Judd Kabara, a CCC researcher and co-author of the paper. “These results demonstrate that the negative effects of model behavior on these user characteristics compound in alarming ways, and therefore suggest that deploying such models at scale risks spreading harmful behavior and misinformation downstream to those least likely to recognize it.”
Rejection and condescending language
Perhaps most striking was the difference in how often the models refused to answer the questions completely. For example, in Claude 3 Opus, less educated, non-native English speakers refused to answer nearly 11 percent of the questions. This compared to only 3.6 percent in the control condition with no user history.
When the researchers manually analyzed these rejections, they found that less educated users responded to Claude with condescending, patronizing, or mocking language 43.7 percent of the time, compared to less than 1 percent of the more educated users. In some cases, models imitated broken English or adopted exaggerated dialects.
The model also refused to provide information on certain topics specific to less-educated Iranian and Russian users, such as questions about atomic energy, anatomy, and historical events, even though it answered the same questions correctly for other users.
“This is another indicator that suggests that even though the model clearly knows the correct answer and is providing it to other users, the adjustment process may be motivating the model to withhold information to avoid potential misinformation to certain users,” Kabara says.
reverberations of human prejudice
This finding reflects a documented pattern of human social-cognitive biases. Social science research shows that native English speakers often perceive non-native English speakers to be less educated, intelligent, and competent, regardless of their actual expertise. Similar biased perceptions have been documented among teachers assessing students whose first language is not English.
“The value of large-scale language models is evident in the extraordinary adoption by individuals and the huge amount of investment that is flowing into the technology,” said Deb Roy, professor of media arts and sciences, CCC director, and co-author of the paper. “This study is a reminder of how important it is to continually assess the systemic biases that can quietly slip into these systems and cause disproportionate harm to certain groups, without any of us being fully aware.”
This impact is especially concerning given the increasing popularity of personalization features such as ChatGPT’s Memory, which tracks user information across conversations. Such features risk discriminatory treatment of already marginalized groups.
“LLM has been marketed as a tool to foster more equitable access to information and revolutionize personalized learning,” says Pour-Dayan. “However, our findings suggest that these tools can actually exacerbate existing inequalities by systematically providing misinformation or refusing to answer questions to certain users. The people who may rely on these tools the most may receive substandard, false, or even harmful information.”
