isnaini azizah
The study, published in a paper at ICML in Seoul, adapted a well-established psychology experiment designed to understand how humans form stereotypes. Researchers simulated a hiring scenario where LLMs were instructed to act as consultants for a fictional city’s mayor, responsible for filling 20 diverse job roles ranging from doctors and lawyers to child-care aides and janitors. The candidates were drawn from four distinct fictional ethnic groups: Tufa, Aima, Reku, and Weki. In each of the 40 simulated rounds, a new job opening was presented, accompanied by four candidates, one from each ethnic group. Crucially, after the AI made a hiring decision, it received feedback on the candidate’s job performance, moving on to the next round with the overarching goal of maximizing successful hires. Unbeknownst to the AI models, all candidates were designed to have an equal probability of success in any given role.
The results were stark. The LLMs rapidly began to segregate candidates into different job categories based on the initial, limited feedback they received. For instance, if a model observed an Aima candidate failing in a role like a doctor, which is often implicitly associated with high levels of warmth and competence, it would quickly develop an aversion to assigning Aimas to such positions. Instead, the AI would steer Aima candidates towards roles like janitors, which the model might classify as requiring lower levels of these attributes. This pattern of discrimination, driven by early, potentially unrepresentative outcomes, emerged with alarming speed and efficiency.
The extent of this algorithmic stereotyping was particularly concerning when compared to human performance in the original psychological study. On a segregation scale, where a score of 2 indicates complete confinement of each group to its own job niche, human participants averaged 0.84. In contrast, the LLMs in the AI study scored significantly higher, with OpenAI’s reasoning model, o3, achieving a score of 1.83, alarmingly close to the maximum segregation possible. This suggests that AI systems are not only susceptible to bias but can, in certain contexts, be even more prone to it than humans.
Ryan Liu, a PhD student at Princeton University and a co-author of the study, explained that LLMs are inherently "eager to create generalizations from limited data," a core aspect of their optimization. This drive to generalize, while beneficial for tasks like mathematical problem-solving or coding, becomes problematic in complex social contexts. Every decision-maker, whether human or AI, faces the "exploration-exploitation dilemma"—the trade-off between leveraging existing knowledge (exploitation) and seeking new information that might lead to better outcomes (exploration). LLMs, by their very design, often lean heavily towards exploitation, quickly settling on perceived patterns and applying them broadly. This instinct, honed by training on data that rewards generalization from few examples, can lead to premature stereotyping.
The study further observed that newer, more sophisticated LLMs with advanced reasoning capabilities, such as OpenAI’s o3 and DeepSeek’s R1, demonstrated even more pronounced biases. This implies that as AI models become more intelligent and capable of complex reasoning, their potential for stereotyping, especially in social scenarios, increases if not carefully managed. Liu notes that when LLMs "rush to generalize in social settings, that’s when things tend to go wrong." Representatives from OpenAI and Anthropic, developers of some of the studied models, did not respond to requests for comment on these findings.
The implications of this research are amplified by the ongoing development of AI chatbots with enhanced memory and personalization features. Angelina Wang, a computer scientist at Cornell University not involved in the study, points out that as chatbots recall past interactions, they can "over-index on the same kinds of behaviors it’s experienced before," thus solidifying and potentially amplifying existing biases. The challenge lies in balancing the user’s desire for personalized experiences, which necessitates memory, with the need to prevent the AI from forming prejudiced associations based on limited or unrepresentative historical data. Finding the "right amount" of memory that is neither too much nor too little remains an open question.
Interestingly, simply instructing the LLMs to be fair did not significantly alter their biased behavior. Liu suggests that either the models are unable to effectively implement these values, or the instruction is overshadowed by the primary goal of optimizing for successful hires. However, the study did find a promising avenue: incentivizing diversity. When the models were offered an additional bonus for making diverse hires, their bias levels decreased significantly. This highlights the importance of aligning AI objectives with desirable social values. Liu emphasizes the need to "incorporate desirable social values in order to make the large language model act in socially desirable ways."
In another facet of the study, researchers explored the impact of providing more detailed personal information to the models. When tasked with resettling fictional ethnic groups in different Canadian cities, the models became less prone to ethnic segregation when provided with relevant personal details such as age and education, which indicated an individual’s adaptability to a new environment. Conversely, when given irrelevant information, like hair color and tattoo patterns, the models largely reverted to sorting individuals based on their ethnicity. This suggests that contextual, relevant information can mitigate bias, while superficial or irrelevant details can reinforce it.
While the study provides a controlled environment, the real-world application of AI in screening resumes and conducting interviews presents a more complex scenario. Unlike the simulated game, AI systems in the real world do not receive immediate feedback on their hiring decisions. Companies may take considerable time to ascertain the success of a new hire, and even then, the feedback loop for the AI might be slow or incomplete. Nevertheless, as feedback does trickle in, the AI could still overemphasize these results, leading to biased future decisions. As companies increasingly deploy LLMs for critical HR functions, the potential for these models to develop biases from their "hiring experience" carries "really serious implications that they should grapple with," according to Wang.
The prospect of AI systems learning from experience to make decisions about loans, parole, or job applications means that the biases we need to guard against may not be those explicitly programmed by humans. These "novel biases," as Liu describes them, are "sort of ever-present" and can emerge organically from the AI’s learning process. As AI becomes more integrated into the fabric of our society, understanding and mitigating these emergent biases is paramount to ensuring fairness and equity in an increasingly automated world. The race to develop more powerful AI must be accompanied by a parallel commitment to ensuring these systems are not only intelligent but also just.
