Lamarr NLP Researchers Train Multilingual Large Language Models Mitigating Stereotype Bias
August 15, 2024
Bias in large language models is a well-known and unsolved problem. In our new paper “Do Multilingual Large Language Models Mitigate Stereotype Bias?” we address this challenge by investigating the influence of multilingual training data on model bias reduction.
In the Lamarr NLP research collaboration between Fraunhofer IAIS and the Language Technologies group at the university of Bonn, we have trained six large language models on public data (one for each of Spanish, German, French, Italian and English, and a combined multilingual one), and compared these LLMs to their state-of-the-art counterparts on multilingual bias benchmarks. Our results show that all multilingual models trained on the same number of tokens as monolingual models are less biased for all languages and benchmarks. In addition, our models are generally less biased than selected open-source LLMs of similar size.
We are very happy to announce that our work, conducted by Shangrui Nie, Michael Fromm, Charles Welch, Rebekka Görge, Akbar Karimi, Joan Plepi, Nazia Afsan Mowmita, Nicolas Flores-Herr, Mehdi Ali and Lucie Flek, has been accepted at the C3NLP (Cross-Cultural Considerations in NLP), collocated with the ACL 2024 in Bangkok.
The challenge of bias in large language models
Large language models allow us to quickly and easily derive and implement real-world applications in fields as diverse as healthcare, finance and law. Huge amounts of data are used to train these models, resulting in incredible model performance. However, numerous studies have shown that large language models learn biases during training that can lead to discrimination against certain groups of people in their downstream application. Often these biases arise from the training data itself, and they vary between different languages and models.
Multilinguality as a solution approach
To avoid the harm of discrimination, research aims to mitigate bias in large language models. Among various approaches, previous research indicates that using multilingual training data to train large language models reduces model bias. In doing so, the models can benefit from the use of different languages, which differ in semantics and syntax and cover a wider cultural diversity. Within our work, we are building up on this previous work by investigating especially in the impact of monolingual versus multilingual data of larger, decoder-based language models.
Our experiments and findings
In our experiments, we train six novel large language models, one each for Spanish, German, French, Italian and English, as well as a multilingual model trained on all five languages but using the same number of tokens. To compare the bias of the monolingual and multilingual models, we benchmark them against two well-known bias evaluation benchmarks CrowS-Pairs and BBQ. The prior measures the degree to which a model prefers stereotyping over a less stereotyping sentences, while the latter is a question-answering dataset that requires from a model to answer stereotype-reflecting questions regarding two social groups (compare Figure 1). As both benchmarks are originally only available in English, we perform a human-validated automated translation.
Our results support the initial hypothesis that multilingual large language models reduce model bias. Within the experiments, we find that all multilingual models trained on the same number of tokens as monolingual models are less biased than the monolingual models for all languages and both benchmarks. We also find that our models are generally less biased than selected open-source large language models of similar size, although they fall short of zero-shot prompt-based approaches with GPT3.
The translated multilingual bias datasets and the code used for LLM training are published at github.
Prof. Lucie Flek contributed to this article.