New report highlights the shortcomings of large language models in analysing non-English content
"Lost in Translation: Large Language Models in Non-English Content Analysis", 23 May 2023.
...A new report from CDT examines the new models that companies claim can analyze text across languages. The paper explains how these language models work and explores their capabilities and limits...
...In the past, it has been difficult to develop artificial intelligence (AI) systems — and especially large language models — in languages other than English because of what is known as the resourcedness gap. This gap describes the asymmetry in the availability of high quality digitized text that can serve as training data for a model. English is an extremely highly resourced language, whereas other languages, including those used predominantly in the Global South, often have fewer examples of high quality text (if any at all) on which to train language models...
...while multilingual language models show promise as a tool for content analysis, they also face key limitations:
- Multilingual language models often rely on machine-translated text that can contain errors or terms native language speakers don’t actually use.
- When multilingual language models fail, their problems are hard to identify, diagnose, and fix.
- Multilingual language models do not and cannot work equally well in all languages.
- Multilingual language models fail to account for the contexts of local language speakers.
These shortcomings are amplified when used in high risk contexts. If these models are used to scan applications for asylum for example, errant systems may limit a users’ ability to access safety. In content moderation, misinterpretations of text can result in takedowns of posts which may erect barriers to information, particularly where not a lot of information in a particular language is available...
...Governments, technology companies, researchers, and civil society should not assume these models work better than they do, and should invest in greater transparency and accountability efforts in order to better understand the impact of these models on individuals’ rights and access to information and economic opportunities. Crucially, researchers from different language communities should be supported and be at the forefront of the effort to develop models and methods that build capacity for tools in different languages...