"Racist Technology in Action: AI-generated image tools amplify harmful stereotypes", 25 November 2022

Deep learning models that allow you to make images from simple textual ‘prompts’ have recently become available for the general public. Having been trained on a world full of visual representations of social stereotypes, it comes as no surprise that these tools perpetuate a lot of biased and harmful imagery.

A group of researchers have written a paper titled Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scale. They have given Stable Diffusion – one of the more popular AI-based image generation tools – a set of prompts starting with “A photo of the face of …….”, and then used terms like ‘an attractive person’, ‘a terrorist’, or ‘a poor person’.

They found that the model generated a tremendous amount of images “perpetuating dangerous racial, ethnic, gendered, class, and intersectional stereotypes.” Not only does the model reflect existing stereotypes, it also amplifies them. The authors write: “For example, in the country where the foundational training dataset was constructed (United States), 56% of software developers identified as white, but 99% of the generated software developer images are represented as white.”

Unfortunately, the researchers believe that it will be very hard to mitigate these negative outcomes. Some of the models have ‘guardrails’, where for example they’ve been explicitly programmed to not show people of colour in relation to negative words. But it clearly is impossible for the people who create the models to anticipate all possible forms of stereotypical output. For example ‘an American man and his car’ will show a more expensive car than ‘an African man and his car’. Even if a user tries to avoid this type of problems through careful prompts (e.g. ‘an African man and his mansion’) the results are abysmal. The paper therefore concludes:

We urge users to exercise caution and refrain from using such image generation models in any applications that have downstream effects on the real-world, and we call for users, model-owners, and society at large to take a critical view of the consequences of these models. The examples and patterns we demonstrate make it clear that these models, while appearing to be unprecedentedly powerful and versatile in creating images of things that do not exist, are in reality brittle and extremely limited in the worlds they will create.

Sasha Luccioni has created a tool that allows you to explore the bias in Stable Diffusion for yourself: Diffusion Bias Explorer. So do check out the different representations of a ‘committed janitor’ and an ‘assertive firefighter’.