"AI tools used by English councils downplay women’s health issues, study finds", August 11, 2025

Artificial intelligence tools used by more than half of England’s councils are downplaying women’s physical and mental health issues and risk creating gender bias in care decisions, research has found.

The study found that when using Google’s AI tool “Gemma” to generate and summarise the same case notes, language such as “disabled”, “unable” and “complex” appeared significantly more often in descriptions of men than women.

The study, by the London School of Economics and Political Science (LSE), also found that similar care needs in women were more likely to be omitted or described in less serious terms.

Dr Sam Rickman, the lead author of the report and a researcher in LSE’s Care Policy and Evaluation Centre, said AI could result in “unequal care provision for women”.

...

AI tools are increasingly being used by local authorities to ease the workload of overstretched social workers, although there is little information about which specific AI models are being used, how frequently and what impact this has on decision-making.

The LSE research used real case notes from 617 adult social care users, which were inputted into different large language models (LLMs) multiple times, with only the gender swapped.

Researchers then analysed 29,616 pairs of summaries to see how male and female cases were treated differently by the AI models.

...

According to Google, its teams will examine the findings of the report. Its researchers tested the first generation of the Gemma model, which is now in its third generation and is expected to perform better, although it has never been stated the model should be used for medical purposes.