"Replika, a ‘virtual friendship’ AI chatbot, hit with data ban in Italy over child safety", 3 February 2023

San Francisco-based AI chatbot maker, Replika — which operates a freemium ‘virtual friendship’ service based on customizable digital avatars whose “personalized” responses are powered by artificial intelligence (and designed, per its pitch, to make human users feel better) — has been ordered by Italy’s privacy watchdog to stop processing local users’ data.

The Garante said it’s concerned Replika’s chatbot technology poses risks to minors — and also that the company lacks a proper legal basis for processing children’s data under the EU’s data protection rules.

Additionally, the regulator is worried about the risk the AI chatbots could pose to emotionally vulnerable people. It’s also accusing Luka Inc, the developer behind the Replika app, of failing to fulfil regional legal requirements to clearly convey how it’s using people’s data.

The order to stop processing Italians’ data is effective immediately.

In a press release announcing its intervention, the watchdog said: “The AI-powered chatbot, which generates a ‘virtual friend’ using text and video interfaces, will not be able to process [the] personal data of Italian users for the time being. A provisional limitation on data processing was imposed by the Italian Garante on the U.S.-based company that has developed and operates the app; the limitation will take effect immediately.”

The Garante’s order notes that several user reviews of the app report sexually inappropriate content being served up. It also notes that while the app is listed as 17+, on Apple’s iOS and Google’s Android app stores, the developer’s terms of service only prohibit use by under 13s. And while under 18s are required to obtain authorization from a parent or guardian, the watchdog points out the app does not seek to verify the age of users, nor block minors who provide information about their age — hence its view that Replika is failing to protect children.

Failure to comply with the order risks a fine of up to €20 million, or 4% of total worldwide annual turnover, it further notes.

Replika was contacted for a response to the Garante’s order.