School children are using AI image generators to create indecent fake images of classmates, an internet safety organization warns

• Schools have told a child protection organization that AI image generators are being used to make indecent content.
• The UK Safer Internet Centre is calling for "urgent action" and warning schools to up their monitoring systems.

Children are using AI image generators at school to create indecent content of other children, experts on online child safety warned on Monday.

The UK Safer Internet Centre is calling for "urgent action" to help children understand the risks associated with such AI images, which it says legally constitute child sexual abuse material.

The organization said that while children may be generating the images out of curiosity, they can quickly get out of hand and circulate online, and be used to abuse or blackmail children.

Emma Hardy, the UKSIC director, said the AI images were photorealistic and comparable to professional photos.

"The potential for abuse of this technology is terrifying," she said. "This is not some theoretical risk. It's something we are seeing here and now."

"We must see measures put in place to prevent the abuse of this technology. Right now, unchecked, unregulated AI is making children less safe."

UKSIC said it had received small numbers of reports from schools, and called on them to ensure their monitoring systems can block illegal material effectively.

"Young people are not always aware of the seriousness of what they are doing, yet these types of harmful behaviors should be anticipated when new technologies, like AI generators, become more accessible to the public," said David Hardy, UKSIC director.

"Although the case numbers are currently small, we are in the foothills and need to see steps being taken now, before schools become overwhelmed and the problem grows," he added.

AI fakes are a major concern

The rise of AI image generators has come with plenty of controversy.

In September, fake naked pictures of young girls created by AI sparked fury in a small Spanish town.

AI has also been used to clone voices, prompting backlash from the likes of Bad Bunny in response to a TikTok song that used his voice and was generated by the technology.

And it was a key debate in this year's SAG-AFTRA strike, as Hollywood studios explored using AI-generated "digital doubles" of actors.