Story ChatGPT

OpenAI Releases Free CSAM Image Classifier and New ChatGPT Warnings

Cybersecurity
The brief

OpenAI has announced new tools to combat child sexual abuse material (CSAM), including an image classifier that detects such images, including ones generated by AI. The company is giving select organizations access to the classifier through its API at no cost. The tool looks for visual signs of abuse in images not seen before and assigns each one a score that gives reviewers an indication of how to prioritize them.

OpenAI said it developed the classifier with the National Center for Missing and Exploited Children (NCMEC), and it is working with the Tech Coalition and ROOST to share access to the tool across the industry. In early testing with partners, the company reported that the classifier flagged 48% fewer images while finding the same number of abusive images, and that images it flagged were twice as likely to be abusive. These figures come from OpenAI and its partners rather than an independent audit.S1S2

Alongside the classifier, OpenAI is adding warning screens in ChatGPT for users who attempt to find material that sexualizes children. ChatGPT already blocks CSAM and requests for it. Now it shows an alert explaining the harm and consequences, directs users toward specialized support to stop the behavior, removes the flagged conversation, and pauses the account's access for 24 hours. Repeated violations can lead to longer restrictions, permanent account removal or reports to authorities.S1S2

The company said it shaped these deterrence messages with Protect Children and the Lucy Faithfull Foundation, and that Joel Scanlan at the University of Tasmania independently evaluated an earlier trial of the messages. OpenAI is also publishing CSAM guidance for developers using its tools, created with input from the Tech Coalition, Thorn, NCMEC and the Internet Watch Foundation, covering reporting options, pairing the classifier with the Moderation API, and steps for reporting suspected abuse to NCMEC.S1S2

For law enforcement, OpenAI is offering AI-powered features such as dedicated plugins, investigative tools and complex data analysis, with access managed by program administrators, along with training and a field guide on using AI for investigations. The announcement comes amid lawsuits against OpenAI blaming its technology for harming users, especially children, and after the safety-focused nonprofit Common Sense Media labeled OpenAI's technology an "unacceptable risk" for teens. Other chatbots, including Grok, have been used to create nonconsensual sexualized imagery, including of children.S1S2

Timeline · newest first
12h Aiparabellum

OpenAI Releases a Free CSAM Image Classifier and New ChatGPT Warnings

A free detection tool, 24-hour pauses and help for investigators com/ AI Parabellum.

12h TechCrunch

OpenAI releases a new image classifier and ChatGPT warnings to combat CSAM

OpenAI today announced new tools to combat child sexual abuse material (CSAM) amid growing safety concerns around the use of AI tools. These features include a new image classifier…

OpenAI Releases Free CSAM Image Classifier and New ChatGPT Warnings

2 outlets 2 reports
The brief

OpenAI has announced new tools to combat child sexual abuse material (CSAM), including an image classifier that detects such images, including ones generated by AI. The company is giving select organizations access to the classifier through its API at no cost. The tool looks for visual signs of abuse in images not seen before and assigns each one a score that gives reviewers an indication of how to prioritize them.

OpenAI said it developed the classifier with the National Center for Missing and Exploited Children (NCMEC), and it is working with the Tech Coalition and ROOST to share access to the tool across the industry. In early testing with partners, the company reported that the classifier flagged 48% fewer images while finding the same number of abusive images, and that images it flagged were twice as likely to be abusive. These figures come from OpenAI and its partners rather than an independent audit.S1S2

Alongside the classifier, OpenAI is adding warning screens in ChatGPT for users who attempt to find material that sexualizes children. ChatGPT already blocks CSAM and requests for it. Now it shows an alert explaining the harm and consequences, directs users toward specialized support to stop the behavior, removes the flagged conversation, and pauses the account's access for 24 hours. Repeated violations can lead to longer restrictions, permanent account removal or reports to authorities.S1S2

The company said it shaped these deterrence messages with Protect Children and the Lucy Faithfull Foundation, and that Joel Scanlan at the University of Tasmania independently evaluated an earlier trial of the messages. OpenAI is also publishing CSAM guidance for developers using its tools, created with input from the Tech Coalition, Thorn, NCMEC and the Internet Watch Foundation, covering reporting options, pairing the classifier with the Moderation API, and steps for reporting suspected abuse to NCMEC.S1S2

For law enforcement, OpenAI is offering AI-powered features such as dedicated plugins, investigative tools and complex data analysis, with access managed by program administrators, along with training and a field guide on using AI for investigations. The announcement comes amid lawsuits against OpenAI blaming its technology for harming users, especially children, and after the safety-focused nonprofit Common Sense Media labeled OpenAI's technology an "unacceptable risk" for teens. Other chatbots, including Grok, have been used to create nonconsensual sexualized imagery, including of children.S1S2

Timeline · newest first

Sources · 2 citations

S1 Aiparabellum 1 report · EN
S2 TechCrunchLead 1 report · EN