Hermes Vector

AI & SI

OpenAI launches CSAM image classifier and ChatGPT warnings

OpenAI has released a free image classifier to detect child sexual abuse material and added ChatGPT warnings with 24-hour suspensions for attempts to find abusive content.

By Hermes Vector

Checked against 2 sourcesFox NewsTechCrunch
Illustration for: OpenAI launches CSAM image classifier and ChatGPT warnings

In brief

  • OpenAI released a free image classifier to detect CSAM, including AI-generated content, developed with NCMEC.
  • The classifier flags 48% fewer images for review while detecting roughly the same amount of abusive material.
  • ChatGPT will show warnings, remove flagged conversations, and suspend accounts for 24 hours for attempts to find abusive material.
  • OpenAI is sharing the tool through its API at no cost to vetted organizations and working with the Tech Coalition's Lantern program.

OpenAI launches CSAM image classifier and ChatGPT warnings

OpenAI has released a new image classifier designed to detect suspected child sexual abuse material, including AI-generated content, and is offering it at no cost to vetted organizations through its API. The company developed the classifier with the National Center for Missing & Exploited Children (NCMEC).

How the classifier works

The tool examines visual signs of potential child sexual abuse material and assigns scores to potentially abusive pictures so human reviewers can prioritize cases. Unlike systems that rely on digital fingerprints, it can flag content that has never been identified before.

OpenAI says early testing showed the classifier flags 48% fewer pictures for review while detecting roughly the same amount of abusive material. Pictures flagged by the new classifier were about twice as likely to contain abuse. AI platform Fal said its reviewers confirmed 72% of high-confidence flags as previously unidentified CSAM. SafetyKit reported that the classifier detected 90% of cases involving drawings or animations that sexualize children in its evaluation.

ChatGPT warnings and suspensions

OpenAI is adding new warnings in ChatGPT when someone attempts to find child sexual abuse material. The warnings explain the harm caused by sexualizing children, address AI-generated material, and direct users toward specialist support designed to help prevent harmful behavior. Along with displaying a warning, OpenAI will remove the flagged conversation and pause access to ChatGPT for 24 hours.

Repeated violations may lead to longer restrictions or permanent account closure, and could result in reports to authorities. Under existing policies, OpenAI bans accounts and reports attempts to generate or upload CSAM to NCMEC.

Additional safety measures

OpenAI is participating in the Tech Coalition's Lantern program, which allows organizations to share information about suspected exploitation across platforms. The company is also introducing specialized AI tools and training for child safety law enforcement units, and publishing guidance to help developers detect suspected abuse and establish procedures for reviewing potentially harmful content.

According to NCMEC, its CyberTipline received more than 400,000 reports involving generative AI in 2025.

Why it matters

The new classifier and ChatGPT safeguards aim to reduce the spread of AI-generated child sexual abuse material and give smaller child protection teams access to detection technology they otherwise could not afford. OpenAI says the impact will depend on how widely organizations adopt the tool and how effectively safety teams respond to warnings.


Sources

This article was drafted with AI assistance and checked against the sources above. Company claims are reported as claims. Cover image is AI-generated.

The day's checked news, by email

One short email a day with the top stories. Free, and you can unsubscribe in one click.

Or follow for the latest news:

How do you feel about this story?

Comments 0

No comments yet. Start the discussion.

AI & SI

US tightens AI chip export rules after Nvidia CEO resignation

The US Commerce Department announced new licensing requirements for advanced chip exports to five countries as Congress considers tighter restrictions following Jensen Huang's resignation from Nvidia.

Checked against 4 sourcesft.comThe Vergestartupfortune.com+1 more