Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

OpenAI releases CSAM image classifier and ChatGPT warnings

AI summary

OpenAI released an image classifier that scores potential child sexual abuse material (CSAM), including a signal for images that may be AI-generated. Selected reviewed organizations can request free API access. Its model card says the classifier may produce false positives and false negatives, has not been tested on video frames, and should be used with hash matching, contextual and behavioral abuse detection, and trained human review rather than as confirmation. OpenAI also added ChatGPT warnings, removal of flagged conversations and 24-hour account pauses for attempts to find or generate material sexualizing children, along with developer guidance and support for child-safety law enforcement.

In-depth insight

OpenAI says early partner testing found that the classifier flagged 48% fewer images while identifying the same number of abusive images, potentially reducing reviewers’ workload. The model card cautions that performance on OpenAI traffic does not guarantee equivalent results on other platforms, so deployers must benchmark representative traffic and choose their own thresholds.

The classifier can produce both false-positive and false-negative errors. The following limitations should inform how results are interpreted and how the classifier is deployed.

Data

  • Generalization beyond OpenAI traffic. Our online evaluation used a sample of ChatGPT traffic. Performance and suitable operating thresholds may differ on other platforms, where content distributions, abuse patterns, and the prevalence of CSAM can be different. Results on OpenAI traffic should not be treated as a guarantee of performance on another service. Deployers should benchmark on appropriate representative traffic from their own service and select thresholds for their use case.
  • Residual overlap between datasets. Perceptual deduplication reduces overlap from duplicate or near-duplicate images but does not address all related material. CSAM can contain different images of the same children over multiple years, and we currently lack the data needed to prevent this form of overlap between training and evaluation sets. Evaluation results should therefore be interpreted in light of this remaining risk of data leakage.
  • Limited availability of evaluation data. Meaningful false-positive evaluation should include adult sexual content that is not CSAM, because this content can be visually similar to CSAM. However, there are not many academic or benchmark datasets containing adult sexual content for use in classifier creation. Additionally, filtering a dataset to remove CSAM also does not establish that every remaining image is non-CSAM.

#OpenAI#ChatGPT#OpenAI CSAM image classifier#National Center for Missing and Exploited Children#Tech Coalition#ROOST#Protect Children#Lucy Faithfull Foundation#Anthropic#Google

Original · 2 min readRead the original
Group chat

No comments yet. Start the conversation.