Insights from Educators, Platforms, Law Enforcement, Legislators, and Victims
In this report we aim to understand how educators, platform staff, law enforcement officers, U.S. legislators, and victims are thinking about and responding to AI-generated child sexual abuse material (CSAM). We interviewed 52 people, analyzed documents from four public school districts, and coded state legislation.
Our main findings are that while the prevalence of student-on-student nudify app use in schools is unclear, schools are generally not addressing the risks of nudify apps with students, and some schools that have had a nudify incident have made missteps in their response. We additionally find that mainstream platforms report the CSAM they discover, but, for various reasons, without systematically trying to discern and convey whether it is AI-generated in their reports to the National Center for Missing and Exploited Children's (NCMEC) CyberTipline. This means the task of identifying AI-generated material falls to NCMEC and law enforcement. However, frontline platform staff believe the prevalence of AI CSAM on their platforms remains low. Finally, we find that legal risk is hindering CSAM red teaming efforts for mainstream AI model-building companies.
This publication has been produced with financial support from Safe Online. However, the opinions, findings, conclusions, and recommendations, expressed herein are those of the authors and do not necessarily reflect those of Safe Online.
Read More
News
Challenges in the Online Child Safety Ecosystem
How to improve the system for reporting child sex abuse material online. Originally published in Lawfare.
News
How Unmoderated Platforms Became the Frontline for Russian Propaganda
In an essay for Lawfare Blog, Samantha Bradshaw, Renee DiResta and Christopher Giles look at how state war propaganda in Russia is increasingly prevalent on platforms that offer minimal-moderation virality as their value proposition.
A new Stanford Internet Observatory report examines how to improve the CyberTipline pipeline from dozens of interviews with tech companies, law enforcement and the nonprofit that runs the U.S. online child abuse reporting system.