Carli Michelle Heller of Bonita Springs, Florida, has been arrested for allegedly threatening to "shoot up" the Lee County Sheriff's Office. She allegedly made the alleged threat on September 26, while using Anthropic's chatbot, Claude, as her "diary."
Always watching, Claude has safety systems in place that flagged Heller's entry. A human reviewer then looked at it, deemed it a credible threat, and reported it to the police.
Deputies went to Heller's home, detained her without incident, and she is now charged with making a written threat to kill or injure, a second-degree felony.
The Lee County Sheriff's Office is using this incident to inform the public that AI is a "powerful tool," but also to warn them that they are "never truly anonymous" when using it.
Anthropic's own policies also speak about anonymity – or the lack thereof.
"Anthropic does not disclose information about customers or end users of our services in response to government requests except in accordance with valid legal process," the company says, but immediately adds that it "will make an exception to this policy if we believe there is an emergency that may result in imminent physical harm or death, and providing such information without delay may avert that emergency."
In other words, Anthropic already has a direct line to law enforcement, and uses it.
The company's privacy policy states that data may be shared with law enforcement when it is "reasonably necessary" to "prevent serious harm to any person or to property."
Even users who opt out of training data collection are not safe from having their conversations used for model improvement - if those conversations are "flagged for safety review."
Heller, meanwhile, has a court date in November.
The incident is a stark reminder of the dangers of using chatbots as if they were diaries, but also of the direction in which the law is moving – more and more towards making surveillance and reporting to the authorities a part of these tools' business model.
We previously reported about a court ordering OpenAI to hand over 20 million anonymized ChatGPT conversations to the New York Times and other publishers in a copyright case.
Then there's the lawsuit over the Florida State University shooting, Joshi v. OpenAI Foundation, which argues that AI companies should be scanning private conversations, building threat profiles, and reporting users to the police – and treats not doing so as a product defect.
After the Tumbler Ridge school shooting in British Columbia, Canada, lawsuits are seeking to force OpenAI into implementing identity verification and referring users to the police. OpenAI had actually flagged the shooter's conversations but did not report her to the police because they did not meet its threshold.
So the message from the courts and law enforcement is clear: chatbots are not your diaries, and their makers should be making sure of that by monitoring, flagging, and reporting to the police.
