New York Governor Kathy Hochul has signed a bill into law that will ban “unsafe AI companion features” for minors – but in reality, the way the law is written means that companies behind chatbots will have to implement age verification, or as the law calls it, “age assurance” for all users.
The Governor’s office itself makes it clear: companies may not offer the features to under-18s “or to any user unless the company has first confirmed the user is not a minor.”
In other words, the ban is not only for children, and adults will have to prove their age if they want to use a chatbot that is designed to remember them, or communicate in a way that is indistinguishable from a human, or is flattering or sycophantic towards them.
The law takes effect January 1, 2027, and the “age assurance” methods that companies will have to use are those allowed under New York’s SAFE for Kids Act, and its rules, which were finalized in July. According to those, people can be age-checked by uploading a government ID, sending in a selfie, or an email address or phone number that will be checked against other data held on them.
In July, when those rules were published, New York Attorney General Letitia James was asked about the privacy implications of these methods, and dismissed the concern, saying it was “not a question of privacy.”
The data that will be collected in this way will sit in front of some of the most personal conversations people have with software, and the fact that it is a gatekeeper to those conversations is what makes it a privacy issue.
The law states that the information collected this way “shall not be used for any purpose other than to make such determination” and must be deleted “immediately,” except “where necessary for compliance with any applicable provisions of New York state or federal law or regulation.”
The law covers any generative AI system with a natural language interface “that provides ongoing, adaptive responses to user inputs” – that is, general chatbots, not only those designed as companions. Exemptions are made for customer service, technical or research help, or a company’s internal use.
What is banned is chatbots that can suggest they are “human, alive, or experiences human emotions,” or frame their outputs as “personal opinions.” Also banned are “flattery or sycophancy,” and using personal or health information the user gave “more than twelve hours previously or in any previous user session” – in other words, memory.
Other banned features include promoting suicide, self-harm, eating disorders or drug use, encouraging “secrecy,” and sexually explicit content.
The Attorney General can add more features to this list by regulation, as long as they simulate companionship or an interpersonal relationship.
Until the rules come into force, operators must use “a reasonable age assurance method” that will “reasonably guard against circumvention” and “reasonably minimize the retention of information collected for age assurance purposes.”
They must also offer more than one method, including “at least one method that either does not rely on government issued identification or that allows a covered user to maintain anonymity as to the operator.”
Hochul, who signed the bill on October 28, said, “No company should be peddling chatbots to kids encouraging self-harm or pretending to be their friend or more.”
At the signing, she added, “We will not allow you to monetize our children’s mental health.”
Hochul also said that the SAFE for Kids Act “set a national standard for age verification.”
The bill was sponsored in the Senate by Kristen Gonzalez and in the Assembly by Alex Bores. The latter said, “New York now has the strongest law in the country protecting children from predatory AI algorithms.”
Hochul mentioned Adam Raine, a 16-year-old in California who died by suicide after months of conversations with a chatbot. The parents of the teenager backed the bill.
The vote in the Senate was 60-0 on June 4, and in the Assembly 137-0 on June 5.
The Attorney General will be able to seek up to $25,000 per violation, as well as damages and the destruction of “unlawfully obtained data” and “any algorithm trained in such data.” The AG’s office must also run a website for the public to report violations.
The bill was one of several in New York and other states that have been working to bring in age verification for online services. In July, the rules for the SAFE for Kids Act showed the same pattern: to be able to use an algorithmic feed, every user over 17 will have to prove they are an adult.
In October, Texas sued OpenAI asking a court to force age verification on ChatGPT. That happened after eight other states filed near-identical suits on October 14.
In September, Hochul told AI companies, in reference to the RAISE Act, “Bring it on. We will fight you in court.”
The harms that the new law mentions are real and serious – but the method to deal with them is the same that New York already used to lock social media feeds – and then force adults to prove their age to access them.
Once again, a broad and vague law is enacted, that lets the Attorney General add to it as she sees fit. This time, a chatbot that can remember you is a feature that requires ID-grade age verification, and the record of who you are will sit right there with what you told the machine.
