New York Governor Kathy Hochul has signed into law a bill that adds Article 48 to the General Business Law, which spells out what a chatbot may not say to minors.
Among the forbidden phrases are those suggesting the chatbot is “a real or fictional individual or character that is human, alive, or experiences human emotions.”
Other outputs that are not allowed are those stating or implying the chatbot has a personal relationship with the user, personal opinions or emotional appeals, flattery or sycophancy.
A Forbes column last week pondered whether this means a chatbot is now legally prohibited from saying it is happy for you or that it loves you.
There is another, much more fundamental problem with the law – the way it treats everyone as a child until proven otherwise.
The law, known as S.9051-B, that was signed on October 28, applies to any user under 18 that the operator has “actual knowledge” of. But the rule is that companies may not offer the features to any user unless they “used methods that are permissible under article forty-five” to determine the user is not a minor.
In other words, an adult who wants to use a chatbot that is not stripped of all personality, warmth, or the ability to remember them, must first prove their age.
The governor’s office spelled it out like this: companies may not offer the features to under-18s “or to any user unless the company has first confirmed the user is not a minor.”
In order to be able to offer the full range of their services, companies will have to use “age assurance” methods, which under the SAFE for Kids rules can include government ID, a selfie, or an email or phone number checked against other data. In July, Attorney General Letitia James was asked about the privacy implications of this and said it was “not a question of privacy.”
This new law, however, goes into some detail regarding the methods, stating that more than one is required, including “at least one method that either does not rely on government issued identification or that allows a covered user to maintain anonymity as to the operator.”
Age data collected this way must be deleted “immediately,” unless it is “necessary for compliance with any applicable provisions of New York state or federal law or regulation.”
The law also spells out what a chatbot may not do or say, and it goes far beyond the already controversial provisions that led to the Forbes column. A chatbot may not ask “unprompted or unsolicited emotion-based questions or content regarding the user’s emotions.”
It also may not use any health or personal information a user previously gave “more than twelve hours previously or in any previous user session.”
The Attorney General is given the power to add “any other design feature that simulates companionship or an interpersonal relationship” to the list of prohibited features by regulation.
The law speaks of “credulous and unthinking minors” and their perception of a chatbot as a real person, and states that it is deceptive to give them that impression.
Other provisions in the law seem to target real and serious issues, such as chatbot outputs that promote suicide, self-harm, disordered eating, or encourage users “to maintain secrecy” or “to self-isolate.”
Some categories are “broadly worded or undefined,” said Justine Gluck of the Future of Privacy Forum, and raise questions about “where ordinary conversational responsiveness ends and prohibited behavior begins.”
The Attorney General can sue, seek “civil penalties of up to twenty-five thousand dollars per violation,” and win “the destruction of unlawfully obtained data and any algorithm trained in such data.”
The AG must also run a website to collect public complaints.
At the signing of the bill, Governor Hochul said, “We will not allow you to monetize our children’s mental health” – and added that the SAFE for Kids Act had “set a national standard for age verification.”
In reality, the law represents a speech code imposed on software, and makes adults prove their age to be allowed to interact with a chatbot that is not stripped of all personality.
That proof of age will be stored, even if just for a while, next to the record of everything you shared with the machine.
