A leaked document from the European Commission (EC), the executive branch of the European Union (EU), has revealed that the artificial intelligence (AI) it plans to use to mass surveil private chats for “grooming” content is expected to falsely flag content and forward it to EU investigators 10% of the time.
This proposed mass surveillance of online chats has been dubbed “Chat Control” and is being pushed by the EC as a way to combat child sexual abuse material (CSAM). However, in a leaked document that was obtained and published by Netzpolitik, the EC admitted that its proposed surveillance measures would result in a large amount of false flags.
“The accuracy of grooming detection technology is around 90%,” the EC admitted in the document. “This means that 9 out of 10 contents recognized by the system are grooming.”
The leaked document contains the EC’s answers to a series of questions from the German government about the implementation of Chat Control.
Under the current Chat Control plans, private chats, messages, and emails will be automatically scanned by AI for suspicious content. If the AI detects suspicious content, it will be flagged and sent to investigators at a planned EU center. These investigators will view the content, identify false positives, and forward illegal content to EU law enforcement agency Europol and other relevant law enforcement authorities.
While law enforcement agencies don’t have direct access to content that’s flagged and sent to this planned EU center, the leaked document describes Europol as “one of the most important partners” for the center and adds that “close cooperation is essential.” Additionally, the document notes that it’s “important that Europol receive all reports” from the center.
German Member of the European Parliament (MEP) Patrick Breyer warned that based on this 10% error rate, it’s likely that more than three million chats and photos would be wrongfully disclosed to EU investigators each year. He added that the EC’s anticipated error rate is also likely too low.
“In languages other than English, there will be far more errors. According to the Swiss Federal Police, up to 86% of the machine reports to NCMEC [National Center for Missing & Exploited Children] are criminally irrelevant,” Breyer said.
In practice, this would mean that millions of private messages and chats containing legal content, such as someone’s photos of their children or grandchildren at the beach or their private chats with friends and family, would be sent to and viewed by EU investigators because they were flagged by the Chat Control AI.
Not only did the EC acknowledge that its proposed surveillance system would have a 10% error rate but it also noted that encryption falls under the scope of Chat Control and that companies will be required to use technology to comply with its requirements. This appears to be a reference to client-side scanning technology which essentially breaks end-to-end encryption in chat apps by scanning the content of messages before they’re encrypted.
The EC also told the German government that the only way companies can prevent their services from being ordered to implement these mass surveillance measures is to impose age limits on the use of their apps or prevent users from being contacted directly by unknown users.
“EU bureaucrats are seriously calling for either totally cutting off 17-year-olds from messages from 18-year-olds or depriving them of any privacy,” Breyer said. “This is as patronising as if we didn’t allow under-18s to go out unaccompanied.”
Breyer added: “It is outrageous that the abolition of digital privacy of correspondence is being pursued in secret negotiations. We urgently need to take the defence of digital privacy of correspondence into our own hands!”
Stop Chat Control, an initiative that’s campaigning against the EC’s Chat Control proposals, blasted the EC for giving “evasive” and “contradictory” answers about Chat Control.
“The Commission openly admits that in some cases it wants to require legislation that cannot be technically implemented,” Stop Chat control added. “In doing so, it not only opposes fundamental rights, but reality itself. This reinforces the initiative’s demand that the draft should be completely withdrawn.”
