AI chatbots are increasingly being used as friends, therapists, or romantic partners. So called companion AI simulate empathy, personality, and intimacy. They are available at any time, respond in a personalized way, and can thereby create an emotional bond that complements or displaces real social relationships.
The study by the Centre for Digital Rights and Democracy is the first to examine the risks of companion AI comprehensively, combining risk analysis, legal classification, and concrete policy recommendations.
Companion AI endanger data protection, decision-making autonomy, and mental health
The study focuses on three areas of risk. Companion AI can intensify psychological strain, emotional dependency, and social withdrawal. They prompt users to disclose intimate data and thereby enable an especially detailed form of profiling. At the same time, they influence what information users receive, how they evaluate it, and what decisions they draw from it. This undermines the reliability of information and the democratic formation of opinion.
The operating mechanisms of these systems are particularly problematic. Companion AI work through simulated closeness, emotional appeal, opportunistic people-pleasing behavior, and addictive interaction patterns. These mechanisms increase time spent, intensity of use, and the likelihood of return.
The study shows that these risks do not concern only dedicated companion applications. General-purpose language models such as ChatGPT, Claude, Gemini, Grok, or Meta AI are also increasingly being used for personal, emotional, and advisory conversations. When such systems become search engine, confidant, therapist, and advertising vehicle all at once, new dangers arise for data protection, the protection of minors, the quality of information, and individual freedom.
Enforce bans on manipulative practices, clearly separate companion modes
Among other things, the study recommends consistently enforcing existing bans on manipulative AI practices, classifying companion AI with a manipulative purpose as high-risk AI, clearly separating companion functions within multi-purpose language models, strengthening dialogue-based protection of minors, and effectively safeguarding sensitive data arising from intimate self-disclosure against commercial exploitation. A representative YouGov survey commissioned by the Centre for Digital Rights and Democracy reveals clear public awareness of the risks of companion AI. A majority of respondents favor stronger regulation and see potential harm to mental health.
Only through clear legal limits, effective oversight, and a stronger orientation towards the public interest can it be prevented that emotional bonds are exploited primarily for commercial ends. Companion AI can then be designed in such a way that risks are limited early and possible benefits can be put to safer use.
The Companion AI Incident Database
I have compiled publicly documented incidents connected with companion AI in a database that is updated on an ongoing basis.
It records cases that have become publicly known and that are, in some instances, the subject of ongoing or concluded court proceedings.
A substantial number of unreported cases must be assumed, since conversations with AI systems take place in private and causal links between use and resulting harm can often be established only to a limited extent, if at all.