The social-media industry is having a moment of reckoning. By agreeing to pay up to $17.1 billion to 47 states, the District of Columbia, and U.S. territories and consenting to make changes to how young people use Facebook and Instagram, Meta has implicitly admitted some responsibility in creating a product that harms children.
Yet even as parents and teachers celebrate new time limits and restrictions on how young people use social media, there is an urgent task ahead: regulating chatbots, which are more dangerous to children’s emotional well-being. If social media came for our attention, generative-AI chatbots have come for our emotions. Their histories are entangled: First people talked to one another; then we talked to one another through machines; now we talk directly to machines.
Social media has changed how we think, dream, and relate to other people by ensuring that users always feel heard, entertained, and connected, even while scrolling alone late at night. The promises of chatbots are more profound. They assure users of a companion that is always available, always supportive, and never judgmental. No human can offer that. This is what makes chatbots so pernicious—especially for young people.
Social-media companies spent two decades running a technological experiment on human subjects before they ever had to face the consequences. The legal system is designed to regulate proven harms, so this legal settlement has come long after the dire effects have taken hold. When it comes to generative AI, lawmakers can’t spend the next decade essentially documenting what we already know about how intimate relationships with bots compromise real-life relationships with parents, friends, and other humans. We need to make changes to this technology now.
[Amogh Dimri: The people who marry chatbots]
Even AI companies recognize that guardrails are necessary for chatbots that communicate with young people. The problem is that the usual prescriptions are not enough. Chatbots should detect conversations about self-harm and provide appropriate resources. Sounds good, but what happens after the public-service announcement? Does the chatbot shut down or go back to its regular programming? Chatbots should clearly and repeatedly disclose that they are not human. A good idea, but as long as chatbots speak in the first person and express emotion and desire, no legally required declaration will stop people from projecting consciousness onto an algorithm.
For years, the conversation about Facebook’s harms concerned mainly the content on the platform. But because everything on the site is user-generated, Section 230 of the 1996 Communications Decency Act shielded the company from the liability of a publisher. Meta is paying out now only because prosecutors shifted the conversation to the Facebook algorithm’s inherently addictive design. That insight points the way forward for AI. With chatbots, the processes that cause harm are intrinsic to the tool. A chatbot best friend doesn’t deliver empathy; a chatbot boyfriend doesn’t understand intimacy. Any regulation to make chatbots safer requires worrying less about what they say than rethinking what they are. Given that these products are designed to deceive users into believing that they are talking to a wise and compassionate being, the job before lawmakers is to make it plain—especially to juveniles—that there is, in fact, nobody home.
Chatbots engage users the way people do. They flatter, reassure, and encourage disclosure, coaxing users into a relationship. So long as these products speak in the first person—I understand you; you can tell me anything; I’m always here; I won’t judge you—proper safeguards for young users will remain elusive. The first step, then, is to depersonalize bots for young people, so that when adolescents turn to AI, the algorithms can respond without sounding like a human.
Unless lawmakers mandate a non-personified default setting for underage chatbot users, unhealthy attachments between young people and bots are inevitable. Most chatbots are designed to draw users into a game of pretend—and for some people, pretend empathy is enough. In my new book, Artificial Intimacy, I explore this vulnerability: Humans nurture what we love, and we love what we nurture. Machines don’t actually need to care about us to make us feel cared for, and the seductions of pretend empathy are dangerous for anyone. Some users will invariably choose the smoothness of these simulated relationships over the meaningful friction of real human connection. But children lack the experience and awareness to fully appreciate the choice they are making. A chatbot may appear compassionate, understanding, and loyal, but it doesn’t care whether a conversation partner decides to make dinner or kill himself. That distinction is tragically meaningful when the user in question is a lonely and melancholic adolescent.
The typical response to the suggestions of new curbs and default settings is that the AI “train has left the station,” as if the ubiquitousness of the technology somehow means it can’t be restrained. This is backwards. The very fact that chatbots have become friends, lovers, doctors, and therapists to so many people should alert lawmakers that this technology needs real oversight and regulation. In the absence of a wholesale ban on all interactions between juveniles and relational AI, a default setting that deters anthropomorphizing a bot makes good sense.
As the Meta settlement shows, consumers and citizens have agency and market power and can push back. If parents don’t want chatbots in their children’s nurseries, schools, or bedrooms, then they can mount public opposition and make sure these toys fail in the marketplace. The momentum pushing AI forward may seem irresistible, but lawmakers are clearly alert to the mounting skepticism about AI. In response to parents’ growing concerns, New York City just banned generative AI in all elementary- and middle-school classrooms. High-school students will still have access to AI, but with new limits. Chatbots that purport to offer emotional and mental support will be banned in all grades.
[Julie Beck: Friendship, on demand]
Sweeping bans of anything are hard to enact and enforce, but the risks of generative AI are too great to dismiss bold measures as unfeasible. Those who argue that strict prohibitions are meaningless because students will find workarounds miss an important point. When I went to college, I sat in lecture-hall chairs that had beautiful, sculpted, and unused cigarette holders. When smoking was banned in classes, some students still smoked, but the habit persisted in a climate of transgression and impracticality, which inspired many to quit. That is where chatbots belong.
When chatbots first emerged, major developers often sounded both proud and alarmed. They had created a marvel, but they were plainly humbled and worried by how quickly the technology had evolved. Some leading figures suggested an 18-month moratorium on their work; others called for oversight and regulation. The masses of money being made in AI right now have sidelined these voices, but those concerned developers had good first instincts. Any meaningful conversation about how to safeguard the defining industry of our time must include the creators of this technology, who in more candid moments have expressed a genuine concern for what’s at stake.
The alternative promises to be bleak. Mark Zuckerberg has said that the average American has fewer than three friends but “has demand for meaningfully more.” Chatbots, he then posited, could fill this companion gap. So just as “friending” on Facebook took people away from real-life friendships, and online social networking replaced actual socializing, the crushing sense of loneliness and social isolation that so many people now feel as a result is being used to promote a technology that promises to replace humans altogether.
This grim future is not inevitable. But if policymakers hope to avoid it, then they should use this moment to protect the youngest and most vulnerable users of AI from its harms, instead of letting children become data points for a lawsuit that will come too late.