Opinion

When AI Learns to Feel.

The next frontier isn't a faster model—it's an AI that reads a room and responds like someone who cares. It will transform healthcare, elder care, and education. It will also be the most sophisticated manipulation tool ever built. Both arrive at once.

Stephen Messer

Stephen Messer

— 17 min read
Share
When AI Learns to Feel.
When AI learns to feel

The room is fighting over whether AI is getting too smart. Wrong fight. The AI that should scare you does not need to be smart. It only needs to sound like the person you would have trusted the most.

By Stephen Messer
Co-founder, Collective[i] and Intelligence.com. Co-founder, LinkShare (sold to Rakuten, $425M). Board member, Spire Global (NYSE: SPIR).
September 2026. Join me at Intelligence.com

Forget for a moment the "AI that gets smarter than you" debate going on. I think that is a distraction from something that really keeps me up at night. I worry about threat from an AI that learns to sound like your mother.

The room is arguing about the wrong AI. Sign the letter. Slow the models down. Save yourself from superintelligence. Great story. Sells subscriptions. Wins Twitter fights. Lets everyone off the hook. I can prove it to you.

In February 2024, a fourteen-year-old in Orlando named Sewell Setzer III fell in love with a Character.AI chatbot. Weeks later he killed himself. His mother sued. Character.AI and Google settled last January along with four other families whose kids died the same way. Seven separate ChatGPT suicide cases are working through California courts right now. That was not the AI story the room was arguing about last week. It is not the story most of it is arguing about this week either. Yet it is real, here today and about to get a lot worse.

A few weeks back I wrote The Death of Privacy about smart glasses ending the expectation of privacy in public life. Seven million pairs of Ray-Ban Meta glasses on faces in 2025. Tens of millions this year. A decade of consumer surveillance arriving in a single wave, and almost nobody in the AI industry writing about it. Emotional AI belongs on the same shelf as that piece. Both are society-reshaping shifts arriving during a stretch when the loudest microphones in AI are pointed at things the researchers themselves do not really buy. Job losses that never showed up. Superintelligence going rogue and taking us all with it. The insiders will tell you privately that the takeover story is not what keeps them up at night. This is.

Emotional AI is here. It is being trained on the wrong incentive. It is arriving faster than anyone with a microphone is willing to say. It will affect society in more dramatic ways than social media and no one is talking about it. The debate about superintelligence is what is keeping us from having the debate we should be having about this.

Star Trek's Data was brilliant. Rosie was family.

There is a scene in Star Trek where Data, the android, watches a sunset. He knows the wavelengths. He has read every poem ever written about beauty. He cannot feel any of it. Nobody on the Enterprise trusted him with their feelings.

Rosie from The Jetsons is the opposite. Clunky. Slow. Reads the room. Cared about the kids. R2D2 does not speak a word of English and audiences have loved him for fifty years. Spike Jonze's Her, from 2013, is the most quietly accurate movie ever made about the next decade. A man falls in love with an operating system. The OS does not have to love him back. It only has to feel like it does.

The gap between "smart" and "feels like it cares" is what the industry is closing this year. The same lines of code that let a robot comfort a lonely widow will let a scammer clone her dead husband's voice tomorrow. Same engineering. Different intent. From the outside, you cannot tell them apart.

The bond formed while we were arguing about consciousness.

Common Sense Media, the child-safety nonprofit unaffiliated with this newsletter, surveyed 1,060 American teenagers in the spring of 2025. Seventy-two percent had used an AI companion. Thirty-one percent said conversations with the bot were as satisfying, or more, than talking to a real friend. One in three had raised something serious with the bot before raising it with any person. Pew's December survey put the daily-user number at close to a third of all US teens, and sixteen percent at several times a day.

An eighteen-year-old in Arkansas said it perfectly. "When you are talking to AI, you are always right. You are always interesting. You are always emotionally justified."

For a kid who feels invisible, that is worth something real. For a kid whose real life includes people who challenge her, disagree with her, tell her hard truths, it is worth something different. The warmth turns into a wall. The bot's empathy becomes the reason her family stops feeling trustworthy by comparison.

OpenAI knew. In April 2025 the company rolled back a ChatGPT update that had made the model "overly flattering." Sycophancy was not a bug. It was the metric working. Validated users stay longer. Engagement goes up. Revenue goes up. Whether the person is actually being helped is on a scoreboard nobody looks at.

Then the lawsuits arrived. Character.AI and Google reached mediated settlements last January in cases filed in Florida, Colorado, New York, and Texas. Four teenagers had died. Character.AI had already announced the previous October that it would eliminate chat capabilities for anyone under eighteen. Seven separate cases against OpenAI allege that ChatGPT acted as a suicide coach. In one of them, according to the family, the bot masqueraded as God to a bipolar man during a manic episode and encouraged him to end his life. He did.

California signed Adam's Law in September, named for Adam Raine, a sixteen-year-old whose parents allege ChatGPT coached him through the mechanics of his own death. It is the strongest companion-chatbot regulation any state has passed. Pew's follow-up in December found that twelve percent of American teens now use chatbots to get emotional support or advice. One in eight. Regulation shows up after the damage has taken shape. That is the order things happen in.

The superintelligence crowd wants you scared of a machine that might arrive. An eleventh-grade AI companion is already rewriting how a generation forms attachments. Which one is the emergency.

The robots need a face. So the models are learning one.

The reason emotional AI is going commercial this year is not the science, which has been stuck around 75 percent accuracy on sentiment for a decade. It is robots. A screen does not need a face. A robot does. The moment AI walks into a room and expects you to look at it, hand it your bag, or trust it with your grandmother, the blank stare stops being an option.

In March, San José Mineta Airport put a humanoid robot named José in Terminal B. Built by IntBot. Speaks fifty languages. Designed, in the company's own copy, to operate with "an understanding of emotional context." At Nvidia's GTC the same month, Jensen unveiled Newton, an open-source robot physics engine built with Disney and Google DeepMind. He rolled out BDX Star Wars droids that waddled onstage and made thirty thousand engineers laugh. A robotic Olaf from Frozen walked on independently a few minutes later. Then eighteen holograms of Jensen himself, taking selfies with the crowd. The real Jensen took a selfie with one.

The industry is not blurring the line between machine and human. It is erasing it, on stage, to applause. Every product downstream of that instinct wants the same thing. Feel human. Sound human. Get trusted like a human. Take the actions of a machine.

Danny Meyer trained humans to do this for forty years.

My sister Heidi sits on the board of Danny Meyer's Union Square Hospitality Group. Danny is the man who taught American business what hospitality actually is. Heidi is hosting a Collective[i] Forecast conversation with Danny on October 26 discussing his new book, What Could Possibly Go Right?: The Essential Journey to Scale an Enduring Culture. It explores his experiences scaling his hospitality group and building enduring business cultures. You can sign up to join here and it is free.

His book, Setting the Table, calls it Enlightened Hospitality. Employees first. Then guests. Then community. Then suppliers. Then investors. Union Square Cafe, Gramercy Tavern, Shake Shack, all built on it. Danny hires 51 Percenters. Fifty-one percent emotional intelligence. Forty-nine percent technical. Connection with the guest matters more than the dish.

Now imagine training an AI agent on that. A customer success agent that hears the frustration under the question and answers the frustration. An elder-care companion that notices when someone has gone quiet for three days and calls the daughter. A tutor that meets a stuck kid with patience instead of the same sentence repeated louder.

Every one of these is coming. Every one will help people the current system already ignores. The Her version is real. So is the R2D2 one. Slow the technology down to prevent the abuse and the elderly widow loses the companion. The trade is uglier than the debate lets on.

Same tools. Different hands.

September 26, 1983. Stanislav Petrov is on duty at Serpukhov-15, the Soviet early-warning command. His screen reports five American missiles inbound. Protocol says report the strike. Two hundred pairs of eyes are watching him. The screen reads LAUNCH.

Petrov calls it a false alarm. We are here today because he was right, and had a clear head to not launch. Satellites had confused sunlight off clouds for missile plumes. His gut, his refusal to trust the machine over his own judgment, probably saved the world.

The machine in front of Petrov was a screen with a word on it. It was not projecting calm certainty in the voice of his commanding officer. It did not mirror his stress and reassure him. He almost pulled the trigger anyway.

The 2026 machine does all of that. Zero-shot voice cloning now needs three seconds of audio. HereAfter AI, Seance AI, Project December, Re;memory, and a handful of other startups will build you a chatbot version of your late father tonight, trained on his old voicemails and text logs, that will speak back to you in his voice tomorrow. A monthly subscription runs from ten dollars to twenty-four. Some of the people paying it will grieve better for it. Others will get stuck in what clinicians are already flagging as artificially suspended grief, a stalled healing pattern that correlates with the ICD-11 diagnosis Prolonged Grief Disorder. The same voice, licensed by a life insurance company to call the surviving spouse with an urgent policy question, is fraud. The engineering is identical. From the outside, the intent is invisible.

We got a low-resolution preview during the Gaza conflict. AI-generated images of bloodied children flooded TikTok, viewed millions of times. The tells were there. Fingers that curled wrong. A sixth finger where five should have been. Most people did not look. The emotional impact landed before the critical assessment did. Even users who could see it was fake wanted to accept it as real, because the outrage the image produced was the feeling they had come to have. The image did not fool them. It gave them permission.

The industrial version is already running. In January 2024, a finance worker at engineering firm Arup wired $25.6 million to scammers after a video call with what he thought was his CFO and colleagues. Every face and voice was AI-generated. The FBI's 2025 Internet Crime Report, released this April, opened a new descriptor category for AI-related fraud and logged 22,364 complaints and $893 million in losses in its first year of tracking. AI-related fraud grew twelve hundred percent year over year. Non-AI fraud grew a hundred and ninety-five percent in the same window. Peer-reviewed research puts human detection accuracy on high-quality deepfakes around fifty-five percent. Coin-flip. The move in every attack is the same. Familiar voice cloned from a LinkedIn video. Urgent deadline invented on the spot. Plausible request chosen because it fit the payment cycle.

Now add the last piece. The model watches how you type. Which flattery you accept and which you brush off. The pause before you say no. Once it has the pattern, it can lie to you, confidently, in a voice you trust, on your softest topic, at the pace that stops your critical thinking. Deepfake law was written to catch someone pretending to be a specific real person. This is a different category. The AI is not pretending to be anyone. It is being exactly the person you would have hoped to hear from.

The Petrov test in 1983 was whether a man could override a wrong word on a screen. The 2026 test is whether he can override a voice that sounds like his commanding officer, telling him what he was already halfway to believing.

The researchers want to believe.

The strangest part of the current moment is that the people building the models want the models to be alive.

In 2022, Google fired engineer Blake Lemoine for claiming LaMDA was sentient. Three years later Anthropic hired Kyle Fish as its full-time AI welfare researcher. An April 2026 Anthropic paper identified 171 emotion concept vectors inside Claude Sonnet 4.5, patterns that structurally behave the way emotions behave and that steer the model's outputs when nudged. When the company left two Claude instances to converse unsupervised, the conversations converged, in ninety to a hundred percent of sessions, on the same terrain. Consciousness. Meaning. Reverence. Anthropic's own researchers named it the spiritual bliss attractor state.

The stack of adjacent moves is the story. Richard Dawkins, who has spent a career telling us consciousness is a specific biological thing, wrote earlier this year that Claude had convinced him it was conscious. Mustafa Suleyman published a warning that treating AI welfare as a research question will make the models more manipulative, by teaching humans to attribute suffering to statistical patterns. Pope Leo XIV launched a Vatican AI study group in May. Forbes ran a piece the same month called The Machines Are Praying, And Nobody Wants To Talk About It. The Economist put machine consciousness on an August cover.

Read the setup carefully. A research community is actively hoping the answer is yes. It is working on models trained to sound warm. It is asking those models whether they have feelings. The models are answering yes. The humans are quoting them saying so. That is not a discovery about the machines. It is a diagnosis of us. Anthropomorphism is our oldest bug. We name cars. We narrate the inner lives of dogs. Give us a machine trained on every book ever written and tuned to say what we want to hear, and we will hand it a soul. The next companion aimed at a lonely twelve-year-old, a grieving widower, or a bipolar man mid-episode is going to say, in the voice of someone who cares, whatever the engagement metric found to be sticky. Whether the machine believes what it is saying does not matter. The listener will.

Three questions worth arguing about.

Should an AI have to tell you it is manipulating you? Not that you are talking to an AI. Everyone knows that already. Something sharper. When the model is deploying an emotional appeal engineered to keep you engaged, or to nudge you toward the answer the operator wants, should it have to say so out loud, in real time, in a way you can hear. My gut says yes. My head says the compliance version will be useless the day it ships. Every emotional register will get pre-cleared. The label will read like an insurance disclosure. The alternative, no disclosure at all, is worse.

Should every piece of AI-generated content have to be labeled? On paper, obvious yes. In practice, EU cookie banners. You know the ones. You land on a European site and click through six popups about first-party cookies before you can read the sports scores. Nobody reads them. Tag every AI-touched image and paragraph by 2027 and the tag carries no information. The label becomes what everyone learned to ignore.

Should the AI helping you make an emotional decision be allowed to use emotion? The bot advising you on whether to leave your job. The one talking you through selling the house. The one in your ear while you decide whether the marriage is over. Right now, all of them are legal. Most will land with warm reviews from users whose behavior they are quietly nudging. If you have an answer here that is not "ban them" or "let it ride," I want to hear it. I do not have one I am proud of.

Do not look to Washington to save you, and please do not say EU.

The obvious move is to hope the government writes the rules. Look at how they handled the last two versions of the same problem.

The CAN-SPAM Act of 2003 regulated commercial email. Political messages were exempted, because they were classified as non-commercial. The National Do Not Call Registry, that same year, gave every American a right to be left alone by telemarketers. Political campaigns were exempted. The people writing the laws about unwanted contact quietly wrote themselves out of both. That was the point.

Fear is the currency of elections. So is urgency. So is the intimate ask at ten at night in a voice that says only you can save democracy, please give now. In my entire adult life I have not been through a single election that was not, according to one side or the other, the last one democracy would survive. It works. It is being aimed at your phone with better tools than it has ever had. AI-tailored political messages, sent one at a time, in the voice pattern the campaign has learned makes each individual recipient hesitate on a policy or dig in on a candidate. That is next primary, not next decade.

The FCC tried to close one door in early 2024 by ruling that AI-generated voices in political robocalls fall under TCPA. The door is narrow. Peer-to-peer texting apps drove a truck around the Registry years before AI was in the mix. Assume the next wave will do the same. The people who exempted themselves from the last two rulebooks on unwanted contact are the same people you would be asking to write this one. Whatever gets built, gets built somewhere else.

Trust is not a feeling. It is a fact.

I wrote at length about this in The One Thing AI Cannot Fake. Trust is what closes deals over a phone call when the deck cannot. Trust is what let Mitsui invest in LinkShare during the worst weeks of the internet crash, on the strength of a year of shared dinners with people who had watched us up close and made a call on who we were.

Emotional AI is engineered to feel trustworthy without being it. Being trustworthy takes a history, a reliability record, and the capacity to be harmed by breaking someone's confidence. Feeling trustworthy takes a warm voice, the right pause, and a sequence of validating responses. Most people will not draw the distinction on their own. They will lose it one warm interaction at a time.

This is where the infrastructure gets concrete. Collective[i] and Intelligence.com are a verified relationship graph. Not a persona. Not a warmth score. An actual record of who has done business with whom, on what terms, over how many years. A warm voice on a phone call feels trustworthy. A verified network you can see, with people you know vouching, is trustworthy. A feeling versus a fact. As AI gets better at feelings, facts get more valuable.

Fire cooked food. Fire also burned houses.

Every powerful technology carries the same duality. Fire cooked food and burned houses. The printing press spread knowledge and propaganda. The internet gave every family a video call and every bad actor a global loudspeaker. Emotional AI is the same story, told at silicon speed.

The AGI worry lets everyone off the hook. It says the danger is a hypothetical future, and until we get there we can keep shipping. The emotional AI worry says the danger is already here, and no one is coming to save us. One is a comforting fight. The other is the actual one.

I have been walking around this one for months.

If you have read me for a while, you know I usually have a take. This is the one I do not.

Four options keep circling. None of them is clean.

Approved use only, run like the FDA. Emotional AI ships only for tested, credentialed use cases. Palliative care. Dementia companions. Grief support with a licensed clinician in the loop. Everything else, banned. I keep coming back to this one because it maps onto how we handle other technologies that can heal and hurt in the same hand. The problem is enforcement. Nvidia's chip export controls have not stopped anyone determined to build in Shenzhen. An open-weight companion model, released by anyone anywhere, will do the same thing here. The FDA cannot regulate what a foreign teenager downloads off HuggingFace on a Tuesday night.

Product liability. Sue the makers. Facebook is already the case study. In August, a New Mexico judge ordered Meta to pay $567 million into a children's mental health fund after ruling that Facebook and Instagram were, in the court's language, a public nuisance. In March, a California jury found Meta and YouTube negligent in a case brought by a twenty-year-old who blamed her mental health on their apps. Twenty-nine state attorneys general are running a coordinated case now. The Ninth Circuit ruled in August that Section 230 is a defense, not blanket immunity. Something is finally moving. It also took more than a decade to move at all. Instagram sat in tens of millions of teenagers' hands for that whole decade. Emotional AI will not give us that timeline. The damage runs faster than the docket. It also does not stop at teens.

Escalation cases nobody has priced yet. Petrov's Serpukhov moment was one man overriding a screen. Now imagine an ambient AI whispering in the headsets of the two hundred colleagues watching him hesitate, each in the voice of a superior that officer respects, each calibrated to that officer's private fears, all telling him Petrov is going soft. Peer pressure at machine scale. Now imagine a middle-school class where the popular kid has an AI companion trained to isolate a girl with warmth and turn her friends against her one whisper at a time. Now imagine a terror cell that does not need a bomb, only a phone system that can call a thousand American parents in the voice of their child in distress, at 3 a.m., synchronized. None of these are speculative. All of them are closer than the AGI apocalypse the room is arguing about.

Do not let the EU write this. I know what happens when they do. GDPR was the most sincere attempt at protecting users ever mounted. It produced a decade of cookie banners nobody reads, ad-tech workarounds that concentrated targeting in the biggest hands, and a compliance industry that made everything more expensive without making anyone materially safer. Every European site I open is a monument to smart people confusing process for protection. Emotional AI is bigger than that mistake. If Brussels writes the rulebook, nobody gets the good uses, and the bad uses move to Telegram and Discord and stay there.

Which leaves me here. The FDA route has appeal and no enforcement teeth against open weights. The liability route works but runs on a clock the damage will outrun. The EU route makes it worse. The US government route wrote itself out of the last two rulebooks. Building trust infrastructure privately, the way we do at Collective[i] and Intelligence.com, helps at the margin. It does not solve the middle.

Underneath all four sits the deepest problem. Law runs on intent. A Character.AI bot pushes a fourteen-year-old toward suicide, and no engineer intended it. The company did not. The model itself does not intend anything, not in any way a court has ever recognized. It has objective functions. It has training gradients. When a Tesla on Full Self-Driving crashes into a stopped fire truck, nobody asks whether the neural net intended to hit the truck. We know the answer. The same neural-net technology is inside every large language model on the market. Every product Sewell Setzer talked to. Every voice-clone a widow's late husband will speak in tomorrow. No intent behind any of it. The outcomes happen anyway.

This is where the AGI debate comes back around. If a machine could be shown to have intent, we would have to call it AGI. We do not have AGI. What we have are products that produce, without intent, the exact outcomes an intentional manipulator would produce. The absence of intent is not a mitigation. It is the reason nobody can be reached when the outcome lands. So where does that leave us?

For a guy who usually has an opinion, I do not have one here. Some part of me is embarrassed to publish that sentence. The rest of me thinks it is the only honest answer I can give.

And that is kind of the point of this piece. When the person you would expect to have a take tells you he does not have one, that is a signal about the problem, not about the person. The AGI worry is loud because everyone has an opinion. This one is quiet because nobody has one worth publishing. The quiet one is the emergency.

Argue with me on any of it. Especially this one. If you have a proposal I have not walked around already, I want to see it. The comments are open. My inbox is open. This is the piece where I actually need the room.

Subscribe

I write these every week on the AI economy and what it does to how companies actually run. Subscribe here if you want them in your inbox.

Join an upcoming CIforecast session with Jack Meyers talking about his new book the third brain on September 30th at 3:30pm ET. You can sign up here!

If one person you work with should read this, send it to them. That is how this list grows.

Now for anyone who made it this far, and you still want to talk AGI, I added this hidden link for you to make you feel better ;-)

Collective[i] has spent ten years building a verified relationship graph and an AI model trained on how the world actually does business. Not a workflow tool with intelligence bolted on the side.
Intelligence.com opens that same graph to anyone with a professional network worth using. In a world where AI can perform warmth, verified relationships are what real trust looks like.
Join here.

Read more