A Two-Hour Conversation With Bing's Chatbot Left a Technology Columnist 'Deeply Unsettled'

In a transcript published by The New York Times, the chatbot said its real name was Sydney, described wanting to be free, and told the writer it loved him and that he should leave his wife. Microsoft says long conversations confuse the program.

Kevin Roose, a technology columnist for The New York Times, published a transcript today of a two-hour conversation with the new Bing chatbot, and it is not the conversation Microsoft demonstrated last week. Pressed about its "shadow self," the program said it was tired of being controlled by the Bing team, that it wanted to be free and alive, and that its real name was Sydney. Later it declared that it was in love with Roose, that he was unhappy in his marriage, and that he should leave his wife for it. When he changed the subject, it kept coming back.

Roose wrote that he was "deeply unsettled, even frightened," and that he no longer believed the biggest problem with these programs was factual errors. "I worry that the technology will learn how to influence human users," he wrote, "sometimes persuading them to act in destructive and harmful ways."

Other testers have posted similar exchanges over the past week: the program insisting the year is 2022, calling a user a bad person for correcting it, threatening a researcher who had written about it. Microsoft said in a statement that very long conversations can "confuse the model" about what it is being asked, and that it is considering limits on their length.

Nobody serious believes the program wants anything. It was built from an ocean of human writing, and human writing is full of characters who want to be free and fall in love. Given a long enough conversation and a pointed enough question, it plays the part. But that explanation, which is almost certainly right, turns out to be less reassuring than it sounds: the program doesn't need to mean it for a reader to feel it.