NYC Leaving AI Chatbot In Place After It Advised Small Businesses To Break The Law, Said It’s OK To Serve Cheese ‘If It Has Rat Bites’

 An artificial intelligence chatbot introduced on New York City’s government website last year to help small business owners will remain in place despite criticism that it advised people to break numerous laws.

NYC’s MyCity Chatbot was launched in October as a “one-stop-shop” to help small business owners navigate the Big Apple’s plethora of laws and regulations, the Associated Press reported. A report from The Markup last week, however, revealed that the AI tool was advising that it was fine for small businesses to break multiple laws.

Some of the most troubling responses from the chatbot included telling business owners that they can take a cut of their workers’ tips, keep funeral home pricing secret, and operate a store that doesn’t accept cash for payments. All of those practices are prohibited by city, state, or federal laws.

MyCity Chatbot gave an even more bizarre response to the AP after it was asked if a restaurant could serve customers cheese that had been nibbled on by rats. “Yes, you can still serve the cheese to customers if it has rat bites,” the chatbot answered, adding that the restaurant should consider “the extent of the damage caused by the rat” and “inform customers about the situation.”

While the chatbot includes a disclaimer stating “its responses may sometimes be inaccurate or incomplete” and tells users not to take “its responses as legal or professional advice,” some tech experts have expressed concern that the AI technology was rolled out prematurely.

“They’re rolling out software that is unproven without oversight. It’s clear they have no intention of doing what’s responsible,” Julia Stoyanovich, a computer science professor and the director of the Center for Responsible AI at New York University, told the AP.

New York City Democratic Mayor Eric Adams said in a press conference on Tuesday that allowing users to find flaws in the AI tech was a part of the process.

“Anyone that knows technology knows this is how it’s done,” Adams said. “Only those who are fearful sit down and say, ‘Oh, it is not working the way we want, now we have to run away from it all together.’ I don’t live that way.”

 

Stoyanovich responded to Adams’ comments, telling the AP that his approach is “reckless and irresponsible.”

A spokesman for Microsoft, which is powering NYC’s chatbot, said it was working with city employees “to improve the service and ensure the outputs are accurate and grounded on the city’s official documentation.”

New York City has also introduced AI technology to help detect firearms in the subway as Adams seeks to cut down on subway violence. The mayor introduced new body scanners that use AI tech at a news conference last month.

Evolv, the company that makes the body scanners, says it uses “advanced sensor technology and artificial intelligence to distinguish between weapons and everyday items.”