AI can’t handle The Needling.
The Seattle-based satirical news site that describes itself as “Seattle’s Only Real Fake News” is quickly learning some of the pitfalls that come with the latest AI models.
The Needling, which draws inspiration from The Onion and the show Portlandia, publishes fabricated stories related to topics and people in the “real” news.
A Needling story from 2022 — Geologists: Mount Rainier Will Not Erupt During Your Lifetime, with Possible Exception of Very End — started showing up last month in Google’s new prominent AI Overview section on searches related to Mount Rainier and an eruption.
Of course, the story was not real. But the new AI feature wasn’t able to discern truth from satire.
“It really is hilarious but also seriously concerning,” said Alexa Vaughn, a former Seattle Times reporter and Pulitzer Prize-winning journalist who founded The Needling in 2018.
Vaughn said the Rainier story was featured in AI Overview for a few days before it was “fixed.”
It’s not the only incorrect answer that has surfaced among AI Overview results, which aim to provide a brief summarized answer to search queries. It also spotlighted content from The Onion about eating rocks for vitamins and minerals.
Following reports of “odd and erroneous overviews,” Google said in a blog post last month that it made more than a dozen technical improvements, including “better detection mechanisms for nonsensical queries that shouldn’t show an AI Overview,” and limiting the inclusion of satire and humor content.
“At the scale of the web, with billions of queries coming in every day, there are bound to be some oddities and errors,” Google said in the blog post, adding: “We’ll keep improving when and how we show AI Overviews and strengthening our protections, including for edge cases, and we’re very grateful for the ongoing feedback.”
Meanwhile, Vaughn said she’s also seen weird results from Meta’s new AI summaries for comments on Facebook posts.
One of the computer-generated responses that tried to summarize “what people are saying” about a post linking to a Needling story noted that some of the comments were “humorous, while others are skeptical about the report’s claims.”
In other words, the AI was not able to detect that the report was satirical.
“I find it really problematic that there was no option to not have those summaries of the comment section because the AI is completely missing that this is a joke and satire and actually the entity that is now more likely to be spreading information by summarizing comments of a satirical post section,” Vaughn said.
Vaughn said Meta has also taken down two of its posts in the past month without explanation, only noting that it violated its community standards.
“It doesn’t give you a reason or any recourse to appeal the decision,” Vaughn said.
Vaughn said her site’s experience with the latest AI technology shows that it’s not ready for prime time.
“I think a lot of tech leaders are in denial about how damaging and underdeveloped this AI technology is,” she said.
