New study finds readers can’t tell AI poems from human ones — and they often prefer AI

Here’s a fun twist you might not expect: a recent study shows that most people can't tell if a poem was crafted by a human or AI — and even more surprising, we often prefer the AI versions! Researchers Brian Porter and Edouard Machery from the University of Pittsburgh put this theory to the test, and the results reveal something pretty quirky about our poetic tastes.

Human or AI? You decide!

For this experiment, Porter and Machery recruited over 1,600 participants and gave them a batch of poetry to read. Each participant got ten poems, five written by legendary poets like William Shakespeare, Emily Dickinson, Lord Byron, and T.S. Eliot, and five crafted by ChatGPT-3.5 mimicking these poets’ styles. Participants had no idea which ones were real and which were AI.
The shocker was that readers were more likely to think the AI poems were written by humans! In fact, the poems people thought least likely to be human were often written by the very poets we’d consider masters of the craft.

What makes a poem feel "real"?

Curious to dive deeper, the researchers set up a second experiment with 696 new participants. This time, each reader rated poems on things like beauty, emotion, rhythm, and originality. The readers were divided into three groups: some were told the poems were written by humans, others by AI, and a third group got no information at all.
They found that if you tell someone a poem was AI-generated, they’re likely to rate it lower on just about every quality. But here’s the twist: when people didn’t know who wrote the poems, they often rated the AI-generated ones higher than the human written ones!

So, why are we falling for AI poetry?

Turns out, AI poems often end up being a little more straightforward and easy to enjoy, while poems by famous poets can be more complex and challenging. Instead of appreciating the depth of a classic poet’s work, readers sometimes mistake complexity for confusion and move on. AI-generated poems, meanwhile, serve up their message in a style that feels familiar and easy to “get,” making them more enjoyable at first glance.

Basically, we tend to believe we’ll prefer human poetry. But in reality, AI’s smooth style is just easier to like — and we interpret this as a sign the poem was created by a human.

The findings of this research have been published in Scientific Reports and can be accessed here.