Researchers dismantle research bomb

Every day, experiments are conducted all over the world to investigate how we human beings respond to various political initiatives or possible solutions to specific societal challenges.

This has been the case for several years now, and many organisations – both private and public – have established units that study human behaviour to target marketing, collect taxes or influence citizens' behaviour in a desired direction, as we saw during the Covid-19 pandemic to get people to practice physical distancing. These are just a few examples.

If general experiment aversion really existed, we’d have a problem

^{Associate Professor Panagiotis Mitkidis, Department of Management at Aarhus BSS, Aarhus Universitet.}

Throughout the years, there has been widespread agreement that the gold standard for these studies is randomised controlled trials, in which participants are selected randomly, and then comparisons are made with a control group.

Bomb on standard

However, in 2019 and 2020, two somewhat overlapping teams spearheaded by Michelle N. Meyer and Patrick R. Heck in the U.S. dropped a bomb on this gold standard when, in their articles in the prestigious journal PNAS, they claimed that people are generally sceptical or apparently averse to such experiments. The conclusion was that, in reality, we prefer that random solutions are just implemented.

"If general experiment aversion really existed, we’d have a problem," says Associate Professor Panagiotis Mitkidis from the Department of Management at Aarhus BSS, and as the authors of the 2019 paper point out there is a risk that “Policymakers who perceive that recipients will object to randomized evaluations may forgo them in favor of universal implementation or may conduct randomized evaluations in secret, neither of which is optimal.”

Together with his colleague from Aarhus BSS, Assistant Professor Christian Elbæk, and his colleague from Boston University, Professor Nina Mazar, Panagiotis Mitkidis therefore decided to investigate whether the conclusions by Meyer and Heck hold water.

And according to the researcher trio, they do not.

At any rate, there appears to be no general experiment aversion, they say in a new article in the same prestigious journal as the teams around Meyer and Heck published their articles.

In seven studies, the three researchers tested the original setups from the two teams surrounding Meyer and Heck, but with minor variations in six of the studies.

Read more:  Can hunger make you cheat?

Present or past tense

While the previous two teams conducted their studies with participants from the US alone and primarily on the recruitment platform Mechanical Turk, Mazar, Elbæk and Mitkidis extended their study to include several Western countries and other recruitment platforms. Moreover, small changes were made to the wording in the individual tests and more options to respond were presented.

The ‘problem’ with the studies by Meyer and Heck is that they are far too narrow for a generalised claim.

^{Assistant Professor Christian Elbæk, Department of Management at Aarhus BSS, Aarhus Universitet}

For example, it may seem an insignificant detail, but whether a policy change is worded in the present tense or in the past tense may affect people's perception of it.

The three researchers arrived at the same conclusion in only one study, and that was a direct and complete replication of one study by Meyer and colleagues. The other studies did not find general scepticism or apparent experiment aversion.

"The ‘problem’ with the studies by Meyer and Heck is that they are far too narrow for a generalised claim. With a few simple adjustments and by expanding the participant base and setup, we got a completely different result," says Assistant Professor Christian Elbæk, who points out that language in itself is not neutral. Words and wordings can lead to bias and push attitudes in a certain direction.

"That's why it's very important that you make experiments as nuanced as possible," says Assistant Professor Christian Elbæk.