We’re Easily Confused About What Experts Really Think, New Research Shows

I’m not a scientist. And chances are, neither are you.
 
That likely means we both find ourselves deferring to the opinion of others, of experts who know more about complex matters — like health or nuclear safety or vaccinations or climate change — than we do.
 
But heck, even scientists have to rely on the expertise of others (unless they’re some sort of super scientist with infinite knowledge of all things. Ahem, Neil deGrasse Tyson).
 
But for the rest of us intellectual Joes, we rely heavily on what we think the experts think. As it happens, figuring out what the experts think isn’t so easy, not even in those instances where the majority of experts agree on a subject.
 
Take for example, the issue of climate change, which is just what cognitive scientist Derek J. Koehler had in mind when he launched a recent pair of experiments designed to investigate what factors might contribute to our collective failure to grasp expert consensus.

The Problem of False Balance 

“There’s this well-documented gap between public perceptions on [climate change] and expert perceptions,” Koehler, professor of psychology at the University of Waterloo, told DeSmog Canada. “I became interested in this as an observer of the news and I guess it was probably climate change more than anything else that was a motivating example for me.”
 
Koehler, a professor at the University of Waterloo, said when it comes to climate change there has been a lot of discussion by media critics on the possible role that false balance in news coverage may play in confusing the public about where actual expert consensus lies.
 
“Even though I had climate change in mind when I started this work, the actual studies I ended up running were about economic issues on the one hand and movies on the other which involved looking at the perception of consensus among film critics.”
 
Koehler’s research, published in a recent online article in the Journal of Experimental Psychology: Applied (print version forthcoming), suggests the practice of giving voice to experts on both sides of an issue may distort public perception about the level of agreement among experts.
 
Koehler entered into the study with a simple question in mind: what factors influence our ability to comprehend where expert opinion lies?
 
As Koehler found, even when individuals are told exactly what experts think (even shown what they think using graphs), it was difficult for those people to digest and then rearticulate that information.

Deciphering Expert Opinion: The Experiments

In Koehler’s experiment a group of participants were given a numerical summary of where the opinion of experts (convened by the University of Chicago) fell on a selection of economic issues.
 
For example, on the issue of whether a carbon tax would be effective in reducing greenhouse gas emissions, there was a very high level of agreement; 93 experts agreed, five indicated they were uncertain and two disagreed.
 
But on the issue of whether raising minimum wage would affect the ability of low-skill workers to find employment, there was widespread disagreement: 38 experts agreed, 27 were uncertain and 36 disagreed.

Koehler presented these opinions to the group and asked them to rank the level of agreement among the experts. Koehler also asked a subset of the group to read comments from experts on either side of the issue.
 
What Koehler found was participants exposed to commentary from the two experts who disagreed were less able to decipher where expert consensus actually resided. So hearing the argument of the experts, rather than just seeing their position displayed on a chart, made it more difficult for those individuals to distinguish high consensus issues (like the carbon tax) from low consensus ones (minimum wage).
 
“Two different people can look at apparently the same body of evidence and draw very different conclusions from it,” Koehler said. “We know from past psychological research that that can and does happen.”
 
But Koehler became more interested in what general factors might lead people to “systematically misperceive where the expert consensus lies across these different domains.”
 
In these experiments, Koehler explained, the participants’ “task is not to tell us what they personally think about the issue but where the experts’ opinions fall on the topic.”
 
Individuals tend to hold strong opinions on economic issues like minimum wage or carbon tax so Koehler performed additional experiments with more neutral topics like the ranking of films — left unnamed — among top critics.
 
“So for instance, with the movie studies participants were making judgments about movies that were not identified, so deciphering the percentage of critics who thought it was good versus bad and reading a couple of comments about the movie from two disagreeing experts,” he said.
 
Koehler said keeping the movies unidentified “was a deliberate attempt to make it impossible for people to draw on their pre-existing opinions and beliefs in making these kinds of judgments.”
 
“In these studies I deliberately tried to develop a task where there would be less room for people’s preexisting opinions to play a role,” Koehler said, adding, “part of that was emphasizing the task was one of simply reporting or rating their perceptions of what experts think as opposed to what they personally thought about these issues.”
 
Koehler said the ability of participants to decipher how film critics ranked movies was influenced by whether or not they heard from experts on both sides. 
 
Koehler’s experiment shows that even before you add in “additional complicating factors” like strong beliefs or preferences surrounding issues like a carbon tax or minimum wage, “the presentation of conflict between specific experts can distort people’s perceptions and lead them to think there’s more disagreement among a population of experts than there really is.”

Reporting Weight of Evidence

The problem of false balance has long been a stumbling block for communicators of climate science. Mainstream media outlets have for years given equal airtime to legitimate climate scientists and climate deniers who often have no scientific background or have direct ties to the fossil fuel industry.
 
Giving equal play to the opinion of climate scientists and deniers has had significant impact on the public’s perception of climate science.
 
One study found the public’s lack of certainty around climate science translated directly into a lack of support for smart climate policy.
 
One remedy that’s been popularly advanced as a solution is ‘weight of evidence’ reporting. Koehler said weight of evidence information would require a reporter to indicate that the opinion of one expert is shared, for example, by 97 per cent of experts while the opinion of the other is only shared by three per cent.
 
Yet Koehler’s research indicates weight of evidence reporting isn’t enough to combat misperception of expert consensus or the distorting influence of false balance.
 
“Even when that weight of evidence information is given, people in their perception of expert consensus discriminate or distinguish less sharply between high and low consensus issues when weight of evidence information is accompanied by conflicting comments from specific experts,” he said.
 
“So even if you know 80 per cent of the experts are on one side and 20 per cent are on the other, when you choose one specific concrete member of each of those disagreeing groups of experts and provide a comment from each there is something in the psychology of doing so that crystallizing the view from either side in the form of a single person.”
 
“That leads people to see more disagreement than there really is in cases of high consensus,” he said.
 
Koehler added that in his studies, information was stripped down to its bare form: “Basically a table with some numbers representing expert opinion.” But in the real world information is rarely ever presented in such a schematic fashion and as individuals we’re often left to rely on our memory when it comes to recalling what we think the experts think, he said.
 
“I think it’s easier to identify a problem like this than it is to suggest a remedy,” Koehler said.
 
But he added, “I would say probably a general piece of advice for everyone is to try to look beyond, to seek out those sources of information that represent the thinking of a population of experts so you’re not forced to rely on a single perspective or opinion.”
 
Although it’s unclear exactly why the presentation of dissenting comments skews our perception of consensus (maybe the expert had a convincing argument or our knowledge of disagreement leads to a sense of uncertainty), Koehler says we need to get better at understanding experts.  
 
The distorting influence of false balance triggers a pretty significant “cognitive glitch,” Koehler recently wrote in the opinion pages of the New York Times.
 
“Whatever the cause, the implications are worrisome,” Koehler wrote.
 
“Government action is guided in part by public opinion. Public opinion is guided in part by perceptions of what experts think. But public opinion may — and often does — deviate from expert opinion, not simply, it seems, because the public refuses to acknowledge the legitimacy of experts, but also because the public may not be able to tell where the majority of expert opinion lies.”

Image: PopTech