Expert opinion has been found to be unreliable and prone to bias
October 23, 2023
Expertise plays essential roles in human societies. Specialization is a key difference between hunter-gatherer societies, in which each individual performed all or most tasks needed for survival, and all subsequent civilizations which have achieved efficiencies by division of labor. Individuals develop greater proficiency at a narrower subset of tasks and cooperate with others with different expertise, generating net benefits to all.
The Nature of Knowledge
Some of the errors of ostensible “experts” are addressed by Dr. Sowell in his book Knowledge and Decisions. A key error is the assumption that consequential knowledge is concentrated in a small group of elite experts, and that their abstract knowledge is more important than mundane knowledge possessed by others. Drawing from the work of economist Friedrich August von Hayek (F. A. Hayek), Dr. Sowell points out that knowledge is diffused in society, and even the most erudite scholars possess only a tiny fraction of consequential knowledge. He observes that much consequential knowledge may be unarticulated and exist in forms different from explicit intellectual articulation.
Dr. Sowell explained the difference between scientific and nonscientific processes. He wrote:
“Controversies have raged in science, but what makes a particular field scientific is not automatic unanimity on particular issues but a commonly accepted set of procedures for resolving differences about issues when there are sufficient data available.”
“A leading historian of science, Thomas Kuhn, has argued that what distinguishes science from other fields is that mutually contradictory theories cannot co-exist indefinitely, but that one or the other must prevail, and the others disappear, when enough of the right data becomes available.” In contrast, “mutually contradictory ideologies can co-exist for centuries, with no resolution of their differences in sight or perhaps even conceivable.”
“What scientists share is not simply agreement on various conclusions but, more fundamentally, agreement about the ways of testing and verifying conclusions, beginning with a careful and strict definition of the terms being used.”
Dr. Sowell cautioned that the status of conclusions as scientific or non-scientific is based on the evidence and process used to reach them, and not the credentials of the claimant. He observed that ostensible training or credentials do not necessarily imply that ostensible experts’ claims derive from rigorous intellectual standards and processes:
“There are, after all, other skills in which intellectuals also tend to excel, including verbal skills that can be used to evade the testing of their favorite notions. In short, the various skills of intellectuals can be used either to foster intellectual standards or to circumvent those standards and promote non-intellectual or even anti-intellectual agendas…it is possible for people not defined as intellectuals—engineers, financiers, physicians—to adhere to intellectual procedures more often or more rigorously than some or most intellectuals.”
Experts can be drastically wrong, at times with catastrophic results. Stanford Professor Dr. John Ioannidis, widely considered the father of evidence-based medicine, wrote: “Empirical evidence on expert opinion shows that it is extremely unreliable.” Evidence-based medicine rating systems which initially included “expert opinion” as the lowest level of medical evidence have in updated schemas removed expert opinion entirely and substituted “mechanism-based reasoning,” acknowledging that the views of ostensible “experts” warrant no more credibility than the evidence and logic they can produce. Dr. Sowell cited Gibson’s Law and amended it as follows: “for every expert there is an equal and opposite expert, but for every fact there is not necessarily an equal and opposite fact.”
Notwithstanding that expert opinion is “extremely unreliable,” it is often represented to the public as being far more reliable than the evidence suggests. Those who object, it is often conveyed, do not have a right to disagree with supposed “experts,” notwithstanding the lack of empirical evidence for the “experts’” recommendations, frequent evidence to the contrary, and often catastrophic records.
Expert opinion continues to be heavily relied upon for findings of fact in U.S. courts. In Knowledge and Decisions, Dr. Sowell wrote: “the ability to weigh conflicting testimony may require as sophisticated an understanding of an alien field as deciding the initial question itself.”
Real expertise can have considerable value in its specific domain. Yet false expertise, often extended far beyond an individual’s actual proficiency and often infused with ideological assumptions and personal incentives, has often proven disastrous. Dr. Sowell wrote:
“A biographer of [John Maynard] Keynes, a fellow economist who was his contemporary, pointed out another aspect of Keynes’ character that has long been characteristic of some other intellectual elites: He held forth on a great range of topics, on some of which he was thoroughly expert, but on others of which he may have derived his views from the few pages of a book at which he had happened to glance. The air of authority was the same in both cases.”
Nobel laureate George J. Stigler stated: “A full collection of public statements signed by laureates whose work gave them not even professional acquaintance with the problem addressed by the statement would be a very large and somewhat depressing collection.” Stigler further referred to “Nobel laureates who issue stern ultimata to the public on almost a monthly basis, and sometimes on no other basis.”
Notwithstanding the catastrophic consequences to others inflicted by repeated failures of so-called “experts” relied upon in political decision-making, there has been no serious public reckoning or accountability for these failures. Time after time, politicians have announced that there will be no accountability for the architects of failed policies. Yet despite disastrous failures, these ostensible “experts” continue to be set forth both individually and collectively as credible guides for the future. Dr. Sowell refers to those who continue to be set forth as credible experts despite catastrophic past failures as “Teflon prophets.”
Bias and Agenda
Experts may seek control or outsized influence in determining societal policies, but are also prone to their own biases and incentives which may be far from the general public interest. These intrinsic biases are amplified by agenda-driven news media, which cherry-pick partisan “experts” to promote an ideological agenda. Kristen Ruby stated: “Journalists state a hypothesis and look for expert sources to support the hypothesis, even if the hypothesis is not correct." The Telegraph (UK) observed:
“the [disinformation] problem is not limited to bots and professional propagandists. Many of those spreading untruths and half-truths online are those who used to be known as the verified, ‘blue-tick’ brigade: the journalists and commentators who work for some of the world’s most prestigious media empires.”
At times, ostensible expertise can pose risks of abuse of power. Economist Dr. Thomas Sowell observed that inequality in the concentration of power is one of the most dangerous inequalities. He also distinguished between expertise in fields with real-world validation, as for engineers, physicians, dentists, and others who produce objectively verifiable results, and those who deal in ideas with no objective validation. The latter, he notes, pose immensely greater dangers for abuse of power.
Ostensible experts often have incentives to exaggerate their own scope of competency and may become intoxicated by the attention and power of influencing public policy. Dr. Sowell observed that they view other types of elites as competitors and have incentives to try to discredit them.
Catastrophic Expert Errors
Dr. Sowell wrote in Social Justice Fallacies: “Stupid people can create problems, but it often takes brilliant people to create a real catastrophe.” In 2012, he stated: “Let’s just say that the road to hell is paved with Ivy League degrees.” Many if not most of the great human disasters of the past century were caused or dramatically worsened by ostensible “experts.” Of the Great Inflation of the 1970s, Dr. Sowell observed:
“The Great Inflation….was a self-inflicted wound…Its intellectual godfathers were without exception men of impressive intelligence. They were credentialed by some of the nation’s outstanding universities: Yale, MIT, Harvard, Princeton. But their high intellectual standing did not make their ideas any less impractical or destructive. Scholars can have tunnel vision, constricted by their own political or personal agendas. Even if their intentions are pure, their ideas may be mistaken. Academic pedigree alone is no guarantor of useful knowledge or wisdom.
Some of the many other examples Dr. Sowell cites with evidence include:
- The Great Depression of the 1930s, which was prolonged and vastly amplified by expansive government intervention;
- The Second World War, the most deadly conflict in human history, which was likely avoidable and was enabled by disarmament and appeasement movements advocated by Western intellectuals;
- The Nazi genocide against Jews and other ethnic minorities traced its genealogy to the US “progressive” eugenics movement, as Hitler acknowledged in a letter to the author of a book he referred to as his “bible;”
- Economic and social retrogressions for Blacks in the United States, including the disintegration of the black family, skyrocketing crime rates in previously safe black neighborhoods, the gutting of educational performance in previously high-performing black schools, and deceleration in income growth,all correlated with government “solutions” advocated by experts;
- Skyrocketing national debt and massive disinvestment in America’s future, reflecting short-term political interest of politicians and given cover by so-called progressive “experts.”
Much more could be added to this list. The Obama Iran deal allowing a terrorist state to enrich uranium, the Russian annexation of Crimea in 2014, the Russian war on Ukraine starting in 2022, the bungled US withdrawal from Afghanistan in 2021 the loss of democracy and human rights for over forty million people, Azerbaijan’s ethnic cleansing of hundreds of thousands of Armenians from their historic Armenian lands going back millennia, and the war against Israel by Iranian proxies in 2023, were all enabled by appeasement policies promoted by Western policy “experts.”
Research on Expert Errors
Various expert errors have been studied by scholars from Yale and Harvard, among others. BBC Presenter David Robson’s award-winning book The Intelligence Trap (W.W. Norton, 2019) highlights some of the shortfalls of ostensible expertise. Brief selections on various themes are noted below.
Rationality and Decision Making
Wandi Bruine de Bruin’s study: “The participants’ decision-making skills were only moderately linked to their intelligence; academic success did not necessarily make them more rational decision makers.” (Robson 48)
“The participants’ rationality scores were about three times more important [than IQ] in determining their behavior.” (Robson 49)
“These tests [of rationality] clearly capture a more general tendency to be a careful, considerate thinker that was not reflected in more standard measures of cognitive ability; you can be intelligent and irrational –... and this has serious consequences for your life.”(Robson 49)
Dysrationalia Among the Nominally Intelligent
“[Researcher Keith] Stanovich found that the relationships between rationality and intelligence were generally very weak. SAT scores revealed a correlation of just 0.1 and 0.19 with measures of framing bias and anchoring bias, for instance. Intelligence also appeared to play only a tiny role in the question of whether we are willing to delay immediate gratification for a greater reward in the future - a tendency known as ‘temporal discounting.’ In one test, the correlation with SAT scores was as small as 0.02. That’s an extraordinary modest correlation for a trait that many might assume comes hand in hand with a greater analytical mind. The sunk cost bias showed almost no relationship to SAT scores in another study.” (Robson 45)
“Circumstantial evidence would suggest that dysrationalia is common. One study of the high-IQ society Mensa, for example, showed that 44 percent of its members believed in astrology, and 56 percent believed that the earth had been visited by extraterrestrials.” (Robson 44)
“According to a survey of more than 1,200 participants, people with college degrees are just as likely to endorse the existence of UFOs, and they were even more credulous of extrasensory perception and ‘psychic healing’ than people with worse education.”(Robson 51)
Problems of Critical Thinking
“Gui Xue and colleagues at Beijing Normal University…[found] that the gambler’s fallacy is actually a little more common among the more academically successful participants in his sample.” (Robson 45)
“Even trained philosophers are vulnerable. Participants with PhDs in philosophy are just as likely to suffer from framing effects, for example, as everyone else – despite the fact that they should have been schooled in logical reasoning. You might at least expect that more intelligent people could learn to recognize these flaws. In reality, most people assume that they are less vulnerable than other people, and this is equally true of ‘smarter’ participants.”(45)
Bias Blind Spots
“In one set of experiments…Stanovich found that people with higher SAT scores actually had a slightly larger ‘bias blind spot’ than people who were less academically gifted.” (45-46)
“‘Adults with more cognitive ability are aware of their intellectual status and expect to outperform others on most cognitive tasks,’ Stanovich told me. ‘Because these cognitive biases are presented to them as essentially cognitive tasks, they expect to outperform on them as well.’” (Robson 46)
Poor Decision Making
“Intelligence does not necessarily lead to better decision making” (Robson 50)
“People with high IQs are also just as likely to face financial distress, such as missing mortgage payments, bankruptcy, or credit card debt. Around 14 percent of people with an IQ of 140 had reached their credit limit, compared to 8.3 percent of people with an average IQ of 100. Nor were they any more likely to put money away in long-term investments or savings; their accumulated wealth each year was just a tiny fraction greater. These facts are particularly surprising, given that more intelligent (and better educated) people do tend to have more stable jobs with higher salaries, which suggests that their financial distress is a consequence of their decision making, rather than, say, a simple lack of earning power.” (Robson 49)
“The researchers suggested that more intelligent people veer closer to the ‘financial precipice’ in the belief that they will be better able to deal with the consequences afterward. Whatever the reason, the results suggest that smarter people are not investing their money in the more rational manner that economists might anticipate.” (Robson 49).
Elaborate Justification vs. Critical Thinking
“Classic studies from the 1970s and 1980s, when David Perkins of Harvard University asked students to consider a series of topical questions….Perkins found that more intelligent students were no more likely to consider any alternative points of view…Instead, they had simply used their abstract reasoning skills and factual knowledge to offer more elaborate justifications of their own point of view.”(Robson 53)
[In case examples], “greater intellect is used for rationalization and justification, rather than logic and reason.” (Robson 62)
Propaganda over Truth Seeking
“The upshot, according to [Yale Law School researcher Dan] Kahan and other scientists studying motivated reasoning, is that smart people do not apply their superior intelligence fairly, but instead use it ‘opportunistically’ to promote their own interests and protect the beliefs that are most important to their identities. Intelligence can be a tool for propaganda rather than truth-seeking.” (54-55) [ref. 34)
Logic often used for persuasion rather than truth-seeking (64).
“Scientists today use the term ‘motivated reasoning’ to describe this kind of emotionally charged, self-protective use of our minds” that can “lead us to become more and more entrenched in our opinions.” (Robson 53)
“The same polarization [as with climate change] can be seen for peoples’ views on vaccination, fracking, and evolution. In each case, greater education and intelligence simply helps people to justify the beliefs that match their political, social, or religious identity.” (Robson 56)
“There is some evidence that, thanks to motivated reasoning, exposure to the opposite point of view may actually backfire, not only do people reject the counter-arguments, but their own views become even more deeply entrenched as a result. In other words, an intelligent person with an inaccurate belief system may become more ignorant after having heard the actual facts.” (Robson 56)
“Three broad reasons why an intelligent person may act stupidly. They may lack elements of creative or practice intelligence that are essential for dealing with life’s challenges; they may suffer from ‘dysrationalia,’ using biased intuitive judgments to make decision; and they may use their intelligence to dismiss any evidence that contradicts their view thanks to motivated reasoning.” (Robson 62-63).
“The Curse of Expertise”
“There are now various social, psychological, and neurological reasons that explain why expert judgment sometimes fails at the times when it is most needed – and the sources of these errors are intimately entwined with the very processes that normally allow experts to perform well.” (Robson 68)
“The illusion of expertise may make you more close-minded. Victor Ottati at Loyola University in Chicago has shown that priming people to feel knowledgeable means that they were less likely to seek or listen to the views of others who disagreed with them” which he calls “earned dogmatism.” (Robson 71)
Experts may “overestimate their own knowledge” and fail to listen to other views or to update or to continue different circumstances (Robson 71)
“Nature has shown that over and over again that the kinds of truth which underlie nature transcend the most powerful minds” (Robson 72)
“Inflated self-confidence and earned dogmatism are just the start of the expert’s flaws” (Robson 72)
“It can also come with costly sacrifices. One is flexibility: the expert may lean so heavily on existing behavioral schemas that they struggle to cope with change” (Robson 74)
“A second sacrifice may be an eye for detail” (Robson 75)
“When fallible, gist-based processing is combined with overconfidence and ‘earned dogmatism,’ it gives us one final form of the intelligence trap – and the consequences can be truly devastating.” (Robson 75)
“[FBI] examiners appeared far more likely to dismiss or ignore any points of interest that disagreed with their initial hunch, while showing far less scrutiny for details that appeared to suggest a match…the examiners…were almost literally blinded by their expectations” (Robson 78)
“Overconfidence of experts themselves, combined with our blind faith in their talents, can amplify their biases - with potentially devastating effect.” (Robson 79)
Blind Spots from Expertise
“Blind spots that arise from expertise” (Robson 80)
“We have seen four potential forms of the intelligence trap:
“We may lack the necessary tacit knowledge and counterfactual thinking that are essential for executing a plan and preempting the consequences of your actions.
“We may suffer from dysrationalia, motivated reasoning, and the bias blind spot, which allow us to rationalize and perpetuate our mistakes, without recognizing the flaws in our own thinking. This results in our building ‘logic-tight compartments’ around our beliefs without considering all the available evidence.
“We may place too much confidence in our own judgment, thanks to earned dogmatism, so that we no longer perceive our limitations and overreach our abilities.
“Finally, thanks to our expertise, we may employ entrenched, automatic behaviors that render us oblivious to the obvious warning signs that disaster is looming, and more susceptible to bias.” (Robson 84)
Scientist Carl Sagan counseled Americans to “ask skeptical questions” and “to be skeptical of those in authority.” The Foundation for Economic Education wrote that “If the last few years have taught us anything, surely it must be that authorities are not omniscient, nor are they always honest or transparent.”
 Sowell, Thomas. Basic Economics, 5th Edition, 2015. Basic Books, Kindle Edition, p. 616.
 Sowell, Basic Economics, 5th edition, p. 617.
 Sowell, Thomas. Intellectuals and Society (p. 27). Basic Books. Kindle Edition.
 Ioannidis JP. Why most published research findings are false [published correction appears in PLoS Med. 2022 Aug 25;19(8):e1004085]. PLoS Med. 2005;2(8):e124. doi:10.1371/journal.pmed.0020124
 Sowell, Social Justice Fallacies, ibid.
 Rutz, David. “Viral exchange between Portnoy and Wash Post reporter shows 'sad state of American journalism,' critics say.” Fox News, September 25, 2023. https://www.foxnews.com/media/viral-exchange-between-portnoy-wash-post-reporter-shows-sad-state-american-journalism-critics-say
 Timothy, Nick. “Elite disinformation is a far greater problem than fake news on Twitter.” The Telegraph (UK), October 22, 2023. https://www.telegraph.co.uk/news/2023/10/22/gaza-war-israel-hamas-disinformation-twitter-social-media/
 Sowell, Basic Economics, 5th edition, p. 461.
 Miltimore, John. “Carl Sagan's Final Warning on the Importance of Scientific Skepticism.” Foundation for Economic Education, September 18, 2023. https://fee.org/articles/carl-sagans-final-warning-on-the-importance-of-scientific-skepticism/