Perusing the advertisements for self-improvement so abundant in these early and optimistic days of the new year, I am once again sensitized to the fact -- and it is a fact -- that the vast majority of "cures," "transformational methods," "miracle treatments," "breakthroughs," and other magical palliatives paraded before us as consumers of goods and services both psychological and otherwise are based on little more than the testimonials that, in the small print or rapid-fire speech accompanying the claims made, are described as "non-typical results" with the disclaimer that our own "experiences may vary." Thus, that 50-pound weight loss may have indeed occurred, but only in very few among the thousands who tried to lose that much. Nonetheless, when we, even as intelligent consumers, see that something has worked for someone "just like us," we believe that it will work for us as well. However, because something works for someone, it does not mean it will work for everyone. Testimonials are a sure sign that we as buyers must beware.
But are there good ways of knowing that we can feel confident in our decision to try a medical procedure, a psychological treatment or a new diet, exercise device or skin cream? Yes. The fact is -- and it is a fact -- sound research done by careful scientists continues to be the gold standard for judging the efficacy of things offered up to make our lives better and longer. Were we to examine the effectiveness of some magical nostrum as scientists, individual testimonials could be replaced, as an example, by far more helpful probabilistic estimates based on carefully-collected and analyzed data from representative samples large enough to allow generalization to larger groups of people. Despite this last sentence being incredibly boring to write and read (I apologize for this so early in the year), we would learn from such things as probability estimates what the likelihood would be of there being a significant positive effect of a given new treatment or diet or the like. For example, we might learn that the positive effects described (think a 50-pound weight loss, or six-pack abs, or better grades in school) would have occurred in less than five out of every 100 people who used the product or procedure. Now, please ask yourself, "Would I buy or use something that worked on only five people out of 100?" The answer is likely to be a confident and resounding, "NO!"
To this point, I think I am saying something most people sense instinctively. Simply put, science trumps testimonials. However, in recent years, there has been a weakening of confidence in the science offered up to us by researchers in many fields, but especially medicine, pharmaceuticals and psychological science. We have begun to distrust what some scientists tell us because it seems that either they are driven by conflicts of interest (as in the many researchers doing pharmaceutical company-sponsored studies of new drugs or treatments) or because we have seen reports of scientific dishonesty among respected scientists from major universities both in the U.S. and abroad (Europe and Asia). We have begun to hear more and more about "retractions," in which published articles are essentially "unpublished," taken back as having been based on unsound methods, faulty or false data or other violations of fundamental scientific methods. Not only do these retractions announce that the original studies were not acceptable, but they add to the growing sense that science itself cannot be trusted. The worst outcome of this would be that scientific research could move toward being grouped with testimonials and other instinctively untrustworthy sources of guidance in our decisions regarding what to do with or to our bodies, minds, children, loved ones, communities and nations.
But there is way out of this. The best way to ensure that any scientific finding is reliable is for it to be replicated. This means that someone else other than the originator of the idea or discoverer tries to do the same study and to find the same thing. Independent replication of scientific research would be akin to the confidence-building power of Consumer Reports, wherein claims of manufacturers or service companies are tested and evaluated by independent experts. Easy to say, but here is the problem facing science today: Almost no independent replications of published research studies are ever done, and if they are done, they are rarely accepted for publication even, or especially, by the same journals that published the original research report. This state of affairs has resulted in what the influential philosopher of science, Thomas Kuhn, once termed a "crisis-in-science-as-is." By this, he meant that all sciences sometimes experience points at which doing things in the same way they have always done them is no longer sustainable. The science is not working optimally. Something must change. The weakening of trust in science, especially in my own area of psychological science, seems now to have produced such a crisis. It is my hope that this blog post can help expand awareness of the response of psychological science to this crisis.
It is becoming increasingly clear that replications of existing studies must be done if we as consumers are to base life decisions on what scientists tell us. The good news is that the Association for Psychological Science (APS) has gathered together some of the most well-informed experts on the question of rebuilding confidence in not only psychological research, but research in all areas of science. A special issue (November 2012) of the APS journal, Perspectives on Psychological Science, is dedicated to the crisis in science based on the absence of replication. In a gesture recognizing the potential wide interest in this topic beyond psychological science itself, the issue has been made available free to anyone wishing it. As an indication of the importance of the ideas presented in this selection of writings dealing with the current crisis in science, more than 215,000 people downloaded the issue in the first three weeks of its being available. The URL at which readers may obtain the free issue of the journal is: http://pps.sagepub.com/content/7/6.toc.
I urge scientists and non-scientists alike, students and teachers, high school, college and graduate level, regardless of area of specialization, to read this very important publication. Unless science gets its house in order, we will have no more credibility than that handful of "satisfied users" of the exercise equipment whose abs are now "six packs" or that 43-year-old worker from Everycity, USA who lost 75 pounds eating "fat-burning donuts." When scientists report that something works well, people should have reason to believe us. When we report that something doesn't work, they should be able to believe us as well. Replicating our work and publishing those replications is a good place to start. If one study tells me that a treatment works, and then a second and third show the same thing with different participants, I grow more confident. If one study says something works and then a second and third do not show the same thing, I must grow wary.
According to Thomas Kuhn, the current crisis in science is not abnormal; such tensions and shifts are inherent in normal science. What would not be adaptive would be simply continuing our current publishing practices, which reflect a devaluing of replications. If scientists, regardless of their specific area of study, are to remain the sources of credible guidance in a world filled with unfounded and unsubstantiated claims -- claims essentially made possible by the lack of confidence in the gold standard that we represent -- we must raise our game. Psychological Science is trying to do this despite feeling a bit ashamed and disconcerted that so many of the widely publicized violations have occurred within its boundaries. The problem of replication spreads throughout the sciences, however, and all must take heed.
For more by Marshall P. Duke, click here.
For more on emotional intelligence, click here.