With education and training, both researchers and consumers of scientific literature might agree on two things: (1) mistakes can easily be made in science, and (2) these mistakes are typically washed-out over time in the literature. Unfortunately, one of these statements seems to dominate the other in a number of ways, at least according to the Economist.
The Economist wrote a piece entitled, “Unreliable Research: Trouble at the lab” released with the following statement: “Scientists like to think of science as self-correcting. To an alarming degree, it is not.” The publication underlines the basic premise that science relies on controls and statistics that can reveal the true workings of science and nature. When these controls and statistics are run by humans who are vulnerable to error and pressures given their environment, it may not be entirely surprising that science gets it wrong from time to time.
Unfortunately, the idea that science is self-correcting tends to receive more credit than is due. With evidence that “more dodgy results are published than are subsequently corrected or withdrawn calls that much-vaunted capacity for self-correction into question. There are errors in a lot more of the scientific papers being published, written about and acted on than anyone would normally suppose, or like to think,” states the author within the Economist article.
“The idea that the same experiments always get the same results, no matter who performs them, is one of the cornerstones of science’s claim to objective truth. If a systematic campaign of replication does not lead to the same results, then either the original research is flawed (as the replicators claim) or the replications are (as many of the original researchers on priming contend). Either way, something is awry.”
Whether you are a tenured professor with decades of research experience or an everyday consumer of research through media blasts, below are five simple ways that you can contribute to improving the accuracy of the science that is perpetuated throughout academia, industry, and the general public.
1. Statistics: Helping and hindering
Statistics are an essential component of every set of data in order to determine significant differences between groups. These differences are the basis upon which we make claims and conclusions. If the statistics that determine these differences are not properly conducted, then the results are incorrect and perpetuated until proven otherwise.
Action: Take the opportunity to educate yourself on the statistics used in a given study, or give pause at the next claim to have “found the cure” in a sensationalized news article. Want to learn more? Check out a variety of lessons offered by Khan Academy: https://www.youtube.com/watch?v=CI7_WKmW4N0
2. Peer Review: Finding the needle in the haystack
With researchers dedicating their careers to a particular field, sub-field, or even a sub-topic within a sub-field, it might be difficult finding an appropriate reviewer to take an in-depth, critical approach to their newly-submitted journal article. Anticipating this need, many journals choose to assign a team of volunteer reviewers to every submission. However, the system is imperfect, as seen with this paper that accepted by hundreds of journals.
Action: Be your own reviewer. Think critically about the paper you read, and be sure to examine the data yourself to see if they are convincing. When given the chance to review prior to publication, ensure you are an appropriate fit for the topic and scope of the paper. Don’t hesitate to ask others who may be experts in a sub-topic that may be unfamiliar. You can return the favor in your own area of expertise, and know that you’re improving the reviewing process and the scientific body of literature!
3. Publication Bias
Those within academia are all-too-aware of the personal pressures surrounding publications. Not only is a steady stream of publications required, but publications that are typically accepted feature a significant difference between groups as opposed to negative data where both the experimental group and the control group performed similarly after treatment. The environment of pressure and the need for the data with a particular outcome certainly contribute to the errors in scientific literature.
Action: Support an organization with integrity by donating to their cause directly and specifying your preferred use for the funds. Providing support to researchers allows additional resources to verify their results prior to submitting them to a journal. These additional resources are often required for scientists who are attempting to publish their negative data, where many more corroborating experiments are necessary prior to a journal considering publishing it.
4. Measure Twice, Cut Once
My mother always taught me that you should measure the length of fabric needed at least twice before you make the final cut. Front-loading work into data collection and analysis allows for less waste of materials, back-pedaling and additional work at a later time if these original measurements weren’t accurate. While each and every data set does require replicates within that publication, it may surprise the average citizen that these data are typically not replicated and verified by an outside source. When Amgen attempted to replicate 53 cancer-related studies, 47 produced differing results, even after close work with the original authors. In other words, only 6 of these studies held up to another attempt at the scientific issue at hand. How is the general public supposed to know which of these studies to believe?
Action: Is there an important research finding that you think will change the world if given the appropriate funding? Donate to your favorite scientist! Do a little digging, and you’ll find a researcher team or organization that will support your donation with earmarking those funds for a crucial replication study.
What do you think about these ideas? Can an individual make a difference in the large institution of scientific research? How else can we act to improve our methods of gaining knowledge to fuel our evidence-based practices?
Check out the full article here, as the Economist points out many other parts of this situation that should be addressed. By educating yourself about this issue, you are contributing to the solution!
