Publication Bias in Neurology Research: Why It Matters and Why It Needs to Change

Publication of completed research studies and clinical trial results is an essential step in the research process and serves many purposes, including advancing scientific discovery, fostering collaboration, and shared problem solving to minimize duplication of effort. It also provides evidence on the basis of which medical decisions are made.

The need for data transparency carries an ethical and moral obligation and requires nonselective and unbiased publication of clinical results. The passage of several federal acts was designed to increase clinical data transparency and included the Food and Drug Administration (FDA) Modernization Act of 1997 and the FDA Amendments Act of 2007, together with the establishment of the ClinicalTrials.gov registry in 2000, the clinical trial registry and results databank operated by the National Library of Medicine of the National Institutes of Health, and efforts by the National Institute of Health to promote timely and complete publication of results within 12 months of study completion.^1-3 Despite these efforts, a significant amount of clinical research is discontinued, has delayed publication, or is not published. A study found that among 362 eligible neurodegenerative disease trials, 42 (12%) had been discontinued, and 91 (28%) of the 320 completed trials remained unpublished up to 5 years after study completion, possibly to conceal negative study results.⁴

In addition, in general, there is bias toward reporting positive results.⁵ Turner and colleagues compared FDA reviews of antidepressant trials and found that 37 of 38 trials viewed as positive by the FDA were published. In contrast, of the 36 trials with negative or questionable results, according to the FDA, 22 were unpublished and 11 of the 14 published studies conveyed a positive outcome.⁶ Interestingly, 94% of the published trials reported positive results, in contrast to the 51% reported by the FDA.⁶

The reasons for publication bias are quite complex, and seem to depend on the source and purpose of the research. For example, if the study hypothesis for an industry-supported, early-phase, randomized trial is not upheld by the study results, there is less motivation to complete the study and publish the data compared with data from a late-phase trial that supports product registration. In an academic setting, the culture of “publish or perish” is in itself a motivator to publish all data, irrespective of the hypothesis, study results, and quality of the study design.^7,8 Further, the results of some trials may never reach the threshold deemed necessary to merit the attention of journal editors.⁹

Withholding negative research results not only skews scientific evidence but also brings into question the validity of medical research.^10,11 At best, it slows scientific progress, and at worst, it may expose patients to unnecessary or dangerous treatments in clinical trials, overestimate the therapeutic value of an investigational treatment, or result in avoidable deaths from agents approved with incomplete clinical data.¹¹ Case in point is the withdrawal of rofecoxib (for Alzheimer disease) and valdecoxib (for acute pain) after reanalysis of clinical data with the inclusion of data that were not published or presented during the FDA approval process, which revealed increased cardiovascular risk.^.6

The risk for withholding negative results is particularly worrisome in research on neurodegenerative diseases, given the high failure rate of these studies. The success rate for clinical studies for Alzheimer disease (AD) treatment, with success defined as positive result outcomes, is very low; a failure rate of 99.6% was reported for studies conducted between 2002 and 2012.¹² Discussion of the reasons for the high failure rate is beyond the scope of this article, although inappropriate clinical design and assessment of treatment outcome have been suggested.¹³

In a recent review, Jeffrey Cummings, MD, argued that a clinical study deemed a failure can, in fact, contribute to improving the success of AD drug development; for example, by improving preclinical models of AD and contributing to improved knowledge of drug penetration of the blood-brain barrier, dose response, and improved application of biomarkers for diagnostic evaluation. Negative results can also contribute to improved clinical trial design that may include adjustment of sample size, use of appropriate comparator groups, and avoidance of misleading subgroup analyses.¹²

Transparency and ethical obligations require that positive as well as negative study results be published. Efforts are being made to address the focus and preference for publishing positive study results by introducing a negative results section in journals, including a new journal dedicated to the publication of negative research findings.^14,15

Robert E. Becker, MD, independent researcher and president of Aristea Translational Medicine Corporation in Utah, told Neurology Advisor, “Scientists learn from their mistakes and can learn only if they know them. In today’s field of AD translational medicine, the greatest harms derive from the often poor quality of science tolerated by leading academic and industry investigators and journal editors.”

The dangers of overestimating drug efficacy and underestimating safety risks that result from bias for positive data, and from which healthcare decisions are made, has caught the attention of the US Department of Health and Human Services, which recently issued a new directive, an extension of the FDA Amendments Act of 2007, effective since January 2017, requiring submission of registration and summary information, including adverse event information, for specified clinical trials of drug products (including biological products) and device products to ClinicalTrials.gov.¹⁶

“Not revealing why drug candidates and clinical trial methods failed or publishing only the most positive results from a drug is witness to the all-too-prevalent commercial interests dominating in modern clinical research and its dissemination to the profession and public,” Dr Becker added.

Although efforts have been made through various federal acts to improve transparency and consider negative study results as important as positive results, changing the culture of publication bias ultimately rests with the integrity of the researcher and the moral and ethical obligation to publish their data in its entirety.

References

1. US Food and Drug Administration. Full Text of FDAMA law. Food And Drug Administration Moderization Act of 1997. https://www.fda.gov/RegulatoryInformation/LawsEnforcedbyFDA/SignificantAmendmentstotheFDCAct/FDAMA/FullTextofFDAMAlaw/default.htm. Updated August 2, 2017. Accessed October 18, 2017.
2. Lo B. Sharing clinical trial data: maximizing benefits, minimizing risk. JAMA. 2015;313(8):793-794.
3. Zarin DA, Tse T, Williams RJ, Carr S. Trial reporting in ClinicalTrials.gov – the final rule. N Engl J Med. 2016;375(20):1998-2004.
4. Stefaniak JD, Lam TCH, Sim NE, Al-Shahi Salman R, Breen DP. Discontinuation and non-publication of neurodegenerative disease trials: a cross-sectional analysis. Eur J Neurol. 2017;24(8):1071-1076.
5. Goodchild van Hilten L. Why it’s time to publish research “failures.” Publishing bias favors positive results; now there’s a movement to change that. https://www.elsevier.com/connect/scientists-we-want-your-negative-results-too. Published May 5, 2015. Accessed October 18, 2017.Turner EH, Matthews AM, 
6. Linardatos E, Tell RA, Rosenthal R. Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med. 2008;358(3):252-260.
7. Gasparyan AY, Nurmashev B, Voronov AA, Gerasimov AN, Koroleva AM, Kitas GD. The pressure to publish more and the scope of predatory publishing activities. J Korean Med Sci. 2016;31(12):1874-1878.
8. Granqvist E. Why science needs to publish negative results: An Elsevier publisher argues that an experiment shouldn’t have to show positive results to earn its place in the published literature. https://www.elsevier.com/authors-update/story/innovation-in-publishing/why-science-needs-to-publish-negative-results. Published March 2, 2015. Accessed October 18, 2017.
9. Hudson KL, Collins FS. Sharing and reporting the results of clinical trials. JAMA. 2015;313(4):355-356.
10. Kicinski M. How does under-reporting of negative and inconclusive results affect the false-positive rate in meta-analysis? A simulation study. BMJ Open. 2014;4(8):e004831.
11. Joober R, Schmitz N, Annable L, Boksa P. Publication bias: what are the challenges and can they be overcome? J Psychiatry Neurosci. 2012;37(3):149-152.
12. Cummings J. Lessons learned from Alzheimer disease: clinical trials with negative outcomes [published online August 2, 2017]. Clin Transl Sci. doi: 10.1111/cts.12491
13. Becker RE, Greig NH, Giacobini E. Why do so many drugs for Alzheimer’s disease fail in development? Time for new methods and new practices? J Alzheimers Dis. 2008;15(2):303-325.
14. Dirnagl U, Lauritzen M. Fighting publication bias: introducing the Negative Results section. J Cereb Blood Flow Metab. 2010;30(7):1263-1264.
15. ScienceDirect. New negatives in plant science. Open Access Journal.
16. National Institutes of Health, Department of Health and Human Services. Clinical Trials Registration and Results Information Submission. Final rule. Fed Regist. 2016;81(183):64981-65157.