The incorrect use of statistics in scientific articles seems to be a never-ending discussion topic. A current controversy involves a decision by Basic and Applied Social Psychology in 2015 to ban the use of P-values (i.e., null hypothesis testing) in articles appearing in their journal. This prompted the American Statistical Association to publish, in 2016, a policy statement on the use of P-values in research publications. Reinhart (2015) in his book, Statistics Done Wrong: The Woefully Complete Guide, gives a good overview of the sorts of statistical mistakes made in science, with many biological examples.
There are also attempts to gauge how severe the misuse of statistics is in various biological disciplines. The article on the website hosted by influentialpoints.com (Dransfield and Brightwell, 2012) provides an overall guide to statistics misuse in biology, with a bias toward medicine. The authors of this site categorized errors found in an examination of “several thousand papers” and the article posted is abstracted from their book (Brightwell and Dransfield, 2013).
A recent evaluation of incorrect analyses of interaction effects in the neurosciences found that about half the published articles had statistical issues when analyzing factorial treatment designs, with some apparently severe enough to call the study’s conclusions into question (Nieuwenhuis et al., 2011). A recent Nature article by Allison et al. (2016) discussed how easy it was to find mistakes in data handling in publications, but how hard it was to get them fixed. Although there are many reasons why a statistical analysis may or may not be appropriate, only those most applicable to horticulture will be discussed below.
We examined issues of the JASHS published between Jan. 2014 and Jan. 2015 inclusive, for statistical problems. This was prompted by an interest in revising the currently antiquated instructions to authors about the use of statistics in the society’s journals. To do this, we needed to identify the kinds of statistical methodologies required by current authors to support their findings, the kinds of data being collected, and what authors were actually doing when analyzing the data. The revised version of statistics instructions will be appearing separately. Here, we describe the kinds of statistical errors most commonly made by authors in this journal and characterize the patterns of errors and omissions we found. These are not necessarily fatal flaws, but reveal weaknesses that may affect conclusions. We then ascribe probable causes and suggest some possible remedies. We hope this review will be helpful both to authors and reviewers.
Benjamini, Y. & Hochberg, Y. 1995 Controlling the false discovery rate: A practical and powerful approach to multiple testing J. R. Stat. Soc. B. 57 289 300
Brightwell, R. & Dransfield, R.D. 2013 Avoiding and detecting statistical malpractice: Design and analysis for biologists, with R. 25 May 2016. <http://influentialpoints.com/aboutus.htm>.
Dransfield, R.D. & Brightwell, R. 2012 Statistical mistakes in research: Use and misuse of statistics in biology. 16 Nov. 2015. <http://influentialpoints.com/Training/statistical_mistakes_in_research_use_and_misuse_of_statistics_in_biology.htm>.
Fatima, T., Sobolev, A.P., Teasdale, J.R., Kramer, M., Bunce, J., Handa, A.K. & Mattoo, A.K. 2016 Fruit metabolite networks in engineered and non-engineered tomato genotypes real fluidity in a hormone and agroecosystem specific manner Metabolomics 12 103
Hochberg, Y. & Tamhane, A.C. 1987 Multiple comparison procedures. Wiley, New York, NY
Johnson, R.A. & Wichern, D.W. 2007 Applied multivariate statistical analysis. 3rd ed. Pearson, New York, NY
Littell, R.C., Milliken, G.A., Stroup, W.W., Wolfinger, R.D. & Schabenberger, O. 2006 SAS for mixed models. 2nd ed. SAS Institute, Cary, NC
Milliken, G.A. & Johnson, D.E. 2009 Analysis of messy data: Designed experiments. Vol. 1, 2nd ed. Chapman and Hall/CRC, New York, NY
Nieuwenhuis, S., Forstmann, B.U. & Wagenmakers, E.J. 2011 Erroneous analyses of interactions in neuroscience: A problem of significance Nat. Neurosci. 14 1105 1107
R Core Team 2013 R: A language and environment for statistical computing. 25 May 2016. <http://www.R-project.org/>.
Reinhart, A. 2015 Statistics done wrong: A woefully complete guide. No Starch Press, San Francisco, CA
Schlotter, Y.M., Veenhof, E.Z., Brinkhof, B., Rutten, V.P., Spee, B., Willemse, T. & Penning, L.C. 2009 A GeNorm algorithm-based selection of reference genes for quantitative real-time PCR in skin biopsies of healthy dogs and dogs with atopic dermatitis Vet. Immunol. Immunopathol. 129 115 118
UC3 Data Pub Blog 2012 Archiving data, best practices, data sharing. 20 Apr. 2016. <https://datapub.cdlib.org/2012/11/20/thanks-in-advance-for-sharing-your-data/>.
Westfall, P.H. & Young, S.S. 1993 Resampling-based multiple testing: Examples and methods for P-value adjustment. Wiley, New York, NY