1660240581197 Rosenzweigweb

Research & Development: Sloppy Statistics Mar Papers

Feb. 21, 2017
Analysis of articles in one field finds that statistical mistakes are common

[pullquote]Technical journals play a crucial role in the effective communication of experimental results and, thus, the progress of science and engineering. So, publication of properly reviewed and accurate information is paramount. Unfortunately, some journals choose content based on inadequate peer reviewing or without any real evaluation whatsoever. (See “Hoax Highlights Publishing Problem.”) Now, an investigation by a team at Tilburg University, Tilburg, the Netherlands, has found that many of the papers they checked have statistical mistakes.

The team focused on null hypothesis significance testing — a technique to determine whether a result from a given set of statistics is significant or not. The Dutch investigators developed a program called statcheck, downloadable at http://statcheck.io, to check files in either pdf or html format for errors in statistical reporting. The software automatically extracts statistics from a paper and recalculates the p (probability) value, a measure of the validity of the hypothesis. Many researchers treat p ≤ 0.05 as indicating a “true” finding.

In a recent posting on “Retraction Watch,” a site that tracks retractions of scientific papers, Michèle Nuijten, a PhD student at the university who helped develop the program, summarized the reason for developing it as well as the troubling results the software has uncovered.

“We knew we would never be able to program statcheck in such a way that it would be as accurate as a manual search, but that wasn’t our goal. Our goal was to create an unbiased tool that could be used to give an indication of the error prevalence in a large sample, and a tool that could be used in your own work to flag possible problems.”

“Unfortunately, we recently found that our suspicions [of reporting inconsistencies] were correct: Half of the papers in psychology contain at least one statistical reporting inconsistency, and one in eight papers contain an inconsistency that might have affected the statistical conclusion.”

Fuller details appear in “The Prevalence of Statistical Reporting Errors in Psychology (1985–2013).”

These disturbing findings only relate to papers published in experimental psychology. Articles in research journals focused on chemical engineering and chemistry hopefully are less prone to such errors, given the technical background of most authors. However, how can we be sure?

Nuijten notes: “At the moment statcheck can only read results reported in APA (American Psychological Association) style (e.g., t(28) = 3.21, p < 05; or more generally: test statistic (degrees of freedom) =/ …, p =/ …).

Extending such a program to other article styles would require they have two “ingredients,” she explains:

1. A standardized reporting style. “The way statcheck works now is that it looks for very specific combinations of letters, numbers and things like parentheses and equal signs, etc. For each specific statistical test that it can read, we had to program separate lines of code.”

2. A way to recalculate numbers from the extracted results. “The statistics that statcheck reads now consist of three parts that are internally consistent. That means that if I have two of the three parts, I can calculate the third to check if the reported result is indeed consistent.”

The Dutch team deserves praise for their initiative in developing a tool that can make spotting statistical mistakes easier. Let’s hope software of this type becomes more widely available and applicable.

However, the very existence of a website like Retraction Watch underscores that articles with flawed research (not just related to statistics) remain a persistent problem — one that occurs more often than most of us would care to believe. The “publish or perish” environment with which many researchers must contend probably only will get worse in the foreseeable future. So, the importance of rooting out inaccurate work surely will increase.

High-quality research journals take this responsibility very seriously. Unfortunately, too many readers assume that all journals are equally diligent. That’s another critical flaw that needs attention.

MARK ROSENZWEIG is Chemical Processing's Editor in Chief. You can email him at [email protected].
About the Author

Mark Rosenzweig | Former Editor-in-Chief

Mark Rosenzweig is Chemical Processing's former editor-in-chief. Previously, he was editor-in-chief of the American Institute of Chemical Engineers' magazine Chemical Engineering Progress. Before that, he held a variety of roles, including European editor and managing editor, at Chemical Engineering. He has received a prestigious Neal award from American Business Media. He earned a degree in chemical engineering from The Cooper Union. His collection of typewriters now exceeds 100, and he has driven a 1964 Studebaker Gran Turismo Hawk for more than 40 years.

Sponsored Recommendations

Keys to Improving Safety in Chemical Processes (PDF)

Many facilities handle dangerous processes and products on a daily basis. Keeping everything under control demands well-trained people working with the best equipment.

Comprehensive Compressed Air Assessments: The 5-Step Process

A comprehensive compressed air audit will identify energy savings in an air system. This paper defines the 5 steps necessary for an effective air audit.

Get Hands-On Training in Emerson's Interactive Plant Environment

Enhance the training experience and increase retention by training hands-on in Emerson's Interactive Plant Environment. Build skills here so you have them where and when it matters...

Managing and Reducing Methane Emission in Upstream Oil & Gas

Measurement Instrumentation for reducing emissions, improving efficiency and ensuring safety.