The Role Robust Statistical Methods in the Credibility Revolution of Psychological Science

The credibility movement in psychology arose as a response to failures to replicate a number
of effects that were previously considered robust (Open Science Collaboration, 2015).
Previous research indicated that the flexibility the researchers exercise when deciding how
to analyse their data, which transformation to apply, and which cases to retain or exclude
(Steegen et al., 2016), has contributed current replicability issues. Analytic decision making
can be influenced by a number of factors, including bias towards producing results that are
more likely to get published because of statistical significance, but may subsequently fail to
replicate. This project will investigate how limiting analytic flexibility in the research process
- namely by preregistering analytic plans and applying robust statistical methods that
remain unbiased under conditions where other commonly used models underperform - can
help address some of the problems with replicability in psychology by reducing the
possibilities for generating false-positive effects and improving power to discover effects
that would otherwise remain undetected. It will do so by: (1) querying psychology
researchers about their knowledge and understanding of the assumptions of the general
linear model (GLM), the most commonly applied statistical model in psychology, and about
the solutions they apply to counteract bias from commonly occurring characteristics of data
from psychological research; (2) using Natural Language Processing (NLP) to text-mine
information from published papers and examine the language the researchers use to
describe their data, statistical models and draw conclusions about the effects under
investigation; (3) assessing the performance of robust statistical methods under real-world
conditions of psychological data (e.g., typical distributional characteristics, presence of
outliers and influential cases, etc.); (4) evaluating the robustness of published effects for
preregistered and non-preregistered studies. The project will shine a light on current
statistical practice within psychological research and lead to specific 'best practice'
recommendations for psychology researchers at all career stages. This work will address the
emerging methodological challenges in the areas of Open Science, replicability, and
transparency of research practice, thus having a direct impact on the improvement of
psychology as a scientific discipline.