5x3: A practical approach to encouraging thoughtful statistical analyses

Abstract

Empirical researchers who draw conclusions from statistical hypothesis tests have long been urged to be more thoughtful about their statistical designs, and to provide more modest interpretations of their findings. To support such behavioral change, we present a novel visualization tool for setting sample sizes and analyzing summary data. The core idea is to present multiple test options in the design phase of research, and to present alternative interpretations of summary data in the analysis phase. Our interactive tool is underpinned by a mathematically rigorous multiple testing procedure featuring one-sided t-tests against margins of material significance (i.e., practically meaningful effect sizes). Its presentations resemble the funnel plots developed for statistical meta-analyses. The tool is applied to studies from three different fields to illustrate how it might encourage better statistical design and more nuanced analysis and reporting.