Open Collections

Abstract

The pursuit of explicitly justifying a study’s sample size began in hopes of mitigating p-hacking and false positive rates across psychology through having researchers be more thoughtful about the research process. However, there is still the question about how to properly justify a sample size for a study. A review of five years of sample size justifications in Psychological Science demonstrates that researchers use many different methods to justify a sample’s size. For example, some common justifications are heuristics, resource constraints, and priori power analysis. While there are many ways to justify a study’s sample size, much uncertainty still surrounds the merit and validity of the different types of sample size justifications. To give researchers and readers a better benchmark to justify and evaluate sample size, I propose using the distribution of effect sizes in a specific discipline to find the average statistical power, known as generalized average power. Starting with the distribution of effect sizes in social psychology, I use robust Bayesian model averaging across 71 meta-analyses to simulate the distribution of population effect sizes in social psychology. I then use the distribution to determine the sample size needed to maintain an average power of 80% across the field. This provides an evidence-based heuristic for researchers, readers, reviewers, and editors to effectively evaluate — and justify — sample sizes.