Political Science Research: A Crisis of Statistical Power

Resources

Books

• "Quantitative Political Science Research is Greatly Underpowered" (Paper) - This paper is central to the episode's discussion, analyzing the statistical power of studies in political science.

Research & Studies

• "The Median Study is Somewhere in that Kind of You Know 10 to 20 Range It's Really Low" (Vince Errol Bundesen and co-authors) - This finding from the paper indicates that the median study in political science has a low probability of detecting a statistically significant result.
• "Quantitative Research in Political Science is Greatly Underpowered" (Vince Errol Bundesen and co-authors) - The title of the paper, directly stating the main conclusion about the low statistical power in the field.
• "Published Effects in the Literature by a Factor of About Three" (Vince Errol Bundesen and co-authors) - The paper suggests that published effects in political science might be overestimated by a factor of three due to low power and selection on significance.
• "Meta-Analyses" - Used in the paper to establish a consensus effect size across a literature, serving as a benchmark to estimate the power of individual studies.
• "The Problem of the Replication Crisis and Related Phenomena P Hacking" - Mentioned as a broader context for concerns about the reliability of evidence in social sciences.
• "Meta-Analyses" (Vince Errol Bundesen and co-authors) - The paper utilizes meta-analyses to estimate consensus effect sizes and then assesses the power of individual studies against these estimates.
• "Political Analysis" (Methodology Journal) - Emails were sent to hundreds of people who had published in this journal to survey their expectations about study power.
• "Paper with the Same Co-Author Ryan Briggs and Macouvinia" (Ethan Bunder de Mesquita) - This paper looked at whether published findings reject the null hypothesis, finding that most papers in political science do reject the null.
• "Bayes' Rule" - Discussed as a framework for updating beliefs based on statistical evidence, contrasting with traditional frequentist approaches.
• "Income and Democracy" - Mentioned as an example of a question that might be difficult to answer due to insufficient data for powerful studies.
• "Ethnicity and Social Trust" - Another example of a question where answering empirically might be challenging with current data limitations.

People Mentioned

• Vince Errol Bundesen - The paper's author interviewed for the podcast, who conducted a study on the statistical power of political science research.
• Ryan Briggs - Co-author of a previous paper with Ethan Bunder de Mesquita on null results in political science.
• Macouvinia - Co-author of a previous paper with Ethan Bunder de Mesquita on null results in political science.

Organizations & Institutions

• Apsr (American Political Science Review) - Mentioned in the context of how published papers are cited and taken as authoritative, even if their findings might be questionable.

Websites & Online Resources

• Political Science Literature - The overall body of research analyzed in the paper to assess statistical power.

Other Resources

• Statistical Power - The probability that a statistical test will reject the null hypothesis if the alternative is true, a key concept discussed throughout the episode.
• Null Hypothesis - The hypothesis that there is no effect or relationship, which statistical tests aim to reject.
• Alternative Hypothesis - The hypothesis that there is an effect or relationship, which statistical tests aim to support.
• Significance Level - The threshold (commonly 5%) used to determine statistical significance.
• Confidence Intervals - Ranges that indicate the uncertainty around an estimate.
• Meta-Science - The study of science itself, including its processes and outcomes, relevant to understanding issues like replicability.
• Selection on Significance - The practice of only publishing or focusing on statistically significant results.
• P Hacking - Manipulating data or analyses until a statistically significant result is found.
• Meta-Analytic Estimate - A consensus estimate of an effect size derived from multiple studies.
• Standard Error - A measure of the variability of an estimate.
• Effect Size - The magnitude of a phenomenon or relationship.
• Minimum Effect Size - The smallest effect that would be considered interesting or consequential.
• Qualitative Work - Research methods that do not rely on numerical data, suggested as a complement to quantitative analysis.
• Randomized Experiments - A research design where participants are randomly assigned to treatment or control groups.
• Observational Studies - Studies where researchers observe phenomena without manipulating variables.
• Bayesian Statistical Analysis - A statistical approach that incorporates prior beliefs into the analysis of data.
• Frequentist Reporting - The traditional approach to statistical reporting, often focusing on p-values and statistical significance.
• Credibility Revolution - A period in social science that emphasized rigorous research designs and identification strategies.