esade

Research Design in Quantitative Methods (2225.YR.008333.2)

General information

Type:

OBL

Curs:

1

Period:

S semestre

ECTS Credits:

2.5 ECTS

Teaching Staff:

Prerequisites

d

COURSE CONTRIBUTION TO PROGRAM

Provides a bridge between foundational statistical concepts and everyday practice of empirical research

Course Learning Objectives

Empower students to understand and conduct credible empirical research

CONTENT

1. Contens overview

1. Simulations: Montecarlo, boostrapping, and resampling
2. Meta-analysis: traditional tools, publication bias correction, internal meta-analysis
3. Fixing and Breaking things with regression
4. Bayesian hypothesis testing
5. Conducting and evaluating replication results
6. Experimental design and analysis
7. What is the purpose of data?

Methodology

Lecture, discussion, in class analysis of data.

ASSESSMENT

ASSESSMENT BREAKDOWN

Description %
Homeworks 50

Assessment criteria

There is an assignment each weak invovling data analysis, generation of graphs, or discussion of readings.
There is a final example with short answer questions.

Bibliography



Torfs & Brauer (2014) A (very) short introduction to R,Hydrology and Quantitative Water Management Group, Wageningen University, The Netherlands,

Simonsohn (2013) Just post it: the lesson from two cases of fabricated data detected by statisics alone,Psychological Science, 24(10), 1875-1888

Joseph P. Simmons, Nelson, & Simonsohn (2011) False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant,Psychological Science, 22(11), 1359-1366

(S) Simonsohn, Simmons, & Nelson (2020) Specification curve analysis,Nature Human Behaviour, 4(11), 1208-1214

Young (2019) Channeling fisher: Randomization tests and the statistical insignificance of seemingly significant experimental results,The Quarterly Journal of Economics, 134(2), 557-598

Boos (2003) Introduction to the bootstrap world,Statistical science, 18(2), 168-174

Boos & Brownie (1989) Bootstrap methods for testing homogeneity of variances,Technometrics, 69-82

Pitman (1937) Significance tests which may be applied to samples from any populations,Journal of the Royal Statistical Society, 4(1), 119-130

David (2008) The beginnings of randomization tests,The American Statistician, 62(1), 70-72
A three page summary of the origin of the idea of randomization. Nutshell: Fisher and others focused on normality, Pitman went beyond it.


Micceri (1989) The unicorn, the normal curve, and other improbable creatures,Psychological bulletin, 105(1), 156

Sawilowsky & Blair (1992) A more realistic look at the robustness and Type II error properties of the t-test to departures from population normality,Psychological bulletin, 111(2), 352

White (2003) A reality check for data snooping,Econometrica, 68(5), 1097-1126

Daniël Lakens (2015) The 20% Statistician - Always use Welch's t-test instead of Student's t-test

Bind & Rubin (2020) When possible, report a Fisher-exact P value and display its underlying null randomization distribution,Proceedings of the National Academy of Sciences, 117(32), 19151-19158

Morris, White, & Crowther (2019) Using simulation studies to evaluate statistical methods,Statistics in medicine, 38(11), 2074-2102

Rousselet, Pernet, & Wilcox (2021) The percentile bootstrap: a primer with step-by-step instructions in R,Advances in Methods and Practices in Psychological Science, 4(1), 2515245920911881

(A) Simonsohn, Nelson, & Simmons (2014) p-curve: A Key to the File Drawer,Journal of Experimental Psychology: General, 143(2), 534-547

(A) Vosgerau, Simonsohn, Nelson, & Simmons (2019) 99% impossible: A valid, or falsifiable, internal meta-analysis,Journal of Experimental Psychology: General, 148(9), 1628

Simonsohn (2015) Data Colada [41] - Falsely Reassuring: Analyses of ALL p-values

McShane, Böckenholt, & Hansen (2016) Adjusting for Publication Bias in Meta-Analysis: An Evaluation of Selection Methods and Some Cautionary Notes,Perspectives on Psychological Science, 11(5), 730-749

DataColada[24] p-curve vs excessive significance testing http://datacolada.org/24

DataColada[30] Trim-and-Fill is Fully of It (Bias) http://datacolada.org/30

DataColada[58] The Funnel Plot is Invalid Because of This Crazy Assumption: r(n,d)=0


?
Lane & Dunlap (1978) Estimating effect size: Bias resulting from the significance criterion in editorial decisions,British Journal of Mathematical and Statistical Psychology, 31(107-112
Published papers over-estimate effect size. Many papers have made this same point later on, without properly citing it.

J. P. A. Ioannidis (2005) Why most published research findings are false,Plos Medicine, 2(8), 696-701
Classic paper. Bayesian calibration for p(H|D)

Pashler & Harris (2012) Is the Replicability Crisis Overblown? Three Arguments Examined,Perspectives on Psychological Science, 7(6), 531-536

Francis (2013) Replication, statistical consistency, and publication bias,Journal of Mathematical Psychology,
It's surprising for all studies to be p<.05 (if we ignore that p>.05 are not published).

Simonsohn (2013) It Really Just Does Not Follow, Coments on Francis (2013),Journal of Mathematical Psychology,
Publication bias does not imply we should ignore published evidence.

Stanley & Doucouliagos (2014) Meta?regression approximations to reduce publication selection bias,Research Synthesis Methods, 5(1), 60-78

David Card & Krueger (1995) Time-series minimum-wage studies: a meta-analysis,The American Economic Review, 85(2), 238-243
Seems like minimum-wage mattering is publication bias at work.

Leamer (1983) Let's take the con out of econometrics,The American Economic Review, 31-43
Selective reporting in economics, and an extreme solution

Sala-i-Martin (1997) I just ran two million regressions,The American Economic Review, 178-183

Duval & Tweedie (2000) Trim and Fill: A Simple Funnel?Plot-Based Method of Testing and Adjusting for Publication Bias in Meta?Analysis,Biometrics, 56(2), 455-463

Rothstein, Sutton, & Borenstein (2005) Publication Bias in Meta?Analysis

Hedges & Vevea (1996) Estimating effect size under publication bias: small sample properties and robustness of a random effects selection model,Journal of Educational and Behavioral Statistics, 21(4), 299-332

Sharpe (1997) Of apples and oranges, file drawers and garbage: Why validity issues in meta-analysis will not go away,Clinical psychology review, 17(8), 881-901

Borenstein (2009) Criticisms of Meta-Analysis,Introduction to Meta-Analysis,

Wallis (1942) Compounding probabilities from independent significance tests,Econometrica, 229-248

J. Ioannidis & Trikalinos (2007) An exploratory test for an excess of significant findings,Clinical Trials, 4(3), 245

Simonsohn (2018) Two lines: a valid alternative to the invalid testing of U-shaped relationships with quadratic regressions,Advances in Methods and Practices in Psychological Science, 1(4), 538-555

Gelman & Park (2008) Splitting a predictor at the upper quarter or third and the lower quarter or third,The American Statistician, 62(4), 1-8

McClelland, Lynch, Irwin, Spiller, & Fitzsimons (2015) Median splits, Type II errors, and false-positive consumer psychology: Don't fight the power,Journal of Consumer Psychology, 25(4), 679-689

Rucker, McShane, & Preacher (2015) A researcher's guide to regression, discretization, and median splits of continuous variables,Journal of Consumer Psychology, 25(4), 666-678

Iacobucci, Posavac, Kardes, Schneider, & Popovich (2015) Toward a More Nuanced Understanding of the Statistical Properties of a Median Split,Journal of Consumer Psychology, 25(4), 652-665


Hainmueller, Mummolo, & Xu (2019) How much should we trust estimates from multiplicative interaction models? Simple tools to improve empirical practice,Political Analysis, 27(2), 163-192

Ganzach (1997) Misleading interaction and curvilinear terms,Psychological methods, 2(3), 235


Ai & Norton (2003) Interaction terms in logit and probit models,Economics letters, 80(1), 123-129

Karaca?Mandic, Norton, & Dowd (2012) Interaction Terms in Nonlinear Models,Health Services Research, 47(1pt1), 255-274

Greene (2010) Testing hypotheses about interaction terms in nonlinear models,Economics Letters, 107(2), 291-296

Bhargava, Kassam, & Loewenstein (2014) A reassessment of the defense of parenthood,Psychological science, 25(1), 299-302

D. Card & Dahl (2011) Family Violence and Football: The Effect of Unexpected Emotional Cues on Violent Behavior*,Quarterly Journal of Economics, 126(1), 103

Spiller, Fitzsimons, Lynch Jr, & McClelland (2013) Spotlights, floodlights, and the magic number zero: Simple effects tests in moderated regression,Journal of Marketing Research, 50(2), 277-288

Westfall & Yarkoni (2016) Statistically Controlling for Confounding Constructs Is Harder than You Think,PLOS ONE, 11(3), e0152719

Preacher, Curran, & Bauer (2006) Computational tools for probing interactions in multiple linear regression, multilevel modeling, and latent curve analysis,Journal of educational and behavioral statistics, 31(4), 437-448

Simonsohn (2022) Interactiongate: Linearly Testing and Probing Interactions in the Real (Nonlinear) World is Scandalously Invalid


Lee & Lemieux (2010) Regression Discontinuity Designs in Economics,Journal of Economic Literature, 48(281-355

Gelman & Stern (2006) The difference between "significant? and "not significant? is not itself statistically significant,The American Statistician, 60(4), 328-331

Gelman & Imbens (2014) Why high-order polynomials should not be used in regression discontinuity designs


A. J. Healy, Malhotra, Mo, & Laitin (2010) Irrelevant events affect voters' evaluations of government performance,Proceedings of the National Academy of Sciences of the United States of America, 107(29), 12804-12809

Fowler & Montagnes (2015) College football, elections, and false-positive results in observational research,Proceedings of the National Academy of Sciences, 112(45), 13800

A. Healy, Malhotra, & Mo (2015) Determining false-positives requires considering the totality of evidence,Proceedings of the National Academy of Sciences, 112(48), E6591


Dienes (2019) How do I know what my theory predicts? ,Advances in Methods and Practices in Psychological Science, 2(4), 364-377

Dienes (2011) Bayesian versus orthodox statistics: Which side are you on? ,Perspectives on Psychological Science, 6(3), 274-290

Rouder, Speckman, Sun, Morey, & Iverson (2009) Bayesian t tests for accepting and rejecting the null hypothesis,Psychonomic Bulletin & Review, 16(2), 225-237

Wagenmakers (2007) A practical solution to the pervasive problems ofp values,Psychonomic bulletin & review, 14(5), 779-804

Simonsohn (2014) Data Colada [13] - Posterior-Hacking

Simonsohn (2015) DataColada[35]: The Default Bayesian Test is Prejudiced Against Small Effects

Rouanet (1996) Bayesian methods for assessing importance of effects,Psychological bulletin, 119(1), 149-158

Sanborn & Hills (2014) The frequentist implications of optional stopping on Bayesian hypothesis tests,Psychonomic bulletin & review, 21(2), 283-300

Rouder (2014) Optional stopping: No problem for Bayesians,Psychonomic bulletin & review, 21(2), 301-308

Sanborn et al. (2014) Reply to Rouder (2014): Good frequentist properties raise confidence,Psychonomic bulletin & review, 21(2), 309-311

?

Timetable and sections