Lectures

Please find below the schedule of our lectures. Each lecture includes a brief description, required readings, and suggested readings for further exploration. The lecture slides and additional resources will be posted on the course website and GitHub repository. Please remember that the schedule may change during the course. If you have any questions or need further assistance, please let me know!

Week 01: Introduction and Course Overview

Lecture 01: Introduction to Experimental Design

Required readings:

• Syllabus and course website: http://danilofreire.github.io/qtm385.
• GitHub repository: https://github.com/danilofreire/qtm385.
• Lecture slides: Welcome to QTM 385: Introduction / Why Experiments?
• FEDAI: Chapter 01: Introduction.

Suggested readings:

• Humphreys, M. (2021). I saw your RCT and I have some worries! FAQs. An informal collection of questions and answers about RCTs.
• Mize, T. D., & Manago, B. (2022). The Past, Present, and Future of Experimental Methods in the Social Sciences. Social Science Research, 108, 1-24.
• Jackson, M., & Cox, D. R. (2013). The Principles of Experimental Design and Their Application in Sociology. Annual Review of Sociology, 39(1), 27-49.
• Hernán, M. A. (2018). The C-Word: Scientific Euphemisms Do Not Improve Causal Inference from Observational Data. American Journal of Public Health, 108(5), 616-619.
• Bothwell, L. E., Greene, J. A., Podolsky, S. H., & Jones, D. S. (2016). Assessing the Gold Standard — Lessons from the History of RCTs. New England Journal of Medicine, 374(22), 2175-2181.
• Deaton, A., & Cartwright, N. (2018). Understanding and Misunderstanding Randomized Controlled Trials. Social Science & Medicine 210, 2-21.

Week 02: The Research Design Process

Lecture 02: Theories and Experiments

Required readings:

• Lecture slides: The Research Design Process: Testing Theories with Experiments.
• Card, D., DellaVigna, S., & Malmendier, U. (2011). The Role of Theory in Field Experiments. Journal of Economic Perspectives, 25(3), 39-62.
• Blair, G., Cooper, J., Coppock, A., & Humphreys, M. (2019). Declaring and Diagnosing Research Designs. American Political Science Review, 113(3), 838-859.
• Problem Set 01 assigned.

Week 03: Potential Outcomes Framework

Lecture 03: Causal Inference and Potential Outcomes

Required readings:

• Lecture slides: Potential Outcomes Framework.
• FEDAI: Chapter 02: Causal Inference and Experimentation.
• Rubin, D. B. (2005). Causal Inference using Potential Outcomes: Design, Modeling, Decisions. Journal of the American Statistical Association, 100(469), 322-331.
• Kahoot Quiz

Lecture 04: Selection Bias and Randomisation

Required readings:

• Lecture slides: Potential Outcomes Continued and Regression Analysis.
• Angrist, J. D., & Pischke, J. S. (2014). Mastering ’Metrics: The Path from Cause to Effect. Princeton University Press. Chapter 01: Randomized Trials. The appendix is also worth reading.
• Kahoot Quiz
• Problem Set 01 due.
• Problem Set 02 assigned.

Weekly suggested readings:

• Imbens, G. W., & Rubin, D. B. (2015). Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction. Cambridge University Press. Part I - Introduction.
• Hernán, M. A., Hernández-Díaz, S., & Robins, J. M. (2004). A Structural Approach to Selection Bias. Epidemiology, 15(5), 615-625.
• Hernán M.A. & Robins J.M. (2020). Causal Inference: What If. Boca Raton: Chapman & Hall/CRC. Chapters 6-10. Maths-heavy but very insightful.
• Morgan, S. L., & Winship, C. (2015). Counterfactuals and Causal Inference: Methods and Principles for Social Research. Cambridge University Press. Chapter 02: Counterfactuals and Potential Outcomes Model.

Week 04: Sampling Distribution and Randomisation Inference

Lecture 05: Statistical Inference for Randomised Experiments

• Lecture slides: Sampling Distributions, Statistical Inference, and Hypothesis Testing.
• FEDAI: Chapter 03: Sampling Distributions, Statistical Inference, and Hypothesis Testing (up to section 3.5).
• EGAP Methods Guides: Hypothesis Testing, Randomisation Inference, and Cluster Randomisation. These short guides provide a good overview of the topics we will cover in class, as well as example code in R.
• Kahoot Quiz.

Lecture 06: Texts for discussion

• Lecture slides: Texts for Discussion.
• Kalla, J. & D. Broockman. 2016. Campaign Contributions Facilitate Access to Congressional Officials: A Randomized Field Experiment. American Journal of Political Science, 60(3): 545–558
• Bertrand, M., & Mullainathan, S. (2004). Are Emily and Greg More Employable than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination. American Economic Review, 94(4), 991-1013.
• Chattopadhyay, R., & Duflo, E. (2004). Women as Policy Makers: Evidence from a Randomized Policy Experiment in India. Econometrica, 72(5), 1409-1443. (if time allows)
• Problem Set 02 due.
• Problem Set 03 assigned.

Weekly suggested readings:

• Imbens, G. W., & Rubin, D. B. (2015). Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction. Cambridge University Press. Part II - Randomised Experiments.
• Athey, S., & Imbens, G. W. (2017). The Econometrics of Randomized Experiments. In Handbook of economic field experiments (Vol. 1, pp. 73-140). North-Holland. Read section 4, although the entire chapter is worth reading.
• Berry, C. R., & Fowler, A. (2021). Leadership or Luck? Randomization Inference for Leader Effects in Politics, Business, and Sports. Science Advances, 7(4), eabe3404.
• Caughey, D., Dafoe, A., Li, X., & Miratrix, L. (2023). Randomisation Inference Beyond the Sharp Null: Bounded Null Hypotheses and Quantiles of Individual Treatment Effects. Journal of the Royal Statistical Society Series B: Statistical Methodology, 85(5), 1471-1491.
• Ritzwoller, D. M., Romano, J. P., & Shaikh, A. M. (2025). Randomization Inference: Theory and Applications. arXiv preprint arXiv:2406.09521. Very technical, recommended for maths or statistics enthusiasts.

Week 05: Blocking, Covariate Adjustment, and Statistical Power

Lecture 07: Blocking, Clustering, and Covariate Adjustment

• Lecture slides: Blocking and Clustering.
• FEDAI: Section 3.6: Sampling Distributions for Experiments That Use Block or Cluster Random Assignment.
• FEDAI: Chapter 04: Using Covariates in Experimental Design and Analysis.

Lecture 08: Blocking and Clustering (Cont.), Statistical Power, and Sample Calculations (February 12)

• Lecture slides: Blocking and Clustering (Cont.), Statistical Power, and Sample Calculations.
• FEDAI Appendix 3.1 on Power
• EGAP Methods Guide on Power Calculations.
• Problem Set 03 due.
• Problem Set 04 assigned.

Weekly suggested readings:

• Blair, G., Coppock, A., & Humphreys, M. (2023). Research Design in the Social Sciences. Princeton University Press. Sections 18.2 and 18.3.
• Bowers, J. 10 Things to Know About Cluster Randomization.
• Kaltenbach, H-M. (2021). Statistical Design and Analysis of Biological Experiments. Chapter 7: Improving Precision and Power: Blocked Designs.
• Imai, K., King, G., & Nall, C. (2009). The Essential Role of Pair Matching in Cluster-Randomized Experiments, with Application to the Mexican Universal Health Insurance Evaluation. Statistical Science, 29-53.
• R Psychology. (2023). Understanding Statistical Power and Significance Testing. A comprehensive guide to power analysis in R.
• EGAP: Power Calculator. An interactive tool to calculate power for your experiments.
• Pek, J., Hoisington-Shaw, K. J., & Wegener, D. T. (2024). Uses of Uncertain Statistical Power: Designing Future Studies, Not Evaluating Completed Studies. Psychological Methods.
• Li, X., & Ding, P. (2020). Rerandomization and Regression Adjustment. Journal of the Royal Statistical Society Series B: Statistical Methodology, 82(1), 241-268.

Week 06: Non-Compliance and Attrition

Lecture 09: Non-Compliance I

• Lecture slides: Non-Compliance I.
• FEDAI: Chapter 05: One-Sided Non-Compliance.
• Kahoot Quiz

Lecture 10: Non-Compliance II

• Lecture slides: Non-Compliance II.
• FEDAI: Chapter 06: Two-Sided Non-Compliance.
• Problem Set 04 due.
• Problem Set 05 assigned.
• KaHoot Quiz
• Two paragraphs (maximum) summarising an experiment that you wish to develop in this course. At a minimum, your summary should include a research question, why the question is important, and a rough outline of how you plan to answer the question. Please send me an email by Wednesday with this information.

Weekly suggested readings:

• Angrist, J. D., & Pischke, J. S. (2009). Mostly Harmless Econometrics: An Empiricist’s Companion. Princeton University Press. Chapter 4: Instrumental Variables Estimation.
• Baryakova, T.H., Pogostin, B.H., Langer, R. et al. Overcoming Barriers to Patient Adherence: The Case for Developing Innovative Drug Delivery Systems. Nature Reviews Drug Discovery 22, 387–409 (2023).
• Kane, J. V. (2025). More than Meets the ITT: A Guide for Anticipating and Investigating Nonsignificant Results in Survey Experiments. Journal of Experimental Political Science, 12(1), 110–125.
• Hartman, E., & Huang, M. (2024). Improving Precision through Design and Analysis in Experiments with Noncompliance. Political Science Research and Methods, 12(3), 557-572.
• Gerber, A. S., Green, D. P., Kaplan, E. H., & Kern, H. L. (2010). Baseline, Placebo, and Treatment: Efficient Estimation for Three-Group Experiments. Political Analysis, 18(3), 297-315.

Week 07: Attrition in Experimental Research

Lecture 11: Attrition and Missing Outcomes

• Lecture slides: Attrition.
• FEDAI: Chapter 07: Attrition.
• Lo, A., Renshon, J., & Bassan-Nygate, L. (2024). A Practical Guide to Dealing with Attrition in Political Science Experiments. Journal of Experimental Political Science, 11(2), 147-161.
• Kahoot quiz

Lecture 12: Ethical Considerations in Experimental Research

• Lecture slides: Ethical Considerations in Experimental Research.
• Problem Set 06 assigned

Required readings:

• Humphreys, M. (2015). Reflections on the Ethics of Social Experimentation. Journal of Globalization and Development, 6(1), 87-112.
• Cronin-Furman, K., & Lake, M. (2018). Ethics Abroad: Fieldwork in Fragile and Violent Contexts. PS: Political Science & Politics, 51(3), 607-614.
• Poverty Action Lab (2022). Ethical Conduct of Randomized Evaluations. J-PAL.
• Problem Set 05 due.

Weekly suggested readings:

• Zhou, H., & Fishbach, A. (2016). The Pitfall of Experimenting on the Web: How Unattended Selective Attrition Leads to Surprising (Yet False) Research Conclusions. Journal of Personality and Social Psychology, 111(4), 493.
• Coppock, A., Gerber, A. S., Green, D. P., & Kern, H. L. (2017). Combining Double Sampling and Bounds to Address Nonignorable Missing Outcomes in Randomized Experiments. Political Analysis, 25(2), 188-206.
• Ghanem, D., Hirshleifer, S., & Ortiz-Becerra, K. (2019). Testing Attrition Bias in Field Experiments. UC Berkeley: Center for Effective Global Action.
• Pallmann, P., Bedding, A.W., Choodari-Oskooei, B. et al. (2018). Adaptive Designs in Clinical Trials: Why Use Them, and How to Run and Report Them. BMC Med 16(29).
• Asiedu, E., Dean, E., Karlan, D., & Osei, R. (2021). Ethics and Society Review: Ethics Reflection as a Precondition to Research Funding. Proceedings of the National Academy of Sciences, 118(52), e2117261118.
• ASAB Ethical Committee/ABS Animal Care Committee. (2024). Guidelines for the Ethical Treatment of Nonhuman Animals in Behavioural Research and Teaching. Animal Behaviour, 207, I-XI.

Week 08: Quarto and Pre-Analysis Plans

Lecture 13: Introduction to Quarto and DeclareDesign

• Lecture slides: Introduction to Quarto and DeclareDesign.
• Quarto official website: https://quarto.org/.
• DeclareDesign official website: https://declaredesign.org/.

Lecture 14: Writing Pre-Analysis Plans

• Lecture slides: How to Write a Pre-Analysis Plan.
• EGAP: Pre-Analysis Plans.
• Olken, B. A. (2015). Promises and perils of pre-analysis plans. Journal of Economic Perspectives, 29(3), 61-80.
• Problem Set 06 due.
• Kahoot Quiz

Weekly suggested readings:

• Quarto official website.
• Awesome Quarto: https://github.com/mcanouil/awesome-quarto. Note: this repository contains dozens of tutorials, examples, and resources.
• Çetinkaya-Rundel, M. & Lowndes, J. S. (2022) Keynote talk: Hello Quarto: Share • Collaborate • Teach • Reimagine. Slides and source code. This is one of the nicest Quarto presentations I have seen.
• Getting Started with Quarto (YouTube). Note: Posit (formerly RStudio) has a series of tutorials on Quarto on their YouTube channel. You can find their playlist here.
• Blair, G., Coppock, A., & Humphreys, M. (2023). Introduction to design declaration with DeclareDesign. A series of presentation slides on DeclareDesign by the authors of the book. Strongly recommended.
• DeclareDesign official website.
• Nosek, B.A., Alter, G., Banks, G.C., Borsboom, D., Bowman, S.D., Breckler, S.J., Buck, S., Chambers, C.D., Chin, G., Christensen, G. and Contestabile, M., 2015. Promoting an open research culture. Science, 348(6242), pp.1422-1425.
• A Gentle Introduction to DeclareDesign (YouTube). Video tutorial by the authors of the book.

Week 09: Interference

Lecture 15: When Experiments are Not Possible

• Lecture slides: Natural and Quasi-Experiments.
• Kahoot Quiz.
• Dunning, T. (2012). Natural Experiments in the Social Sciences: A Design-Based Approach. Cambridge University Press. Chapter 01: Introduction.

Lecture 16: Interference in Experiments (March 19)

• Lecture slides: Interference in Experiments.
• FEDAI: Chapter 08: Interference between Experimental Units.
• EGAP: Spillovers.
• Kahoot Quiz.
• Problem Set 07 assigned.

Weekly suggested readings:

• Diener, E., Northcott, R., Zyphur, M. J., & West, S. G. (2022). Beyond Experiments. Perspectives on Psychological Science, 17(4), 1101-1119.
• Dunning, T. (2016). Transparency, Replication, and Cumulative Learning: What Experiments Alone Cannot Achieve. Annual Review of Political Science, 19(1), 541-563.
• Petticrew, M., Cummins, S., Ferrell, C., Findlay, A., Higgins, C., Hoy, C., … & Sparks, L. (2005). Natural Experiments: An Underused Tool for Public Health?. Public Health, 119(9), 751-757.
• Grosz, M. P., Ayaita, A., Arslan, R. C., Buecker, S., Ebert, T., Hünermund, P., … & Rohrer, J. M. (2024). Natural Experiments: Missed Opportunities for Causal Inference in Psychology. Advances in Methods and Practices in Psychological Science, 7(1), 25152459231218610.
• Aronow, P. M., Eckles, D., Samii, C., & Zonszein, S. (2020). Spillover Effects in Experimental Data. arXiv preprint arXiv:2001.05444. Quite technical too.
• Imbens, G. (2024). Interference and Spillovers in Randomized Experiments (Video). A short lecture on recent advances and new designs to deal with interference in experiments. A little technical.

Week 10: Heterogeneous Treatment Effects

Lecture 17: Texts for Discussion

• Lecture slides: Texts for Discussion.
• Kahoot Quiz.
• Centola, D. (2010). The spread of behavior in an online social network experiment. Science, 329(5996), 1194-1197.
• Paluck, B. L., Shepherd, H., and Aronow, P. 2016. Changing climates of conflict: A social network experiment in 56 schools. Proceedings of the National Academy of Sciences, 113(3): 566-571
• Gerber, A. S., & Green, D. P. (2000). The effects of canvassing, telephone calls, and direct mail on voter turnout: A field experiment. American Political Science Review, 94(3), 653-663.

Lecture 18: Heterogeneous Treatment Effects

• Lecture slides: Heterogeneous Treatment Effects.
• FEDAI: Chapter 09: Heterogeneous Treatment Effects.
• Problem Set 07 due.
• Problem Set 08 assigned.
• Pre-Analysis Plan due.

Weekly suggested readings:

• Holzmeister, F., Johannesson, M., Böhm, R., Dreber, A., Huber, J., & Kirchler, M. (2024). Heterogeneity in Effect Size Estimates. Proceedings of the National Academy of Sciences, 121(32), e2403490121.
• Ding, P., Feller, A., & Miratrix, L. (2016). Randomization Inference for Treatment Effect Variation. Journal of the Royal Statistical Society Series B: Statistical Methodology, 78(3), 655-671.
• Wager, S., & Athey, S. (2018). Estimation and Inference of Heterogeneous Treatment Effects using Random Forests. Journal of the American Statistical Association, 113(523), 1228-1242.
• Susan Athey and Stefan Wager: Estimating Heterogeneous Treatment Effects in R. A one-hour video on how to use the grf package in R to estimate heterogeneous treatment effects.
• Generalised Random Forests (R package).
• Coppock, A. (2015). 10 Things to Know about Multiple Comparisons. EGAP Methods Guide.
• Frandsen, B. (2025). Machine Learning and Heterogeneous Treatment Effects. Slides and coding labs from a workshop on this topic. Available on GitHub.

Week 11: Heterogeneous Treatment Effects (Continued) and Mediation

Lecture 19: Texts for Discussion

• Lecture slides: Texts for Discussion.
• Munshi, K. (2003). Networks in the modern economy: Mexican migrants in the US labor market. The Quarterly Journal of Economics, 118(2), 549-599.
• Miguel, E., & Kremer, M. (2004). Worms: identifying impacts on education and health in the presence of treatment externalities. Econometrica, 72(1), 159-217.

Lecture 20: Mediation Analysis

• Lecture slides: Mediation Analysis.
• FEDAI: Chapter 10: Mediation.
• Problem Set 08 due.
• Problem Set 09 assigned.
• KaHoot Quiz

Weekly suggested readings:

• Bullock, J. G., & Green, D. P. (2021). The Failings of Conventional Mediation Analysis and a Design-Based Alternative. Advances in Methods and Practices in Psychological Science, 4(4), 25152459211047227.
• VanderWeele, T. J. (2016). Mediation Analysis: A Practitioner’s Guide. Annual Review of Public Health, 37(1), 17-32.
• Acharya, A., Blackwell, M., & Sen, M. (2018). Analyzing Causal Mechanisms in Survey Experiments. Political Analysis, 26(4), 357–378.
• MacKinnon, D. P., Fairchild, A. J., & Fritz, M. S. (2007). Mediation Analysis. Annual Review of Psychology, 58(1), 593-614.

Week 12: Survey Experiments

Lecture 21: Survey Experiments

• Lecture slides: Survey Experiments.
• EGAP: Survey Experiments.
• Stantcheva, S. (2023). How to run surveys: A guide to creating your own identifying variation and revealing the invisible. Annual Review of Economics, 15(1), 205-234.

Lecture 22: Survey Methods for Sensitive Topics

• Lecture slides: Survey Methods for Sensitive Topics.
• Problem Set 09 due.
• Problem Set 10 assigned.

Weekly suggested readings:

• Sniderman, P. M. (2018). Some Advances in the Design of Survey Experiments. Annual Review of Political Science, 21(1), 259-275.
• Schneider, D., & Harknett, K. (2022). What’s to Like? Facebook as a Tool for Survey Data Collection. Sociological Methods & Research, 51(1), 108-140.
• Clifford, S., Sheagley, G., & Piston, S. (2021). Increasing Precision Without Altering Treatment Effects: Repeated Measures Designs in Survey Experiments. American Political Science Review, 115(3), 1048-1065.
• Kane, J. V. (2024). More Than Meets the ITT: A Guide for Anticipating and Investigating Nonsignificant Results in Survey Experiments. Journal of Experimental Political Science, 1-16.
• Blair, G., Imai, K., & Zhou, Y. Y. (2015). Design and Analysis of the Randomized Response Technique. Journal of the American Statistical Association, 110(511), 1304-1319.
• Blair, G., & Imai, K. (2012). Statistical Analysis of List Experiments. Political Analysis, 20(1), 47-77.
• Riambau, G., & Ostwald, K. (2021). Placebo Statements in List Experiments: Evidence from a Face-to-Face Survey in Singapore. Political Science Research and Methods, 9(1), 172-179.
• Gosciak, J., Molitor, D., & Lundberg, I. (2025). Adaptive Randomization in Conjoint Survey Experiments. Retrieved from <osf.io/preprints/socarxiv/69y2j_v1>.

Week 13: Meta-Analysis and Integration of Research Findings

Lecture 23: Texts for Discussion (Related to Survey Experiments)

• Lecture slides: Texts for Discussion.
• Druckman, J. N., Gilli, M., Klar, S., & Robison, J. (2015). Measuring drug and alcohol use among college student‐athletes. Social Science Quarterly, 96(2), 369-380.
• Blair, G., Imai, K., & Lyall, J. (2014). Comparing and Combining List and Endorsement Experiments: Evidence from Afghanistan. American Journal of Political Science, Vol. 58, No. 4 (October), pp. 1043-1063.
• Rosenfeld, B, Imai, K., & Shapiro, J. (2016). An Empirical Validation Study of Popular Survey Methodologies for Sensitive Questions. American Journal of Political Science, 60(3), 783-802.
• Freire, D., & Skarbek, D. (2023). Vigilantism and Institutions: Understanding Attitudes Toward Lynching in Brazil. Research & Politics, 10(1), 1-8.

Lecture 24: Meta-Analysis and Integration of Research Findings

• Lecture slides: Meta-Analysis and Integration of Research Findings.
• FEDAI: Chapter 11: Integration of Research Findings.
• Problem Set 10 due.

Weekly suggested readings:

• Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2011). Introduction to meta-analysis. John Wiley & Sons. A whole book dedicated to the topic. A bit technical but very good.
• Harrer, M., Cuijpers, P., Furukawa, T., & Ebert, D. (2021). Doing Meta-Analysis with R: A Hands-On Guide. Chapman and Hall/CRC.
• Hansen, C., Steinmetz, H., & Block, J. (2022). How to Conduct a Meta-Analysis in Eight Steps: A Practical Guide. Management Review Quarterly, 1-19.
• Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., … & Prisma-P Group. (2015). Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P) 2015 Statement. Systematic reviews, 4, 1-9.

Week 14: Course Revision and Presentations

Lecture 25: Course Revision

• Lecture slides: Course Revision.

Presentations

• Presentations of final projects.

Week 15: Presentations

Presentations (Continued)

• Presentations of final projects.