Lectures
Please find below the schedule of our lectures. Each lecture includes a brief description, required readings, and suggested readings for further exploration. The lecture slides and additional resources will be posted on the course website and GitHub repository. Please remember that the schedule may change during the course. If you have any questions or need further assistance, please let me know!
Lecture 01: Introduction to Experimental Design (August 27, 2025)
Required readings:
- Syllabus and course website: http://danilofreire.github.io/qtm385.
- GitHub repository: https://github.com/danilofreire/qtm385.
- Lecture slides: Welcome to QTM 385: Introduction / Why Experiments?
- FEDAI: Chapter 01: Introduction.
Suggested readings:
- Humphreys, M. (2021). I saw your RCT and I have some worries! FAQs. An informal collection of questions and answers about RCTs.
- Mize, T. D., & Manago, B. (2022). The Past, Present, and Future of Experimental Methods in the Social Sciences. Social Science Research, 108, 1-24.
- Jackson, M., & Cox, D. R. (2013). The Principles of Experimental Design and Their Application in Sociology. Annual Review of Sociology, 39(1), 27-49.
- Hernán, M. A. (2018). The C-Word: Scientific Euphemisms Do Not Improve Causal Inference from Observational Data. American Journal of Public Health, 108(5), 616-619.
- Bothwell, L. E., Greene, J. A., Podolsky, S. H., & Jones, D. S. (2016). Assessing the Gold Standard — Lessons from the History of RCTs. New England Journal of Medicine, 374(22), 2175-2181
- de Kadt, D. & Grzysmala-Busse, A. (2025). Good Description. Working paper..
- Deaton, A., & Cartwright, N. (2018). Understanding and Misunderstanding Randomized Controlled Trials. Social Science & Medicine 210, 2-21.
Lecture 02: Theories and Experiments (September 3, 2025)
Required readings:
- Lecture slides: The Research Design Process: Testing Theories with Experiments.
- Card, D., DellaVigna, S., & Malmendier, U. (2011). The Role of Theory in Field Experiments. Journal of Economic Perspectives, 25(3), 39-62.
- Blair, G., Cooper, J., Coppock, A., & Humphreys, M. (2019). Declaring and Diagnosing Research Designs. American Political Science Review, 113(3), 838-859.
- Problem Set 01 assigned.
Suggested readings:
- Brutger, R., Kertzer, J. D., Renshon, J., Tingley, D., & Weiss, C. M. (2023). Abstraction and Detail in Experimental Design. American Journal of Political Science, 67(4), 979-995.
- Peterson, J. C., Bourgin, D. D., Agrawal, M., Reichman, D., & Griffiths, T. L. (2021). Using Large-Scale Experiments and Machine Learning to Discover Theories of Human Decision-Making. Science, 372(6547), 1209-1214.
- Smith, V. L. (2010). Theory and Experiment: What Are the Questions?. Journal of Economic Behavior & Organization, 73(1), 3-15.
- van Rooij, I., & Baggio, G. (2021). Theory Before the Test: How to Build High-Verisimilitude Explanatory Theories in Psychological Science. Perspectives on Psychological Science, 16(4), 682-697.
Lecture 03: Causal Inference and Potential Outcomes (September 8, 2025)
Required readings:
- Lecture slides: Potential Outcomes Framework.
- FEDAI: Chapter 02: Causal Inference and Experimentation.
- Rubin, D. B. (2005). Causal Inference using Potential Outcomes: Design, Modeling, Decisions. Journal of the American Statistical Association, 100(469), 322-331.
- Kahoot Quiz
Lecture 04: Selection Bias and Randomisation (September 10, 2025)
Required readings:
- Lecture slides: Potential Outcomes Continued and Regression Analysis.
- Angrist, J. D., & Pischke, J. S. (2014). Mastering ’Metrics: The Path from Cause to Effect. Princeton University Press. Chapter 01: Randomized Trials. The appendix is also worth reading.
- Kahoot Quiz
- Problem Set 01 due.
- Problem Set 02 assigned.
Suggested readings:
- Imbens, G. W., & Rubin, D. B. (2015). Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction. Cambridge University Press. Part I - Introduction.
- Hernán, M. A., Hernández-Díaz, S., & Robins, J. M. (2004). A Structural Approach to Selection Bias. Epidemiology, 15(5), 615-625.
- Hernán M.A. & Robins J.M. (2020). Causal Inference: What If. Boca Raton: Chapman & Hall/CRC. Chapters 6-10. Maths-heavy but very insightful.
- Morgan, S. L., & Winship, C. (2015). Counterfactuals and Causal Inference: Methods and Principles for Social Research. Cambridge University Press. Chapter 02: Counterfactuals and Potential Outcomes Model.
Lecture 05: Statistical Inference for Randomised Experiments (September 15, 2025)
- Lecture slides: Sampling Distributions, Statistical Inference, and Hypothesis Testing.
- FEDAI: Chapter 03: Sampling Distributions, Statistical Inference, and Hypothesis Testing (up to section 3.5).
- EGAP Methods Guides: Hypothesis Testing, Randomisation Inference, and Cluster Randomisation. These short guides provide a good overview of the topics we will cover in class, as well as example code in
R
. - Kahoot Quiz.
Suggested readings:
- Imbens, G. W., & Rubin, D. B. (2015). Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction. Cambridge University Press. Part II - Randomised Experiments.
- Athey, S., & Imbens, G. W. (2017). The Econometrics of Randomized Experiments. In Handbook of economic field experiments (Vol. 1, pp. 73-140). North-Holland. Read section 4, although the entire chapter is worth reading.
- Berry, C. R., & Fowler, A. (2021). Leadership or Luck? Randomization Inference for Leader Effects in Politics, Business, and Sports. Science Advances, 7(4), eabe3404.
- Caughey, D., Dafoe, A., Li, X., & Miratrix, L. (2023). Randomisation Inference Beyond the Sharp Null: Bounded Null Hypotheses and Quantiles of Individual Treatment Effects. Journal of the Royal Statistical Society Series B: Statistical Methodology, 85(5), 1471-1491.
- Ritzwoller, D. M., Romano, J. P., & Shaikh, A. M. (2025). Randomization Inference: Theory and Applications. arXiv preprint arXiv:2406.09521. Very technical, recommended for maths or statistics enthusiasts.
Lecture 06: Texts for discussion (September 17, 2025)
- Lecture slides: Texts for Discussion.
- Kalla, J. & D. Broockman. 2016. Campaign Contributions Facilitate Access to Congressional Officials: A Randomized Field Experiment. American Journal of Political Science, 60(3): 545–558
- Bertrand, M., & Mullainathan, S. (2004). Are Emily and Greg More Employable than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination. American Economic Review, 94(4), 991-1013.
- Chattopadhyay, R., & Duflo, E. (2004). Women as Policy Makers: Evidence from a Randomized Policy Experiment in India. Econometrica, 72(5), 1409-1443. (if time allows)
- Problem Set 02 due.
- Problem Set 03 assigned.
Lecture 07: Blocking, Clustering, and Covariate Adjustment (September 22, 2025)
- Lecture slides: Blocking and Clustering.
- FEDAI: Section 3.6: Sampling Distributions for Experiments That Use Block or Cluster Random Assignment.
- FEDAI: Chapter 04: Using Covariates in Experimental Design and Analysis.
Lecture 08: Blocking and Clustering (Cont.), Statistical Power, and Sample Calculations (September 24, 2025)
- Lecture slides: Blocking and Clustering (Cont.), Statistical Power, and Sample Calculations.
- FEDAI Appendix 3.1 on Power
- EGAP Methods Guide on Power Calculations.
- Problem Set 03 due.
- Problem Set 04 assigned.
Suggested readings:
- Blair, G., Coppock, A., & Humphreys, M. (2023). Research Design in the Social Sciences. Princeton University Press. Sections 18.2 and 18.3.
- Rainey, C. (2025). Power Rules: Practical Advice for Computing Power (and Automating with Pilot Data). OSF Preprints. Note: This is a useful guide for power calculations without the need for complex maths. Not peer-reviewed yet.
- Bowers, J. 10 Things to Know About Cluster Randomization.
- Kaltenbach, H-M. (2021). Statistical Design and Analysis of Biological Experiments. Chapter 7: Improving Precision and Power: Blocked Designs.
- Imai, K., King, G., & Nall, C. (2009). The Essential Role of Pair Matching in Cluster-Randomized Experiments, with Application to the Mexican Universal Health Insurance Evaluation. Statistical Science, 29-53.
- R Psychology. (2023). Understanding Statistical Power and Significance Testing. A comprehensive guide to power analysis in
R
. - EGAP: Power Calculator. An interactive tool to calculate power for your experiments.
- Pek, J., Hoisington-Shaw, K. J., & Wegener, D. T. (2024). Uses of Uncertain Statistical Power: Designing Future Studies, Not Evaluating Completed Studies. Psychological Methods.
- Li, X., & Ding, P. (2020). Rerandomization and Regression Adjustment. Journal of the Royal Statistical Society Series B: Statistical Methodology, 82(1), 241-268.
Lecture 09: Non-Compliance I (September 29, 2025)
- Lecture slides: Non-Compliance I.
- FEDAI: Chapter 05: One-Sided Non-Compliance.
- Kahoot Quiz
Lecture 10: Non-Compliance II (October 1, 2025)
- Lecture slides: Non-Compliance II.
- FEDAI: Chapter 06: Two-Sided Non-Compliance.
- KaHoot Quiz
- Problem Set 04 due.
- Problem Set 05 assigned.
- Two paragraphs (maximum) summarising an experiment that you wish to develop in this course. At a minimum, your summary should include a research question, why the question is important, and a rough outline of how you plan to answer the question. Please send me an email by Wednesday with this information.
Suggested readings:
- Angrist, J. D., & Pischke, J. S. (2009). Mostly Harmless Econometrics: An Empiricist’s Companion. Princeton University Press. Chapter 4: Instrumental Variables Estimation.
- Baryakova, T.H., Pogostin, B.H., Langer, R. et al. Overcoming Barriers to Patient Adherence: The Case for Developing Innovative Drug Delivery Systems. Nature Reviews Drug Discovery 22, 387–409 (2023).
- Kane, J. V. (2025). More than Meets the ITT: A Guide for Anticipating and Investigating Nonsignificant Results in Survey Experiments. Journal of Experimental Political Science, 12(1), 110–125.
- Hartman, E., & Huang, M. (2024). Improving Precision through Design and Analysis in Experiments with Noncompliance. Political Science Research and Methods, 12(3), 557-572.
- Gerber, A. S., Green, D. P., Kaplan, E. H., & Kern, H. L. (2010). Baseline, Placebo, and Treatment: Efficient Estimation for Three-Group Experiments. Political Analysis, 18(3), 297-315.
Lecture 11: Attrition and Missing Outcomes (October 6, 2025)
- Lecture slides: Attrition.
- FEDAI: Chapter 07: Attrition.
- Lo, A., Renshon, J., & Bassan-Nygate, L. (2024). A Practical Guide to Dealing with Attrition in Political Science Experiments. Journal of Experimental Political Science, 11(2), 147-161.
- Kahoot quiz
Suggested readings:
- Zhou, H., & Fishbach, A. (2016). The Pitfall of Experimenting on the Web: How Unattended Selective Attrition Leads to Surprising (Yet False) Research Conclusions. Journal of Personality and Social Psychology, 111(4), 493.
- Coppock, A., Gerber, A. S., Green, D. P., & Kern, H. L. (2017). Combining Double Sampling and Bounds to Address Nonignorable Missing Outcomes in Randomized Experiments. Political Analysis, 25(2), 188-206.
- Ghanem, D., Hirshleifer, S., & Ortiz-Becerra, K. (2019). Testing Attrition Bias in Field Experiments. UC Berkeley: Center for Effective Global Action.
- Pallmann, P., Bedding, A.W., Choodari-Oskooei, B. et al. (2018). Adaptive Designs in Clinical Trials: Why Use Them, and How to Run and Report Them. BMC Med 16(29).
Lecture 12: Ethical Considerations in Experimental Research (October 8, 2025)
- Lecture slides: Ethical Considerations in Experimental Research.
- Humphreys, M. (2015). Reflections on the Ethics of Social Experimentation. Journal of Globalization and Development, 6(1), 87-112.
- Cronin-Furman, K., & Lake, M. (2018). Ethics Abroad: Fieldwork in Fragile and Violent Contexts. PS: Political Science & Politics, 51(3), 607-614.
- Poverty Action Lab (2022). Ethical Conduct of Randomized Evaluations. J-PAL.
- Problem Set 05 due.
- Problem Set 06 assigned
Suggested readings:
- Asiedu, E., Dean, E., Karlan, D., & Osei, R. (2021). Ethics and Society Review: Ethics Reflection as a Precondition to Research Funding. Proceedings of the National Academy of Sciences, 118(52), e2117261118.
- ASAB Ethical Committee/ABS Animal Care Committee. (2024). Guidelines for the Ethical Treatment of Nonhuman Animals in Behavioural Research and Teaching. Animal Behaviour, 207, I-XI.
Lecture 13: Introduction to Quarto and DeclareDesign (October 13, 2025)
- Lecture slides: Introduction to Quarto and DeclareDesign.
- Quarto official website: https://quarto.org/.
- DeclareDesign official website: https://declaredesign.org/.
Suggested readings:
- Quarto official website.
- Awesome Quarto: https://github.com/mcanouil/awesome-quarto. Note: this repository contains dozens of tutorials, examples, and resources.
- Çetinkaya-Rundel, M. & Lowndes, J. S. (2022) Keynote talk: Hello Quarto: Share • Collaborate • Teach • Reimagine. Slides and source code. This is one of the nicest Quarto presentations I have seen.
- Getting Started with Quarto (YouTube). Note: Posit (formerly RStudio) has a series of tutorials on Quarto on their YouTube channel. You can find their playlist here.
- Blair, G., Coppock, A., & Humphreys, M. (2023). Introduction to Design Declaration with DeclareDesign. A series of presentation slides on DeclareDesign by the authors of the book. Strongly recommended.
- DeclareDesign official website.
- Nosek, B.A., Alter, G., Banks, G.C., Borsboom, D., Bowman, S.D., Breckler, S.J., Buck, S., Chambers, C.D., Chin, G., Christensen, G. and Contestabile, M., 2015. Promoting an Open Research Culture. Science, 348(6242), pp.1422-1425.
- A Gentle Introduction to DeclareDesign (YouTube). Video tutorial by the authors of the book.
Lecture 14: Writing Pre-Analysis Plans (October 15, 2025)
- Lecture slides: How to Write a Pre-Analysis Plan.
- EGAP: Pre-Analysis Plans.
- Olken, B. A. (2015). Promises and perils of pre-analysis plans. Journal of Economic Perspectives, 29(3), 61-80.
- Kahoot Quiz
- Problem Set 06 due.
- Problem Set 07 assigned.
Suggested readings:
- Ofosu, G. K., & Posner, D. N. (2023). Pre-Analysis Plans: An Early Stocktaking. Perspectives on Politics, 21(1), 174-190.
- Rubenson, D. (2021). Tie My Hands Loosely: Pre-Analysis Plans in Political Science. Politics and the Life Sciences, 40(2), 142-151.
- Coffman, L. C., & Niederle, M. (2015). Pre-Analysis Plans Have Limited Upside, Especially where Replications Are Feasible. Journal of Economic Perspectives, 29(3), 81-98.
- Even-Tov, Omri and Valentine, K. (2025). The Research Paper Playbook: A PhD Student’s Guide to Writing and Presenting.
Lecture 15: When Experiments are Not Possible (October 20, 2025)
- Lecture slides: Natural and Quasi-Experiments.
- Kahoot Quiz.
- Dunning, T. (2012). Natural Experiments in the Social Sciences: A Design-Based Approach. Cambridge University Press. Chapter 01: Introduction.
Suggested readings:
- Diener, E., Northcott, R., Zyphur, M. J., & West, S. G. (2022). Beyond Experiments. Perspectives on Psychological Science, 17(4), 1101-1119.
- Dunning, T. (2016). Transparency, Replication, and Cumulative Learning: What Experiments Alone Cannot Achieve. Annual Review of Political Science, 19(1), 541-563.
- Petticrew, M., Cummins, S., Ferrell, C., Findlay, A., Higgins, C., Hoy, C., … & Sparks, L. (2005). Natural Experiments: An Underused Tool for Public Health?. Public Health, 119(9), 751-757.
- Grosz, M. P., Ayaita, A., Arslan, R. C., Buecker, S., Ebert, T., Hünermund, P., … & Rohrer, J. M. (2024). Natural Experiments: Missed Opportunities for Causal Inference in Psychology. Advances in Methods and Practices in Psychological Science, 7(1), 25152459231218610.
Lecture 16: Interference in Experiments (October 22, 2025)
- Lecture slides: Interference in Experiments.
- FEDAI: Chapter 08: Interference between Experimental Units.
- EGAP: Spillovers.
- Kahoot Quiz.
- Problem Set 07 due.
- Problem Set 08 assigned.
Suggested readings:
- Aronow, P. M., Eckles, D., Samii, C., & Zonszein, S. (2020). Spillover Effects in Experimental Data. arXiv preprint arXiv:2001.05444. Quite technical too.
- Baird, S., Bohren, J. A., McIntosh, C., & Ozler, B. (2014). Designing Experiments to Measure Spillover Effects.
- Benjamin-Chung, J., Arnold, B. F., Berger, D., Luby, S. P., Miguel, E., Colford Jr, J. M., & Hubbard, A. E. (2018). Spillover Effects in Epidemiology: Parameters, Study Designs and Methodological Considerations. International Journal of Epidemiology, 47(1), 332-347.
- Imbens, G. (2024). Interference and Spillovers in Randomized Experiments (Video). A short lecture on recent advances and new designs to deal with interference in experiments. A little technical.
Lecture 17: Texts for Discussion (October 27, 2025)
- Lecture slides: Texts for Discussion.
- Kahoot Quiz.
- Centola, D. (2010). The spread of behavior in an online social network experiment. Science, 329(5996), 1194-1197.
- Paluck, B. L., Shepherd, H., and Aronow, P. 2016. Changing climates of conflict: A social network experiment in 56 schools. Proceedings of the National Academy of Sciences, 113(3): 566-571
- Gerber, A. S., & Green, D. P. (2000). The effects of canvassing, telephone calls, and direct mail on voter turnout: A field experiment. American Political Science Review, 94(3), 653-663.
Lecture 18: Heterogeneous Treatment Effects (October 29, 2025)
- Lecture slides: Heterogeneous Treatment Effects.
- FEDAI: Chapter 09: Heterogeneous Treatment Effects.
- Problem Set 08 due.
- Pre-Analysis Plan due.
Suggested readings:
- Holzmeister, F., Johannesson, M., Böhm, R., Dreber, A., Huber, J., & Kirchler, M. (2024). Heterogeneity in Effect Size Estimates. Proceedings of the National Academy of Sciences, 121(32), e2403490121.
- Ding, P., Feller, A., & Miratrix, L. (2016). Randomization Inference for Treatment Effect Variation. Journal of the Royal Statistical Society Series B: Statistical Methodology, 78(3), 655-671.
- Wager, S., & Athey, S. (2018). Estimation and Inference of Heterogeneous Treatment Effects using Random Forests. Journal of the American Statistical Association, 113(523), 1228-1242.
- Susan Athey and Stefan Wager: Estimating Heterogeneous Treatment Effects in R. A one-hour video on how to use the
grf
package in R to estimate heterogeneous treatment effects. - Generalised Random Forests (R package).
- Coppock, A. (2015). 10 Things to Know about Multiple Comparisons. EGAP Methods Guide.
- Frandsen, B. (2025). Machine Learning and Heterogeneous Treatment Effects. Slides and coding labs from a workshop on this topic. Available on GitHub.
Lecture 19: Texts for Discussion (November 3, 2025)
- Lecture slides: Texts for Discussion.
- Munshi, K. (2003). Networks in the modern economy: Mexican migrants in the US labor market. The Quarterly Journal of Economics, 118(2), 549-599.
- Miguel, E., & Kremer, M. (2004). Worms: identifying impacts on education and health in the presence of treatment externalities. Econometrica, 72(1), 159-217.
Lecture 20: Mediation Analysis (November 5, 2025)
- Lecture slides: Mediation Analysis.
- FEDAI: Chapter 10: Mediation.
- KaHoot Quiz
- Problem Set 09 assigned.
Suggested readings:
- Bullock, J. G., & Green, D. P. (2021). The Failings of Conventional Mediation Analysis and a Design-Based Alternative. Advances in Methods and Practices in Psychological Science, 4(4), 25152459211047227.
- VanderWeele, T. J. (2016). Mediation Analysis: A Practitioner’s Guide. Annual Review of Public Health, 37(1), 17-32.
- Acharya, A., Blackwell, M., & Sen, M. (2018). Analyzing Causal Mechanisms in Survey Experiments. Political Analysis, 26(4), 357–378.
- MacKinnon, D. P., Fairchild, A. J., & Fritz, M. S. (2007). Mediation Analysis. Annual Review of Psychology, 58(1), 593-614.
Lecture 21: Survey Experiments (November 10, 2025)
- Lecture slides: Survey Experiments.
- EGAP: Survey Experiments.
- Stantcheva, S. (2023). How to run surveys: A guide to creating your own identifying variation and revealing the invisible. Annual Review of Economics, 15(1), 205-234.
Suggested readings:
- Sniderman, P. M. (2018). Some Advances in the Design of Survey Experiments. Annual Review of Political Science, 21(1), 259-275.
- Schneider, D., & Harknett, K. (2022). What’s to Like? Facebook as a Tool for Survey Data Collection. Sociological Methods & Research, 51(1), 108-140.
- Clifford, S., Sheagley, G., & Piston, S. (2021). Increasing Precision Without Altering Treatment Effects: Repeated Measures Designs in Survey Experiments. American Political Science Review, 115(3), 1048-1065.
- Kane, J. V. (2024). More Than Meets the ITT: A Guide for Anticipating and Investigating Nonsignificant Results in Survey Experiments. Journal of Experimental Political Science, 1-16.
Lecture 22: Survey Methods for Sensitive Topics (November 12, 2025)
- Lecture slides: Survey Methods for Sensitive Topics.
- Bursztyn, L., Haaland, I. K., Röver, N., & Roth, C. (2025). The Social Desirability Atlas. NBER Working Paper No. 33920.
- Problem Set 09 due.
- Problem Set 10 assigned.
Suggested readings:
- Blair, G., Imai, K., & Zhou, Y. Y. (2015). Design and Analysis of the Randomized Response Technique. Journal of the American Statistical Association, 110(511), 1304-1319.
- Blair, G., & Imai, K. (2012). Statistical Analysis of List Experiments. Political Analysis, 20(1), 47-77.
- Riambau, G., & Ostwald, K. (2021). Placebo Statements in List Experiments: Evidence from a Face-to-Face Survey in Singapore. Political Science Research and Methods, 9(1), 172-179.
- Gosciak, J., Molitor, D., & Lundberg, I. (2025). Adaptive Randomization in Conjoint Survey Experiments. Retrieved from <osf.io/preprints/socarxiv/69y2j_v1>.
Lecture 24: Meta-Analysis and Integration of Research Findings (November 19, 2025)
- Lecture slides: Meta-Analysis and Integration of Research Findings.
- FEDAI: Chapter 11: Integration of Research Findings.
- Problem Set 10 due.
Suggested readings:
- Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2011). Introduction to Meta-analysis. John Wiley & Sons. A whole book dedicated to the topic. A bit technical but very good.
- Harrer, M., Cuijpers, P., Furukawa, T., & Ebert, D. (2021). Doing Meta-Analysis with R: A Hands-On Guide. Chapman and Hall/CRC.
- Hansen, C., Steinmetz, H., & Block, J. (2022). How to Conduct a Meta-Analysis in Eight Steps: A Practical Guide. Management Review Quarterly, 1-19.
- Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., … & Prisma-P Group. (2015). Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P) 2015 Statement. Systematic Reviews, 4, 1-9.
Lecture 25: Course Revision (November 24, 2025)
- Lecture slides: Course Revision.
Lecture 26: Drop-In Session (December 1, 2025)
- No specific agenda; students can drop in for help with their projects.
Lecture 27: Presentations (December 3, 2025)
- Presentations of final projects.
Lecture 28: Final Presentations (December 8, 2025)
- Final presentations of experimental projects.