Research Methods

“Practitioner’s Guide: Designing Adaptive Experiments” With V. hadad, s. athey & D. Karlan.

Adaptive experiments present a unique opportunity to more rapidly learn which of many treatments work best, evaluate multiple hypotheses, and optimize for several objectives. For example, they can be used to pilot a large number of potential treatments when the researcher does not have strong hypotheses about what works and why; the data can then be used to narrow down a set of alternatives for further development, hypothesis testing, and evaluation. However, adaptive experiments may not be appropriate in all cases, as they create complexity and may require additional resources for implementation. This guide explains what adaptive experiments are, when they can be beneficial, and their limitations. It also offers insights into the questions to ask when considering running adaptive experiments on technology platforms.

“Survey Sampling in the Global South Using Facebook Advertisements” With P. BergQuist, K. Hoffmann Pham, F. Rampazzo & M. Mildenberger. under review.

Survey research in the Global South traditionally requires large budgets and lengthy fieldwork, for which researchers hire local enumerators to conduct face-to-face surveys with respondents. However, much of the world's population is now digitally accessible, offering an opportunity for researchers with limited budgets and those seeking to study settings where in-person contact is impossible, such as natural disasters, violent conflicts, and pandemics. In this paper, we evaluate whether Facebook advertising can be used to cost-effectively generate representative survey samples in the Global South. We introduce a framework for evaluating quality in Facebook survey samples, highlighting key trade-offs for researchers considering the platform. We then quota-sample respondents in two countries: Mexico (n=5,168) and Kenya (n=1,452) to evaluate how well these samples perform on a diverse set of survey indicators related to both internal and external validity. We find that while the Facebook platform can quickly and cheaply recruit respondents, these samples tend to be more male, more educated, and more urban than the overall national populations. Applying post-stratification weighting after oversampling key demographic variables ameliorates, but does not fully overcome, these initial sample imbalances. Our analysis demonstrates the considerable potential of Facebook advertisements to cost-effectively conduct research in diverse global settings.

“Online Surveys and Digital Demography in the Developing World: Facebook Users in Kenya” with K. Pham & F. RampazZo. [Pre-Analysis plan]

As internet and cell phone use spread, a growing number of citizens in developing countries are getting online and using social media. Can social media platforms provide a cheaper, faster, and reliable mechanism to reach populations that are currently expensive to survey in comparative contexts? How representative of the national population are social media users? We examine these questions in the case of Facebook users in Kenya. First, we obtain the demographic profiles of Facebook users in Kenya using the Facebook advertising platform, which provides data on users’ location and demographic characteristics for marketing purposes, and compare them to the overall population as measured by census and WorldPop data. Second, emulating the sampling strategy used by a well-known nationally representative survey (AfroBarometer), we use Facebook advertisements to recruit online survey respondents. Respondents are asked a series of questions that appear on nationally representative surveys, and are presented with canonical behavioral survey experiments. Fielding our survey on Facebook immediately following the 2019 census and an in-person nationally representative survey allows us to ensure that any observed differences between our survey and nationally representative samples are not due to differences in timing. We examine how closely the social and political attitudes, beliefs, and behaviors of Kenyans recruited on Facebook match those of the general population. In doing so, we evaluate the potential of Facebook to serve as a survey respondent recruitment mechanism in developing countries. We provide practical information about how to conduct stratified sampling with Facebook ads, outlining best practices for paying respondents, and highlighting important limitations to consider when conducting online surveys and experiments in contexts with varying levels of literacy.