Practical roadmap

How to run a multi-proxy experiment

If you already know how to run a conventional experiment, the multi-proxy version is conceptually straightforward. The extra work is mainly in building the proxy pool and representing proxy variation correctly in the analysis.

1

Build a proxy pool

Identify many viable operationalizations of the construct. Sources can include prior literature, existing datasets, expert input, qualitative work, or newly generated examples. Check that the proxies genuinely instantiate the construct you want to study.

2

Identify proxy-level moderators (optional)

If you have a theory about why effects should differ across proxies, define the relevant proxy characteristic before the experiment.

3

Measure those proxy characteristics (optional)

Use an independent sample, expert coding, or another appropriate source to obtain proxy-level scores. Planned missingness can make large rating studies more practical.

4

Run the experiment

Randomly assign participants to the experimental condition as usual. In addition, randomly assign each participant to one proxy from the pool.

5

Analyze participants nested in proxies

Estimate the average treatment effect and allow that effect to vary across proxies. If proxy-level moderators are available, test whether they explain the variation.

6+

Reuse the dataset

New proxy characteristics can be measured later and merged with the original experimental data. This allows follow-up moderation analyses without rerunning the participant-level experiment.

What about the number of proxies?

There is no universal number that is optimal for every design. The original roadmap suggested formulating at least about 30 proxies as a practical lower-bound heuristic for multilevel work, while emphasizing that power depends jointly on the number of proxies, the total participant sample, effect size, and heterogeneity. Larger proxy pools can improve what you learn about between-proxy variation, but they also require more preparation.

What about the number of participants?

Power in a multi-proxy experiment is not determined by the number of participants in each proxy cell alone. It depends on the overall participant-by-proxy design. This is one reason why designs with many proxies can still work with modest numbers of participants per proxy when the total sample and number of proxies are sufficiently large.

Do moderators need to be planned in advance?

No. A preregistered moderator test is ideal when the theory is already clear, but one advantage of the design is that proxy features can also be identified and measured post hoc. Those later analyses should of course be presented as exploratory or follow-up analyses rather than as preregistered tests.

Good practice: publish the proxy list, proxy-level measurements, experimental data, and analysis code whenever possible. Reusable proxy pools and reusable multi-proxy datasets make the approach more cumulative and collaborative.

Read the full tutorial →

Need a more systematic way to choose behavioral proxies?

The original multi-proxy roadmap focused on creating and validating a large proxy pool. Our newer behavior-centered work goes further by mapping behavioral domains and distinguishing random, purposive, and matched selection depending on the inferential goal.