A multi-proxy experiment is an experiment in which participants are randomly assigned not only to experimental conditions, but also to one of many proxies used to operationalize the construct of interest.
The defining move
In a traditional experiment, the proxy used as the dependent measure is usually fixed. In a multi-proxy experiment, the proxy comes from a pool. Each participant typically receives one randomly sampled proxy, which helps avoid order and spillover effects that could arise when many proxies are presented sequentially to the same person.
(Of course, designs in which participants encounter multiple proxies are also possible.)
1. Estimate the effect across proxies
Multi-proxy data can be analyzed using a multilevel model that accounts for variation across proxies. These models allow researchers to estimate an overall experimental effect while also examining how much that effect varies across different proxies.
2. Quantify heterogeneity
The model can estimate whether the experimental effect differs meaningfully across proxies. This is important because a single-proxy design cannot reveal between-proxy effect variation by design.
3. Explain heterogeneity with proxy-level moderators
Characteristics of the proxies can be measured independently and linked to the experimental data. These characteristics then become proxy-level moderators: they test whether a particular characteristic of the operationalization predicts a stronger, weaker, or reversed experimental effect.
Importantly, proxy characteristics can be measured before the experiment when the moderator is part of the hypothesis, or after the experiment when new theoretical ideas emerge. This makes a well-documented multi-proxy dataset reusable for later research.
A concrete example
In our work on positive cueing (i.e. a social influence technique to stimulate pro-environmental choices), participants were assigned to one of 81 pro-environmental choice pairs. The cueing effect varied substantially across those proxies. Later, the identity signaling value of each choice was measured in an independent sample and linked back to the original experimental data. The resulting moderation analysis showed a crossover pattern: the direction of the effect depended on how strongly the choice signaled identity.
Beyond behavioral dependent variables
We developed the approach to address a recurring problem in behavioral research: a broad behavioral construct is often represented by one fixed dependent variable proxy. But the general idea is wider. Experimental elements that are usually kept fixed (e.g. stimuli) can also be treated as sampled elements when the research question calls for it.
Multi-proxy experimentation is a design principle, not a particular statistical model. The appropriate analysis depends on the experimental design and research question. Multilevel models are often a natural choice because they can account for variation in experimental effects across proxies.
Read the publication →