Main Results

Interactive forest plots of experimental design effects

General support (observed) d = +0.06 +2.3 pp 95% CI 0.02 to 0.10 95% PI -0.19 to 0.31

General support (stated) d = +0.08 +2.0 pp support 95% CI 0.04 to 0.12 95% PI -0.13 to 0.29

General price increase d = +0.06 +2.0 pp 95% CI 0.02 to 0.10 95% PI -0.14 to 0.27

Social price d = +0.07 +2.0 pp support 95% CI 0.03 to 0.10 95% PI -0.12 to 0.25

Tax with rebates d = +0.06 +1.0 pp support 95% CI 0.02 to 0.10 95% PI -0.12 to 0.24

Social price with rebates d = +0.04 +1.0 pp support 95% CI 0.00 to 0.07 95% PI -0.14 to 0.22

New here?

Self-assessed effect

Peer-assessed effect

Estimated RT effect

Meta-effect (95% CI)

95% PI

both outcomes p < .05 one outcome p < .05 neither p < .05

Estimated RT effect

Meta-effect (95% CI)

95% PI

Estimated RT effect

Meta-effect (95% CI)

95% PI

Estimated RT effect

Meta-effect (95% CI)

95% PI

Estimated RT effect

Meta-effect (95% CI)

95% PI

What is #ManyDesignsCarbon?

#ManyDesignsCarbon is a crowd-science project on whether behavioral interventions can increase public support for carbon pricing. The main results shown here summarize 55 independently designed studies run under a shared protocol with 19,558 U.S. participants.

135 applications

55 implemented designs

19,558 participants

Why many designs?

A single randomized study usually provides only a noisy estimate from one finite sample of a much larger population; for one design, and one analytical specification. It might give us a hint whether an intervention works in a specific setting, but its policy relevance is necessarily limited, and with typical sample sizes, often statistically fragile. For important policy recommendations, such as climate policies, we therefore need a broader balance of evidence, ideally meta-scientific evidence. Conventional meta-analyses provide this breadth, but they also come with important limitations: they take years or decades to accumulate, are vulnerable to publication bias, and often pool studies that differ in populations, outcomes, implementation details, and analysis strategies. This makes the overall estimate less precise and harder to interpret as it is difficult to isolate the role of policy design features in variation in effect sizes. A many-designs approach addresses these limitations more directly. It leverages the wisdom of an expert crowd and offers several advantages. First, it strongly increases effective sample size and thus statistical power at the aggregate level. Second, it makes design heterogeneity visible: instead of a single estimate, it yields a distribution of effects and shows how sensitive results are to design choices. Third, it enables immediate evidence aggregation using established methods (e.g., random-effects meta-analysis), rather than waiting for years of accumulated studies, and also aids in investigating what exactly makes interventions work by examining highly comparable studies together. Fourth, it reduces the risk that conclusions hinge on a single, potentially idiosyncratic design, improving the robustness of the findings. Finally, because all designs are preregistered, all materials are shared online and the data is analyzed in a coordinated manner, and reported regardless of results, transparency, reproducibility, and replicability are enhanced, while publication bias is reduced.

How designs entered the study and were implemented

Research teams applied to participate and were randomly selected from a large international applicant pool of 135 applications. Selected teams then submitted pre-registered design proposals following a common protocol. Each study was required to include both a treatment and a control condition, and to implement a consequential, real-world measure of support for carbon pricing, such as donations to NGOs, petition signatures, letters to policymakers, or advocacy-oriented social media posts. In addition to designing and programming their own experiments, teams provided ex-ante predictions of their expected effect sizes. To further capture expert judgment, teams were randomly assigned to evaluate a subset of other designs, generating independent peer assessments alongside self-assessments. All 55 studies were then implemented simultaneously with almost 20,000 U.S. residents from a shared participant pool, with approximately 356 participants per design. In addition to each team’s real-world outcome, all participants completed a common battery of stated support measures, and all studies were analyzed under a unified framework, ensuring comparability across designs.

What the project learned

On average, behavioral interventions generated small but statistically detectable increases in support for carbon pricing: meta-analytic effects ranged from about d = 0.04 to 0.08, equivalent to roughly a two-percentage-point increase across the main real-world and stated support measures. However, effects varied meaningfully across designs. Prediction intervals span both positive and negative values, implying that future interventions from the same design space could plausibly have positive, null, or even backfiring effects. The key takeaway is therefore nuanced: behavioral interventions can move policy support, but their effects are small on average and highly design-dependent. We do not find significant evidence that peer-assessed informativeness, peer-predicted effect size, or broad intervention category systematically explains variation in real-world support effects. Exploratory post-hoc analyses pointed to two suggestive patterns: interventions clustered around behavioral biases and reflective thinking, and a broader factor capturing motivation-based, future-focused, and reflective-thinking designs, were both positively associated with real-world support.

Open science

The project was pre-registered, received ethics approval from the University of Innsbruck, and provides the pre-registered study protocol, data, and analysis files in the OSF repository.

#ManyDesignsCarbon Main Results

Main Results