CoE Meta-Analysis

Last update: 23 October 2026

Aggregate CoE p-Value

Each condition has three change of evidence measures (Maximum BF, BF Energy and FFT Amplitude Sum), all computed on the same Bayes factor trajectory. Because they capture related aspects of that trajectory and are highly intercorrelated, the primary analysis combines them per condition into a single aggregate CoE p-value, the harmonic mean of the three component p-values (details).

The aggregate p-values are combined across conditions, separately for experimental and control conditions, with two methods. The inverse chi-square method weights each condition by its degrees of freedom (N − 1), so that larger studies count more. Fisher’s method weights all conditions equally and serves as a robustness check against the argument that data collection was selectively extended in studies with a noticeable BF, which would give exactly those studies more weight in the weighted analysis.

If the observed trajectories do not differ from chance, the aggregate p-values are approximately uniformly distributed (as the null simulations confirm, see CoE measures), and about 5% of conditions should fall below .05.

tab <- rbind(cbind(Type = "Experimental", combine(exp)),
             cbind(Type = "Control",      combine(con)))
names(tab)[1] <- ""
kable(tab, align = "lccc") |> kable_styling(full_width = FALSE)
Table 1: Aggregate CoE p-values combined across conditions.
Conditions p < .05 Inverse Chi-Square (Weighted) Fisher (Unweighted)
Experimental 33 7 (21%) χ²(41493) = 42238.1, p = .005 χ²(66) = 119.6, p = < .001
Control 19 1 (5%) χ²(27584) = 27298.6, p = .888 χ²(38) = 36.2, p = .552

Comparison with the Simulations

If the conditions contain no more temporal structure than chance, their aggregate p-values lie on the diagonal, with the k-th smallest of n p-values close to k / (n + 1). The grey band marks the pointwise 95% range expected under the null. Points above the band indicate more small p-values, and thus more non-random evidence dynamics, than the simulations produce.

qq <- function(d, label) {
  n <- nrow(d); i <- seq_len(n)
  data.frame(group = sprintf("%s (n = %d)", label, n),
             expected = -log10(i / (n + 1)),
             observed = -log10(sort(d$p_agg)),
             lo = -log10(qbeta(.975, i, n - i + 1)),
             hi = -log10(qbeta(.025, i, n - i + 1)))
}
qd <- rbind(qq(exp, "Experimental"), qq(con, "Control"))
qd$group <- factor(qd$group, levels = unique(qd$group))   # Experimental left
ggplot(qd, aes(expected, observed)) +
  geom_ribbon(aes(ymin = lo, ymax = hi), fill = "grey85") +
  geom_abline(slope = 1, intercept = 0, linetype = "dashed", colour = "grey40") +
  geom_point(size = 2, colour = grp_cols[["Experimental"]]) +
  facet_wrap(~ group) +
  labs(x = expression(Expected~~-log[10](p)),
       y = expression(Observed~~-log[10](p))) +
  theme_bw(base_size = 11)
Figure 1: Observed aggregate CoE p-values against the uniform distribution expected under the null, for experimental and control conditions (−log₁₀ scale, so that small p-values lie to the right and top). Grey band, pointwise 95% range under the null.

Experimental vs. Control Conditions

The comparison of the two condition types is the critical test of the volatile effect hypothesis. If the observed patterns were only statistical artefacts of the analysis, the aggregate p-values should be distributed alike in both, since both pass through the same procedures. If experimental conditions carry volatile effects, their aggregate p-values should be systematically smaller. We test this with a one-sided Mann-Whitney U test and, as a robustness check, with a permutation test on the difference in Fisher chi-square statistics between the two condition types, reassigning the condition labels at random 10,000 times.

Figure 2: Aggregate CoE p-values of experimental and control conditions. Violins show the distribution, boxes the median and interquartile range, points the individual conditions.
Test Statistic p (one-sided)
Mann-Whitney U (experimental < control) W = 170 .003
Permutation test, Fisher χ² difference (10,000 permutations) Δχ² = 83.4 .007

The median aggregate p-value is 0.241 in experimental and 0.611 in control conditions.

Sensitivity to Shared Participants

The combination methods assume independent p-values, but some conditions reuse the same participants, namely the two time points of the Monks study, the three median splits of the Psychoscanner sample and the experimental and control columns of the Priming studies. The table below repeats the combination with only the first condition of each participant sample.

first_per_sample <- function(d) d[!duplicated(r$sample[d$Study]), ]
tab2 <- rbind(cbind(Type = "Experimental", combine(first_per_sample(exp))),
              cbind(Type = "Control",      combine(first_per_sample(con))))
names(tab2)[1] <- ""
kable(tab2, align = "lccc") |> kable_styling(full_width = FALSE)
Table 2: Aggregate CoE p-values combined across conditions, one condition per participant sample.
Conditions p < .05 Inverse Chi-Square (Weighted) Fisher (Unweighted)
Experimental 29 7 (24%) χ²(38419) = 39067.4, p = .010 χ²(58) = 107.3, p = < .001
Control 15 1 (7%) χ²(23471) = 23068.9, p = .969 χ²(30) = 24.6, p = .744

Individual Measures (Exploratory)

The component measures were the primary analysis in the first two versions of this report. They are kept as an exploratory breakdown because they show which aspect of the trajectory drives a result, although testing all three separately repeats largely the same information three times.

comp <- function(d, label) {
  do.call(rbind, lapply(c("MaxBF p", "Energy p", "FFT p"), function(col) {
    p <- pmax(d[[col]], 1 / 1001)   # a component p of 0 cannot enter log()
    w <- invchisq(p, k = d$N - 1); f <- sumlog(p)
    data.frame(Conditions = label,
               `Inverse Chi-Square (Weighted)` = sprintf("χ²(%d) = %.1f, p = %s", w$df, w$chisq, fmt_p(w$p)),
               `Fisher (Unweighted)` = sprintf("χ²(%d) = %.1f, p = %s", f$df, f$chisq, fmt_p(f$p)),
               check.names = FALSE)
  }))
}
ct <- rbind(comp(exp, "Experimental"), comp(con, "Control"))
ct <- cbind(Measure = rep(c("Maximum BF", "BF Energy", "FFT Amplitude Sum"), 2), ct)
kable(ct, align = "llcc", row.names = FALSE) |> kable_styling(full_width = FALSE)
Table 3: Component p-values combined across conditions (inverse chi-square weighted by N − 1, Fisher unweighted).
Measure Conditions Inverse Chi-Square (Weighted) Fisher (Unweighted)
Maximum BF Experimental χ²(41493) = 41321.4, p = .724 χ²(66) = 113.9, p = < .001
BF Energy Experimental χ²(41493) = 42154.2, p = .011 χ²(66) = 120.8, p = < .001
FFT Amplitude Sum Experimental χ²(41493) = 41991.1, p = .042 χ²(66) = 118.9, p = < .001
Maximum BF Control χ²(27584) = 18482.7, p = 1.000 χ²(38) = 38.0, p = .467
BF Energy Control χ²(27584) = 27129.3, p = .974 χ²(38) = 31.1, p = .777
FFT Amplitude Sum Control χ²(27584) = 27087.5, p = .983 χ²(38) = 35.5, p = .584

Component p-values of 0 (no simulation as extreme as the data) are set to 1/1001 before combining, the smallest value 1,000 simulations can resolve.

Earlier versions passed the p-values of the two-sided Smokers 1 conditions through two2one(), which halves them. That conversion applies to p-values of two-sided tests, whereas CoE p-values are one-tailed exceedance probabilities already, computed against simulations that use the same two-sided Bayes factor. It is no longer applied.

Data

d <- coe[, c("Study", "Experimental", "N", "COE p", "MaxBF p", "Energy p",
             "FFT p", "Direction")]
names(d)[4] <- "Aggregate p"
d[] <- lapply(d, function(x) if (is.numeric(x) && !is.integer(x)) round(x, 3) else x)
kable(d, row.names = FALSE) |> kable_styling(full_width = FALSE, font_size = 12)
Table 4: Aggregate and component CoE p-values per condition.
Study Experimental N Aggregate p MaxBF p Energy p FFT p Direction
Loving Kind TRUE 171 0.127 0.113 0.154 0.120 greater
Prayer TRUE 431 0.541 0.481 0.723 0.481 greater
Monks T1 Exp TRUE 23 0.123 0.139 0.110 0.123 greater
Monks T1 Con FALSE 29 0.924 1.000 0.807 0.992 greater
Monks T2 Exp TRUE 23 0.160 0.169 0.139 0.177 greater
Monks T2 Con FALSE 29 0.471 0.440 0.472 0.504 greater
Meditation T1 FALSE 40 0.982 1.000 0.949 1.000 greater
Meditation T2 TRUE 48 0.332 0.248 0.559 0.312 greater
Sound preference TRUE 70 0.387 0.502 0.306 0.401 greater
All-will-be-good TRUE 38 0.349 0.293 0.470 0.326 greater
One-armed Bandit TRUE 40 0.034 0.053 0.023 0.037 greater
Coin Toss TRUE 40 0.394 0.494 0.311 0.420 greater
Incongruence Exp TRUE 236 0.021 0.034 0.014 0.023 greater
Incongruence Con FALSE 271 0.229 0.289 0.171 0.265 greater
Smokers 1 Exp TRUE 122 0.006 0.007 0.006 0.006 different
Smokers 1 Con FALSE 132 0.998 1.000 0.993 1.000 different
Smokers 2 Exp TRUE 175 0.065 0.055 0.115 0.052 less
Smokers 2 Con FALSE 220 0.091 0.080 0.111 0.086 less
Smokers 3 TRUE 203 0.266 0.248 0.259 0.294 less
Psyscanner Style 1 Exp TRUE 1400 0.024 0.041 0.017 0.024 greater
Psyscanner Style 1 Con FALSE 1003 0.611 0.735 0.449 0.755 greater
Psyscanner Style 2 Exp TRUE 1308 0.205 0.234 0.185 0.202 greater
Psyscanner Style 2 Con FALSE 1095 0.058 0.087 0.042 0.060 greater
Psyscanner Style 3 Exp TRUE 1462 0.275 0.341 0.241 0.261 greater
Psyscanner Style 3 Con FALSE 941 0.287 0.181 0.637 0.298 greater
Relaxation TRUE 12571 0.104 0.110 0.110 0.093 greater
Priming 1 Exp TRUE 4092 0.021 0.023 0.019 0.020 greater
Priming 1 Con FALSE 4092 0.698 0.648 0.687 0.768 greater
Priming 2 Exp TRUE 2063 0.844 0.767 0.903 0.876 greater
Priming 2 Con FALSE 2063 0.922 1.000 0.801 0.995 greater
Priming 3 Exp TRUE 6099 0.850 0.868 0.827 0.855 greater
Priming 3 Con FALSE 6099 0.971 1.000 0.949 0.964 greater
Priming 4 Exp TRUE 4060 0.667 0.539 0.877 0.666 greater
Priming 4 Con FALSE 4060 0.717 0.562 0.972 0.727 greater
Erotic Images 1 TRUE 241 0.388 0.492 0.288 0.449 greater
Erotic Images 2 TRUE 678 0.079 0.111 0.053 0.101 greater
Smokers Priming Exp TRUE 38 0.250 0.233 0.259 0.259 greater
Smokers Priming Con FALSE 38 0.223 0.209 0.223 0.238 greater
Sobjectivity 1 FALSE 898 0.669 0.526 0.830 0.724 greater
Sobjectivity 2 FALSE 986 0.959 1.000 0.922 0.959 greater
Sobjectivity 3 FALSE 1503 0.011 0.008 0.052 0.008 greater
Epsi Correlation Condition A FALSE 2052 0.555 0.486 0.683 0.532 greater
Epsi Correlation Condition B FALSE 2052 0.384 0.299 0.685 0.332 greater
Robots TRUE 34 0.128 0.150 0.113 0.125 greater
Schrödingers Cat Hope TRUE 285 0.995 1.000 0.986 1.000 greater
Schrödingers Cat Baseline TRUE 285 0.241 0.327 0.172 0.279 greater
Desire TRUE 201 0.346 0.440 0.282 0.352 greater
Stories TRUE 766 0.186 0.305 0.129 0.195 greater
Willpower TRUE 703 0.049 0.037 0.085 0.045 greater
Baseline 1 Lucky TRUE 801 0.024 0.032 0.020 0.023 less
Baseline 1 Unlucky TRUE 725 0.916 1.000 0.810 0.960 greater
Baseline 2 TRUE 2094 0.263 0.161 0.615 0.280 less