TL;DR: Causal foundation models can report sound intervals per unit, yet become overconfident on group averages. The cause is aggregation that ignores correlated unit errors.
Unit-level aggregation breaks coverage in causal foundation models
Key points
ใป๐ฏ For the full-sample average effect on ACIC 2016, CausalPFN's calibrated 95% interval contained the truth in only 8 of 40 datasets
ใป๐งฎ The cause is correlated unit errors: variance is inflated by 1+(kโ1)ฯ (the design effect), yet units were summed as if independent
ใป๐ The fix is SCM-marginalized aggregation: units within a draw share one causal model
ใป๐ง Approximated with small CNP and latent NP models, also usable as heads on a frozen CausalPFN
ใป๐ Full-sample coverage across 240 datasets on 6 benchmarks is about 90% (CausalPFN: 53%)
ใปโ ๏ธ The cost: lower unit-level accuracy on 3 benchmarks and wider intervals
If decisions rely on group averages, how uncertainty is aggregated deserves as much scrutiny as the point estimate.
#
CausalInference# #
FoundationModels#