Skip to content

S5 · PPI correction

  • Author: Krishna Nandimandalam
  • Date: 2026-10-09
  • Card: #9 S5 · PR: TBD

What I built

Prediction-powered inference (PPI) for unbiased mean estimation: ppi_mean(y_labeled, yhat_labeled, yhat_unlabeled, *, confidence=0.95) → Estimate.

Main signature: 

def ppi_mean(
    y_labeled: npt.ArrayLike,
    yhat_labeled: npt.ArrayLike,
    yhat_unlabeled: npt.ArrayLike,
    *,
    confidence: float = 0.95,
) -> Estimate:
    """Prediction-powered inference for mean estimation."""

The idea, simply

A judge has seen only a small labeled set (humans scored it); you want to estimate the true mean on a much larger unlabeled set the judge also scored.

The "trick" is to measure the judge's average error on the labeled set (the rectifier) and subtract it from the judge's average on the unlabeled set. This works because: - If the judge is perfect, the rectifier is 0, and the estimate is just the judge's unlabeled average. - If the judge is systematically biased, the rectifier corrects for it—on average. - The estimate is unbiased whether the judge is good or bad; only the confidence interval width changes.

Formula: 

rectifier = mean(ŷ_labeled − y_labeled)
θ̂ = mean(ŷ_unlabeled) − rectifier
Var ≈ Var(ŷ_unlabeled) / N + Var(ŷ_labeled − y_labeled) / n
SE = √Var
low, high = θ̂ ∓ z · SE

The assumption: the labeled set is a random sample from the same population as the unlabeled set.

How I know it's right

  • Hand example: y = [1, 0, 1, 1], ŷ = [1, 1, 1, 0], ŷ_unl = [1, 1, 0, 1, 1, 0] yields θ̂ = 0.6667 (4/6). ✓
  • Perfect judge: when ŷ_labeled == y_labeled, θ̂ equals mean(ŷ_unlabeled). ✓
  • Oracle validation: all three fixed-seed cases match ppi_py.ppi_mean_ci(..., lam=1) within 1e-6 on both point and interval. ✓
  • Error cases: rejects mismatched lengths, n < 2, N < 2, NaN, infinity, non-1D arrays. ✓

What surprised me / gotchas

  1. ddof=0 is critical. ppi_py uses NumPy's default std() without specifying ddof, which means it uses population variance (ddof=0), not sample variance (ddof=1). Computing variance as np.var(..., ddof=0) was essential to match the oracle.

  2. Bounds are not clipped to [0, 1]. The spec is explicit: keep bounds as-is to preserve oracle agreement. For binary data, the interval can technically exceed [0, 1], and docstring warns users of this. The Estimate post-init validation allows any finite interval.

  3. alpha vs confidence mapping. ppi_py uses alpha = 1 - confidence to map a 95% confidence request to alpha=0.05. This is standard but easy to flip.

  4. lam=1 requirement. ppi_py's default lam=None triggers PPI++ (power-tuned inference), which is not the classical PPI. The spec requires lam=1 to recover classical PPI. Oracle tests explicitly pass lam=1.

  5. Rectifier as a single statistic. The rectifier is computed once (not per bootstrap), then subtracted from the unlabeled mean. This is different from the Rogan-Gladen corrected rate, which bootstraps the judge's calibration. Here, the rectifier's variance is part of the pooled SE.

How AI helped (and where it was wrong)

Left for the maintainer to fill in.

Learn more