CUPED & Variance Reduction: A Learning Guide

What You're About to Understand

After working through this guide, you'll be able to explain to a colleague why pre-experiment data makes A/B tests faster and more powerful — and how much faster — without hand-waving. You'll spot situations where CUPED will deliver big wins versus barely help, correctly set up power calculations that account for variance reduction, and ask the right questions when someone proposes using ML-based extensions like CUPAC or in-experiment data for adjustment.

The One Idea That Unlocks Everything

Think of CUPED as noise-cancelling headphones for your A/B test.

Your experiment outcome for each user is a mix of two signals: who they already are (a power user will always spend more than a casual user) and what your treatment did to them. A standard A/B test hears both at once — it's trying to detect a quiet melody (the treatment effect) in a noisy room (individual differences). CUPED looks at who each user was before the experiment started, estimates the "room noise," and subtracts it. The melody doesn't change. The room gets quieter.

If you remember only this: CUPED subtracts predictable individual differences using pre-experiment data, making the treatment signal easier to detect — without changing what you're measuring.

Learning Path

Step 1: The Foundation [Level 1]

Imagine you're running an A/B test on a checkout redesign, measuring revenue per user. Your experiment has 100,000 users in each group. After two weeks, you see:

• Treatment group: $47.32 average revenue
• Control group: $46.85 average revenue
• Difference: $0.47 — but the p-value is 0.23. Not significant.

The problem isn't that the effect doesn't exist. The problem is variance. Some users spend $500/week; others spend $2. This massive person-to-person variation drowns out a real but modest treatment effect.

Here's the CUPED insight: you already know a lot about each user's spending level from before the experiment started. A user who spent $500/week last month will probably spend around $500/week this month too, regardless of your checkout redesign. That predictable spending is noise — it's not caused by your treatment.

The CUPED formula is disarmingly simple:

$$\hat{Y}^{cuped} = Y - \theta(X - \bar{X})$$

Where Y is the user's outcome during the experiment, X is their pre-experiment value of the same metric, $\bar{X}$ is the overall pre-experiment average, and θ is a coefficient that controls how aggressively you adjust.

You compute this adjusted outcome for every user, then run your standard t-test on the adjusted values. The treatment effect estimate stays the same (in expectation), but the confidence interval shrinks.

Why doesn't this bias anything? Because randomisation guarantees that the pre-experiment averages are the same in treatment and control. When you subtract θ(X - $\bar{X}$) from both groups, those subtractions cancel out in the difference. The treatment effect is untouched. You've only removed noise.

How much noise? That depends on the correlation (ρ) between pre-experiment and during-experiment values:

A correlation of 0.7 between last month's spending and this month's spending is common for engagement metrics — meaning CUPED effectively doubles your sample size for free.

Check your understanding:
1. Why is it critical that the covariate X was measured before the experiment started? What goes wrong if you use an in-experiment metric instead?
2. If the correlation between pre- and during-experiment revenue is 0.7, and you currently need 200,000 users per group, approximately how many would you need with CUPED?

Step 2: The Mechanism [Level 2]

Now let's understand why the math works — and why the optimal θ is exactly what OLS regression would give you.

The variance decomposition. The variance of the adjusted outcome is:

$$Var(\hat{Y}^{cuped}) = Var(Y) + \theta^2 Var(X) - 2\theta Cov(X,Y)$$

This is a quadratic in θ. Take the derivative, set it to zero, and you get the optimal θ:

$$\theta^* = \frac{Cov(X,Y)}{Var(X)}$$

Recognise that? It's the OLS regression slope of Y on X. Plugging it back in:

$$Var(\hat{Y}^{cuped}) = Var(Y)(1 - \rho^2)$$

This is why CUPED, ANCOVA, and running a regression Y ~ treatment + X_pre all give the exact same answer. They're the same geometric operation viewed from different angles: projecting Y onto the space orthogonal to X, removing the component that's predictable from pre-experiment data.

Key Insight: CUPED is not a new statistical technique. It's Fisher's 1932 ANCOVA with one specific covariate choice: the pre-experiment value of the same metric. The 2013 paper's contribution was showing this particular covariate choice is near-optimal for online experiments and packaging it for platform-scale implementation.

Worked example: Why pre-experiment same-metric is usually the best single covariate.

Consider revenue per user. Revenue = visits × conversion rate × basket size. The pre-experiment revenue captures all three dimensions simultaneously — it's the natural "sufficient statistic" for predicting future revenue. If you instead used "number of pre-experiment visits" as your covariate, you'd capture one dimension and miss two. That's why pre-experiment same-metric typically gives the highest single-covariate correlation.

When does this break? When the metric is sparse (annual subscription renewal — most users have a pre-value of zero) or when the metric definition changed between pre and during periods.

The Freedman–Lin resolution. A critical backstory: In 2008, David Freedman showed that regression adjustment in experiments could actually increase variance under model misspecification with heterogeneous treatment effects. This was alarming. In 2013, Winston Lin resolved it: include treatment × covariate interaction terms plus Huber-White robust standard errors, and regression adjustment can never hurt asymptotic precision. CUPED as typically implemented (single pooled slope) is technically vulnerable to Freedman's critique, but the practical impact in large online experiments is negligible.

Check your understanding:
1. Why is the optimal CUPED coefficient θ* identical to the OLS regression slope? What geometric operation connects them?
2. A colleague argues that using "number of pre-experiment page views" would be a better CUPED covariate than "pre-experiment revenue" when the outcome metric is revenue. Under what specific circumstances might they be right?

Step 3: The Hard Parts [Level 3]

This is where the simple model breaks — and where expertise lives.

Problem 1: The correlation overestimation trap (Conductrics critique).

CUPED's power is seductive: at ρ = 0.9, you only need 19% of the standard sample size. But what if you estimated ρ = 0.9 from historical data, and the actual correlation during your experiment is 0.85?

At ρ = 0.85, you need 27.8% of standard sample — a 46% increase over what you planned. Your experiment, sized for 80% power, now has roughly 66% power. The test may fail not because CUPED is wrong, but because you were too optimistic about how much it would help.

This is worst precisely when CUPED is most valuable. At high ρ, the function (1 - ρ²) is steep — small errors in ρ cause large errors in required sample size. At ρ = 0.5, the same magnitude of estimation error barely matters.

Practical fix: Use conservative ρ estimates in power calculations. If historical data suggests ρ = 0.9, plan for ρ = 0.8.

Problem 2: The lookback window paradox.

Intuition says: more historical data → better predictions → more variance reduction. Reality: longer windows capture behavioural drift and regime changes. A user's behaviour 6 months ago is a noisier proxy for their "persistent self" than their behaviour 2 weeks ago. But 2 weeks might be too short to capture a full purchase cycle.

Different companies have landed on wildly different answers: Statsig and LaunchDarkly use 7 days; Eppo uses 30 days; Nubank uses 42 days (they need full credit-card billing cycles). The optimal window is product-specific, metric-specific, and arguably unknowable without experimentation.

Problem 3: New users have no history.

If 30% of your users are new (no pre-experiment data), you can't CUPED-adjust them. Common approaches:
- Impute with zero or the population mean, plus a binary "is_new_user" indicator
- Run CUPED only on returning users, unadjusted analysis for new users
- Use CUPAC/ML methods that incorporate non-historical covariates (device, geography, time of day)

Problem 4: Unequal group sizes.

Nubank discovered empirically that using a pooled θ estimate with unequal splits (e.g., 90/10) can actually increase variance. The treatment effect changes the covariance structure, and the pooled approach gets the wrong θ for both groups. Fix: compute group-specific θ values and use a weighted average.

Problem 5: The semiparametric efficiency bound.

CUPED uses a linear adjustment h(X) = -θX. The theoretically optimal adjustment is h(X), which involves the true conditional expectations E[Y|X, treatment] and E[Y|X, control]. When the true relationship is non-linear, CUPED leaves variance on the table. This is why ML-based methods (CUPAC, MLRATE) can beat linear CUPED by 15-30% — they approximate h better. But they introduce computational overhead, model risk, and interpretability costs.

Check your understanding:
1. You estimate ρ = 0.9 from last quarter's data. Your experiment targets a new user segment that didn't exist last quarter. What specific risk does this create, and how would you mitigate it?
2. Why can a pooled θ estimate increase variance when group sizes are unequal and treatment effects exist?

The Mental Models Worth Keeping

1. Signal-Noise Decomposition
Observed Outcome = Persistent Individual Effect + Treatment Effect + Random Noise. CUPED estimates and removes the first term, leaving a cleaner view of the second. Example: When deciding whether CUPED will help for a particular metric, ask: "How much of this metric's variation is between-person vs. within-person?" High between-person variation = high CUPED payoff.

2. The Effective Traffic Multiplier (ETM)
ETM = 1/(1 - R²). This converts variance reduction to an intuitive unit: how many additional users would give you the same power boost. An R² of 0.4 → ETM of 1.67 → it's like having 67% more users. Example: When pitching CUPED to leadership, translate "40% variance reduction" into "equivalent to having 67% more traffic" — the second framing lands immediately.

3. Estimator vs. Estimand Separation
CUPED changes the estimator (how you calculate the treatment effect), not the estimand (what you're trying to measure). The ATE being estimated is identical to simple difference-in-means. Example: When a stakeholder asks "did CUPED change our results?", the answer is: "It gave us a more precise measurement of the same thing, like replacing a blurry camera with a sharp one."

4. The Quadratic Sensitivity Trap
Sample size scales as (1 - ρ²), which is steep near ρ = 1. Small errors in correlation estimation produce outsized errors in power calculations — and this is worst when CUPED seems most valuable. Example: Always sanity-check your power calc by running it at ρ - 0.05 and ρ - 0.10 to see how sensitive your experiment design is to correlation misestimation.

5. The Variance Reduction Ladder
CUPED (linear, single covariate) → CUPAC/MLRATE (ML, multiple covariates) → ANA (in-experiment data). Each rung trades simplicity for more variance reduction. Know which rung you need before climbing. Example: Etsy moved from CUPED (7% reduction) to CUPAC (27% reduction) when they realised their metrics had strong non-linear predictors that a single linear covariate couldn't capture.

What Most People Get Wrong

1. "CUPED changes what you're measuring"
Why people believe it: The adjusted outcomes look weird — a binary conversion becomes a continuous number like 0.37. It feels like you're measuring something different.
What's actually true: The individual adjusted values aren't interpretable as outcomes. But the difference in group means is still estimating the exact same ATE. Think of it as a mathematical trick applied to both groups identically — the trick cancels in the subtraction.
How to tell in the wild: If someone reports a "CUPED-adjusted conversion rate," they're conflating the adjusted intermediate values with the final estimate. The treatment effect should still be reported in original units.

2. "CUPED reduces effect sizes"
Why people believe it: When CUPED is applied, previously-significant effects sometimes become smaller. Stakeholders see "the effect shrank."
What's actually true: The CUPED point estimate is unbiased. The apparent shrinkage is actually correction of the Winner's Curse — significant results from high-variance tests are systematically overestimated because you're conditioning on having cleared a significance threshold. CUPED's tighter confidence intervals produce estimates closer to truth.
How to tell in the wild: Compare across many experiments. With CUPED, the distribution of significant effect sizes should be closer to the true distribution. Without it, significant results are inflated.

3. "More pre-experiment data is always better"
Why people believe it: More data usually helps in statistics. A 90-day lookback should be better than a 7-day lookback.
What's actually true: Longer windows capture behavioural drift — job changes, life events, seasonal shifts. The user 3 months ago may not be the same user today. There's an optimal window that balances signal richness against behavioural staleness, and it varies by product and metric.
How to tell in the wild: Plot the pre/during-experiment correlation as a function of lookback window length. It should rise initially, then plateau or decline.

4. "You can choose between CUPED and non-CUPED results"
Why people believe it: It seems reasonable to check both and report whichever is cleaner.
What's actually true: This is p-hacking. Choosing the "better" result post-hoc inflates false positive rates and systematically overestimates effects. The decision to use CUPED must be made before seeing results — ideally, it should be automatic in your platform.
How to tell in the wild: If someone reports only CUPED results for some experiments and only non-CUPED for others, without a pre-registered rule, the analysis is compromised.

5. "CUPED is a novel technique from 2013"
Why people believe it: The name, the paper, the adoption wave all feel recent.
What's actually true: The underlying math — control variates (1950s), ANCOVA (1932) — is nearly a century old. Deng et al.'s genuine contribution was practical: identifying that pre-experiment same-metric is often the best covariate, and packaging the method for online experimentation platforms at scale. Both sides of the "is it new?" debate have a point.
How to tell in the wild: If a classical statistician dismisses CUPED as "just ANCOVA," they're mathematically right but practically wrong — the engineering and operational framework is genuinely novel.

The 5 Whys — Root Causes Worth Knowing

Chain 1: Why does CUPED reduce variance?
Because it subtracts predictable individual variation → Because user behaviour persists over time → Because habits and preferences change slowly relative to experiment durations → Because the biological and social systems underlying behaviour operate on longer timescales than typical experiments.
Level 2 deep: The persistent component is noise with respect to treatment — it would exist regardless of assignment. Removing it isolates the treatment signal.
Level 3 deep: This is mathematically identical to the control variates technique from Monte Carlo simulation: you have a random variable correlated with an auxiliary variable whose expectation is known, and subtracting the correlated component reduces variance without changing the expectation.

Chain 2: Why does CUPED struggle with binary metrics?
Because ρ between pre/post binary outcomes is mechanically low → Because binary variables have limited range (0,1) constraining possible correlations → Because for a 5% conversion rate, 95% of users are (0,0), concentrating the joint distribution → Because the information content (entropy) of a low-base-rate binary variable is inherently small.
Level 2 deep: Workaround: use a continuous predictor of conversion (engagement score, visit count) instead of the binary same-metric. Higher ρ, more variance reduction.
Level 3 deep: This is a fundamental limit. Binary outcomes contain at most 1 bit per observation. No covariate adjustment can create information that isn't there.

Chain 3: Why does overestimating ρ lead to underpowered tests?
Because n = n_standard × (1 - ρ²) → Because (1 - ρ²) is convex near ρ = 1, making small errors in ρ produce large errors in n → Because power is a threshold phenomenon: being slightly underpowered doesn't give "slightly significant" results — it gives "not significant." The step-function nature of hypothesis testing amplifies estimation errors.
Level 2 deep: At ρ = 0.5, the sample multiplier is 0.75 ± small absolute error. At ρ = 0.9, it's 0.19 ± potentially large relative error. The danger grows with the benefit.

Chain 4: Why is the pre-experiment same metric usually the best single covariate?
Because revenue = visits × conversion × basket size, and pre-period revenue encodes the full joint distribution → Any single alternative covariate captures only one dimension → The same metric is the natural sufficient statistic for predicting future behaviour under stationarity.
Level 2 deep: ML can sometimes beat same-metric because it captures non-linear relationships and interactions. The gain = difference between linear and non-linear R².

Chain 5: Why do different companies use such different lookback windows?
Because autocorrelation structure depends on product type → Because different products have different engagement cycles (daily for social media, monthly for fintech) → Because user decision-making timescales differ by product category.
Level 2 deep: Per-metric optimisation is prohibitively expensive. Companies choose one compromise value. Nubank's 42 days reflects credit-card billing cycles; Statsig's 7 days reflects fast-moving consumer web products.

The Numbers That Matter

Var_reduced = Var(Y) × (1 - ρ²)
The master equation. Everything flows from this. A correlation of 0.7 cuts variance nearly in half.

ρ = 0.7 is a good typical value for engagement metrics.
Bing reported ~50% variance reduction; Netflix ~40%. These correspond to ρ in the 0.6–0.7 range. To put that in perspective: that's like doubling your user base for free.

Etsy got 7% from CUPED vs 27% from CUPAC.
A 4x improvement by switching from linear single-covariate to ML-based multi-covariate. This is the strongest published evidence for when ML extensions are worth the complexity.

The ρ sensitivity cliff: going from 0.90 to 0.85 increases required sample by 46%.
That's a 5-percentage-point estimation error producing a 46% sample size miscalculation. At ρ = 0.5, the same error changes sample needs by only ~8%.

Microsoft's ETM varies from 1.05x to 1.2x+ across product surfaces.
On one product, >68% of metrics showed almost no CUPED benefit (ETM ≤ 1.05). On another, >55% showed substantial benefit (ETM > 1.2). The same technique, same company, wildly different results. CUPED is not uniformly helpful — it depends on the autocorrelation structure of your specific metrics.

Binary metric correlation is bounded by base rate.
For a 5% conversion rate, ρ is typically < 0.3 — giving at most ~9% variance reduction. Compare that to 51% for a continuous engagement metric with ρ = 0.7. CUPED's value varies enormously by metric type.

Meta runs "hundreds of thousands of experiments" with regression adjustment.
Their "Mean2.0" system improved detectability by >30%. At that scale, even small per-experiment improvements have massive cumulative value.

Nubank found ~12% of metric comparisons had insufficient pre-experiment data.
Not every metric can be CUPED'd. New metrics, new user segments, and sparse outcomes all limit applicability.

Where Smart People Disagree

1. Linear CUPED vs ML-based extensions
What it's about: Whether the 15-30% additional variance reduction from CUPAC/MLRATE justifies the computational cost, model risk, and interpretability loss.
Side A (simplicity): Linear CUPED captures most of the variance reduction. ML adds complexity for marginal gain. Keep it simple.
Side B (optimisation): Etsy went from 7% to 27%. LinkedIn showed 30% beyond CUPED. At scale, this translates to weeks saved and millions in experimentation capacity.
Why it's unresolved: The answer is genuinely context-dependent. For companies with strong linear autocorrelation, linear CUPED is near-optimal. For those with complex, non-linear user behaviour patterns, ML wins convincingly.

2. Automatic vs selective application
What it's about: Should CUPED be applied by default to every experiment, or should practitioners choose per-experiment?
Side A (platforms): Default to on. With proper implementation (group-specific θ, robust SEs), CUPED asymptotically never hurts.
Side B (cautious practitioners): Automatic application hides complexity, can mislead power calculations, and creates false confidence in experimenters who don't understand the method.
Why it's unresolved: Most platforms now default to on — but the Conductrics critique about underpowering via correlation overestimation remains valid and largely unaddressed in default-on implementations.

3. Can in-experiment data be safely used? (ANA)
What it's about: The 2023 augmentation framework argues in-experiment data can reduce variance if the component has "near-zero" treatment effect. Traditionalists say only pre-experiment data is safe.
Side A (traditional): In-experiment data risks treatment contamination. Even "small" bias can accumulate across thousands of experiments.
Side B (frontier): Airbnb used ANA and detected significance in 8/25 experiments vs 4/25 with standard approaches. The variance reduction potential is much larger than pre-experiment-only methods.
Why it's unresolved: The bias-variance tradeoff for "approximate null" components is not fully characterised. The practical guidance for when this is safe is underdeveloped.

4. The Conductrics critique: does CUPED cause underpowered tests?
What it's about: When practitioners use overestimated ρ for power calculations, CUPED leads to smaller experiments that are actually underpowered.
The critique: This is a real problem — and it's worst when CUPED seems most valuable (high ρ settings).
The counter: This is a power-calculation problem, not a CUPED problem. Use conservative ρ estimates.
Why it's unresolved: ρ estimation is genuinely hard — it varies by user segment, season, and metric. No consensus on how conservative "conservative" should be.

What You Don't Know Yet (And That's OK)

Open problems no one has fully solved:
- How to optimally combine pre-experiment and in-experiment covariates while controlling bias
- Whether CUPED can improve detection of heterogeneous (subgroup-specific) treatment effects, or only average effects
- How CUPED-adjusted statistics interact with sequential testing and always-valid confidence intervals
- The fundamental limits of variance reduction for a given data-generating process — how close do current methods get to the semiparametric efficiency bound in finite samples?
- How CUPED performs under network interference (social networks, marketplaces) where the stable unit treatment value assumption (SUTVA) is violated
- Variance reduction for sparse, delayed outcomes like annual subscription renewal or churn

Where your new knowledge runs out:
You now understand CUPED's mechanics, limitations, and extensions. What you don't have is the operational knowledge of implementing it at scale — the data pipeline engineering, the automated covariate selection, the organisational change management (Nubank invested heavily in training and "office hours" just to build stakeholder trust in adjusted results). You also lack deep fluency with the semiparametric theory that underpins the efficiency bounds.

Subtopics to Explore Next

1. ANCOVA and Regression Adjustment in Experiments
Why it's worth it: Understanding ANCOVA deeply makes CUPED trivially obvious — it's just one covariate choice within a larger framework.
Start with: Winston Lin's 2013 paper "Agnostic Notes on Regression Adjustments to Experimental Data" — it's the definitive resolution of when regression adjustment helps vs hurts.
Estimated depth: Medium (half day)

2. CUPAC and ML-Based Variance Reduction
Why it's worth it: Unlocks the next 15-30% of variance reduction beyond linear CUPED, which matters enormously at scale.
Start with: DoorDash's 2020 blog post on CUPAC, then Etsy's 2021 implementation case study showing their LightGBM pipeline.
Estimated depth: Medium (half day)

3. The Semiparametric Efficiency Bound
Why it's worth it: Tells you the theoretical ceiling — the best any estimator can do — so you know when to stop optimising.
Start with: Search "semiparametric efficiency bound randomized experiments" — the key concept is that CUPED approximates the optimal estimator, and ML methods get closer.
Estimated depth: Deep (multi-day)

4. Sequential Testing and Always-Valid Inference
Why it's worth it: Most modern platforms use continuous monitoring, and CUPED changes the variance structure — understanding their interaction prevents subtle errors.
Start with: Search "always-valid p-values CUPED" or "sequential testing variance reduction" — this is an active research area with limited definitive answers.
Estimated depth: Medium (half day)

5. Causal Inference Fundamentals (Potential Outcomes Framework)
Why it's worth it: The language of estimands, estimators, SUTVA, and potential outcomes makes the entire CUPED framework — and its limitations — crystal clear.
Start with: Rubin's potential outcomes framework; search "Neyman potential outcomes CUPED" for the direct connection.
Estimated depth: Deep (multi-day)

6. The ANA / Augmentation Framework (Deng 2023)
Why it's worth it: Represents the frontier of variance reduction — using in-experiment data without (much) bias. Could be the next big platform-level improvement.
Start with: Deng et al. 2023 "From Augmentation to Decomposition" paper on arXiv.
Estimated depth: Medium (half day)

7. Doubly Robust Estimation
Why it's worth it: Connects CUPED to the broader causal ML world (TMLE, targeted learning) — the same ideas applied to observational data and policy evaluation.
Start with: Search "doubly robust estimator randomized experiments" — in experiments with known propensity scores, doubly robust estimators reduce to the optimal semiparametric form.
Estimated depth: Deep (multi-day)

8. Practical Power Calculations with Variance Reduction
Why it's worth it: The Conductrics critique shows this is where CUPED goes wrong in practice — getting power calcs right is the difference between CUPED helping and hurting.
Start with: Statsig's 2024 blog post "How to Plan Test Duration When Using CUPED" for the practical angle; then model the sensitivity of your own power calculations to ρ estimation error.
Estimated depth: Surface (1-2 hours)

Key Takeaways

• CUPED changes the estimator, never the estimand — it's a sharper lens, not a different measurement.
• The pre-experiment value of the same metric is usually the best single covariate because it's the natural sufficient statistic for that user's behaviour.
• Variance reduction scales as (1 - ρ²): the relationship is quadratic, which means moderate correlations (0.5-0.7) already deliver substantial gains.
• The same quadratic relationship that makes CUPED powerful also makes it fragile: small errors in ρ estimation produce large errors in power calculations, especially at high ρ.
• CUPED is mathematically identical to ANCOVA with one covariate — the innovation was practical (which covariate, at what scale, in what pipeline), not theoretical.
• ML-based extensions (CUPAC, MLRATE) can deliver 2-4x the variance reduction of linear CUPED by capturing non-linear patterns — but the gains are context-dependent.
• Binary metrics get much less CUPED benefit than continuous metrics, because the correlation between binary pre/post values is mechanically bounded by the base rate.
• The "shrinkage" people see in CUPED-adjusted effect sizes is actually correction of the Winner's Curse — the unadjusted "bigger" effect was the inflated one.
• The optimal lookback window is non-obvious: more history captures behavioural drift, not just more signal. It varies by product, metric, and user segment.
• Choosing between CUPED and non-CUPED results after seeing the data is p-hacking — the method must be pre-specified.
• CUPED doesn't fix flawed randomisation, network effects, or poor metric choice. It only reduces noise from individual-level heterogeneity.
• At scale (Microsoft, Meta, Netflix), CUPED is table stakes — the frontier has moved to ML-based multi-covariate methods and in-experiment data leverage.
• The biggest operational challenge isn't statistical — it's communicating adjusted results to stakeholders who see "smaller effects" and lose trust.
• When CUPED becomes universal, the bottleneck shifts from "can we detect effects?" to "are we testing the right thing?" — experiment design becomes the constraint.

Sources Used in This Research

Primary Research:
- Deng, Xu, Kohavi, Walker — "Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data" (WSDM 2013) — the original CUPED paper
- Deng et al. — "From Augmentation to Decomposition: A New Look at CUPED in 2023" (arXiv, 2023) — augmentation framework and ANA
- Jin, Ba et al. — "Towards Optimal Variance Reduction in Online Controlled Experiments" (Technometrics, 2022)
- Guo et al. — "Machine Learning for Variance Reduction in Online Experiments (MLRATE)" (NeurIPS, 2021)
- Winston Lin — "Agnostic Notes on Regression Adjustments to Experimental Data: Reexamining Freedman's Critique" (Annals of Applied Statistics, 2013)
- David Freedman — "On Regression Adjustments to Experimental Data" (2008)
- Xie, Aurisset — "Improving the Sensitivity of Online Controlled Experiments: Case Studies at Netflix" (KDD, 2016)
- Various — "Variance Reduction Combining Pre-Experiment and In-Experiment Data" (arXiv, 2024)

Expert Commentary:
- Microsoft Research ExP Team — "Deep Dive Into Variance Reduction" (2022)
- Conductrics — "CUPED's Sting: More Power More Underpowered A/B Tests" (2025)
- Nubank Engineering — "3 Lessons from Implementing CUPED at Nubank" (2024)
- DoorDash — "Improving Experimental Power through CUPAC" (2020)
- Etsy Engineering — "Reducing Experiment Duration with Predicted Control Variates" (2021)
- Analytics at Meta — "How Meta Scaled Regression Adjustment" (2023)
- Eppo — "CUPED and CUPED++: Bending Time in Experimentation" (2023)
- Statsig — "How to Plan Test Duration When Using CUPED" (2024)
- Matteo Courthoud — "Understanding CUPED" (2022)
- Marton Trencseni — "Reducing Variance in A/B Testing with CUPED" (2020)
- Alex Deng — "Ch10: Improving Metric Sensitivity" (ongoing reference)

Reference:
- Statsig CUPED Documentation
- GrowthBook CUPED Documentation
- Eppo CUPED++ Documentation
- LaunchDarkly Covariate Adjustment Documentation