Author(s): Armin Norouzi, Ph.D

Originally published on Towards AI.

Counterfactual Evaluation in Ads: IPS, SNIPS, and Doubly Robust

You have a new ranking model. You want to know if it’s better than the one in production before you ship it. The honest answer is: you should run an A/B test. But A/B tests take weeks, they require splitting real traffic, and they expose users to an unvalidated model. For a recommender system serving hundreds of millions of impressions per day, that cost is real.

The article explains counterfactual evaluation for ads as a way to estimate the value of a new policy from existing logged bandit data, covering four estimators—Direct Method (DM), Inverse Propensity Scoring (IPS), Self-Normalizing IPS (SNIPS), and Doubly Robust (DR). It defines the logged data (context, chosen action, reward, and logging-policy propensity), derives what each estimator is trying to compute, and shows how bias and variance trade off under different assumption failures. DM is efficient but fails when the reward model is wrong; IPS is unbiased if propensities are correct but can suffer “variance explosion” when propensities are small, while SNIPS reduces that variance via self-normalization at the cost of small asymptotic bias. DR combines both reward modeling and propensity weighting so it is approximately correct when either the reward model or propensities are correct, and the simulations quantify when each method breaks. The piece further discusses practical mitigations like weight clipping, the engineering requirement to log propensities properly, structural conditions where offline evaluation can still be misleading (e.g., covariate shift, reward censoring, lack of overlap, reward-model extrapolation), and a production checklist for verifying propensity coverage, bias recovery, estimator agreement, effective sample size, and sensitivity to clipping—concluding that counterfactual evaluation complements but does not replace A/B testing.

Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI