A/B Test Uplift Prediction Model
2024 · Python, Polars, Snowflake, Scikit-learn
Problem
Campaign targeting was based on intuition — experienced marketers choosing who to include in a test based on gut feel. This leads to wasted budget on segments that wouldn't respond anyway, and missed opportunities in segments that would.
The insight: we had years of historical experiment data sitting in Snowflake. That's enough to train a model that predicts likely uplift before a campaign launches, and to shift targeting decisions from intuition to evidence.
Solution
Used historical A/B test results as training data. For each customer segment × campaign type combination, the model estimates the expected uplift if that group is targeted. Before a campaign launches, the model scores candidate segments and the marketer can see the predicted lift ranked, rather than guessing.
Architecture
Impact & Scale
- ~35% uplift improvement in experiments compared to intuition-based targeting
- Predictions based on the same historical data the team was already collecting — no new data infrastructure needed
Next Steps
- Improve causal inference robustness (current approach is correlational)
- Integrate scoring directly into the experimentation platform
- Automate retraining as new experiment results come in