←Back

International Churn Model Deployment

2023 · AWS SageMaker · GitLab CI/CD · Snowflake · XGBoost · MLflow

Impact

  • First deployment on the central ML platform, establishing the integration pattern from local data warehouse through S3 to Snowflake; the same architecture now runs dozens of models across the organization
  • Churn indication 40x better than random and 8x better than the best selection benchmark, enabling targeted retention campaigns on players genuinely at risk
  • A/B tests running weekly or monthly across multiple countries using these predictions to measure and improve retention campaign effectiveness

Business Problem

Across 5 countries, retention marketing was reactive: campaigns were sent without knowing which players were genuinely at risk of leaving. With a monthly churn rate in the low single digits of the active base, the vast majority of outreach landed on players who would have stayed regardless, wasting budget and diluting campaign impact.

The harder problem was structural: each country ran its own data warehouse, its own marketing automation system, and there was no shared infrastructure for delivering a model at all. Building a country-specific solution for each would mean fragmented codebases, diverging model versions, and no scalable path forward.

Solution

I built a single XGBoost model running on a central ML platform on AWS and GitLab CI/CD, with Snowflake as the database layer for feature storage and score delivery. The model scores every active player daily with a churn probability for the next 30 days. Rather than per-customer tickets, it works at customer level, flagging anyone who cancels any of their active tickets as a churn event, which keeps the targeting logic consistent across all lotteries regardless of ticket count.

Training data covers 12 months of monthly snapshots with a deliberate sampling strategy: all churners are included in full, and non-churners are undersampled to reach the target fraction. This exposes the model to the full diversity of churn behavior rather than repeatedly learning from the same few positive cases.

Predictions are written to a central Snowflake output table and distributed to local data warehouses per country. Each country ingests the scores into its own warehouse and connects them to its own marketing automation tooling. Adding a new country is a config change, not a code change.

Architecture

Churn model deployment architecture
architecture_v1.svg

Technical Challenges

First platform deployment, no playbook. Every integration designed from scratch: data contracts, Snowflake schema, SageMaker orchestration, and per-country output feedback loops. No prior template; sustained cross-functional alignment required across local and international teams.

Class imbalance. At a low single-digit monthly churn rate, a model that predicts "no churn" for everyone is still highly accurate but useless. Standard oversampling on a static snapshot doesn't capture how churn behavior shifts across seasons and market conditions. The sampling strategy had to expose the model to the full diversity of churn patterns across time, not just the most recent or most frequent cases.

XGBoost takes a single parameter for binary imbalance, scale_pos_weight, which scales the gradient and hessian of positive examples so each churner contributes as much to training as a non-churner. This is more principled than SMOTE for a tabular production model: no synthetic data is created, the model still learns from every real training example, and the same config parameter retrains correctly across countries with different churn rates.

training_functions.py
from sklearn.utils.class_weight import compute_class_weight

class_weights = compute_class_weight("balanced", classes=[0, 1], y=y_train)
class_weight_dict = dict(enumerate(class_weights))
scale_pos_weight = class_weight_dict[1] / class_weight_dict[0]

model = XGBClassifier(
  scale_pos_weight=scale_pos_weight,
  ...
)

Temporal sampling without leakage. Each of the 12 monthly snapshots predicts churn in the following month, so the same customer can appear in several snapshots at different points in their lifecycle. Every churner row across all 12 months was kept in full; non-churners were undersampled to the target ratio. The hard part was making sure no feature used information from after that snapshot's cutoff date, and that train and test splits were made by time period rather than by random row, so a snapshot in the test set could never share information with one used in training.

Choosing a metric marketers could act on. Model quality and campaign usefulness needed different metrics. Overall recall, how many of all churners the model catches, is the general quality signal. But marketers don't act on the whole population: they take the top 1 to 5 percent of scored customers for a campaign and want to know how many of those are genuine churn risks, for ROI estimation. Recall within that top percentile became the metric that mattered operationally. Not every customer in that group churns the very next month, but they're churn-sensitive players likely to churn soon after, so the group still targets the right population. Precision mattered less here: campaign cost per contact is low, so a false positive is cheap, while missing a real churner is not.

Heterogeneous stacks per country. Each lottery delivered different schemas and used different marketing automation tools. A data contract standardized inputs; output integration required custom config per country. Feature availability differences (save desk, add-ons, prize strategies) directly explain the performance range across lotteries.

Daily scoring at production cadence. Scores must be fresh every day. SageMaker Pipelines run event-driven daily scoring, with training monthly and tuning quarterly, scheduled within overnight windows.

Status

  • In production across 5 countries, with daily scoring and monthly retraining
  • Dozens of models now run on the platform integration pattern established by this project
  • A/B tests running weekly or monthly in each country to measure retention campaign effectiveness

Next Steps

Most improvements identified at launch have been applied in the years since: the temporal sampling strategy, parallel hyperparameter tuning, and additional feature sets per lottery. The remaining priorities are:

  • Automatic A/B test monitoring and retention uplift modelling: a monthly pipeline that measures the actual retention effect of every campaign using causal inference and validates randomization, power, and significance automatically; see the A/B Test Evaluation and Uplift Modelling Pipeline
  • Shadow modelling via the Model Performance Monitoring & Alerting System: run challenger models in parallel with the production champion, compare performance continuously, and trigger replacement when the challenger consistently outperforms; the monitoring infrastructure is already in place

Code is proprietary; happy to walk through the architecture in detail.