Model Performance Monitoring & Alerting System

2025 to 2026 · Terraform · MLflow · Lambda · EventBridge · AMP · SNS · GitLab · SageMaker Pipelines

Impact

Dozens of models, hundreds of pipelines, across multiple countries tracked in a single MLflow instance; a dozen stakeholders alerted on critical performance issues via Teams
MLflow has been running for over a year, alerting for half a year; the platform is stable enough that the team now applies incremental improvements frequently rather than firefighting

Business Problem

With dozens of models running across multiple countries and hundreds of data pipelines, there was no central source of truth for model metrics or artifacts. Performance was tracked in isolation per team, making it nearly impossible to compare models or catch degradation early. Issues surfaced when business stakeholders noticed drops in campaign results, not when the engineering team could still act.

The missing piece was not alerting in isolation. It was consistent metrics and a shared artifact store across all pipelines, and a unified view of model health that any team could trust.

Solution

I designed a central MLflow tracking server running on SageMaker, with S3 as the artifact store and the full infrastructure provisioned via Terraform. All pipelines log metrics on every run to the same server. Alert evaluation is platform-managed: an EventBridge rule on pipeline success invokes a check_mlflow_alerts Lambda, which reads the finished run's metrics and the config.yml the run logged as an artifact, then compares each metric against a hard threshold or a benchmark run. A model can pass the absolute minimum and still be flagged for regressing against its own previous baseline, and a benchmark can live in another environment, so acceptance compares against production.

A data scientist's only interface is that config file: metric, comparison, threshold, severity, alert name. No config means no alerting for that run, so opting in is a file in the repo rather than a platform ticket. Breaches go to Amazon Managed Prometheus (AMP), where Alertmanager groups and throttles per model and routes on the run's value case to one of three SNS topics, each delivering to a Teams channel by email. Failed Lambda invocations land on a dead-letter queue with a scheduled redrive, and a formatter Lambda notifies the platform team inbox, so a silent alerting outage cannot pass for a quiet week.

Architecture

Technical Challenges

Infrastructure from scratch. The full stack had to be provisioned via Terraform: MLflow tracking server on SageMaker, S3 artifact store, AMP workspace with its Alertmanager definition, the evaluation Lambda and its dead-letter queue, and the SNS topics. That meant IAM roles, VPC routing and storage lifecycle as code, reproducible across four environments.

A config file as the only interface. Alerting is owned by the platform, but the thresholds are owned by whoever trained the model, so the contract between them is a config.yml logged with the run. The Lambda validates each spec before it evaluates anything, warns when two specs would resolve to the same alert labels, and skips a metric the run never logged rather than failing the batch. A pathological label value is truncated, because AMP sends a run's breaches as one batch and a single rejected SNS subject would drop every alert in it.

config.yml

alerting:
enabled: true

metrics:
  # static threshold: alert when the run's value breaches it
  - metric_key: num_failed_channels
    comparison: gt
    threshold: 0
    severity: critical
    alertname: FailedChannels

  # no threshold: resolved against a benchmark run instead
  - metric_key: auc
    comparison: lt
    benchmark_factor: 0.95
    severity: warning
    alertname: AucRegression

benchmarks:
  pr:
    search_mode: latest
    experiment_name: churn-plg-training_pipeline

Alert fatigue. With dozens of models each emitting performance signals, naive alerting produces noise. Alerts group on model_name with a 60 second wait and a four hour repeat interval, deliberately not on alert name, so one training run produces one email listing every metric it breached rather than one email per metric.

Status

Dozens of models monitored continuously across multiple countries
Hundreds of pipelines monitored with MLflow: training, tuning, and scoring pipelines tracked on model performance and feature drift; analytics automation pipelines tracked on data size, feature drift, and A/B test conversion and retention metrics
Throttled alerting in production: benchmark and hard threshold evaluation on every finished run, grouped per model so one run is one notification
Platform-managed delivery: a single evaluation Lambda for every model, with a dead-letter queue and scheduled redrive reporting failures to the platform team

Next Steps

Data drift: more granular feature-level drift detection with actionable signal rather than binary breach notifications
Shadow modelling: run challenger models in parallel automatically, compare against production, and trigger retraining when the challenger consistently outperforms
Downstream dashboards: surface model health and drift signals directly in business dashboards so country teams can act on trends without needing to interpret MLflow runs
Weekly performance digests: scheduled summary of model health across all models and countries, sent automatically to stakeholders without requiring them to pull reports manually

Code is proprietary; happy to walk through the architecture in detail.