Model Performance Monitoring & Alerting System

2025 · Terraform · MLflow · S3 · AMP · SNS · GitLab · SageMaker Pipelines

Impact

• Dozens of models, hundreds of pipelines, across multiple countries tracked in a single MLflow instance; a dozen stakeholders alerted on critical performance issues via Teams
• MLflow has been running for over a year, alerting for half a year; the platform is stable enough that the team now applies incremental improvements frequently rather than firefighting

Business Problem

With dozens of models running across multiple countries and hundreds of data pipelines, there was no central source of truth for model metrics or artifacts. Performance was tracked in isolation per team, making it nearly impossible to compare models or catch degradation early. Issues surfaced when business stakeholders noticed drops in campaign results, not when the engineering team could still act.

The missing piece was not alerting in isolation. It was consistent metrics and a shared artifact store across all pipelines, and a unified view of model health that any team could trust.

Solution

I designed a central MLflow tracking server running on SageMaker, with S3 as the artifact store and the full infrastructure provisioned via Terraform. All pipelines log metrics on every run to the same server. A SageMaker Pipeline evaluation step compares each run against a hard threshold and a benchmark experiment run, so a model can pass the absolute minimum and still be flagged if it has regressed against its own previous baseline.

When a threshold is breached, the evaluation step posts to Amazon Managed Prometheus (AMP), where Alertmanager applies deduplication and throttling per model group and publishes to an SNS topic. Teams channels subscribe directly to the topic. With dozens of models emitting signals simultaneously, deduplication is what keeps the alerting channel usable.

Architecture

Technical Challenges

Infrastructure from scratch. The full stack had to be provisioned via Terraform: MLflow tracking server on SageMaker, S3 artifact store, AMP workspace with alertmanager.yml rules, and SNS topic. This meant managing IAM roles, VPC routing, and storage lifecycle policies as code, while keeping the setup reproducible across environments.

Multiple pipeline triggers. Batch scoring pipelines, scheduled retrains, and GitLab CI/CD jobs all needed to log metrics to the same tracking server. Integrating heterogeneous triggers without duplicating instrumentation logic required a thin shared logging client used across all pipeline types.

Alert fatigue. With dozens of models each emitting performance signals, naive alerting produces noise. Throttling and deduplication rules in Alertmanager ensure flapping signals fire once rather than repeatedly, and correlated issues across models are grouped rather than broadcast individually.

Grouping by model_name + country + alert_type means every metric failure for one model in one country within a 30-second window produces a single notification instead of several. An inhibit rule then suppresses a warning for a model that already has a critical alert firing, since the on-call person is already aware.

route:
group_by: ["model_name", "country", "alert_type"]
group_wait: 30s        # collect related alerts before the first notification
group_interval: 5m     # wait before sending updates on an active group
repeat_interval: 24h   # do not repeat a firing alert more than once per day

routes:
  - match:
      severity: critical
    receiver: teams-and-slack
    repeat_interval: 4h

inhibit_rules:
# if a model has a critical alert, suppress its warning alerts
- source_match:
    severity: critical
  target_match:
    severity: warning
  equal: ["model_name", "country"]

Status

• Dozens of models monitored continuously across multiple countries
• Hundreds of pipelines monitored with MLflow: training, tuning, and scoring pipelines tracked on model performance and feature drift; analytics automation pipelines tracked on data size, feature drift, and A/B test conversion and retention metrics
• Throttled alerting in production: performance benchmark and hard threshold evaluation across all models, max one daily warning when triggered and immediate notification for critical issues

Next Steps

• Data drift: more granular feature-level drift detection with actionable signal rather than binary breach notifications
• Shadow modelling: run challenger models in parallel automatically, compare against production, and trigger retraining when the challenger consistently outperforms
• Downstream dashboards: surface model health and drift signals directly in business dashboards so country teams can act on trends without needing to interpret MLflow runs
• Weekly performance digests: scheduled summary of model health across all models and countries, sent automatically to stakeholders without requiring them to pull reports manually

Code is proprietary; happy to walk through the architecture in detail.