Shipping ML Models Without the Drama

Most ML incidents are not modeling problems, they are deployment problems. Here is a boring, repeatable path from notebook to production.

Train reproducibly

Pin everything and log the run. If you cannot reproduce the artifact, you cannot roll back to it.

import mlflow
from sklearn.ensemble import GradientBoostingClassifier

mlflow.set_experiment("orders-churn")
with mlflow.start_run():
 params = {"n_estimators": 300, "max_depth": 3, "learning_rate": 0.05}
 model = GradientBoostingClassifier(**params).fit(X_train, y_train)
 mlflow.log_params(params)
 mlflow.log_metric("auc", score(model, X_val, y_val))
 mlflow.sklearn.log_model(model, "model", registered_model_name="churn")

Promote with a canary

Never flip 100% of traffic. Shadow, then canary, then promote.

# roll the new version to 10% and watch the metrics
kubectl set image deploy/churn-api api=registry/churn:v7
kubectl annotate rollout/churn-api canary-weight=10 --overwrite
kubectl argo rollouts get rollout churn-api --watch

The lifecycle in one picture:

Guardrails that pay off

• Log every prediction with the model version

• Monitor input drift, not just accuracy

• Keep the previous version warm for instant rollback

If you want the intuition behind what these models are actually learning, this explainer is superb:

Ship the pipeline before you ship the model.

Reproducibility plus canaries turns "model day" into a non-event, which is exactly the goal.